Locally Invariant Explanations: Towards Stable and Unidirectional Explanations through Local Invariant Learning
Authors: Amit Dhurandhar
Karthikeyan Natesan Ramamurthy
Kartik Ahuja
Vijay Arya
Venue: Submitted to ICLR 2023 (2023)
Keywords: explainable AI

Abstract: Locally interpretable model agnostic explanations (LIME) method is one of the most popular methods used to explain black-box models at a per example level. Although many variants have been proposed, few provide a simple way to produce high fidelity explanations that are also stable and intuitive. In this work, we provide a novel perspective by proposing a model agnostic local explanation method inspired by the invariant risk minimization (IRM) principle -- originally proposed for (global) out-of-distribution generalization -- to provide such high fidelity explanations that are also stable and unidirectional across nearby examples. Our method is based on a game theoretic formulation where we theoretically show that our approach has a strong tendency to eliminate features where the gradient of the black-box function abruptly changes sign in the locality of the example we want to explain, while in other cases it is more careful and will choose a more conservative (feature) attribution, a behavior which can be highly desirable for recourse. Empirically, we show on tabular, image and text data that the quality of our explanations with neighborhoods formed using random perturbations are much better than LIME and in some cases even comparable to other methods that use realistic neighbors sampled from the data manifold. This is desirable given that learning a manifold to either create realistic neighbors or to project explanations is typically expensive or may even be impossible. Moreover, our algorithm is simple and efficient to train, and can ascertain stable input features for local decisions of a black-box without access to side information such as a (partial) causal graph as has been seen in some recent works.

Under review as a conference paper at ICLR 2023
LOCALLY INVARIANT EXPLANATIONS: TOWARDS STABLE AND UNIDIRECTIONAL EXPLANATIONS THROUGH LOCAL INVARIANT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Locally interpretable model agnostic explanations (LIME) method one of the most popular methods used to explain black-box models at per example level. Although many variants have been proposed, few provide a simple way to produce high fidelity explanations that are also stable and intuitive. In this work, we provide a novel perspective by proposing a model agnostic local explanation method inspired by the invariant risk minimization (IRM) principle originally proposed for (global)
generalization to provide such high fidelity explanations that are also stable and unidirectional across nearby examples. Our method is based on a game theoretic formulation where we theoretically show that our approach has strong tendency to eliminate features where the gradient of the black-box function abruptly changes sign in the locality of the example we want explain, while in other cases more careful and will choose more conservative (feature) attribution, a behavior which can be highly desirable for recourse. Empirically, we show on tabular, image and text data that the quality of our explanations with neighborhoods formed using random perturbations are much better than LIME and in some cases even comparable to other methods that use realistic neighbors sampled from the data manifold. This is desirable given that learning a manifold to either create realistic neighbors or to project explanations is typically expensive may even be impossible. Moreover, our algorithm is simple and efficient to train, and can ascertain stable input features for local decisions of a black-box without access to side information such (partial) causal graph has been seen in some recent works.
INTRODUCTION
Deployment and usage of neural black-box models has significantly grown in industry over the last few years creating the need for new tools to help users understand and trust models (Gunning (017). Even well-studied application domains such as image recognition require some form of prediction understanding in order for the user to incorporate the model into any important decisions (Simonyan
et al. 2013 Lapuschkin et al. 2016). An example of this could be a doctor who is given a cancer image scan. Since the doctor holds responsibility for the final diagnosis, the model must provide sufficient reason for its prediction. Even new text categorization tasks (Feng al. 2018) are becoming important with the growing need for social media companies provide better monitoring of public content. Twitter recently began monitoring tweets related to COVID-19 in order to label tweets containing misleading information, disputed claims, or unverified claims (Roth& & Pickles 2020). Laws will likely emerge requiring explanations for why red flags were or fact, the General Data Protection and Regulation (GDPR)
diagnosis based
were not raised in many examples. (Yannella & Kagan 2018) act passed in Europe already requires automated systems that make decisions affecting humans to be able to explain them. Given this acute need, a number of methods have been proposed to explain local decisions (i.e. example specific decisions) of classifiers (Ribeiro) et al. 2016 Lundberg & Lee 2017: Simonyan et al. 2013 Lapuschkin et al. 2016 Dhurandhar et 2018a). Locally interpretable model-agnostic explanations (LIME) is arguably the most
well-known local explanation method that requires only query (or black- box) access to the model.
Although LIME is popular method, it is known to be sensitive certain design choices such as

---

Under review as a conference paper at ICLR 2023
i) (random) sampling to create the (perturbation) neighborhood" ii) the size of this neighborhood (number of samples) and iii) (local) fitting procedure to lear the explanation model (Molnar 2019: Zhang et al., 2019b). The first, most scrious issue could lead to nearby examples having
drastically different exp explanations making effective recourse a challenge. One possible mitigation is to increase the neighborhood size, however, one cannot arbitrarily do as it not only leads to higher computational cost, but in today's s cloud computing- implications where every query to a black-box model has an associated cost (Dhurandhar et al. 2019) There have been variants suggested to overcome some of these limitations (Botari et al.) 2020; Shrotri et al. 2021 Plumb et al. 2018) primarily through mechanisms that create realistic neighborhoods or through adversarial train- ing (Lakkaraju et al.) 2020), however, their efficacy is
world it could have direct monetary
Confficient inconsistency for LIME
4.5
sepal width (cm)
4.0
restricted to certain settings and modalities based on their
assumptions and training strategies.
3.0
88
In this paper we introduce a new method called Locally INvariant EXplanations (LINEX) inspired by the invariant risk minimization (IRM) principle (Arjovsky et al.] 2019), that produces explanations in the form of feature attribu- tions that are robust to neighborhood sampling and can recover faithful (i.e. mimic black-box behavior), stable (i.e. similar for closeby examples) and unidirectional (i.e. same sign attributions a.k.a. feature importances) for closeby examples, see section 4.1) explanations across tabular, im- age, and text modalities. In particular, we show that our method performs better than the competitors for random as well as realistic neighborhood generation, where in some cases even with the prior strategy explanation quality is close to methods that employ the latter. Qualitatively, our method highlights (local) features important that in the particular locality i) have consistently high gradient with respect (w.r.t.) the black-box function and ii) where the gradient does not change significantly, especially in sign. Such stable behavior for LINEX is visualized in Figure [] where we get similar explanations for nearby examples in the IRIS dataset. The (in)fidelity of LINEX is still similar to LIME (see Table 2). but of course our explanations are much more stable.
2.5
2,0
4.5
5.0
5.5
6.5
7.0
7.5
8.0
sepal length (cm)
Confficiant inconsistancy for LINEX
4.5
sepalwie width (em)
4.0
3.5
3.0
2.5
2.0
5.0
6.5 sepal length (cm)
5.5
8.0
Figure 1: Above visualize for the IRIS dataset the Coefficient Inconsis- tency (CI) (see Section 5 for exact def- inition and setup details) between the explanation (top two features) for an ex- ample and its nearest neighbor in the dataset. Each circle denotes an exam- ple and a rainhow colormap depicts the degree of inconsistency w.r.t. its nearest neighbor where red implies least incon- sistency, while violet implies the most. As can be seen LINEX explanations much more consistent than LIME's.
2 RELATED WORK
Posthoc explanations can typically be partitioned into two broad categories global and local. Global explainability avers to trying to understand a black-box model at a holis-
tie level where the typical tact is knowledge transfer (Hinton et al. 2015 Dhurandhar et al. 8b 2020) where (soft/hard) labels of the black-box model are used to train an interpretable model such as a decision tree or rule list (Rudin) 2019). Local explanations on the other hand avers to understanding individual decisions. These explanations are typically in two forms, either exemplar based feature based. For exemplar based as the name suggests similar but diverse examples (Kim et al. 2016 explanations for the input in question. While for feature based (Ribeiro et al. 2016:) Lundberg & Lee| 2017] Dhurandhar et al., 2018a Lapuschkin et al., 2016: Zhao a 2021). D. which is the focus of this work, important features are returned as being important for the decision made for the input. There are some methods that do both (Plumb et al.) 2018). Moreover, there are methods which provide explanations that are local, global as well as at a group level (Ramamurthy et al.) 2020). All of these methods though may not still provide stable and robust local feature based explanations which can be desirable in practice (Ghorbani et al.) 2019).
Gurumoorthy et al. 2019) are provided
By perturbation neighborhood - referred to simply neighborhood - we mean neighborhoods generated for local explanations. By exemplar neighborhood, we mean nearest examples in dataset to given example.
2

---

Under review as a conference paper at ICLR 2023
Given this there have been more recent works that try to learn either robust even causal explanations. In (Lakkaraju et al. 2020) the authors try to learn robust and stable local explanations relative to distribution shifts and adversarial attacks. However, the distribution shifts they consider are linear shifts and adversarial training is performed which can be slow and sometimes unstable (Zhang et al. 2019ap. Moreover, the method seems to be applicable primarily to tabular data. Works causal explanations (Frye et al.) 22000] Heskes et al. 2020) mainly modify SHAP and assume access to a partial causal graph. Some others (Vig et al. (020) assume white-box access. In this work we do not assume availability of such additional information. There are also works which show that creating realistic neighborhoods by learning the data manifold for LIME (Botari et al.) 20200 /hhotti et al. 2021) can lead to better quality explanations, where in particular work (Anders et al.) (020) it is
suggested that projecting explanations themselves on to the manifold can also make them more robust. The need for stability in exemplar neighborhood for LIME like methods has been highlighted in
(Zhang et al. (019b),. with the general desire for stable explanations being also expressed in (Ych
ct al. 2019 Visani et al. 2020).
Given that our approach is inspired from IRM we now describe, how it is novel w.r.t. to it. It is important realize that IRM approaches such as (Ahuja et al. (2021 2020) are designed for the out-of-distribution (OOD) generalization, which learn global models directly from the data. The main similarity of these works to ours is only that they also are game theory based approaches, but with the details being quite different, For one, they assume accessibility to environments which (ideally) correspond to different interventional distributions and with assumptions on the structural causal model derive results on how the true causal factors could be divulged. In our case, we propose ways to generate environments as they are not given, and have l and los constraints on the entire and environment specific parts of the model respectively. which is not the case with these prior works. As such those algorithms do not produce sparse unidirectional models that are also consumable. Moreover, the perspective we provide is novel in the context of local posthoc explanations where priori it is not obvious that approaches from OOD generalization could be extended and adapted. Additionally, propose novel metric Unidirectionality which is not part of any of these works, but as we have argued it is a desirable property for explanations.
3 PRELIMINARIES
Invariant Risk Minimization: Given a collection of training datasets D = (De)cee, gathered 11 is the dataset gathered from environment from set of environments Esr, where D. {or y s=1
E Exy and ne is the number of points in environment e. The feature value for data point is a E X and the corresponding label is y E V. where X' R2 and y S R. Each point (et vt) in environment e is drawn i.i.d from a distribution Pe. Define a predictor f X' + R. The goal of IRM is to use these collection of datasets D to construct a predictor f that performs well across many unseen environments Eatl. where E.tl 2 &tr. Define the risk achieved by f in environment e as (f) = F. (x.), Ya)]. where is the square loss when f(X.) is the predicted value and Y, is the corresponding label, (Xe, Ye) ~ P. and the expectation E. is defined w.r.t. the distribution of points in environment e.
An invariant predictor is composed of two parts a representation & E dxd and predictor (with the constant term) w E Rdx1 We say that a data representation & elicits an invariant predictor w T o across the set of environments E, if there a predictor w that achieves the minimum risk for all the environments w E arg migu Re (w TT), Ve E Etr. IRM may be phrased as the following constrained optimization problem (Arjovsky et al. 2019):
min T #ERUX WERdx1
Σ Re b) s.t. w arg min Re (zo Re.TT)), Ve &ir eEE.
weRdx1
If wT T & solves the above, then it is invariant predictor across the training environments Eyy.
Nash Equilibrium (NE): To understand how certain key aspects of our method function let us revisit the notion of Nash Equilibrium (Dutta) 1999). A standard normal form game is written as tuple Ω = (N, { hev, (S,)ee)), where N is a finite set of players. Player é E A takes actions from a strategy set S.. The utility of player i is u, S + E, where write the joint set of actions of all the players as S = S, The joint strategy of all the players is given as s E S, the strategy of player i 8; and the strategy of the rest of players is s (8x) )x
3

---

Under review as a conference paper at ICLR 2023
Definition 1. A strategy st S is said be pure strategy Nash equilibrium (NE) if it satisfies, (s(8), sL,,) 2 uz (k: s s),), Ak E S Vi E N, where uz(8), 8 (.( 8 (x) uz(s)).
NE thus identifies a state where each player is using the best possible strategy in response to the rest of the players leaving incentive for any player to alter their strategy. In seminal work by (Debreu 1952) it was shown that for a special class of games called concave games such a pure NE always exists. This is relevant because the game implied by Algorithm falls in this category.
METHODOLOGY
We first define desirable properties we would like our explanation method to have. The first three have been seen in previous works, while the last Unidirectionality is something new we propose. We then describe our method where the goal is to explain a black-box model f X + R for individual inputs z based on predictors w by looking at their corresponding components, also termed as feature attributions. We take inspiration from IRM since, our goal here too is to extract robust features that are ideally stable and unidirectional.
4.1 DESIRABLE PROPERTIES
We now discuss certain properties we would like our explainability method to have in order to provide robust explanations that could potentially be used for recourse. Let D, denote (test) dataset with examples (2, y) where yb(x)) is the black-box models prediction on and y (ar) is the prediction on x (E X) using the explanation model at x'. The feature attributions (or coefficients) for the explanation model at x are denoted by cg, and N₂ denotes the exemplar neighborhood of with llland denoting cardinality.
Fidelity: This is the most standard property which all proxy model based explanation methods are evaluated against (Ribeiro et al. 2016)] Lundberg & Lee 2017 Lakkaraju et al.] 2020) as it measures how well the proxy model simulates the behavior of the black-box (a.k.a. faithfulness
to the black box) it is attempting to explain. Here we define inverse of it, that is Infidelity (INFD), as the MAE between the black-box and explanation model predictions across all the test points: INFD [Delem (x,))),, (x) - vo. (z)).
We also define another metric here called Generalized Infidelity (GI), which also been used in previous works (Ramamurthy et al. (020) to measure the generalizability of local explanations to neighboring
test points. is defined as: GI DDelani E(z.)eD
N2 card EVeNs (1)()) ye (z)).
Stability:
This is also a popular notion (Hancox-Li 2020 Ramamurthy et al. 2020 Yeh et al. 2019)
to evaluate robustness of explanations. Largely, stability can be measured at three levels. One is prediction stability, which measures how much the predictions of an explanation model change for the same example subject to different randomizations within the method or across close by examples. The second is the variance in the feature attributions again for the same or close by examples. It is good for a method to showcase stability w.r.t. both even though in many cases the latter might imply the former. An interesting third notion of stability is the correlation between the feature attributions of an explanation model and average feature values of examples belonging to a particular class. This measures how much does the explanation method pick features that important for the class, rather than spurious ones that seem important for the example of interest. Given this we define two stability metrics.
Coefficient Inconsistency (CI): This notion has been used before (Hancox-L.) 2020) to measure an explanation methods robustness. It can be defined as the MAE between the attributions of the test. points and their respective neighbors: CI
Dr lears Σ (a,))E,, IN E.'e,, 1- cg' |t.
Class-Atribution Consistency (CAC): For local explanations of classification black-boxes, we expect certain important features to be highlighted across most of the explanations of a class. This is codified by this metric which is defined as follows: CAC set of class labels in the dataset, My the mean (vector) of all inputs in class y E V. μa the mean explanation for class y and the Pearson's correlation coefficient.
Tyle Eyey r(u2, py), where y denotes the
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Locally Invariant EXplanations (LINEX) method.
Input:
example to explain æ. black-box predictor f(.,, number of environments to be created k.
(1.0) threshold 7 0, (4) threshold > 0 and convergence threshold >>
Initialize: vi E {1. {1,... k} wb; and A =
Let (i(), (x) be k environment creation functions described in section 4.2.2 do
for 1 t0 k do u _j j) w wheee w
w, arg min E 2
(f(i) u =x=z) s.t. |w0 21/2 S and |2ū) S
A max (/20, 2, A)
end
while
2 E
Output: w (ie(l W
Black-box Invariance: This is the same as implementation invariance defined in (Sundararajan et al. 2017). Essentially, if two models have exactly the same behavior on all inputs then their explanations should also be the same. Since, method is model agnostic with only query access to the model it is easy to see that satisfies this property the same environments are created.
Unidirectionality: This is a new property, but argue that this is natural one have. Loosely speaking, unidirectionality would measure how consistently the sign the predictor for a feature is maintained for the same close by examples by explanation method. This is natural metric (Miller 2018), which from an algorithmic recourse (Karimi et al.) 2021) perspective is also highly desirable. For instance, recommending a person to increase their salary to get loan and then recommending to another person with very similar profile to decrease their salary for the same outcome makes little sense.
We define the unidirectionality T as a measure of how consistent the sign of the attribution for particular feature in local explanation when varying neighborhoods for the same example when considering different close by examples. In particular, given m attributions for each of d features denoted by w (1) (d) the unidirectionality metric for example is:
r
sgn
md
where I.| stands for absolute value. Clearly, the more consistent the signs for the attribution of particular feature across m attributions the higher the value, where the maximum value can be one. If equal number of attributions have different signs for all features then Y will be zero, the lowest possible value. This property thus measures how intuitively consistent (ignoring magnitude) the explanations are. Given its sole focus on the sign of the attributions it compliments the above metrics along with attributional robustness metrics (Chen et al. 2019 Sarkar et al. 2021).
4.2 METHOD
4.2.1 DESCRIPTION
In Algorithm we show the steps of our method LINEX. The input to the method the example want explain æ, the black-box predictor, few thresholds that we describe next and & (local) environments whose creation is described in Section 4.2.2
In the algorithm we iteratively learn constrained least squares predictor for each environment, where the final (local) linear predictor is the sum of these individual predictors. In each iteration when com- puting the contribution of environment e; the final summed predictor, the most recent contributions of the other predictors are summed and the residual is optimized subject to the constraints. The first constraint is standard lasso type constraint which tries to keep the final predictor sparse as in LIME.
5

---

Under review as a conference paper at ICLR 2023
Why lo constraint? The second constraint is more unique and is a loc constraint the predictor of just the current environment. This constraint as we prove in Section 4.3] is essential for obtaining robust predictors. To intuitively understand why this is the case consider we have two environments. In this case if the optimal predictors for a feature in each environment have opposite signs, then the Nash equilibrium (NE) is when each predictor takes +7 or -?: values as they try force the sum to have the same sign as them. In other words, features that have a disagreement in even the direction of their impact are eliminated by our method. LIME type methods on the other hand would simply choose some form of average value of the predictors which may be a risky choice especially for actionability/recourse given that the directions change so abruptly. On the other hand, if the optimal predictors for feature in the two environments have the same sign, the lower absolute valued predictor would be chosen (assuming 7 is greater) making a careful choice. The reasoning for this and discussion involving more than two environments is given in Section
The overall algorithm resembles a (simultaneous) game where each environment is a player trying to find the best predictor for its environment given all the other predictors and constraints. Formally, for i {1, .l,, k- the players are N = ] their strategy space is S; = [-7, and their utility us (w,, w solved by each player is convex as norms are convex.
(f(Ø) w w. - = Also note that the optimization problem
E (x)
4.2.2 CREATING LOCAL ENVIRONMENTS
In the standard IRM framework environments are assumed to be given, however, in our case of local explainability we have to decide how to produce them. We offer few options for the environment creation functions& &, vi(1,.., k} in Algorithm |
This possibly is the simplest approach and similar what LIME employs. We could perturb the input example by adding zero mean gaussian noise to create the base environment (used by LIME) and then perform bootstrap sampling to create the k different environments. This will efficiently create neighbors in each environment, although they may be unrealistic in the sense that they could correspond to low probability points w.r.t. the underlying distribution.
Random Perturbation:
Realistic Generation/Selection: One could also create neighbors using data generators such as done in MeLIME (Botari et al., 2020) or select neighboring examples from the training set as done in MAPLE (Plumb et al., 2018) to create the base environment following which bootstrap sampling could be done to form the different environments. This approach may provide more realistic neighbors than the previous one, but may be much more computationally expensive.
Other than bootstrapping one could also over sample and try to find the optimal hard/soft partition through various clustering type objectives (Aggarwal & Reddy 2013) Creager et al., 2020).
4.3 THEORETICAL RESULTS
In this section, we analyze the output of Algorithm 1 when there are two environments. The extension to multiple environments is discussed following this result, where the general intuition is still maintained but some special cases arise depending whether there are an even or odd number of environments. To prove our main result we make two assumptions.
The feature values for each of the dimensions in the samples created forming the local environments are independent.
Assumption 1
This assumption is satisfied by the most standard way of creating neighhorhoodkeenvrnnneeii where random gaussian noise is used create them as described in Section 4.2.2]
Assumption 2 2 yd, where d is the dimensionality of the feature vector:
Making this assumption ensures that we closely analyze the role of the to penalty, which is one of the main novelties in our method.
Definition 2 Let the explanation that each environment & arrives at for an example xr based on unconstrained least squares minimization be wn where,
i arg min Exee w = WER4

---

Under review as a conference paper at ICLR 2023
The expectation is taken w.r.t the environment generation distribution.
Theorem 1. The output of Algorithm
7 under Assumptions 1, 2 and equation is given by:
w i
lej [ + uu @ 1 /x711//55)
(3)
where @ is element wise product and 1 is the indicator finction..
Proof Sketch. The above expression describes the NE of the game played between the two local environments each trying move towards their least squares optimal solution. Given assumptions I and 2, we witness the following behavior of our method. Let the ith feature of the predictors v0 and W2 from Algorithm be Wi; and W2; respectively. Let the corresponding least squares optimal predictors for the t feature have the following relation: wi; > 2 and |w > |- Then the two environments will push the ensemble predictor, W1; + W2g in opposite directions during their turns, with the first environment increasing its weight, Wii, and the second environment decreasing its weight, W2.. Eventually, the environment with a higher absolute value (5i 1 since |wi. > |w5;]] D reaches the boundary (vi1 7) and cannot move any further due to the loo constraint. The other environment &2 best responds, where it either hits the other end of the boundary (w²2 = -7), in which case the weight of the ensemble for component i is zero, a case which occurs if wi, and wa: have opposite signs; or gets close to the other boundary while staying in the interior (w's = w5: - 7), in which case the weight of the ensemble for feature is w situation which occurs if wf, and w have the same sign.
Implications of the Theorem I The following are the main takeaways from Theorem (1) If the signs of the explanations for unconstrained least squares for the two environments differ for some feature, then the algorithm outputs a zero as the attribution for that feature. (2) If the signs of the explanations for the two environments are the same, then the algorithm outputs the lesser magnitude of the two. These two properties are highly desirable from an algorithmic recourse or actionability perspective, where the first biases us not rely on features where the black-box function changes direction rapidly (unidirectionaaity). The second, provides a reserved estimate so that we do not incorrectly over rely on the particular feature (stability). Based on similar logic presented in the proof sketch the behavior for more than two environments for LINEX is discussed in Appendix C
5 EXPERIMENTS
We test our method on five real world datasets covering all three modalities: IRIS (Tabular) (Dheeru & Karra Taniskidou 2017), Medical Expenditure Panel Survey (Tabular) (Agency for Healthcare Re- search and Quality 2019). Fashion MNIST (Image) (Xiao et al. 2017), CIFARIO (Image) Krizhevsky (2009) and Rotten Tomatoes reviews (Text) (Pang et al. 2002) with LIME-like random (rand) and McLIME-like realistic neighborhood generation (real) MAPLE- like realistic neighborhood se- lection (mpl). The summary of black-box classifier accuracies, and type of realistic perturbation used for the datasets are provided in Table in the appendix. In other cases except FMNIST and CIFARIO which come with their own test partition we randomly split the datasets into 80/20% train/test partition and average results for the local explanations over this test partition. For LINEX we produce two environments where the two environments are formed by performing bootstrap sampling on the base environment which is created either by rand, real or mpl type neighborhood generation. Thus in all cases the union of the environments is the same as single neighborhood used to produce explanations for the competitors making a fair comparison. LINEX behavior with more environments is in Appendix F.
Given the neighborhood generation schemes we compare LINEX with LIME, Smoothed LIME (S-LIME), MeLIME and MAPLE, where for S-LIME we average the explanations of LIME across the LINEX environments. SHAP's results are in Appendix H since it is not a natural fit here. Nor are methods such as saliency maps, gradcam, integrated gradients etc. as they are white-box methods requiring access to a differentiable model.
Metrics: We evaluate using five simple metries: Infidelity (INFD), Generalized Infidelity (GI), Coefficient Inconsistency (CI), Class Attribution Consistency (CAC) and Unidirectionality (Y), which are defined in section 4. The first two evaluate faithfulness, the next two stability and the last goodness for recourse.
We report the above metrics in Table Each result in Table 2 is mean + standard error of the mean over five kerne sizes T v generally, where 7 {0.05, 0.1, 0.25, 0.5, 0.753. Test neighborhoods do
7

---

Under review as a conference paper at ICLR 2023
MeLIMEir LINEX:/ 0.633
[4c) Original
[d] MeLIME:r m 0,297
(el LINEXI- 0.962
(1) Original
14-
16
13 35
20
23
Figure 2: Sample results using FMNIST dataset for two classes. (a-c): Class Dress, (d-f): Class Sandal. (a, d): MeLIME explanations. (b, d): LINEX explanations. (c, f): Original images. We observe that LINEX explanations capture important artifacts and thus exhibit significantly higher correlation with the original images for the same level of sparsity, where in aggregate too the correlations are high w.r.t. images belonging to particular class, thus showcasing higher stability (i.e. high CAC) as is seen in Table 2 More examples are shown in Appendix G
Original
LINEX
MeLIME
Original
LINEX
MeLIME
1.0
0.8
Festure Attriletion
0.4
0.2
Sample results using CIFARIO dataset for dog and bird class. As can be seen LINEX focuses more on salient features such as head and legs for the dog, and wings for the bird (rather than also the background). More examples are shown in Appendix G
Figure 3:
not make sense for random perturbations with FMNIST, CIFAR10 and Rotten Tomatoes because the features (viz. superpixels) used by neighboring test examples are different. Also, do not use realistic perturbations with MEPS since KDE and VAE generators do not work well with categorical data. In addition, since MEPS data uses regression black-box, CAC cannot computed. Also for CIFARIC 10 images in class are not aligned so CAC is inapplicable. All these justify the missing entries in Table The results were generated on Linux machines with 56 cores and 242 GB RAM. More details regarding the exact perturbation schemes for LIME/MeLIME/MALLE the perturbation neighborhood sizes and the time taken by the different methods are in Appendices A and D
see that in terms of CAC, LINEX is better than baselines in all cases which indicates that on average the local explanations with LINEX highlight the important features characterzing the entire class making them more stable. This is further verified by looking at the and CI metrics where LINEX is similar or better than others. For GI and INFD metrics, the results are more evenly spread which implies that LINEX's main advantage is obtaining stable and unidirectional explanations that are faithful to similar degree. Ablation studies showing superiority of LINEX over McLIME on the FMNIST dataset where we have significantly higher INFD than
Observations:
Quantitatively,
MeLIME are given in Appendix I.
An interesting observation is that when comes the stability metrics (CI and CAC) and unidirec- tionality LINEX with even random perturbation model is better than MeLIME in some cases. This very promising as it means LINEX could be potentially be trusted without the need to generate realistic perturbations which may be computationally expensive or not even possible.
Qualitatively, we see in Figures 2 and 3, that LINEX explanations are more coherent and highlight more salient features compared to McLIME. Even the text data more reasonable attributions in Table where "masterpiece", "moving" and "audacious" are highlighted as the most important words indicative of positive sentiment in the three examples.
We also performed qualitative error analysis on FMNIST where our INED is much worse than McLIME and is described in Appendix ! We see that even where LINEX has high infidelity it invariably still focuses on salient features ignoring superfluous features that may not be critical for correct identification, but focusing which may result in lower infidelity for the specific example.
8

---

Under review as a conference paper at ICLR 2023
Table Below we see three example positive sentiment sentences from the Rotten Tomatoes dataset. Green and red indicate the most important word highlighted by MeLIME and LINEX respectively. As can be seen LINEX highlights stronger positive sentiment words. More examples Appendix
Example
Example
Example 3
moving tale of love and destruction in unexpected places, unexamined lives
one-of-a- mear-masterpiece
spare yet audacious
Table 2:
Comparison of the different methods based on infidelity (INFD), generalized infidelity (GI), coefficient inconsistency (CI), class attribution consistency (CAC) and unidirectionality (I). indicates higher value for the metric is better, and J. indicates lower is better. Statistically significant results based on paired t-test are bolded. LINEX is better than baselines in 21 out of 40 cases, and worse only in 5 cases. Plots showing behavior with varying neighborhood size, number of environments and kernel width are in Appendix
Dataset
Method
INFD 1
G
C
Tt
CAC f
LIME
0.015 + 0.011
0.132 1:22:00 0.042
0.319 = 0.132
0.646 - 0.040
0.667 0.167
S-LIME
0.015 0.010
0.077
0.143
0.878 0.007 0.033
LINEX/rand
0.013 0.009
0.052 : 0.008
0.044 + 0.013
0.802 0.921 0.043
0.042
IRIS
McLIME
0.008 + 0.003
0.049:#0000
0.
0.019
0.464 +.16440.100
LINEX/real
0.009 + 0.003
0.029 = 0.003
0.024 0.002
0.744 +
0.942
0.023
MAPLE
0.009 - 0.001
0.038 0.004
0.261 = 0.033
0.458 0.032
0.586
0.035
LINEX/mpl
0.013 + 0.000
0.020 = 0.000
0.026 = 0.002
0.694 - 0.008
0.929 0.004
LIME
0.158 0.066
0.214 + 0.041
0.005 + 0.001
0.981 0.006
S-LIME
0.158 + 0.066
0.214 + 0.042
0.005 + 0.001
0.974 + 0.008
NA
MEPS
LINEX/rand
0.130 1 0.052
0.164 + 0.021
0.003 + 0.001
0.979 1 0.006
MAPLE
0.063 0.000
0.067
0.000
0.007 1 0.000
0.957 0.000
NA
LINEX/mpl
0.098 0.001
0.094 0.001
0.007 1 0.000
0.950 0.000
LIME
0.162 0.003
S S-LIME 0.142 1 0.003
NA
NA
NA
NA
FMNIST
LINEX/rand
0.149 + 0.002
McLIME
0.001 + 0.000
0.277 = 0.000
0.007 = 0.000
0.769 = 0.000
0.327 - 0.000
LINEX/real
0.100 0.002
0.304 0.001
0.002 + 0.000
0.780 + 0.000
0.649 0.001
LIME
0.191 + 0.005
S-LIME
0.185 0.002
NA
NA
NA
NA
CIFARIO
LINEX/rand
0.186 0.002
MeLIME
0.100 1 0.003
0.412 1 0.007
0.014 = 0.000
0.546 + 0.003
NA
LINEX/real
0.090 1 0.005
0.279 1 0.001
0.006 + 0.000
0.679 + 0.004
LIME
0.079 + 0.036
S-LIME
0.075 0.035
NA
NA
NA
NA
Rotten
LINEX/rand
0.069 0.032
Tomatoes
McLIME
0.029 + 0.001
0.391 + 0.000
0.000 + 0.000
0.999 + 0.000
0.909 + 0.000
/INEX/real
0.053 + 0.000
0.361 + 0.000
0.000 + 0.000
1.000 + 0.000
0.953 + 0.001
The goodness of these features identified by LINEX can be further verified by looking at other metrics such as GI, CAC, CI and Y in Table where either comparable or better than MeLIME.
6 DISCUSSION
In this paper we have provided method based on game theoretic formulation and inspired by the invariant risk minimization principle to provide faithful, stable and unidirectional explanations. We have defined the latter property and argued that it somewhat of necessity (may not be sufficient) for recourse. We have theoretically shown that our method has strong tendency to be stable and unidirectional as we will mostly eliminate features where the black-box models gradient changes abruptly and in other cases choose conservative value. Empirically, we have verified this where we outperform competitors in majority of the cases on these metrics. An interesting observation is also that in some cases our method provides more stable and unidirectional explanations with just random perturbation model relative to more expensive methods that use realistic neighbors.
In the future, it would be worth experimenting with more varied strategies to form environments and possible find the optimal ones (Creager et al.) 2220),, which may lead to picking even more relevant features that are "causal" to the local decision.

---

Under review as a conference paper at ICLR 2023
ETHICS AND REPRODUCIBILITY STATEMENT
With the wide adoption of deep learning tcchnologies, explaining or understanding the reasons behind their decisions has become extremely important in many critical applications (Arya al. 2019).
Numerous explainability methods have been proposed in literature to explain individual decisions
of black-bor box models (Ribeiro al. 2016)] Plumb et al., 2018 Botari al., 2020 Dhurandhar
ett al. 2018a; 8a, Lundberg & Lee 2017). Although LINEX is more stable and undirectional than other competing approaches, it still is posthoe explainability method that may be completely faithful to the black-box model. This of course is not just a limitation of our approach, but nonetheless should be taken into account before a user makes a decision. Our method could also be used to divulge information by exposing the inner workings of the black-box leading to privacy concerns. One possible mitigation strategy this case would be to keep the sensitive attributes hidden from the explainer.
Experimental details are provided Section Sfo the main paper and Appendix D All datasets are public. Code will be provided during the discussion phase through anonymized link.
REFERENCES
Agency for Healthcare Research and Quality. Medical Expenditure Panel Survey (MEPS). https;
/ //meps.. , www..aao.....a.a.ooooooooooaoooo /mepsweb// 2019. Last accessed 2019 2019-08.
C. Aggarwal and C. Reddy. Data Clustering: Algorithms and Applications. CRC Press, 2013.
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization game. In International Conference on Muchine Leurning, 2020.
Kartik Ahuja, Karthikeyan Shanmugam, and Amit Dhurandhar. Linear regression games: Conver- gence guarantees to approximate out-of-distribution solutions. In AISTATS, 2021.
Elvio Amparore, Alan Perotti, and Paolo Bajardi. To trust or not to trust an explanation: using leaf to evaluate local linear xai methods. PeerJ Computer Science, 7:e479, 2021.
Christopher J. Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus- Muller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In Insl. Conference on Machine Learning (ICML), 2020.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 997.02299, 2019.
Vijay Arya, Rachel E. Bellamy, Pin- Yu Chen, Amit Dhurandhar, Michael Hind, Samuel Hoff- man, Stephamie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. One explanation does not fit all: A toolkit and taxonomy of Al explainability techniques. MiX::1909.030111 2019.
Tiago Botari, Frederik Hvilshoj, Rafael Izbicki, and Andre C. Meaningful local explanation for machine learning models, 2020.
de Carvalho. Melime:
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization. NeurIPS, 2019.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In e ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning, 2020.
Gerard Debreu. A social equilibrium existence theorem. Proceedings of the National Academy of Sciences, 38(10)8886 893, 1952.
Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository, 2017. URL http 7/archive. ics wcs.sci........................c edu/ml
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, pp. 592-603, 2018a.
10

---

Under review as a conference paper at ICLR 2023
Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, and Peder Olsen. Improving simple models with confidence profiles. In Advances in Neural Information Processing Systems, pp. 10296-10306, 2018b.
Amit Dhurandhar, Tejaswini Pedapati, Avinash Balakrishnan, Kartik Ahuja Pin-Y Yu Chen, Karthikeyan Shanmugam, and Ruchir Puri. Model agnostic contrastive explanations for structured 17, 2019.
data. https://aaxiv.org/ab///99//
Amit Dhurandhar, Karthikeyan Shanmugam, and Ronny Luss. Enhancing simple models by exploiting what they already know. In Intl. Conference on Machine Learning, 2020.
Prajit K. Dutta. Strategies and games: theory and practice. MIT Press, 1999.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3719-3728, Brussels, Belgium, Octoher-November 2018. Association for Computational Linguistics.
Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. NeurIPS, 2020.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile, 2019.
David Gunning. Explainable artificial intelligence (xai). In Defense Advanced Re-
search Projects Agency, 2017.
URL https://wwes www.w.dara..omm mi 1/program/
explainable-aute haw.haabbbe-attiiiiiaaa artificiaal www.iaaliineeiiienne....
Karthik Gurumoorthy, Amit Dhurandhar, Guillermo Cecchi, and Charu Aggarwal. Efficient data representation by selecting prototypes with importance weights. In Proceedings of the IEEE International Conference Data Mining, 2019.
Leif Hancox-Li. Robustness in machine learning explanations: does it matter? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, 2020.
Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances in Neural Information Processing Systems, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
Karimi, Bernhard Schölkopf. and Isabel Valera. Algorithmic recourse: from coun- terfactual explanations to interventions. In ACM conference on Fairness, Accountability and 2021.
Transparency (FAccT), 2021.
Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples are not enough, learn criticize! Criticism for interpretability. In In Advances of Neural Inf. Proc. Systems, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. ICML., 2020.
Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. The Irp toolbox for artificial neural networks. Journal of Machine Learning Research, 17 (114):1-5, 2016. URL http::/ /jmrrjmmr.r./..
618 html
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 2018.
Christoph Molnar. Interpretable machine learning, 2019. URL httptta://chiittppppptoto github io/interpretable-e w indexx html

---

Under review as a conference paper at ICLR 2023
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP, pp. 79-86, 2002.
Gregory Plumb, Denali Molitor, and Ameet Talwalkar. Model agnostic supervised local explanations. In Advances in Neural Information Processing Systems, pp. 2520-2529, 2018.
Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. Model agnostic multilevel explanations, 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 1135-11444 2016.
Yoel Roth and Nick Pickles. Updating approach
misleading information. URL httts:://blog..twttee..... http:///wwa.napppppp/ppop/tt////
2020.
updating our http:pppprpcatttttaaaaaaaaaaaaaa
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Mach. Intell., 155:200-2 May 2019.
Anindya Sarkar, Anirban Sarkar, and Vineeth N Balasubramanian. Enhanced regularizers for attributional robustness. AAAI, 2021.
Aditya Aniruddha Shrotri, Nina Narodytska, Alexey Ignatiev, Joao Marques-Silva, Kuldeep S. Meel, and Moshe Vardi. Constraint-driven explanations of black-box {ml} models, 2021.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv:1312.6034 2013.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICML, 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. NeurIPS, 2020.
Giorgio Visani, Enrico Bagli, Federico Chesani, Alessandro Poluzzi, and Davide Capuzzo. Sta- tistical stability indices for lime: obtaining reliable explanations for machine learning models. 1171 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rashion-mmist: novel image dataset for benchmarking machine learning algorithms, 2017.
Philip N. Yannella and Odia Kagan. Analysis: Article 29 working party guidelines on automated
decision making under gdpr. 2018. https//www.cybbnndiistbblggg 29 99-wworking-paae party- wwwpriiin-ooonnttmtteeeeeeeeeede
wwt.e/llloooooo//o11//oo com/201 8/0 /a0laalyyiis-articl wwwpriiioooiiiinuunnnnneeeeeeeee under--ddrrrrr
Chih-Kuaa Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. In Advances in Neural Information Processing Systems, pp. 10965-10976, 2019.
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S. Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. In International Conference on Learning Representations (ICLR), 2019a.
Yujia Zhang, Kuangyan Song. Yiming Sun, Sarah Tan, and Madeleine Udell. Why should you trust my explanation? ICML-Al Social Good, 2019b.
Xingyu Zhao, Wei Huang, Xiaowei Huang, Valentin Robu, and David Flynn. Baylime: Bayesian local interpretable -anel-agi explanations. UAI, 2021.
12

---

wi w If the optimal solutions to both environments the convex set [-7, 71 are the same, then in the first iteration itself where fit to the first environment we would have reached the optimal solution to our problem where w = wi. This is because in the second iteration where we fit the second environment to the residual from the previous fit w O and the algorithm would terminate. Under review as a conference paper at ICLR 2023
Table 3: Datasets, models and neighborhoods used in experiments. RF-+ Random Forest, NN-
Neural Network, ResNet-> Residual Network and NB- Naive Bayes.
Dataset
Modality
Black-bov model acc/Ht'.
Realistic neighborhood creation methods
IRIS
tabular
RF classifier, 93%
KDEGen (Botari et al. 2020). RF (Plumb
et al.
2018
MEPS FMNIST
tabular
RF regressor, 0.325
(Plumb et al.
2018
image
NN classifier, 87%
VAEGen (Botari
2020
CIFAR 10
image ResNet18, 95%
VAEGen (Botart ct al.
2020)
Rotten Tomatocs
text
NB classifier, 75%
Word2VeeGen (Botan cf al.
2020
A EFFICIENCY OF LINEX
It is important note that the query complexity (i.e. number of times we query the black box to obtain an explanation) of LINEX is the same as that of LIME since the union of the environments is the same LIME perturbation neighborhood. This is important in todays cloud-driven world where models may exist on different cloud platforms and posthoc explanations are an independent service where each call the model has an associated cost. In terms of running time for two environments, convergence was fast and running time was approximately 2.5 times that of LIME (LINEX took 2.5 seconds on IRIS for 30 examples as opposed to 1 second by LIME, LINEX took 47 seconds on MEPS for 500 examples as opposed to 18 seconds by LIME), which is very similar to Smoothed LIME (S-LIME) (took 2.3 seconds on IRIS and 40 seconds on MEPS) that we still outperform in majority of the cases.
Realistic neighborhood generation can be time consuming especially for McLIME since generators have to be trained which may take up an hour using single GPU for datasets such as FMNIST. After the generator is trained and neighborhood sampled MeLIME takes the same amount of time as LIME since the model fitting procedure is the same. MAPLE took 1.5 seconds for the IRIS dataset for 30 examples and 27 seconds for 500 MEPS examples.
A way to further speed up LINEX would be to implement it through embarrassing parallelism which can easily be done across explanations. This will prevent scaling of the running time in the number of examples when many explanations we would need efficiency because if only few explanations were desired the slightly higher running time of LINEX would not be an issue.
needed. The setting with many explanations is anyway where
B PROOF OF THEOREM
Expanding on the proof sketch provided in the main paper we now provide a case wise analysis to prove Theorem 1.
This would imply the output of algorithm would be w = wi.
i w³: When the optimal solutions for the two environments not equal consider the following two cases:
Opposite sign attributions: If the 1th component of unj and un have opposite signs, then the 7th components of the ensemble predictor, Wis and t2i are both at the boundary y and -y respectively w > O. This is because both try to push the ensemble (i.c. their sum) towards the sign they have where eventually they reach the boundary +-y and have no incentive to deviate. Any deviation from these values will lead to a higher least squares error in their environment, thus making this NE.
Same sign attributions: If the tth component of w and w have same signs, then the th component of ensemble predictor constructed from the NE is set to the least squares attribution with smaller absolute value, i.e., w; wio where |wiil | Without
13

---

Under review as a conference paper at ICLR 2023
loss of generality assume 0 < w9; < w2; the attribution of the environments' predictors in NE, then W1; and Wyy have opposite signs, ie., (يy = Y and w1, = wi: - - where the ensemble predictor for the 7th component would be ui = W1s + d2.. = wi, - - = wie since any deviation from this would lead to worse least squares loss for the corresponding environment. This shows that ensemble predictor is conservative and selects the smaller least squares attribution.
C BEHAVIOR FOR MORE THAN TWO ENVIRONMENTS
Given Assumptions and 2 we now discuss the behavior of our method for more than two environ- ments. If the number of environments is odd, then using similar logic to that discussed in the proof sketch one can see that the feature attribution would be equal to the median of the feature attributions across all the environments. Essentially, all environments with optimal least squares attributions above the median would be at +7, while those below it would be at -7 The one at the median would remain so with no incentive for any environment to alter its attribution making it a NE. This is a stable choice that is also likely to be faithful as we have no more information to decide otherwise. On the other hand if we have an even number of environments the final attribution in this case depends on the middle two environments in the same manner as the two environment case proved in Theorem I Thus, if the optimal least squares attributions of the middle two environments have opposite sign, then the final attribution is zero, else its the lower of the two attributions in terms of the numerical value. This happens because the NE for the other environments is +7 depending on if their optimal least squares attributions are above/below those of the middle two environments. This again is stable and likely to be faithful choice, where also unidirectionality is preferred.
D EXPERIMENTAL DETAILS
D.1 DATASET DETAILS AND HYPERPARAMETER SPECIFICATIONS
We describe the datasets and the hyperparameters used for each. We set perturbation neighborhood sizes 10 (IRIS). 500 (MEPS), 100 (FMNISS-randdmm, 500 (FMNIST-realliticc). 100 (CIFARIO- random), 500 (LIFARIO-realittic) 100 (Rotten tomatoes) for generating local explanations. We also use 3, 10, 10, 10, 5 as exemplar neighborhood sizes to compute GI, CI and Y metries for the five datasets respectively. We also use 5-sparse explanations for all cases except FMNIST and CIFARIO with realistic perturbations where we follow McLIME and generate a dense explanation using ridge penalty with penalty multiplier value of 0.001. The foo bound in Algorithm I is set as the maximum absolute value of linear coefficient computed by running LIME/MeLIME in the two individual environments. Please look IRIS dataset first since it contains some of the common details used across others.
IRIS (Tabular): This dataset has 150 instances with four numerical features representing the sepal and petal width and length in centimeters. The task is to classify instances of Iris flowers into three species: setosa, versicolor, and virginica. A random forest classifier was trained with train/test. split of 0.8/0.2 and yielded a test accuracy of 93%. We provide local explanations for the prediction probabilities for class setosa. For both random and realistic perturbations, we use perturbation neighborhood size of n. For random perturbations, we used the same approach followed by LIME and sample from Gaussian around each data point. Realistic perturbations (with the same number n) were generated using KDEGen Botari et al. (2020). a kernel density estimator (KDE) with the Gaussian kernel fitted on the training dataset to sample data around sample point. For both random and realistic perturbations, we weight the neighhorhood using Gaussian kernel of width r rvd, where d is the dimension of the feature vector and = {0.05, 0.1, 0.25. 0.5, 0.75), and this corresponded to kernel widths 10.1, 0.2, 0.5, 1.0, 1.5}. We also perform a weighted version of realistic selection where we use MAPLE Plumb et al. (2018) to assign weights to all the test examples and pick the top n weighted examples to use as the perturbation neighborhood. For random/realistie perturbations and realistic selection, the corresponding environments (of size n each) for LINEX are created by drawing k bootstrap samples where = {2, 3, 4, 5) in our experiments. We test for n = {10, 20, 30, 40, 50) with this dataset.
14

---

Under review as a conference paper at ICLR 2023
Medical Expenditure Panel Survey (Tabular): The Medical Expenditure Panel Survey (MEPS) dataset is produced by the US Department of Health and Human Services. It is a collection of surveys of families of individuals. medical providers, and employers across the country. We choose Panel 19 of the survey which consists of a cohort that started in 2014 and consisted of data collected over 5 rounds of interviews over 2014 2015. The outcome variable was composite utilization feature that quantified the total number of healthcare visits of a patient. The features used included demographic features, perceived health status, various diagnosis, limitations, and socioeconomic factors. We filter out records that had utilization (outcome) of O, and log- log-transformed the outcome for modeling. These pre-processing steps resulted in a dataset with 11136 examples and 32 categorical features. We train random forest regressor that has a test Rt² of 0.325 in this dataset. We provide local explanations of the predictions. With MEPS, we do not use realistic perturbations since KDE and VAE generators do not work well with categorical data. Otherwise the setting is similar as IRIS data, except that we use n = {50, 100, 200, 300, 400, 5003. The kernel widths in this case were {0.28, 0.57, 1.41, 2.83, 4.24]. We use k = {2, 3, 4, 5) for this dataset.
Fashion MNIST (Images): This dataset has 28 x 28 grayscale images of fashion articles with 60,000 train and 10,000 test samples. The task is to classify these into 10 classes corresponding to coat, shoe, and so on. A neural network trained with test accuracy of 87%. Explanations are generated for the prediction probabilities corresponding to the predicted class for each example. We choose 1000 test examples to generate explanations. Realistic perturbations were generated using VAEGen [Botari et al. (2020), Variational Auto Encoder (VAE) fitted on the training dataset. For random perturbations, we chose n from {50, 100, 200, 300, 400, 500} and kernel sizes were {0.43, 0.85, 2.14, 4.27, 6.41 J. For realistic perturbations chose n from (250, 500, 750, 1000} and the kernel widths were {1.4, 2.8, 7.0, 14.0, 21.03. We use & = {2, 4, 5) for this dataset.
CIFARIO (Images): This dataset has 32 x 32 colored images belonging to 10 different classes. The dataset has 50,000 train and 10,000 test samples. The task to classify these into 10 classes cor- responding to dog, bird, and so on. A residual network with 18 units (ResNet18) was trained with test accuracy of 95%. Explanations are generated for the prediction probabilities corresponding to the predicted class for each example. We choose 1000 test examples to generate explanations. Realistic perturbations were generated using VAEGen Botari al. (2020), Variational Auto Encoder (VAE) fit- ted on the training dataset. For random perturbations, we chose n from (50, 100, 200, 300, 400, 500} and kernel sizes were {0.43, 0.85, 2.14, 4.27, 6.413. For realistic perturbations we chose n from {250, 500, 750, 1000} and the kernel widths were {1.4, 2.8, 7.0, 14.0, 21.0}. We use k {2, 3, 4, for this dataset.
Rotten Tomatoes (Text): This dataset contains 10662 movie reviews from rotten tomatoes website along with their sentiment polarity, i.e., positive or negative reviews and the task is classify the sentiment of the reviews into positive or negative. The review sentences were vectorized using CountVectorizer and TädfTransformer and a sklearn Naive Bayes classifier was fitted on train- ing dataset which yielded a test accuracy of 75% Explanations are generated for the prediction probabilities corresponding to the predicted class for each example. Realistic perturbations were generated using Word2VecGers [Botari al. 2020), wherein word2vec embeddings are first trained using the training corpus and new sentences are generated by randomly replacing a sentence word whose distance in the embedding space lies within the radius of the neighbourhood. For both random and realistic perturbations, n was chosen from (25, 50, 75, 1003. The kernel sizes were (0.42, 1.06, 221 3.18) for random perturbations (kernel size 0.21 resulted in numerical issues), and (0.21, 0.42, 1.06, 2. 2, for realistic perturbations. We use l = {2,3, 4, 5) for this dataset.
E RESULTS WITH ALL DATASETS AND HYPERPARAMETER COMBINATIONS FOR RANDOM AND REALISTIC PERTURBATIONS
We present results with all hyperparameter combinations for random and realistic perturbations. Results for LIME with random perturbations (LIME), smoothed LIME (S-LIME), LINEX with random perturbations (LINEX/rand). MeLIME (MeLIME), LINEX with MeLIME-like realistic neighborhoods (eeNEX/rea),, MAPLE (MAPLE), LINEX with MAPLE-like realistic neighborhoods (LINEX/mpl) are presented in figures 5 19
The legend for these figures are given in Figure
15

---

Under review as a conference paper at ICLR 2023
For the five datasets, we perform ablations by varying one of perturbation neighborhood size (Figures
9.. number of environments (Figures T0 14), and kernel width (Figures 19). Each point in these figures are averaged over all possible values for the two parameters that are not ablated. For example, each point in Figure averaged over all possible values for kernel widths and number of environments for given perturbation neighborhood size. Standard emrors of the mean are also plotted in the same color with lesser opacity. Lower values of Infidelity (INFD), Generalized Infidelity (GI), Coefficient Inconsistency (CI) are better whereas for Unidirectionality (T) and Class Attribution Consistency (CAC) higher values are better.
Figures 5 show ablations with respect to perturbation neighborhood sizes. Considering all datasets, the stability/recours metrics (CI. T. CAC) are clearly better for LINEX compared to its counter- parts. For LINEX methods (IINEE/rannd, LINEX/real, L.INEX/mp)), the metrics get better or stays approximately the same generally as perturbation neighborhood size increases keeping with the intuition that larger perturbation neighborhood sizes should produce explanations that are more stable in the exemplar neighborhood. T for FMNIST and CIFARIO 10 are already good for small perturbation neighborhood sizes possibly because of the quality of MeLIME perturbations.
Turning to the fidelity metrics (INFD and GI) in tabular datasets, we see that the results still favor LINEX, but less heavily compared to the stability/recoouse metrics. This is in line with what we observe in T In IRIS and MEPS, LINEX is close to outperforms the corresponding baselines in the GI measure (except for I LINEX/mpl with MEPS). This gap closes a bit with INFD, but we note that GI is better measure since it estimates how faithful explanations are in a exemplar neighborhood. With the text dataset, LINEX variants are slightly more favored, whereas with the image dataset, the baselines have an edge.
Considering Figures 10 14 we see that variations are less stark with respect to number of environ- ments overall for LINEX variants. Note that except for S-LIME, other baselines do not use multiple environments, and hence stay constant. The slight variations in MAPLE are due to the effect of random seeds. In the stability/recourse metrics, again LINEX variants emerge as the clear winner across datssets. With the faithfulness metrics (GI and INFD), in the text dataset, LINEX variants generally perform better, whereas the baselines have better performance in the image dataset.
Finally, we study the variation of the performance measures with respect to kernel width in Figures We see that the stability/recourse metrics flatten out all cases with large kernel widths. This behaviour holds true for faithfulness metrics (GI and INFD) as well except in some cases. Gl and INFD measures also increase before they flatten out since the fit becomes poorer at larger kernel widths. The stability/recourse metrics become better remain approximately the same since explanations generally improve or preserve their stability properties as kernel widths increase. Note that very small kernel widths can lead to unexpected behavior that does not fit the trend as seen with the tabular datasets since explanations can become hyper-local. MAPLE and LINEX/mpl stay the same at different kernel widths since they use a different weighting scheme. As with other ablations, we see that LINEX variants are similar or better in stability/recourse metrics overall, while with the faithfulness metries the results are more mixed.
15 [9
Note that we do not compute MeLIME perturbations with MEPS since KDE and VAE generators do not work well with categorical data, and do not use compute CAC since the task is regression. Further, the features used in explanations for different test examples are not comparable for random perturbations with FMNIST, CIFAR10 and Rotten Tomatoes, hence we cannot compute CAC for those cases as well. This explains the missing curves/plots.
16

---

Under review as a conference paper at ICLR 2023
LIME
S-LIME
LINEX/rand
MeLime
LINEX/real
MAPLE
LINEX/mpl
Figure 4: Legend for figures 19
06822
8.00
20
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
(c) CIFARIO
Figure 5: Coefficient inconsistency (CI) vs. Perturbation neighborhood size.
1000
Bergherttedd
(a) Iris
(b) FMNIST
(c) Rotten Tomatoes
Figure 6: Class attribution consistency (CAC) vs. Perturbation neighborhood size.
8755
k779
050446
84 Reigborkeed abr
Weghberbao Eb4
Negtborhoc dbs
BeighbaTooo 604
(a) Iris
MEPS
FMNIST
(d) Rotten Tomatoes
835
654
(e) CIFARIO
Figure Unidirectionality (Y) vs. Perturbation neighborhood size.
17

---

Under review as a conference paper at ICLR 2023
3155
130
190
Seighbrfeed
20 303
40
337 433
Relybbuhood
09
90 3030
Heigherhood
53
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatocs
(e) CIFARIO
Figure 8: Generalized infidelity (GI) vs. Perturbation neighborhood size.
322
0.50
035
(a) Iris
(b) MEPS
(c) FMNIST (random)
(d) FMNIST (realistic)
0/20
(e) Rotten Tomatoes
(I) Rotten Tomatoes (realistic)
(g) CIFARIO
(random)
Figure 9: Infidelity (INFD) vs. Perturbation neighborhood size.
040220
04822
0005
25 24
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatocs
(e) CIFAR10
Figure 10: Coefficient inconsistency (CI) vs. Number of environments.
F EXAMPLE FEATURE ATTRIBUTIONS IN TEXT DATA: MELIME VS LINEX
Below sample attributions by the two methods along with the magnitude of the attributions. Attribution magnitudes are printed with a precision 10 - and shown along with the corresponding words in descending order.
18

---

Under review as a conference paper at ICLR 2023
6.50
1.0
Monkeraf 4.0
Hondor
(a) Iris
(h) FMNIST
(c) Rotten Tomatnes
Figure 11: Class attribution consistency (CAC) vs. Number of environments.
A75
03833
335
0.72
A776
3.79
272
3955
379
03097
Rundwr dnnnroo
Number derrronmenn
Number of anrrooeeee
(a) Iris
MEPS
FMNIST
(d) Rotten Tomatoes
035
(e) CIFARIO
Figure 12: Unidirectionality (Y) vs. Number of environments.
35 4.0
4.5
5.0
20
22
30
35
6.0
56
25 40
45
5.
25
30
35
40
56
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
632
625
Hondow
(e) CIFARIO
Figure 13: Generalized infidelity (GI) vs. Number of environments.
F.1 POSITIVE SENTIMENT
enticing and often funny documentary
MeLIME:
documentary funny and enticing often documentary funny often enticing and 0.517 0. 446 0.333 0.317 0.311
LINEX :
MeLIME:
LINEX :
0.416 0. 0.342 0.331 0.330
one-of-a-kind near-masterpiece MeLIME: kind near masterpiece masterpiece kind one 0.832 6.5 0.182 LINEX 0.212 0.384 0.381
LINEX :
MeLIME:
fast funny highly enjoyable movie
19

---

Under review as a conference paper at ICLR 2023
132
2.2
3003
3000
123
133
322
23
20
35
20
25
40
25
43
5.
25
20
25
60
52
(a) Iris
(b) MEPS
(c) FMNIST (random)
(d) FMNIST (realistic)
9695
0030
(e) Rotten Tomatoes
(I) Rotten Tomatoes
CIFARIO
(random)
(realistic)
Figure 14: Infidelity (INFD) vs. Number of environments.
3.00
038551
K77
332
KT7
3774
038532
577
03899)
Koreel
Semel mids
31.
with
20
25
20
(a) Iris
MEPS
(c) FMNIST
(d) Rotten Tomatoes
6.0
650
030
(c) CIFARIO
Figure 15: Unidirectionality (Y) vs. Kernel width.
3030
3.01
300e
Garedt
Kome with
(a) Iris
MEPS
FMNIST
Rotten Tomatoes
Caneel warth
CIFARIO
Figure 16: Coefficient inconsistency (CI) vs. Kernel width.
MeLIME:
enjoyable highly funny fast movie LINEX : enjoyable highly fast funny movie
MeLIME:
0.550 4.2 0. 412 0.389 0.198 LINEX 0.409 3.389 0. 3.2 3.350 3.326
ferrara's strongest and most touching movie of recent years.
20

---

Under review as a conference paper at ICLR 2023
133
25
10.8
2013
Gereet mid
wilts
(a) Iris
(h) FMNIST
(c) Rotten Tomatoes
Figure 17: Class attribution consistency (CAC) vs. Kernel width.
3.5
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
833
(e) CIFARIO
Figure 18: Generalized infidelity (GI) vs. Kernel width.
3.92
3.84
20
280
Sandl mids
Eerrec wits
Romel mids
(a) Iris
(b) MEPS
(c) FMNIST (random)
(d) FMNIST (realistic)
0.13
0.00
0.00
0.135
330
(e) Rotten Tomatoes
(i) Rotten Tomatoes (realistic)
(g) CIFARIO
(random)
Figure 19: Infidelity (INFD) vs. Kernel width.
MeLIME: touching years most strongest and touching most recent strongest and MeLIME:
LINEX :
0.735 4.9 0.450 0.443 0. 427 0.490 488 0.450 0.444 0.407
LINEX :
saved from being merely way-cool by a basic , credible compassion MeLIME: cool basic credible merely from LINEX: ccol credible merely compassion from
MeLIME: 1.514 0.0 0.040 0.9 0.026 LINEX 0.858 0.308 0.304 0.299 0.293
really quite funny
MeLIME: funny quite really
LINEX funny quite really
21

---

Under review as a conference paper at ICLR 2023
MeLIME: 0.559 0.417 0.233 0.462 0.368 0.275
LINEX :
spare yet audacious
MeLIME: spare yet audacious
LINEX :
audacious spare yet
MeLIME:
626 0.4 0.395 LINEX 0.501 0. 431 0.422
engrossing and infectiousie enthusiastic documentary MeLIME: engrossing documentary and enthusiastic an LINEX engrossing documentary an enthusiastic and
MeLIME: 0. 0.455 0.358 0.354 0.333
LINEX 0.461 0.407 0.374 0.357 0.350
wildly funny prison caper
MeLIME: funny caper wildly prison funny caper prison wildly MeLIME:
LINEX :
0.541 0. 0.214 0.193 403 0.335 0.245 0.239
LINEX :
this charming but slight tale has warmth wit and interesting characters compassionately portrayed MeLIME: charming compassionately and interesting portrayed
LINEX charming compassionately has tale portrayed
MeLIME: 0.690 0.507 0.456 0.444 0.424
LINEX 0.464 0.435 0.431 0.430 0.429
thoughtful provocative and entertaining MeLIME: fu entertaining and provocative
LINEX : thoughtful entertaining and provocative
MeLIME: 0. 612 0.517 0.402 0.395
LINEX 505 0.4 0.415 0.404
the film is quiet threatening and unforgettable MeLIME: quiet unforgettable and film the LINEX : unforgettable quiet film and is
MeLIME: 0.597 0.483 0.412 0.325 0.303 0.421 4.415 0.388 0.378 0.338
LINEX :
moving tale of love and destruction in unexpected places unexamined lives MeLIME: unexpected moving love tale lives moving unexpected places lives in
LINEX :
MeLIME:
0.692 0.662 0.577 0.538 0.499
LINEX :
0.538 0.530 0.521 0.513 0.501
though frodo's quest remains unfulfilled determined new zealanders has proved its creative mettle
hardy group of
MeLIME: creative group proved has new
LINEX : creative quest its proved determined
MeLIME: 0.602 4.1 0. 424 0.402 0.393
LINEX :
0.410 0.392 0. 390 0.385 0.381
F.2 NEGATIVE SENTIMENT
originality is sorely lacking
MeLIME:
lacking sorely is originality
LINEX : lacking sorely originality is
MeLIME: 0.543 0.381 0.296 0.278
22

---

Under review as a conference paper at ICLR 2023
LINEX 0.430 3.356 0.314 0.271
ugly pointless stupid movie
MeLIME: stupid point.less ugly movie an LINEX stupid pointless ugly movie an MeLIME: 0.543 0.499 0.385 0.365 0.276
LINEX 0.446 0.411 0.373 0.360 0.350
so devoid of pleasure or sensuality that cannot even dubbed hedonistic MeLIME:
devoid even be dubbed of devoid so dubbed be cannot
LINEX :
MeLIME:
0.666 0.416 0.413 0.372 0.344 0.400 0.392 0.387 0.380 0.368
LINEX :
neither revelatory nor truly edgy--merely crassly flamboyant and comedically labored
MeLIME: edgy neither nor labored revelatory
LINEX :
edgy neither nor labored truly
MeLIME:
1.256 0.338 0.277 0.204 0.021 0.439 0.398 0.398 0.369 0.349
LINEX :
occasionally funny sometimes inspiring often boring. MeLIME: boring occasionally inspiring sometimes often boring occasionally sometimes often inspiring
LINEX :
MeLIME: 0.669 0.242 0.218 0.210 0.182
LINEX :
0.377 0.265 0.266 0.250 0.236
cumbersome and cliche-ridden movie greased with every emotional device known to cliche every device movie with cliche every man cumbersome emotional
MeLIME:
LINEX :
MeLIME: 0. 0.449 0.327 0.280 0.268
LINEX :
3.5 0.361 0.354 0.349 0.309
ponderous plodding soap opera disguised as a feature film. MeLIME: plodding soap ponderous opera disguised plodding soap film ponderous feature MeLIME: 0.579 0.522 0.421 0.408 0.382
LINEX :
LINEX :
0.442 4.4 0.418 0.406 0.377
kitschy flashy overlong soap opera
MeLIME: soap flashy opera overlong kitschy soap flashy opera overlong kitschy MeLIME:
LINEX :
0.499 0.397 0.391 0.358 0.230 0.389 0.362 0.360 0.346 0.300
LINEX :
[a] poorly executed comedy MeLIME: poorly comedy executed poorly conedy executed
LINEX :
MeLIME: 0. 653 0.348 0.257
LINEX :
0. 0.3 0.309
bad movie that happened good actors MeLTME: bad happened movie to that
LTNEX : bad happened to movie acters
MeLIME: 0. 692 0.396 0.371 0.367 0.242 0.442 0.384 0.367 0.361 0.344
LINEX :
complete waste of time
23

---

Under review as a conference paper at ICLR 2023
MeLIME: waste complete time of waste complete time of
LINEX :
MeLIME: 0.614 0.425 0.313 0.247 0.480 0. 381 0.348 0.278
LINEX :
don't t waste your money MeLIME: waste money don your
LINEX :
waste money don your
MeLIME: 0.592 0.497 0.408 0.309
LINEX :
0.483 0.450 0.411 0.337
witless and utterly pointless
MeLIME: pointless witless and utterly
LINEX pointless witless utterly and
MeLIME: 0.6 0.491 0.263 0.245
LINEX 0.506 0.444 0.311 0.269
G EXAMPLE FEATURE ATTRIBUTIONS IN IMAGE DATA: MELIME VS LINEX
We show feature attributions for individual example images with McLIME and LINEX with MeLIME perturbations in Figure 20 In Figure 21 we show class-wise mean feature attributions along with mean images. Figure 22 we see examples from CIFARTO. LINEX explanations seem provide more mcaningful feature attributions.
H RESULTS FOR ALL METHODS INCLUDING SHAP
In Table 4 we provide the results for SHAP along with all methods for easy comparison. Note that SHAP does not have standard errors since it is computed only once per test point. The INFD values for SHAP
miniscule since SHAP values add up the predictions by definition. In order to compute GI, CI. Y, CAC, we convert the SHAP values to SHAP attributions Amparore et al. 20021 first and follow the same approach used by other explanation methods.
I ERROR ANALYSIS OF LINEX
We perform error analysis for LINEX to gain better understanding about the method. We choose FMNIST dataset for doing this since, LINEX/real under performs MeLIME in terms of the INFD measure here (see Table more heavily compared to other datasets and so we wanted to investigate the reasons for this. This also happens to one of the higher dimensional datasets that is intuitive visualize and understand.
We start by observing that even though LINEX/real underperforms in the INFD metric, the gap not so great in the GI metric, which suggests that McLIME may be overfitting explanations here. We also note that in terms of CI, and CAC metrics, LINEX/real clearly outperforms McLIME.
We now choose a sample of images from the dataset where LINEX/real has highest instance- level infidelity numbers and display them in Figure 23 Just looking at the explanations and the corresponding original images visually, it is evident that LINEX/real highlights the prominent features like sleeves and collar shirt, handles of the bags, outlines of the boots/shoes, even though the infidelity values are high. However, MeLIME misses out on some of these prominent features and focuses only on optimizing the local fit. The fact that LINEX zeroes in on important features also provides additional evidence for the closeness of GI metries between the two methods, and the better performance of LINEX/real with CI, Y, and CAC metrics.
This conclusion is also verified when we look at the performance of LINEX at class level. In Figure we see two classes one where the infidelity of LINEX is low (i.e. Trousers class) and the other where its infidelity is high (i.e Shirt class). As can be seen since the Trousers class has examples with less superfluous features (viz. varied designs) focusing on which might reduce infidelity but are not critical for determination of the class, LINEX does better in terms of infidelity the prior. However,
24
24

---

Under review as a conference paper at ICLR 2023
Original
[2a] MeLIMEY (26)
(2c) Original
28-
25
27-
(3a) ELIIME 0.259 (3b) LINEX:r 0.711
(3c) Original
(4a) MeLIME/ 0.292 (4b) LINEX:r - 0.633
(4c) Original
13-
18-
16
15
26
20
(5a)
0.271
(5b)
Original
(6a)
(6b)
(6c) Original
20
(7a) McLIME:r= 0.482 (7b) LINEX = 0.820
(7c) Original
(Ba) McLIME: 0.177 (8b) LINE XYX 0.788
(Bc) Original
15-
10-
(9a) : 0.247 (9b)
(9c) Original
(10a) MeLIME 0.381(1bb) LINEXI 0 0.792
(10c) Original
35-
29
28
20
10
23
Figure 20: Results using individual samples for realistic perturbations for FMNIST dataset for all classes:1-10 (T-shirt/top. Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag and Ankle boot). (a) MeLIME feature attributions for image. (b) LINEX feature attributions for an image. (c) Original image the class. The values show Pearson's correlation between feature attributions and the original image from the respective class. We observe that LINEX attributtinn//epllaaatton exhibit significantly higher correlation with the original image belonging to particular class (i.c.
high CAC).
although infidelity is higher for the latter Shirt class it does much better other metrics such GI, CAC, CI and Y indicating that LINEX truly focuses on robust features.
J ABLATION ANALYSIS OF IMPORTANT FEATURES FOR VARIOUS EXPLANATION METHODS
We wanted to analyze the most challenging case for us in the reported experiments which is on the FMNIST dataset where we are more worse than MeLIME in terms INFD than any of the other setups. We thus assess if the features deemed important- those with the largest coefficients by the explanation methods are indeed important for the black box model to make their predictions. To assess this, we set the set a fraction of features (pixel values) corresponding the top coefficients of McLIME and LeNEX/realistic to a baseline value and run the modified images again through the black box model this is what mean by ablation here. The baseline value here was chosen to be -1 since that is the value of the background pixels. We then used two measures to assess the quality of explanations higher values being better for both. The first measure mean absolute error
25

---

Under review as a conference paper at ICLR 2023
(1a) (1b)
Original
(2a)
(2c) Original
25
25
(3a) -
[3b] LINEX//00611
(3c) Original
(4a) MeLIME:/-0A41
(4b) LINEX:/=0.860
(4c) Original
13
18-
16
15
20
25
20
20
(5a) (5b)
Original
(6a)
(6c) Original
20
20
(7a) Me.
(7b)
(7c) Original
(Ba) MeLIME:r=0.443
(Bb) LINEX:=0..a
(Bc) Original
15-
10-
10-
11-
15
24
(9a) MeLIME:/=0.372
(9b) LINEX>=0.751
(9c) Original
(10a) MeLIME:r= 0.1 (10b) LINEX:r=0.851
(10c) Original
23
23
28
22
23
Figure 21: Results using realistic perturbations for FMNIST dataset with mean feature importances for all classes:1-10 (T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag and Ankle boot). (a) Mean feature attributions of all images in the class using MeLIME. (b) Mean feature attributions of images in the class using LINEX. (c) Mean all images show Pearson's 's correlation between average feature attributions and mean of the original images from the respective classes. We observe that LINEX explanations//ttribtiio
class. The r values
exhibit significantly higher correlation with the original images belonging to a particular class (i.c. high CAC).
between the predicted scores before and after ablation, corresponding to the original predicted class. The second measure is the fraction of images that changed their predicted class after ablation. We see from Figure 25 that LINEX/realistic substantially outperforms MeLIME in both these measures, clearly demonstrating the relevance of features chosen by our method to the black box.
K ERROR ANALYSIS OF LINEX BASED ON ABLATION
Highlighting stable features for examples near con-linearities is a key strength of LINEX. However, in some cases for examples near class boundaries it may ignore sensitive features show in this demonstration.
In Figure 26 we show 6 examples that appear to be close to class boundaries. We ablate pixels corresponding to top 15% of important features chosen by MeLIME and LINEX/realistic using the approach discussed in Section Ablation based MeLIME importances meaningfully changes classes, whereas ablation by LINEX importances does not. The changes in prediction McLIME ablation for the six images respectively from Dress Trouser, Sneaker to Sandal, Pullover
26

---

Under review as a conference paper at ICLR 2023
Original
LINEX
MeLIME
Original
LINEX
MeLIME
1.0
Original
LINEX
MeLIME
Original
LINEX
MeLIME
0.8
Feature Attribution
0.6
0.4
Original
LINEX
MeLIME
Original
LINEX
MeLIME
0.2
0.0
Figure 22: Results using realistic perturbations for CIFARIO dataset. We above images of dog, a horse, a truck, a bird, boat and a dog again randomly selected from CIFARIO. The original images greyed out here that the (normalized) attributions attributions seem to consistently focus on salient features as compared to MeLIME. For example for the first dog image we highlight the head, ears and leg, while MeLIME focuses more on the neck and some of the background. For horse too LINEX focuses on head and body, while MeLIME focuses on the legs and neck. For truck both seem to focus on important features. For bird LINEX hones in on the wings, while MeLIME although giving importance to wings also attributes some of the background. The boat image LINEX focuses on the center of the boat, while Melime on the edges and some of the water around the boat. For the dog face image LINEX focuses on the nose, eyes and ears, while Melime focuses on the ears and neck.
clearly visible. As be seen LINEX
Dress, Sneaker to Sandal, Bag to Pullover, and Sneaker to Sandal. The new class assignment looks reasonable looking at the ablated images. We also that the changes in class probabilities for the original class (p) are much higher after MeLIME ablation compared to LINEX/realistic ablation.
MeLIME ablated images for the first example has structures that look like trouser legs, for the second, fourth and sixth examples the area around the heel is more open making the original sneaker look like sandal, for the third example, there hole in the hooded part of the pullover making it resemble dress. The fifth example is classified as pullover possibly because of the elongated structures on the sides that look like hands.
Note that such cases of LINEX under performing are rare though as is confirmed by its superior performance in Figure
L UNDERSTANDING BEHAVIOR OF LIME AND LINEX WITH SYNTHETIC DATA
We consider explaining the behavior of a function of two variables x and y with Class sandwiched between Class 0 (sce Figure 27). The third (or vertical) axis denotes the probability of being in Class Clearly, æ is the only important feature here that determines the class label.
From Figure 27 (left), we see that the LIME (here MeLIME would be the same as LIME since the space is flat and all points are realistic) feature attributions at points a. b, and will provide importance to feature for small well large kernel width (1 and 2 respectively) neighborhoods. For point c, in the interior of the Class 0 the attributions are stable across kernel widths. However for points and close the boundary of classes, the attributions for small kernel width and large
27

---

Under review as a conference paper at ICLR 2023
Table Comparing the different methods (including SHAP) using metrics infidelity (INFD), gen- eralized infidelity (GI), coefficient inconsistency (CT), class attribution consistency (CAC) and unidirectionality (T).
Dataset
Method
INFD
G
C
Tt
CAC
LIME
0.015 1 0.011
0.132 0.042
0.319 0.132
0.646 0.040
0.667 0.167
S-LIME
0.015 + 0.010
+007=
0.143
0.704 0.007
0.878 = 0.034
LINEX/rand
4.033
0.052 0.008
0.044
0.802
0.921 1:21=0000 0.042
NB/rand
0.040 0.010
0.067 + 0.003
0.319 = 0.132
0.646 0.040
0.667 + 0.167
IRIS
MeLIME
0.008 0.003
0.049 0.018
0.219 0.108
0.629 1.011
0,464 0.100
LINEX/real
0.009 + 0.003
0.029 0.003
0.024 0.002
1
0.942
0.023
NB/real
0.058 0.022
0.034 + 0.000
0.219 + 0.108
0.629 0.013
0.464
0.100
MAPLE
0.009 = 0.001
0.038 = 0.004
0.261 = 0.033
U.458 = 0.032
0.586
0.035
LINEX/mpl
0.013 + 0,000
0.020 0.000
0.026 = 0.002
0.694 = 0.008
0.929 0.004
SHAP
0.007
0.197
0.248
0.664
0.524
LIME
0.158 + 0.066
0.214 0.041
0.005 = 0.001
0.981 = 0.006
S-LIME
0.158 0.066
0.214 0.042
0.005 = 0.001
0.974 0.008
NA
LINEX/rand
0.130 0.052
0.164 + 0.021
0.003 + 0.001
0.979
MEPS
NB/rand
0.275 + 0.062
0.311 + 0.079
0.005 + 0.001
0.981 + 0.006
MAPLE
0.063 + 0.000
0.067 + 0.000
0.007 + 0.000
0.957 = 0.000
NA
LINEX/mpl
0.098 0.001
0.094 = 0.001
0.007 = 0.000
0.950 = 0.000
SHAP
0.000
0.091
0.009
0.940
LIME
0.162 0.003
S-LIME
0.142 0.003
FMNIST
NA
NA
NA
NA
LINEX/rand
0.149 + 0.002
NB/rand
0.207 + 0.000
McLIME
0.001 0.000
0.277
0.000
0.007 + 0.000
0.769
0.000
0.327
0.000
LINEX/real
0.100
0.002
0.304 + 0.001
0.002 + 0.000
0.780
0.000
0.649
0.001
NB/real
0.017 0.000
0.446 + 0.000
0.007 = 0.000
0.769 + 0.000
0.327 + 0.000
SHAP
0.000
1.962
0.589
0.551
0.038
LIME
0.191 + 0.005
S-LIME
0.185 + 0.002
NA
NA
NA
NA
CIFARIO
LINEX/rand
0.186 0.002
NB/rand
0.208 0.001
McLIME
0.100 + 0.003
0.412 + 0.007
0.014 + 0.000
0.546 + 0.003
LINEX/real
0.090 + 0.005
0.279 + 0.001
0.006 + 0.000
0.679 + 0.004
NA
NB/real
0.103 0.002
0.004 0.398
0.014 0.000
0.546 + 0.003
SHAP
0.003
1.376
0.398
0.512
LIME
0.079 0.036
Rotten
S-LIME
0.075 0.035
NA
NA
NA
NA
Tomatoes
LINEX/rand
0.069 + 0.032
NB/rand
0.241 0.007
McLIME
0.029 0.001
0.391
0.000
0.000 + 0.000
0.999
0.000
0.909
0.000
LINEX/real
0.053 0.000
0.361
+ 0.000
0.000 + 0.000
1.000
0.000
0.953
0.001
NB/rcal
0.035 + 0.000
0.535 + 0.000
0.000 + 0.000
0.999 + 0.000
0.909 0.000
SHAP
0.000
0.384
0.008
0.999
0.015
kernel width neighborhoods differ significantly along the direction. This shows the instability of LIME explanations near boundaries of classes for different kernel widths.
In contrast in Figure 27 (right), see that the LINEX explanation constructed for the two kernel
widths provides stable feature attributions for all points a, b, c. For a and b, LINEX will conservatively pick a smaller feature attribution along the direction since the function changes rapidly in its neighborhood. As such though LINEX will still pick the feature in the r direction in this scenario.
M VARIATION OF FEATURE ATTRIBUTIONS WITH
Based the proof of Theorem if for feature the optimal attributions have opposite sign for each of the two environments, then 7 can be made arbitrarily small (except 0) or large and the output of Algorithm should still be the same which is 0 as the Nash Equilibrium is t If the optimal
28

---

Under review as a conference paper at ICLR 2023
attributions the same sign then we should still get the same output from Algorithm | as long as y 2 min([wn, (2zil) since the attribution from our algorithm is the minimum of those values. When y unn([wyd (u2d]) then the feature attributions will smoothly reduce
reduces.
We demonstrate this behavior in Figure 28 using an example from the IRIS dataset with random perturbations using the same setting in Section In the experiments in Section 5 set 0.329 which is the maximum absolute value based on a linear fit to each environment. As increases beyond 0.329, the attributions are unchanged demonstrating robustness, Same holds true while reducing ? up to 0.165 beyond which we see smooth reduction in the attribution values. Qualitatively, similar behavior seen for other examples too. Because we set y pessimistically (ignoring constraints) to high value, we can expect our reported performances in the paper to be robust across many values of y.
N CONVERGENCE OF LINEX PROCEDURE AND COMPARISONS
We demonstrate based on a synthetic example how Algorithm D and provides a unidirectional explanation. We generate synthetic data using function in 8 (Figure 29 [efft). The function gently rises with increasing y values, and along z it is flat first, then rises abruptly and then falls gradually. We want obtain robust attributions of this function at the point 1.0. y 0.0, which is close to the end of the rising edge along x direction.
As can imagine, since the slope changes abruptly along direction near the point, it should be ideally excluded from an explanation intended towards recourse based on a linear proxy. Otherwise, the explanation will not generalize in the neighborhood of this point. On the other hand, the direction should be included since the function changes smoothly along y throughout.
To generate explanations We first create environments centered the example explain with variances 0.5 and 2.0. Now independently fitting to these environments leads to feature attributions that are 1-0.033, 0.098} and {0.084, 0.102). Appending the two environments the attributions are {0.029, 0.095]. whereas with LINEX, the attributions would be {0.0,0.093) Thus, LINEX effectively eliminates the feature with high variability or abrupt changes. The behavior of the coefficients for each environment as LINEX converges is shown in Figure 29 right). As such, one can also see the convergence is fast.
O LIMITATIONS
Like any other posthoe explainable AI method there is no way to surely say that LINEX exactly reflects the true reasoning behind black box classifier in arbitrary applications. It also is somewhat slower than LIME as shown section given the game theoretic nature of the algorithm, where its stability and unidirectionality hopefully offsets the additional time required. On the flip side, given its favorable properties in terms of recovering explanations could be used to violate privacy which may be concerning from social standpoint.
29

---

Under review as a conference paper at ICLR 2023
MoLime: INFD=0.001 INFD-0.169
Original
MeLime: r=0.093, INF D=0.001
LNEX/rea:: r=0.338, INFD=0.216
Original
MeLime: r=0.112, INFD=0.00) 001 LNEX/real:: r=0.539, INFD=0.201
Original
MeLime: r10. NFD=0.001 UNEX/real: r=0.412, INFD-0.189
Original
MoLime: r-0.393, INDD00.001 LINE/rraal: r-0.713 INFD-0.180
Original
IND=0.601 LNEE/rall: r-0.528, INFD-0.161
Original
Figure 23: Error analysis for a chosen set of examples in FMNIST using MeLIME and LINEX/real methods. The three columns the MeLIME feature attributions, LINEX/real feature attributions, and the original images. The rows correspond to different examples. We show the Pearson's correlation coefficient between feature attributions and mean of the original images from the respective classes (r) and instance-level infidelity (INFD) measures. LINEX seems to highlight important features like stripes the t-shirt, handles of the bags, outlines of the boots/shoes more prominently, while MeE seems to overfit to the data while missing out on highlighting some key features prominently.
30

---

Under review as a conference paper at ICLR 2023
Trouser class less infidelity (INFD)
0
0
Shirt class more infidelity (INFD)
Figure 24: We see above that infidelity is lower for Trousers class for LINEX compared with the Shirts class. A reason for this is that the trousers are more plain with less superfluous features such as the different designs shirts. Since LINEX focuses on robust features focusing excessively on the designs is not critical for it to determine shirt, albeit focusing
these designs might reduce infidelity. Advantage of it relying on robust features is however apparent when we look other metrics such GI, CAC, CI and Y seen in Table 2 where it is much closer to superior to McLIME.
Mean absolute error
ablated pred. probs)
LIEE//rea
MeLIME
Average predicted class changes
0.5
LINEX/real
0.4
0.4
(original
0.2
0.1
0.1
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Fraction important features ablated
Fraction of important features ablated
Figure 25: Ablation analysis to determine the features deemed important by the explanation methods are actually considered important for prediction by the black box model. We see that features chosen by LINEX impact the prediction of the black box model much more than those chosen by MeLIME, This is true with respect both MAE measure (left) between
predicted probabilities before and after ablation for winning (or argmax) class, and the change in predicted classes (right) before and after ablation. Higher values here mean that the features chosen by the explanations are more relevant for the black box to make its predictions. The maximum value of both measures is 1.0.
31

---

Under review as a conference paper at ICLR 2023
VeLime Coofta
UIFXiea Coafi
Original 2-4.9967. ds.
KcJME 1bite
p-0.1472
UNEKecol abite:
p-0 p-0.7400 dz
UNExireal Cootts
Original pmo 9oDu,
ailate jmn) st6,,
UNiXeeal pmill
Coetts
9105,
Me.MM 343
UNEXeeal
Volime Cootss
LIFEXted Coelis
Orighal p-0.9965 ds.
KeJME ablote p-0.4782.
UNEKrcal abite 9862. di
peil
Vel me Coetts
Unitmraa Colffs
Onginal p:0 5860, ds
MIJME state p:0 2787,
UNEkreal abobe p=0 5850, ds.
Figure 26: Error analysis for a chosen set of examples in FMNIST using McLIME and LINEX/realittic methods, using ablation of important features. Each row shows results for particular image. The columns show the: (a) MeLIME coefficients, (b) LINEX/realistic coefficients, (c) the original image along with its predicted class (cls.) and predicted probability for that class (p), (d) the image after MeLIME ablation along with the predicted probability for the original class (p) and the new class prediction (cls.), and the image after LINEX/realistic ablation along with the predicted probability for the original class (p) and the new class prediction (cls.). The changes in prediction for MeLIME ablation for the six images are respectively from Dress to Trouser, Sneaker to Sandal. Pullover to Dress, Sneaker to Sandal, Bag to Pullover, and Sneaker to Sandal. No changes in classes are seen for LINEX ablation.
32

---

Under review as a conference paper at ICLR 2023
Class
Class
0.-
0.4
2,4
0.2
0.2
a
a
Class
Class
Class 0
Class 0
Figure 27: LIME (left) and LINEX (right) feature attributions for three points (a, b, c) for synthetic data where have Class sandwiched between Class O. For LIME, the different colors pink and blue correspond to feature attributions obtained with the small and large kernel width neighborhoods. Note how explanations for LIME change significantly (in magnitude) by kernel widths near the class boundaries, whereas the LINEX explanation remains stable, where it still picks up the important feature.
0.2
Wo
feature attribution
W1
W2
W3
0.0
-0.1
-0.2
0.1
0.2
0.3
0.5
0.6
Figure 28: Feature attributions for the four features for an example in the IRIS dataset are shown above when varying y. We used the same setting as in Section[5foo this experiment. The attributions increase smoothly as y increases and stay constant after 7 2 min[[wrd, [Vii))Vi.
33

---

Under review as a conference paper at ICLR 2023
Environment coefficients
0.3
Wi.o
0.2
W1.1
1.0
W2.0
0.1
0.0
y
0.0
-0.
0.3
120
Iterations
Figure 29: Left side: Explaining scalar function R22 the point indicated by the triangle. The point is centered at r 1.0, y = 0.0. The two environments created by sampling multivariate normals with variances 0.5 and 2.0 respectively (samples not shown) centered at this point. Right side: Convergence of individual environment attributions. The attributions for first feature (z), 11o0 and W2.0, converge to y and -y leading to the optimal attribution 0. For the second feature (y) the optimal attribution (W1,1 + 12,1) converges positive value.
34


Reviews:
[{"review_id": "Pn64nFMHu9r", "reviewer": "Reviewer_uJGF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The article describes a black box perturbation based explainability approach. It introduces an evolution of the LIME algorithm where the linear local predictor used to identify feature attribution is estimated using an extra $l_\\infty$ constraint. The algorithm is justified by a Game Theory interpretation and inspired by an Invariant Risk Minimization approach. In addition to fidelity and stability, it proposes a new evaluation metric for explanation based on computing the sign consistency of attribution values and called unidirectionality. The approach is evaluated on a series of small size datasets containing text, tabular and image data. A large appendix (20 pages) with complementary experiments  is added to the main paper.\n", "summary_of_the_review": "The paper describes a well motivated evolution of the classical LIME approach, with a thorough experimental analysis but on small size data. The novelty is incremental but gives good performance. The theoretical justification of the algorithm is interesting  but should be improved.", "strength_and_weaknesses": "== Strengths ==\n\n- Simple but efficient evolution of  LIME algorithm for explainability\n\n- Thorough evaluation and good results on five small size datasets for five metrics\n\n== Weaknesses ==\n\n- The novelty is incremental (an evolution of LIME)\n\n- The theoretical justification of the algorithm is interesting but should be improved.\n\n- Evaluation limited to small dimension data.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Oidv5gr8aG", "reviewer": "Reviewer_8tif", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new explanation device, i.e. LINEX, that requires only black-box access to the target model, motivated by Invariant Risk Minimization (IRM). To compute the feature attribution score locally at an input x, LINEX iterates over different environments until the overall approximation error is less than the threshold, with constraints encouraging a Nash Equilibrium (NE) over all environments. To show the effectiveness of LINEX, the paper compares it with several LIME-based baselines on 5 different metrics. ", "summary_of_the_review": "I vote for rejection for now as the presentation of the work and the lack of sufficient experiments outrun the novelty of the method so that I don’t think the current paper is ready for publication. ", "strength_and_weaknesses": "### Strength\nOverall I think the methodology, minimizing IRM with constraints incorporating NE, is novel and sound. It provides the means to capture the model’’s behavior at a particular instance in a broader neighborhood (i.e. a set of environments). As a result, the method is automatically justified compared to some other baselines.  \n\n### Weakness\nThere are two major weaknesses of this work. \n\n**Presentation.** The current presentation of the story is so confusing and hard to follow. \n\nIn Section 2 (Related Work), the paper gives an overview of explanation methods without defining the term “explanation”. There is a large body of explanation types, e.g. feature attribution, instance-based explanations, rule extractions, etc., and the scope of the paper should be specified. Important setups like “In this work we do not assume availability of such additional information” (access to parameters) should appear somewhere else instead of the section discussing related work. Moreover, I have to read the entire paper before reading the last paragraph of Section 2 because a lot of contexts are missing if I don't read the rest. \n\nSection 4.1 is most confusing. It introduces a lot of metrics that neither motivates the proposed method nor provides any insight why the new method would do better on them until the experiment section shows up. It is explaining metrics that are only discussed in the empirical part. Why not put this into the experiment section?\n\nFinally, the way LINEX is introduced is strange to me. The authors directly present Algorithm 1 after introducing a number of metrics that do not motivate the algorithm. My questions then follow: (1) what is the motivation of this method. Namely, what are the problems that existing work does not solve and LINEX does?; and (2) why LINEX may do well on the metrics defined in Section 4.1 compared to baselines? Namely, how do IRM and NE constraints relate to these metrics while baselines are not or not relating in a tight way? The theoretical motivations of Algorithm 1 is present after the introduction of the algorithm in Theorem 1 with an example of two environments; Therefore, to understand Algorithm 1 I have to read the paper in a reversed order. \n\n**Experimental Results.**. Firstly, The family of LIME is not the only one that provides attribution scores. For example, NB and Square in Yeh et al (2019) [1] and Occlusion [2]. Secondly, the paper claims that LINEX is more stable and unidirectional and faithful in the same degree. It seems this argument is only true for IRIS because on other datasets sometimes the results are usually within the ballpark (and there are a lot of NAs in the table or the results on CIFAR10 only works between MeLIME and LINEX). Also, within the axis of fidelity, LINEX is better than MAPLE on GI but worse on INFD on IRIS dataset. Can authors explain what this means? Why for the same axis we need two metrics, e.g. INFD and GI. Lastly, I would appreciate more analytical discussions on why the proposed method should be better on these metrics. As a result, I know what to expect from the empirical results and it helps to convince me that the proposed method solves the problems in the existing ones. The empirical results so far are a bit dense without a clear interpretation.\n\nAlso, it would be nice to provide the run-time comparisons. \n\n\n[1] Yeh, Chih-Kuan et al. “On the (In)fidelity and Sensitivity of Explanations.” NeurIPS (2019).\n\n[2] Zeiler, Matthew D., and Rob Fergus. \"Visualizing and understanding convolutional networks.\" European conference on computer vision. Springer, Cham, 2014.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N2vQ4ledm1b", "reviewer": "Reviewer_xs2D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes LINEX, an explainable AI(XAI) method. In a typical categorization of XAI, LINEX deals with a method to explain a given black-box predictor locally with features, i.e. providing an explanation of a given data with important features. LINEX employs IRM and game theory in its formulation and the optimization of the explanation model fitting objective. In addition to existing desired properties for the explanation model, the authors also introduce unidirectionality, which is persuasive. The LINEX is compared with existing approaches on various modalities and show competitive or better performance with respect to various metric including the proposed unidirectionality.", "summary_of_the_review": "Overall, the idea proposed in the paper and its empirical validation are convincing, except for the lack of rigorous proof of the Thm.1. It seems that even without its rigorous proof, there are meaningful contributions of the paper. However, if the authors want to include the theoretical contribution, I think it is better to be self-contained to a certain level. If rigorous proof or a more detailed sketch with a certain level of rigor can be provided, I would increase the score.", "strength_and_weaknesses": "### Strengths\n- The paper combines IRM with game theory to find a local explanation model.\n- A novel property for the feature-based local explanation model, unidirectionality, has been proposed.\n- LINEX is shown to outperform existing methods with different ways to generate local environments.\n- In experiments with various modalities, LINEX generally shows better performance than existing methods on various metrics.\n\n\n### Weaknesses\n- Lack of rigorous proof of the main theoretical contribution.\n    - Even though a sketch is provided in the proof, it is quite difficult to check whether the claim is correct. I cannot find proof in the appendix, either. This may be kind of trivial to someone with appropriate knowledge of game theory. If so, I hope that the authors provide some references to check the correctness of the proof.\n    - From the sketch, I can get kind of a rough picture of how $l_{\\inf}$ affects. However, since $l_{\\inf}$ regularization is claimed as one of the core contributions of the paper, the proof at least for the 2-players case should be given rigorously.\n    - Since it is a sketch, it doesn't have to be complete. Still, it does not seem to cover cases exhaustively.\n    - If the authors provide rigorous proof, I will check it and re-evaluate on this weakness.\n\n\n### Questions\n- Sensitivity to $\\gamma$, $l_{\\inf}$ norm bound\n    - It seems that $l_{\\inf}$ regularization seems to play an important role. It seems that $\\gamma$ is chosen reasonably as specified in Appendix C.1. Still, I am curious how sensitive the performance of LINEX with respect to the choice of $\\gamma$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8EqMrzJqIX", "reviewer": "Reviewer_Tzhp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new explanation method that improves the robustness of LIME to variations in perturbation methods and sample size. Technically, it leverages the invariant risk minimization (IRM) framework to design a new method for fitting the approximation model. Specifically, it first defines the environment concept for the explanation task, which is an exemplary neighborhood around the example that needs to be explained. Different environments correspond to neighborhoods generated via different algorithms (i.e., random perturbation or realistic generation). Once the environments are generated, it learns its specific linear classifier for each environment and sums these models together to get a final model, and uses the final model as the explanation. The paper gives a theoretical analysis of why such a method could provide high-fidelity explanations that are also stable and unidirectional explanations across nearby examples. Empirically, the paper verifies the effectiveness of the proposed method on five datasets in three different modalities (i.e., tabular, image, and text).", "summary_of_the_review": "This paper proposes a novel method to provide more robust explanations. Technically, the proposed method makes minor changes to the existing explanation method, and the empirical advantage is not that significant. As such, I do not rank this work as making enough contributions to be accepted by ICLR.\n", "strength_and_weaknesses": "Strength:\n+ The experimental design is relatively comprehensive.\n+ The proposed method is overall clearly presented.  \n\nWeaknesses:\n- My first major comment is the overall novelty of the proposed technique. Different from existing works that learn one approximation model on collected data, it just learns separate models and adds them together. Since the linear model is additive, I am wondering why not just learn one approximation model using data collected from different environments. Would that give similar results to the proposed model? Besides, the evaluation shows relatively marginal improvements. In some cases, the proposed method even gives worse performance than the baseline approaches. This makes me doubt the effectiveness of the proposed design. \n\n- It is not that clear how NE helps with the development of the proposed technique. I understand the proposed technique is developed based on IRM loss in Section 3. But I am not sure how NE guides the development process. \n\n- In section 4.2.1, the author argues, “features that have a disagreement with even the direction of their impact are eliminated by our method .”I wonder whether this property is good or not for the stable explanation. Intuitively, if some attribute is sensitive, it may vary or even change signs for different environments since different environments correspond to the different neighborhoods we generated around the sample we need to explain. However, according to Algorithm 1, the coefficient of this attribute would be set to zero, which means this attribute is not important anymore for our explanation. Could the authors provide more evidence for this implicit assumption (i.e., why they are unimportant)? \n\n- In Table3, we could notice that in terms of INFD metric, the proposed method performs not well compared to other baselines in some datasets. Could the author give some explanations about this result?\n\n- I would suggest the authors further highlight the advantages and benefits brought by the proposed method. Say with the proposed method, the number of needed neighborhoods is reduced a lot.  \n\n- I do not fully agree with the authors' argument about the computational expansiveness of using realistic neighbors. One can use Gaussian blur or mean feature value to generate these neighbors. A pretrained generative model can also be used for neighbor generation. All the methods are not that expensive. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject