Locally Invariant Explanations: Towards Causal Explanations through Local Invariant Learning
Authors: Amit Dhurandhar
Karthikeyan Natesan Ramamurthy
Kartik Ahuja
Vijay Arya
Venue: ICLR 2022 Submitted (2022)
Keywords: explainable AI

Abstract: Locally interpretable model agnostic explanations (LIME) method is one of the most popular methods used to explain black-box models at a per example level. Although many variants have been proposed, few provide a simple way to produce high fidelity explanations that are also stable and intuitive. In this work, we provide a novel perspective by proposing a model agnostic local explanation method inspired by the invariant risk minimization (IRM) principle -- originally proposed for (global) out-of-distribution generalization -- to provide such high fidelity explanations that are also stable and unidirectional across nearby examples. Our method is based on a game theoretic formulation where we theoretically show that our approach has a strong tendency to eliminate features where the gradient of the black-box function abruptly changes sign in the locality of the example we want to explain, while in other cases it is more careful and will choose a more conservative (feature) attribution, a behavior which can be highly desirable for recourse. Empirically, we show on tabular, image and text data that the quality of our explanations with neighborhoods formed using random perturbations are much better than LIME and in some cases even comparable to other methods that use realistic neighbors sampled from the data manifold, where the latter is a popular strategy to obtain high quality explanations. This is a desirable property given that learning a manifold to either create realistic neighbors or to project explanations is typically expensive or may even be impossible. Moreover, our algorithm is simple and efficient to train, and can ascertain stable input features for local decisions of a black-box without access to side information such as a (partial) causal graph as has been seen in some recent works.

Under review as a conference paper at ICLR 2022
LOCALLY INVARIANT EXPLANATIONS: TOWARDS CAUSAL EXPLANATIONS THROUGH LOCAL INVARI- ANT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Locally interpretable model agnostic explanations (LIME) method one of the most popular methods used to explain black-box models at per example level. Although many variants have been proposed, few provide a simple way to produce high fidelity explanations that are also stable and intuitive. In this work, we provide a novel perspective by proposing a model agnostic local explanation method inspired by the invariant risk minimization (IRM) principle originally proposed for (global) out-of-distribution generalization to provide such high fidelity explanations that are also stable and unidirectional across nearby examples. Our method is based on a game theoretic formulation where we theoretically show that our approach has strong tendency to eliminate features where the gradient of the black-box function abruptly changes sign in the locality of the example we want explain, while in other cases more careful and will choose more conservative (feature) attribution, a behavior which can be highly desirable for recourse. Empirically, we show on tabular, image and text data that the quality of our explanations with neighborhoods formed using random perturbations are much better than LIME and in some cases even comparable to other methods that use realistic neighbors sampled from the data manifold, where the latter is popular strategy to obtain high quality explanations. This is a desirable property given that learning a manifold to either create realistic neighbors or to project explanations is typically expensive or may even be impossible. Moreover, our algorithm is simple and efficient train, and ascertain stable input features for local decisions of black-bor box without access to side information such as a (partial) causal graph as has been seen in some recent works.
INTRODUCTION
Deployment and usage of neural black-box models has significantly grown in industry over the last few years creating the need for new tools to help users understand and trust models (Gunning 2017). Even well-studied application domains such as image recognition require some form of prediction understanding order for the user to incorporate the model into any important decisions (Simonyan et al. 2013 Lapuschkin et al. 2016). An example of this could be doctor who is given cancer diagnosis based on an image scan. Since the doctor holds responsibility for the final diagnosis, the model must provide sufficient reason for its prediction. Even new text categorization tasks (Feng et al. 2018) are becoming important with the growing need for social media companies to provide better monitoring of public content. Twitter recently began monitoring tweets related to COVID-19 in order to label tweets containing misleading information, disputed claims, or unverified claims (Roth & Pickles 2020). Laws will likely emerge requiring explanations for why red flags were or were not raised in many examples. In fact, the General Data Protection and Regulation (GDPR) (Yannella & Kagan| 2018) act passed in Europe already requires automated systems that make decisions affecting humans to be able to explain them. Given this acute need, number of methods have been proposed to explain local decisions (i.e. example specific decisions) of classifiers (Ribeiro et al. 2016 Lundberg & Lee, 2017: SSionnyan al. 2013 Lapuschkin et al. 2016 Dhurandhar ct al. 2018a). Locally interpretable aonel-agnostic explanations (LIME) is arguably the most well-known local explanation method that requires only query (or black-box) access to the model.

---

Under review as a conference paper at ICLR 2022
Although LIME is popular method, it is known to be sensitive to certain design choices such as i) (random) sampling to create the (perturbation) neighborhood" ii) the size of this neighborhood (number of samples) and iii) (local) fitting procedure to learn the explanation model (Molnar) 2019: Zhang et al., 2019b). The first, most serious issue could lead to nearby examples having drastically different explanations making effective recourse challenge. One possible mitigation is to increase the neighborhood size, however, one cannot arbitrarily do so as it not only leads to world it could have direct monetary
higher computational cost, but in today's cloud computing- implications where every query to black-box model has an associated cost (Dhurandhar et al. 2019). There have been variants suggested to overcome some of these limitations (Botari et al. 20200 Shrotri et al. 2021] Plumb et al. 2018) primarily through mechanisms that create realistic neighborhoods or through adversarial train- ing (Lakkaraju ct al. 2020), however, their efficacy is
Coofficient inconsistency for LIME
sapal width (cm)
4.0
3.5
restricted to certain settings and modalities based on their assumptions and training strategies.
3.0
In this paper we introduce new method called Locally INvariant EXplanations (LINEX) inspired by the invari- ant risk minimization (IRM) principle (Arjovsky et al., 2019), that produces feature based explanations that are robust neighborhood sampling and can recover faithful (i.c. mimic black-box behavior), stable (i.e. similar for closeby examples) and unidirectional (i.e. same sign attri- butions for closeby examples, see section 4 explanations across tabular, image, and text modalities. In particular, we show that our method performs better than the competitors for random as well as realistic neighborhood generation, where in some cases even with the prior strategy our planation quality is close to methods that employ the latter. Qualitatively, method highlights (local) features important that in the particular locality i) have consistently high gradient with respect to (w.r.t)) the black-box function and ii) where the gradient does not change significantly, especially in sign. Such stable behavior for LINEX is visualized in Figure[] II where we get similar explanations for nearby examples in the IRIS dataset. The (in)fidelity of LINEX is still similar to LIME (see Table 3) but of course our explanations are much more stable.
2.0
4.5
5.0
5.5
6.0
6.5
7.0
7.5
8.0
sepal length (cm)
Coefficient inconsistency for LINEX
sepal wied width (cm)
4.0
3.5
3.0
2.0
4.5
5.0
5.5
6.0
6.5
7.0 7.5 0.0
sepal length (cm)
Figure 1: Above we visualize for the IRIS dataset the Coefficient Inconsis- tency (CI) (see Section B for exact def- inition and setup details) between the explanation (top two features) for an ex- ample and its nearest neighbor in the dataset. Each circle denotes an exam- ple and rainbow colormap depicts the degree of inconsistency w.r.t. its nearest neighbor where red implies least incon- sistency, while violet implies the most. As can be seen LINEX explanations are much more consistent than LIME's.
2 RELATED WORK
Posthoc explanations can typically be partitioned into two broad categories global and local. Global explainabil- ity avers to trying to understand black-box model a holistic level where the typical tact is knowledge transfer
(Hinton et al.) 2015, Dhurandhar et al. 20088) 2020) where (soft/hard) labels of the black-box model are used to train an interpretable model such as decision tree or rule list (Rudin) 2019). Local explanations on the other hand avers to understanding individual decisions. These explanations are typically in two forms, either exemplar based or feature based. For exemplar based as the name suggests similar but diverse examples (Kim et al., 2016: Gurumoorthy et al. 2019) are provided as explanations for the input in question. While for feature based (Ribeiro al. 2016 Lundberg & Lee 2017; Dhurandhar et al. 2018a; Lapuschkin et al., 2016). which is the focus of this work, important features are returned being important for the decision made for the input. There some methods that do both (Plumb et al., 2018). Moreover, there are methods which provide explanations that are local, global as well as at a group level (Ramamurthy et al. 2020). All of these methods though may not still provide stable and robust local feature based explanations which can be desirable in practice (Ghorbani et al. 2019)
By perturbation neighhorhood referred to simply neighborhood we mean neighborhoods generated for local explanations. By exemplar neighborhood, we mean nearest examples in dataset to given example.
2

---

Under review as a conference paper at ICLR 2022
Given this there have been more recent works that try to learn either robust even causal explanations. In (Lakkaraju et al. 2020) the authors try to learn robust and stable local explanations relative to distribution shifts and adversarial attacks. However, the distribution shifts they consider are linear shifts and adversarial training is performed which can be slow and sometimes unstable (Zhang et al. 2019a). Moreover, the method seems to be applicable primarily to tabular data. Works causal explanations (Frye et al. 22000] Heskes et al. 2020) mainly modify SHAP and assume access to a partial causal graph. Some others (Vig et al. (020) assume white-box access. In this work we do not assume availability of such additional information. There are also works which show that creating realistic neighborhoods by learning the data manifold for LIME (Botari al.) 20200 Shhotri et al. 2021) can lead to better quality explanations, where in particular work (Anders et al.) (020) it is suggested that projecting explanations themselves on to the manifold can also make them more robust. The need for stability in exemplar neighborhood for LIME like methods has been highlighted in (Zhang et al. (0199))) with the general desire for stable explanations being also expressed in prior works (Yhh et al. 2019 Visani et al. 2020).
3 PRELIMINARIES
Invariant Risk Minimization: Given collection of training datasets D {De gathered from a set of environments E.,, where De = {x ye V is the dataset gathered from environment Eep and ne the number of points in environment e. The feature value for data point is 2 E X and the corresponding label is y E V/ where X' C IR and y C R. Each point (a) v in environment e is drawn i.i.d from a distribution Pe.. Define a predictor f X' - of IRM is to use these collection of datasets D to construct a predictor f that performs well across many unseen environments Ealt, where Eall 2 Etr. Define the risk achieved by f in environment e as (f( = E._ [x(f(x.), Y.)], where É is the square loss when f(X.) is the predicted value and Y, is the corresponding label, (X Ye) ~ Pe and the expectation E, is defined w.r.t. the distribution of points in environment e.
R. The goal
An invariant predictor is composed of two parts representation & E Rx and predictor (with the constant term) E _dx1 We say that a data representation across the set of environments &sr if there is predictor w that achieves the minimum risk for all the environments w E arg min Re (w) (w)) Φ), Ve E Etr. IRM may be phrased as the following constrained optimization problem (Arjovsky et al. 2019):
elicits an invariant predictor w T &
min + GER WERRX
Σ eEEr
Re(w )))
s.t. w E arg min (w [T) Ve E Eer WER4X1
(1)
If wp T & solves the above, then it is an invariant predictor across the training environments Egy.
Local explainability setup vs IRM setup: There are multiple differences between IRM setup and the setup we have in this paper. First, IRM is meant to learn global models directly from the dataset. This also applies to the game theoretic version proposed in (Ahuja et al., 2020), while for local explainability we are trying to learn local models that explain (per example) a given black-box model. Second, given that we want to explain black-box model which typically is function of the input features, we want to highlight features in our explanation that may be spurious from the domain perspective, but nonetheless the black-box model uses them to make decisions. Third, the IRM model does not have to be interpretable as its learned representation 4 can be arbitrary. Fourth, the IRM learning procedure is quite inefficient highly non-convex constraint. Fifth, IRM assumes that one has access to multiple environments (viz. multiple data sources), but in our case we have figure out how to appropriately create them for each example that we want to explain. Last, since black-box model is trained on the input features all or subset of them are causal to its prediction, unlike the standard IRM setup where predictor expected to model the true underlying mechanism. Hence in our case, it is sufficient to limit ourselves to some interpretable representation, which in many cases is just the input feature space (i.e. & = X').
tries to solve a bi-level optimization problem with a
Thus IRM cannot be directly applied to our problem and we propose novel game theoretic approach that is suitable for local explainability consistent with the above points. With our approach it is interesting that we able to obtain similar style theoretical results for our setting that have been seen in out-of-disse di-ddribbutiin generalization type of works (Ahuja et al. (02T)) thus justifying our choice of leveraging such frameworks.
3

---

Under review as a conference paper at ICLR 2022
Nash Equilibrium (NE): To understand how certain key aspects of our method function let revisit the notion of Nash Equilibrium (Dutta) 1999). A standard normal form game is written as a tuple ? = (N, {u, Nev, (Sr)e)), where .N is fimite set of players. Player i . takes actions from strategy set S.. The utility of player i is u, S + R, where we write the joint set of actions all the players S S. The joint strategy of all the players is given as E S, the strategy of player i is 8; and the strategy of the rest of players is 8-4 (8 ((
Definition 1. A strategy 8 E ES is said to be pure strategy Nash equilibrium (NE) if it satisfies
u, sL 2 s..), S.
NE thus identifies a state where each player is using the best possible strategy in response to the rest of the players leaving no incentive for any player to alter their strategy. In seminal work by (Debreu 1952) it was shown that for special class of games called concave games such a pure NE always exists. This is relevant because the game implied by Algorithm falls in this category.
METHODOLOGY
We first define certain desirable properties we would like our explanation method to have. The first three have been seen in previous works, while the last Unidirectionality is something new propose. We then describe our method where the goal is to explain a black-box model f X - IR for individual inputs æ based on predictors w by looking at their corresponding components, also termed as feature attributions.
4.1 DESIRABLE PROPERTIES
We now discuss certain properties we would like our explainability method have in order to provide robust explanations that could potentially be used for recourse.
This is the most standard property which all proxy model based explanation methods are evaluated against (Ribeiro et al., 2016 Lundberg & Lee 2017 Lakkaraju et al. (020) as it measures how well the proxy model simulates the behavior of the black-box is attempting to explain.
Fidelity:
This is also a popular notion (HancoxLii 2020 Ramamurthy et al. 2020 Yeh et al. 2019) to evaluate robustness of explanations. Largely, stability can be measured at three levels. One is prediction stability, which measures how much the predictions of explanation model change for the same example subject to different randomizations within the method or across close by examples. The second is the variance in the feature attributions again for the same close by examples. is good for a method to showcase stability w.r.t. both even though in many cases the latter might imply the former. An interesting third notion of stability is the correlation between the feature attributions of an explanation model and average feature values of examples belonging to a particular class. This measures how much does the explanation method pick features that important for the class, rather than spurious ones that seem important for the example of interest.
Stability:
This is the same as implementation invariance defined (Sundararajan et al. 2017). Essentially, if two models have exactly the same behavior on all inputs then their explanations should also be the same. Since, our method is model agnostic with only query access to the model is easy to see that it satisfies this property if the same environments are created.
Black-box Invariance:
This a new property, but as we argue that this is a natural one have. Loosely speaking, unidirectionality would measure how consistently the sign of the predictor for a feature is maintained for the same or close by examples by an explanation method. This natural metric (Miller 2018), which from an algorithmic recourse (Karimi et al. 2021) perspective is also highly desirable. For instance, recommending a person to increase their salary to get a loan and then recommending another person with very similar profile to decrease their salary for the same outcome makes little sense.
Unidirectionality:
We define the unidirectionality T as measure of how consistent the sign of the attribution for particular feature in a local explanation is when varying neighborhoods for the same example or when considering different close by examples. In particular, given m attributions for each of d features
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Locally Invariant EXplanations (LINEX) method.
Input: example to explain æ. black-bos predictor f(.,, number of environments to be created k.
(1s0) threshold 7 0, (4) threshold > 0 and convergence threshold >>
Initialize: vi E {1, .I.... k} wb; and A =
Let (i(), (x) be k environment creation functions described in section 4.2.2 do
for 1 to k do . _j.(L,....)) j) w wheee w;
(f(i) u =x=z) s.t. Iw0 21/2 S and | S A max (/20, 2 A)
W, arg min E 2
end
while
A 2 E
Output: w Eie(l
denoted by w/ ,.. (a) the unidirectionality metric for an example is:
T
sgn
W
(2)
md
where |.| stands for absolute value. Clearly, the more consistent the signs for the attribution of a particular feature across m realizations/explaaaaiios the higher the value, where the maximum value can be one. If equal number of attributions have different signs for all features then T will be zero, the lowest possible value. This property thus measures how intuitively consistent (ignoring magnitude) the explanations are and compliments the other properties mentioned above.
4.2 METHOD
4.2.1 DESCRIPTION
In Algorithm 1 we show the steps of our method LINEX. The input to the method is the example we want to explain æ, the black-box predictor, a few thresholds that we describe next and (local) environments whose creation is described in Section 4.2.2
In the algorithm we iteratively learn a constrained least squares predictor for each environment, where the final (local) linear predictor is the sum of these individual predictors. In each iteration when com- puting the contribution environment & to the final summed predictor, the most recent contributions of the other predictors summed and the residual is optimized subject to the constraints. The first constraint is a standard lasso type constraint which tries to keep the final predictor sparse as is also seen in LIME.
Why aa constraint? The second constraint is more unique and is a loc constraint on the predictor of just the current environment. This constraint we prove in Section 4.3 is essential for obtaining robust predictors. To intuitively understand why this is the case consider we have two environments. In this case if the optimal predictors for a feature in each environment have opposite signs, then the Nash equilibrium (NE) is when each predictor takes +y or -? values as they try to force the even the
sum to have the same sign as them. In other words, features that have a disagreement direction of their impact are eliminated by our method. LIME type methods on the other hand would simply choose some form of average value of the predictors which may be risky choice especially for actionability/recourse given that the directions change abruptly. On the other hand, if the optimal predictors for a feature in the two environments have the same sign, the lower absolute valued predictor would be chosen (assuming 7 is greater) making it careful choice. The reasoning for this and a discussion involving more than two environments is given in Section
The overall algorithm resembles a (simultaneous) game where each environment is player trying to find the best predictor for its environment given all the other predictors and constraints. Also note that the optimization problem solved by each player is convex norms convex.
5

---

Under review as a conference paper at ICLR 2022
4.2.2 CREATING LOCAL ENVIRONMENTS
In the standard IRM framework environments are assumed be given, however, in our case of local explainability we have to decide how to produce them. We offer a few options for the environment creation functions & Vi{1, ..., k.} in Algorithm
This possibly is the simplest approach and similar to what LIME employs. We could perturb the imput example by adding zero mean gaussian noise to create the base environment (used by LIME) and then perform bootstrap sampling to create the k different environments. This will efficiently create neighbors in each environment, although they may unrealistic the sense that they could correspond to low probability points w.r.t. the underlying distribution.
Random Perturbation:
Realistic Generation/Selection One could also create neighbors using data generators such as done in MeLIME (Botari et al., 2020) or select neighboring examples from the training set as done in MAPLE (Plumb et al. 2018) 8) to create the base environment following which bootstrap sampling could be done to form the k different environments. This approach may provide more realistic neighbors than the previous one, but may be much more computationally expensive.
Other than bootstrapping one could also over sample and try to find the optimal hard/soft partition through various clustering type objectives (Aggarwal & Reddy 2013 Creager et al. 2020).
4.3 THEORETICAL RESULTS
In this section, we analyze the output of Algorithm 1 when there are two environments. The extension to multiple environments is discussed following this result, where the general intuition is still maintained but special cases arise depending whether there of environments. To prove our main result we make two assumptions.
or odd number
Assumption The feature values for each of the dimensions in the samples created forming the local environments are independent.
This assumption is satisfied by the most standard way of creating neighborhoodssenvinonnnnss where random gaussian noise is used to create them as described in Section 4.2.2
Assumption yd, where d is the dimensionality the feature vector.
Making this assumption ensures that we closely analyze the role of the 6o penalty, which is one of the main novelties in method.
Definition 2 Let the explanation that each environment & arrives at for an example x based on unconstrained least squares minimization be wi where,
Wi arg min (E 2) WERd
(3)
The expectation is taken w.r.t the environment generation distribution. Theorem 1. The output of Algorithm 7 under Assumptions 2 and equation is given by:
w w
1wll//|1 + w @ 1 | [//l)//0) 1 @w220
(4)
where: 3 element wise product and the indicator function.
Proof Sketch. The above expression describes the NE of the game played between the local environments each trying to move w towards their least squares optimal solution. Given assumptions 1 and 2, we witness the following behavior of our method. Let the 7th feature of the predictors ú and W2 from Algorithm|1 be W1: and W2; respectively. Let the corresponding least squares optimal predictors for the 7th feature have the following relation: wi; > w5, and ww;;; > |wžzl- Then the two environments will push the ensemble predictor, W1; + U22, in opposite directions during their turns, with the first environment increasing its weight, w and the second environment decreasing its weight, W2. Eventually, the environment with a higher absolute value (5 1 since |w] > | D reaches the boundary (w), = 7) and cannot move any further due to the Lac constraint. The other environment &2 best responds, where it either hits the other end of the boundary (wzz -?), in which case the weight of the ensemble for component is zero, a case which occurs if wf, and ww; have opposite signs; or gets close the other boundary while staying in the interior (was whe - 7), in which case the weight of the ensemble for feature i is w a situation which occurs if wf, and w have the same sign.

---

Under review as a conference paper at ICLR 2022
Table 1: Datasets, models and neighborhoods used in experiments. RF- Random Forest, NN- Neural Network and -B-+ Naive Bayes.
Dataset
Modality
Black-bo model acc/Ht".
Realistic neighborhood creation methods
IRIS
tabular
RF classifier, 93%
KDEGen (Botari et al. 2020). RF (Plumb
et al.
2018
MEPS
tabular
RF regressor, 0.325
(Plumb et al.
2018
FMNIST
image
NN classifier, 87%
VAEGen (Botar 2020
Rotten Tomatocs
text
NB classifier, 75%
Word2VeeGen (Botan ct al.
2020)
Implications of the Theorem I The following are the main takeaways from Theorem (1) If the signs of the explanations for unconstrained least squares for the two environments differ for some feature, then the algorithm outputs zero as the attribution for that feature. (2) If the signs of the explanations for the two environments are the same, then the algorithm outputs the lesser magnitude of the two. These two properties are highly desirable from an algorithmic recourse or actionability perspective, where the first biases us to not rely on features where the black-box function changes direction rapidly (unidirectionality). The second, provides a reserved estimate so that we do not incorrectly over rely on the particular feature (stability).
Behavior for More than Two Environments: Given Assumptions 1 and we now discuss the behavior of our method for more than two environments, If the number of environments is odd, then using similar logic to that discussed in the proof sketch one can see that the feature attribution would be equal to the median of the feature attributions across all the environments. Essentially, all environments with optimal least squares attributions above the median would be at +7, while those below it would be at -y- The one at the median would remain so with no incentive for any environment to alter its attribution making it a NE. This is a stable choice that is also likely to be faithful as we have no more information decide otherwise. On the other hand if we have an even number of environments the situation is a bit different. The final attribution in this case depends on the middle two environments in the same manner as the two environment case proved in Theorem [ Thus, if the optimal least squares attributions of the middle two environments have opposite sign, then the final attribution is zero, else its the lower of the two attributions in terms of the numerical value. This happens because the NE for the other environments is + or -y depending on if their optimal least squares attributions are above or below those of the middle two environments. This again is a stable and likely to be faithful choice, where also unidirectionality is preferred.
EXPERIMENTS
We test our method on four real world datasets covering all three modalities: IRIS (Tabular) (Dheeru & Karra Taniskidou 2017), Medical Expenditure Panel Survey (Tabular) (Agency for Healthcare
Research and Quality 2019), Fashion MNIST (Image) (Xiao et al., 2017). p. and Rotten Tomatoes reviews (Text) (Pang al. 2002) with LIME-like random (rand) and MeLiME-like realistic neigh- borhood generation (real) or MAPLE-like realistic neighborhood selection (mpl). The summary of black-box classifier accuracies, and type of realistic perturbation used for the datasets are provided in [ In all cases except FMNIST which comes with its own test partition we randomly split the datasets into 80/20% train/test partition and average results for the local explanations over this test partition. We chose FMNIST over other higher resolution image datasets as McLIME performs best w.r.t. it, making an acid test for our method. For LINEX we produce two environments where the two environments are formed by performing bootstrap sampling on the base environment which is created either by rand, real or mpl type neighborhood generation described above. Thus all cases the union of the environment is the same as single neighborhood used to produce explanations for the competitors making it a fair comparison. Behavior of LINEX with more environments is in Appendix C
Given the neighborhood generation schemes we compare LINEX with LIME, Smoothed LIME (S-LIME), MeLIME and MAPLE, where for S-LIME we average the explanations of LIME across the LINEX environments. SHAP's results are in Appendix FF since is not a natural fit here.
Metrics: We evaluate the methods using five simple metrics: Infidelity (INFD), Generalized Infidelity (GI), Coefficient Inconsistency (CI), Class Attribution Consistency (CAC) and Unidirectionality (T).
The first two evaluate faithfulness, the next two stability and the last goodness for recourse.
7

---

Under review as a conference paper at ICLR 2022
MeLIMEir (4b) LINEX:/ 0.633
[4c) Original
[d] MeLIME:r 0,297
(el LINEXI 0.962
(1) Original
14-
16
13 35
18
23
30
20
20 20
Figure 2: Sample results using FMNIST dataset for two classes. (a-c): Class Dress, (d-f): Class Sandal. (a, d): MeLIME explanations. (b, d): LINEX explanations. (c. f): Original images. We observe that LINEX explanations capture important artifacts and thus exhibit significantly higher correlation with the original images, where in aggregate too the correlations are high w.r.. images belonging to a partieular class, thus showcasing higher stability (i.e. high CAC) as is seen in Table 3 More examples are shown in Appendix
Table Below we see three example positive sentiment sentences from the Rotten Tomatoes dataset. Green and red indicate the most important word highlighted by MeLIME and LINEX respectively, As can be seen LINEX highlights stronger positive sentiment words. More examples in Appendix
Example
Example
Example 3
moving tale of love and destruction in unexpected places unexamined lives
one-of-a-kind near-masterpiece
spare yet audacious
Let De denote (test) dataset with examples (z; y) where (o(x)) is the black-box models prediction on and yr (x) is the prediction on x (E X) using the explanation model at æ'. The feature attributions (or coefficients) for the explanation model at x are denoted by cz and N, denotes the exemplar neighborhood of with I. card denoting cardinality. We now define the above metrics.
Infidelity (INFD): This is the most commonly used metric to validate the faithfulness of explanation models (Ribeiro et al.] 2016). Here we define as the MAE between the black-box and explanation model predictions across all the test points: INFD
De card DangeE (vb(ar) y2 (z)].
Generalized Infidelity (GI): This metric has also been used in previous works (Ramamurthy et al. 2020 to measure the generalizability of local explanations to neighboring test points. It is defined as:
GI
D L(ayyEED Te Ex'N, (16( - ye (x)]-
Coefficient Inconsistency (CI): This notion has been used before (Hancox-L) 2020) to measure how robust an explanation method is. We define it be the MAE between the the linear coefficients of all the test points and their respective neighbors: CI Dell E(sn)) [aa EveN, 1c c |1.
Clais-Auriuution Consistency (CAC): For local explanations classification black-boxes, we expect certain important features to be highlighted across most of the explanations of class. This is codified by this metric which is defined as follows: CAC = P>lea Σyes' r(u py), where y denotes the set of class labels in the dataset, By the mean (vector) of all inputs in class E y, ne the mean explanation for class y and the Pearson's 's correlation coefficient. This metric quantifies the consistency between the important features for a class and attributions provided by the explanations.
i
Coefficient Unidirectionality (f): This is motivated and defined in equation 2 in section
We report the above metrics in Table 3 Each result in Table is mean + standard error of the mean over five kernel sizes 7 rdd generally, where r = {0.05, 0.1, 0.25, 0.5, 0.75). Test neighborhoods do not make sense for random perturbations with FMNIST and Rotten Tomatoes because the features (viz. superpixels) used by neighboring test examples are different. Also, we do not use realistic perturbations with MEPS since KDE and VAF generators do not work well with categorical data. In addition, since MEPS data uses regression black-box, CAC cannot be computed. All these justify the missing entries in Table 3 The results were generated on Linux machines with 56 cores and 242 GB RAM. More details regarding the exact perturbation schemes for MAME/MMLMME/MAPPP perturbation neighborhood sizes and the time taken by the different methods are Appendix A and B
the
Observations: We see that in terms of CAC, LINEX is better than baselines in all cases which indicates that on average the local explanations with LINEX highlight the important features charac- terzing the entire class making them more stable. This is further verified by looking at the Y and CI
8

---

Under review as a conference paper at ICLR 2022
Table 3: Comparing the different methods using metrics infidelity (INFD), generalized infidelity (GI), coefficient inconsistency (CI), class attribution consistency (CAC) and unidirectionality (T). indicates higher value for the metric better, and | indicates lower is better. Results better by 1% are bolded. LINEX is better than baselines in 18 out of 35 cases, and worse only in 5 cases. Plots showing behavior of these metrics with varying neighborhood size, number of environments and kernel width are in Appendix C
Dataset
Method
INFD L
G
C
71
CAC 1
LIME
0.015 + 0.011
0.132 + 0.012
0
0.646 0.040
0.667 + 0.167
S-LIME
0.015
0.077 +
0.143
0.701
0.037
0.878 0.034
LINdX/rand
0.052 0.008
0.044
0.802
0.044
0.921 = 0.042
IRIS
McLIME
0.008 0.003
0.049 0.018
0.219
0.629:000113
0.464 0.100
LINEX/real
0.009 - 0,003
0.029 + 0.003
0.024 + 0.002
0.744 0.044
0.942 + 0.023
MAPLE
0.009 0.001
0.038 - 0.004
= 0
0.586 + 0.035
LINEX/mpl
0.013 + 0.000
0.020 = 0.000
0.026 = 0.002
0.694 + 0.008
0.929 + 0.004
LIME
0.158 + 0.066
0.214 + 0.041
0.005 + 0.001
0.981 + 0.006
S-LIME
0.158 0.066
0.214 0.042
0.005 + 0.001
0.974 + 0.008
NA
MEPS
LINEX/rand
0.130 0.052
0.164 + 0.021
0.003 + 0.001
0.979 0.006
MAPLE
0.063 0.000
0.067 1 0.000
0.007 = 0.000
0.957 = 0.000
NA
LINEX/mpl
0.098 1 0.001
0.094 + 0.001
0.007 + 0.000
0.950 + 0.000
LIME 0,162 + 0,003
S-LIME
0.142 + 0.003
NA
NA
NA
NA
FMNIST
LINEX/rand
0.149 0.002
McLIME
0.001 0.000
0.277 + 0.000
0.007 + 0.000
0.769 0.000
0.327 0.000
LINEX/real
0.100 + 0.002
0.304 + 0.001
0.002 + 0.000
0.780 + 0.000
0.649 + 0.001
LIME
0.079 1 0.036
S-LIME
0.075 + 0.035
NA
NA
NA
NA
Rotten
LINEX/rand
0.069 + 0.032
Tomatoes
McLIME
0.029 0.001
0.391
0.000
0.000 = 0.000
0.999 0.000
0.909 0.000
LINEX/real 0.053 1 0.000
0.361 + 0.000
0.000 + 0.000
1.000 + 0.000
0.953 + 0.001
metrics where LINEX is similar better than others. For GI and INFD metrics, the results more evenly spread which implics that LINEX's main advantage is obtaining stable and unidirectional explanations that are faithful to similar degree.
We show an example MeLIME and LINEX/realistic explanation on FMNIST in Figure 2 where we that LINEX explanations more coherent and also delineate the outline of the image more clearly compared to MeLIME explanations leading to better CAC. Even on the text data we see more reasonable attributions in Table where "masterpiece", "moving" and "audacious" are highlighted as the most important words indicative of positive sentiment in the three examples.
An interesting observation is that when comes the stability metrics (CI and CAC) and unidirec- tionality LINEX with even random perturbation model is better than MeLIME in some cases. This very promising as it means LINEX could be potentially be trusted without the need to generate realistic perturbations which may be computationally expensive or not even possible.
6 DISCUSSION
In this paper we have provided method based on game theoretic formulation and inspired by the invariant risk minimization principle to provide faithful, stable and unidirectional explanations. We have defined the latter property and argued that it somewhat of necessity (may not be sufficient) for recourse. We have theoretically shown that our method has strong tendency to be stable and unidirectional as we will mostly eliminate features where the black-box models gradient changes abruptly and in other cases choose conservative value. Empirically, we have verified this where we outperform competitors in majority of the cases on these metrics. An interesting observation is also that in some cases our method provides more stable and unidirectional explanations with just random perturbation model relative to more expensive methods that use realistic neighbors.
In the future, it would be worth experimenting with more varied strategies to form environments and if possible find the optimal ones (Creager et al.) 2220), which may lead to picking even more relevant features that are "causal" to the local decision.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
With the wide adoption of deep learning technologies, explaining understanding the reasons behind their decisions has become extremely important in many critical applications (Arya et al. 2019). Numerous explainability methods have been proposed in literature explain individual decisions of black-box models (Ribeiro et al. 2016, Plumb et al., 2018) Botari et al., 2020 Dhurandhar al. 20118a] Lundberg & Lee 2017). Although LINEX more stable and undirectional than other competing approaches, still is posthoe explainability method that may not be completely faithful the black-hox model. This of course is not just limitation of approach, but nonetheless should be taken into account before a user makes a decision. Our method could also be used to divulge information by exposing the inner workings of the black-box leading to privacy concerns. One possible mitigation strategy in this case would be to keep the sensitive attributes hidden from the explainer.
REPRODUCIBILITY STATEMENT
Experimental details are provided Section of f the main paper and Appendix B All datasets are public. Code will be provided during the discussion phase through an anonymized link.
REFERENCES
Agency for Healthcare Research and Quality. Medical Expenditure Panel Survey (MEPS). https
imeps.ahrg www.aaaw......aaoooooooooooe gov/mepsweb/ 2019. Last accessed 2019-08.
C. Aggarwal and C. Reddy. Data Clustering: Algorithms and Applications. CRC Press, 2013.
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization game. In International Conference on Machine Learning, 2020.
Kartik Ahuja, Karthikeyan Shanmugam, and Amit Dhurandhar. Linear regression games: Conver- gence guarantees to approximate out-of-distribution solutions. In AISTATS, 2021.
Elvio Amparore, Alan Perotti, and Paolo Bajardi. To trust not to trust explanation: using leaf to evaluate local linear xai methods. PeerJ Computer Science, 7:e479, 2021.
Christopher Anders, Plamen Pasliev,
Dombrowski, Robert Muller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In Intl, Conference on Machine Learning (ICML), 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paze Paz. Invariant risk minimization. arXiv preprint 11Xhv:190. 2019.
Vijay Arya, Rachel K. F. Bellamy, Pin- Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoff- man, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. One explanation does not fit all: A toolkit and taxonomy of Al explainability techniques. arXivv1909.0300
2019.
Tiago Botari, Frederik Hvilshøj, Rafael Izbicki, and Andre C. P. L. F. de Carvalho, Melime: Meaningful local explanation for machine learning models, 2020.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In e ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning. 2020.
Gerard Debreu. A social equilibrium existence theorem. Proceedings of the National Academy of Sciences, (8(10))88--89 1952.
Dua Dheeru and Ef Karra Taniskidou. UCI machine learning repository, 2017. URL http: 7/archive cs. uci. edu/m)
10

---

Under review as a conference paper at ICLR 2022
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun- Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, pp. 592-603, 2018a.
Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, and Peder Olsen. Improving simple models with confidence profiles. In Advances in Neural Information Processing Systems, pp. 10296-10306, 2018b.
Amit Dhurandhar, Tejaswini Pedapati, Avinash Balakrishnan, Kartik Ahuja Pin-Yu Yu Chen, Karthikeyan Shanmugam, and Ruchir Puri. Model agnostic contrastive explanations for structured data. https:/aariv.orr//b///1966 1ttp://///9.000000011 17, 2019.
Amit Dhurandhar, Karthikeyan Shanmugam, and Ronny Luss. Enhancing simple models by exploiting what they already know. In Intl. Conference on Machine Learning, 2020.
Prajit K. Dutta. Strategies and games: theory and practice. MIT Press, 1999.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3719-3728, Brussels, Belgium, Octoher-November 2018. Association for Computational Linguistics.
Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. NeurIPS, 2020.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile, 2019.
David Gunning. Explainable artificial intelligence (xai). In Defense Advanced Re- search Projects Agency, 2017. URL bttpps:/www.t wwwpa.arra..oo mi mil/program/
explainable artiaiiiaaliiiiiiaaaaaitaaiaaaia wttiiiialliteeliieen..
Karthik Gurumoorthy, Amit Dhurandhar, Guillermo Cecchi, and Charu Aggarwal. Efficient data representation by selecting prototypes with importance weights. In Proceedings of the IEEE International Conference Data Mining, 2019.
Leif Hancox-Li Robustness in machine learning explanations: does it matter? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, 2020.
Tom Heskes, Evi Sijben, loan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances in Neural Information Processing Systems, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
Amir- Karimi, Bernhard Schölkopf. and Isabel Valera. Algorithmic recourse: from coun- terfactual explanations to interventions. In ACM conference on Fairness, Accountability and Transparency (FAceT), 2021.
Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples are not enough, learn criticize! Criticism for interpretability. In In Advances of Neural Inf. Proc. Systems, 2016.
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. ICML, 2020.
Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Rober Müller, and Wojeiech Samek. The Irp toolbox for artificial neural networks. Journal of Machine Learning Research, 17 (114):1-5, 2016. URL TtRLhtp::// //jmr http://l....rrrrreeeeeeeoeeeeeee org/ pwwpssapppsspppaa /vl1vv17 vel/v/l////// /15- 618. html
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial
Intelligence, 2018.
11

---

Under review as a conference paper at ICLR 2022
Christoph Molnar. Interpretable machine learning, 2019. [LL::tttss////chstttoomm
1o/ interpretablee www.salleee---oooooeeeeeoee html
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP, pp. 79-86, 2002.
Gregory Plumb, Denali Molitor, and Ameet Talwalkar. Model agnostic supervised local explanations. In Advances in Neural Information Processing Systems, pp. 2520- 2529, 2018.
Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. Model agnostic multilevel explanations, 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, 2016.
Yoel Roth and Nick Pickles. Updating our approach to misleading information. 2020. Pub- lished http://ee..ttaaaaaooooootottttt. http://wuuurrppppppp/ppprpo/////
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Mach. Intell., 165):206-215, May 2019.
Aditya Aniruddha Shrotri, Nina Narodytska, Alexey Ignatiev, Joao Marques-Silva Kuldeep S. Meel, and Moshe Vardi. Constraint-driven explanations black-box {ml} models, 2021.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv:1312.6034, 2013.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICMI., 2017.
Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. NeurIPS, 2020.
Giorgio Visani, Enrico Bagli, Federico Chesani, Alessandro Poluzzi, and Davide Capuzzo. Sta- tistical stability indices for lime: obtaining reliable explanations for machine learning models. arXiv:2001. 11757, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Philip N. Yannella and Odia Kagan. Analysis: Article 29 working party guidelines on automated decision making under gdpr. 2018. https//wwww.cbberatiisstt https/eerrooo.ooo////o/ooe com/201 8/01 /a0llyyyssii--rtiillle wwwking-pprty---ooee
29- 29-working- party- sarty-guidelines-on antoomattdd-eeeiiioonnmiiiii woaiiiinn--nndg-gggee under-gdpr/.
Chih-Kuar Kuan Ych, Cheng- Yu Hsich, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. In Advances in Neural Information Processing Systems, pp. 10965-10976, 2019.
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S. Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. In International Conference on Learning Representations (ICLR), 2019a.
Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. Why should you trust my explanation? ICML-AI for Social Good, 2019b.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A EFFICIENCY OF LINEX
important note that the query complexity (i.e. number of times we query the black box to obtain an explanation) of LINEX is the same as that of LIME since the union of the environments is the same as LIME perturbation neighborhood. This is important in todays cloud-driven world where models may exist on different cloud platforms and posthoe explanations are an independent service where each call to the model has an associated cost. terms of running time for two environments, convergence was fast and running time was approximately 2.5 times that of LIME (LINEX took 2.5 seconds on IRIS for 30 examples as opposed to 1 second by LIME, LINEX took 47 seconds on MEPS for 500 examples as opposed to 18 seconds by LIME), which is very similar to Smoothed LIME (S-LIME) (took 2.3 seconds on IRIS and 40 seconds on MEPS) that we still outperform in majority of the cases.
important
Realistic neighborhood generation can be time consuming especially for McLIME since generators have to be trained which may take up an hour using single GPU for datasets such as FMNIST. After the generator is trained and neighhorhood sampled MeL.ME takes the same amount of time as LIME since the model fitting procedure is the same. MAPLE took 1.5 seconds for the IRIS dataset for 30 examples and 27 seconds for 500 MEPS examples.
B EXPERIMENTAL DETAILS
B.1 DATASET DETAILS AND HYPERPARAMETER SPECIFICATIONS
We describe the datasets and the hyperparameters used for each. We set perturbation neighborhood sizes 10 (IRIS), 500 (MEPS), 100 (FMNNST-random), 500 (FMNIST-realistic) 100 (Rotten tomatoes) for generating local explanations. We also use 3, 10, 10, 5 as examplar neighborhood sizes to compute GI, CI and T metrics for the four datasets respectively. We also use 5- sparse explanations for all cases except FMNIST with realistic perturbations where we follow MeLIME and generate a dense explanation using ridge penalty with penalty multiplier value of 0.001. The &o bound in Algorithm iis set as the maximum absolute value of linear coefficient computed by running LIME/McLIME in the two individual environments. Please look at IRIS dataset first since it contains some of the common details used across others.
IRIS (Tabular): This dataset has 150 instances with four numerical features representing the sepal and petal width and length in centimeters. The task is to classify instances of Iris flowers into three species: setosa, versicolor, and virginica. A random forest classifier was trained with a train/test split of 0.8/0.2 and yielded test accuracy of 93%. We provide local explanations for the prediction probabilities for class setosa. For both random and realistic perturbations, we use a perturbation neighborhood size of n. For random perturbations, and sample from a Gaussian around each data point. Realistic perturbations (with the same number n) were generated using KDEGen Botari al. (202)), kernel density estimator (KDE) with the Gaussian kernel fitted on the training dataset to sample data around a sample point. For both random and realistic perturbations, we weight the neighhorhood using Gaussian kernel of width r rvd, where {0.05, 0.1, 0.25, 0.5, 0.75), and this corresponded to kernel widths 10.1, 0.2, 0.5, 1.0, 1.5]. J. We also perform a weighted version of realistic selection where we use MAPLE Plumb al. (2018) assign weights all the test examples and pick the top n weighted examples to use as the perturbation neighborhood. For random/realistic perturbations and realistic selection, the corresponding environments (of size n each) for LINEX are created by drawing k bootstrap samples where k- = {2, 3, 4, 5} in our experiments. We test for n = {10, 20, 30, 40, 50} with this dataset.
used the same approach followed by LIME
d is the dimension of the feature vector and
Medical Expenditure Panel Survey (Tabular): The Medical Expenditure Panel Survey (MEPS)
dataset is produced by the US Department of Health and Human Services. It is a collection of surveys of families of individuals. medical providers, and employers across the country. We choose Panel 19 of the survey which consists of a cohort that started in 2014 and consisted of data collected over rounds of interviews over 2014 4 2015. The outcome variable composite utilization feature that
13

---

Under review as a conference paper at ICLR 2022
quantified the total number of healthcare visits of a patient. The features used included demographic features, perceived health status, various diagnosis, limitations, and socioeconomic factors. We filter out records that had utilization (outcome) of O, and log- trgaasasffrred the outcome for modeling. These pre-processing steps resulted in a dataset with 11 11136 examples and 32 categorical features. We train random forest regressor that has a test. R of 0.325 in this dataset. We provide local explanations of the predictions. With MEPS, we do not use realistic perturbations since KDE and VAE generators do not work well with categorical data. Otherwise the setting is similar as IRIS data, except that we use = (50, 100, 200, 300, 400, 500). The kernel widths in this case were {0.28, 0.57, 1.41, 2.83, 4.24). We use k = (2, 4, 5) for this dataset.
Fashion MNIST (Images): This dataset has 28 x 28 grayscale images of fashion articles with 60,000 train and 10,000 test samples. The task is to classify these into 10 classes corresponding to coat, shoe, and so on. A neural network trained with test accuracy of 87%. Explanations are generated for the prediction probabilities corresponding to the predicted class for each example. We choose 1000 test examples to generate explanations. Realistic perturbations were generated using VAEGen (Botari et al., 2020), a Variational Auto Encoder (VAE) fitted on the training dataset. For random perturbations, the we chose from {50, 100, 200, 300, 400, 500} and kernel sizes were {0.43, 0.85, 2.14, 4.27, 6.41). For realistic perturbations we chose n from {250, 500, 750, 1000} and the kernel widths were {1.4, 2.8, 7.0, 14.0, 21.0}. We use k = {2, 3, 4. 4.5) for this dataset.
Rotten Tomatoes (Text): This dataset contains 10662 movie reviews from rotten tomatoes website along with their sentiment polarity, i.e., positive or negative reviews and the task is to classify the sentiment of the reviews into positive or negative. The review sentences were vectorized using CountVectorizz and ThidTTansformer and sklearn Naive Bayes classifier was fitted on train- ing dataset which yielded a test accuracy of 75%. Explanations are generated for the prediction probabilities corresponding to the predicted class for each example. Realistic perturbations were generated using Word2 (Botari et al., 2020), wherein word2vec embeddings are first trained using the training corpus and new sentences are generated by randomly replacing a sentence word whose distance in the embedding space lies within the radius of the neighbourhood. For both random and realistic perturbations, n was chosen from {25, 50, 75, 100). The kernel sizes were (0.42, 1.06, 2. 12 3.18} for random perturbations (kernel size 0.21 resulted in numerical issues), and (0.21, 0.42, 1.06, 2. 12, 381 for realistic perturbations. We use k-
{2, 3, 4, 5} for this dataset.
C RESULTS WITH ALL DATASETS AND HYPERPARAMETER COMBINATIONS FOR RANDOM AND REALISTIC PERTURBATIONS
We present results with all hyperparameter combinations for random and realistic perturbations. Results for LIME with random perturbations (LIME), smoothed LIME (S-LIME), LINEX with random perturbations (LINEX/rand), MeLIME (MeLIME), LINEX with MeLIME-like realistic neighborhoods (LINEX/rea),, MAPLE (MAPLE), LINEX with MAPLE-like realistic neighborhoods (LINEX/mpI) are presented in figures 18 The legend for these figures are given in Figure 3
For the four datasets, we perform ablations by varying one of perturbation neighborhood size (Figures 48. number of environments (Figures (3), and kernel width (Figures) [8]. Each point in these figures are averaged over all possible values for the two parameters that are not ablated. For example, each point in Figure is averaged over all possible values for kernel widths and number of environments for given perturbation neighborhood size. Standard errors of the mean are also plotted in the same color with lesser opacity. Lower values of Infidelity (INFD), Generalized Infidelity (GI), Coefficient Inconsistency (CI) are better whereas for Unidirectionality (Y) and Class Attribution Consistency (CAC) higher values are better.
Figures 4 8 show ablations with respect to perturbation neighborhood sizes. Considering all datasets, the stabifity/recouss metrics (CI, Y. CAC) are clearly better for LINEX compared to its counterparts. For LINEX methods (LINEX/rand, LINEX/real, LINEX/mpl), the metrics get better or stays approxi- mately the same generally as perturbation neighborhood size increases keeping with the intuition that larger perturbation neighborhood sizes should produce explanations that more stable in the exemplar neighborhood. Y for FMNIST is somewhat of an exception which could be because of the high dimensionality as well as the quality of MeL.ME perturbations.
14

---

Under review as a conference paper at ICLR 2022
Turning to the fidelity metrics (INFD and GI) in tabular datasets, see that the results still favor LINEX, but less heavily compared to the stability/recourse metrics. This is line with what we observe in T In IRIS and MEPS, LINEX is close to outperforms the corresponding baselines in the GI measure (except for LINEX/mpl with MEPS). This gap closes a bit with INFD, but we note that Gl is better measure since it estimates how faithful explanations in exemplar neighborhood. With the text dataset, LINEX variants are slightly more favored, whereas with the image dataset, the baselines have an edge.
Considering Figures 1 we see that variations are less stark with respect to number of environments overall for LINEX variants. Note that except for S-LIME, other baselines do not use multiple environments, and hence stay constant. The slight variations in MAPLE are due to the effect of random seeds. the stability/recourse metries, again LINEX variants emerge as the clear winner across datssets. With the faithfulness metrics (GI and INFD), in the text dataset, LINEX variants generally perform better, whereas the baselines have better performance in the image dataset.
Finally, we study the variation of the performance measures with respect to kernel width in Figures 1418 We see that the stability/recourse metrics flatten out all cases with large kernel widths. This behaviour holds true for faithfulness metrics (GI and INFD) as well except in some cases. GI and INFD measures also increase before they flatten out since the becomes poorer at larger kernel widths. The stability/reccouse metrics become better remain approximately the same since explanations generally improve or preserve their stability properties as kernel widths increase. Note that very small kernel widths lead to unexpected behavior that does not fit the trend as seen with the tabular datasets since explanations can become hyper-local. MAPLE and LINEX/mpl stay the same at different kernel widths since they use a different weighting scheme. As with other ablations, we see that LINEX variants are similar or better in stability/recourse metrics overall, while with the faithfulness metries the results are more mixed.
Note that we do not compute MeLIME perturbations with MEPS since KDE and VAE generators do not work well with categorical data, and do not use compute CAC since the task is regression. Further, the features used in explanations for different test examples are not comparable for random perturbations with FMNIST and Rotten Tomatoes, hence we cannot compute CAC for those cases as well. This explains the missing curves/plots.
LIME
S-LIME
LINEX/rand
MeLime
LINEX/real
MAPLE
LINEX/mpl
Figure 3: Legend for figures 18
02822
2000
8000
300
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
Figure& 4: Coefficient inconsistency (CT) vs. Perturbation neighborhood size.
15

---

Under review as a conference paper at ICLR 2022
1210
Sorggh
Berghbrbbed
(a) Iris
(h) FMNIST
(c) Rotten Tomatnes
Figure 5: Class attribution consistency (CAC) vs. Perturbation neighborhood size.
camuss
25
20
25
56 35
239
weighoorhood Bib
276
30
50
030
K6 1290
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
Figure 6: Unidirectionality (Y) Perturbation neighborhood size.
3254
8112
3 3335
2144
3015
3005
Weihberfest 60D4
Relgbbohood dtra
Neigborbood FON
(a) Iris
(h) MEPS
FMNIST
Rotten Tomatoes
Figure 7: Generalized infidelity (GI) vs. Perturbation neighborhood size.
232
1.59
km
335
(a) Iris
(h) MEPS
(c) FMNIST (random)
(d) FMNIST (realistic)
0.0.
(e) Rotten Tomatoes
(I) Rotten Tomatoes
(random)
(realistic)
Figure 8: Infidelity (INFD) vs. Perturbation neighborhood size.
3.5
830
3004
5000
Hunder
(a) Iris
MEPS
(c) FMNIST
Rotten Tomatoes
Figure 9: Coefficient inconsistency (CI) VS. Number of environments.
16

---

Under review as a conference paper at ICLR 2022
6.50
1.0
Monkeraf 4.0
Hondor
(a) Iris
(h) FMNIST
(c) Rotten Tomatnes
Figure 10: Class attribution consistency (CAC) vs. Number of environments.
0015
5,02
375
4214
0770
3.5
3.09
A776
01850
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
Figure 11: Unidirectionality (T) vs. Number of environments.
9.39
(a) Iris
MEPS
FMNIST
Rotten Tomatoes
Figure 12: Generalized infidelity (GI) vs. Number of environments.
3028
133
3005
3005
22
322
3008
(a) Iris
MEPS
(c) FMNIST (random)
(d) FMNIST (realistic)
25
20
(c) Rotten Tomatoes (random)
(f) Rotten Tomatoes
(realistic)
Figure 13: Infidelity (INFD) vs. Number of environments.
3700
68
8716
337
3779
A774
03952
A770
A776
03847
Samil mURS
Kor mAR0
Boreal
20
(a) Iris
MEPS
FMNIST
(d) Rotten Tomatoes
Figure 14: Unidirectionality (T) vs. Kernel width.
17

---

Under review as a conference paper at ICLR 2022
3005
100
24
(a) Tris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
Figure 15: Coefficient inconsistency (CI) vs. Kernel width.
34
3.0
100 125 15.0 375 303
(a) Iris
(b) FMNIST
(c) Rotten Tomatoes
Figure 16: Class attribution consistency (CAC) vs. Kernel width.
3.5
2.3
(a) Iris
(b) MEPS
(c) FMNIST
(d) Rotten Tomatoes
Figure 17: Generalized infidelity (GI) vs. Kernel width.
3.32
131
Emnnl widh
Exmmed wiRs
(a) Tris
MEPS
FMNIST (random)
(d) FMNIST (realistic)
8006
331c
3015
8003
03
12
1.2
335
(e) Rotten Tomatoes
(f) Rotten Tomatoes
(random)
(realistic)
Figure 18: Infidelity (INFD) vs. Kernel width.
D EXAMPLE FEATURE ATTRIBUTIONS IN TEXT DATA: MELIME VS LINEX
Below we see sample attributions by the two methods along with the magnitude the attributions. Attribution magnitudes are printed with a precision of 10-3 and shown along with the corresponding words in descending order.
D.1 POSITIVE SENTIMENT
enticing and often funny documentary
MeLIME: documentary funny and enticing often
LINEX : documentary funny often enticing and
MeL 0.517 0.446 0.333 0.317 0.311
18

---

Under review as a conference paper at ICLR 2022
LINEX
0.416 0.377 0.342 0.331 0.330
one-of-a-kind near-masterpiece
MeLIME: kind near masterpiece LINEX masterpiece kind
MeLIME: 0.832 0. 695 0.182 LINEX 0.212 0.384 0.381
fast funny highly enjoyable movie MeLIME: enjoyable highly funny fast movie enjoyable highly fast funny movie
LINEX :
IeLIME:
0.550 0.432 0.412 0.389 0.198 0.409 0.389 0.372 0.350 0.326
LINEX :
ferrara's strongest and most touching movie of recent years
MeLIME: touching years most strongest and touching most recent strongest and 0.735 0.490 0.450 0.443 0.427
LINEX :
MeLIME:
LINEX :
0.490 0.488 0.450 0.444 0.407
saved from being merely way-cool by a basic credible compassion MeLIME: cool basic credible merely from LINEX: ccol credible merely compassion from
MeLIME: 1.514 0.050 0.040 0.029 0.026
LINEX : 358 0.308 0.304 0.299 0.293
really quite funny MeLIME: funny quite really funny quite really
LINEX :
0.559 0.417 0.233 LINEX 0.462 0.368 0.275
MeLIME:
spare yet audacious MeLIME: spare yet audacious audacious spare yet
LINEX :
MeLIME: 0. 626 0.447 0.395
LINEX : 0. 501 4.1 0.422
an engrossing and infectiously enthusiastic documentary MeLIME: engrossing documentary and enthusiastic an engrossing documentary an enthusiastic and 0.593 0.455 0.358 0.354 0.333
LINEX :
MeLIME:
LINEX :
0.461 0.407 0.374 0.357 0.350
wildly funny prison caper MeLIME: funny caper wildly prison funny caper prison wildly MeLIME:
LINEX :
5.541 0.364 0.214 0.193 0.403 0.335 0.245 0.239
LINEX :
this charming but slight tale has warmth wit and interesting characters compassionately portrayed charming compassionatell and interesting portrayed LINEX : charming
MeLIME:
irmpaassionatell has tale portr
MeLIME: 0. 690 0.507 0. 0.444 0. .24
LINEX :
464 0.4 0. 431 0. 430 0.9
tho provocative and entertaining MeLIME: thoughtful entertaining and provocative
19

---

Under review as a conference paper at ICLR 2022
LINEX thought entertaining and provocative
MeLIME: 0.612 0.517 0.402 0.395
LINEX 0.505 0.461 0.415 0.404
the film is quiet threatening and unforgettable MeLIME: quiet unforgettable and film the unforgettable quiet film and is MeLIME: 0.597 0.483 0.412 0.325 0.303 021 0.416 0.388 0.378 0.338
LINEX :
LINEX :
moving tale of love and destruction in unexpected places unexamined lives MeLIME: unexpected moving love tale lives moving unexpected places lives in
LINEX :
MeLIME:
0.692 0.662 0.577 0.538 0.499 0.538 0.530 0.521 0.513 0.501
LINEX :
though frodo's s quest remains unfulfilled determined new zealanders has proved its creative mettle.
hardy group of
MeLIME: creative group proved has new
LINEX : creative quest its proved determined
MeLIME: 0.602 0.441 0. 424 0.402 0.393 0.410 0.392 0.390 0.385 0.381
LINEX :
D.2 NEGATIVE SENTIMENT
originality is sorely lacking lacking sorely is originality lacking sorely originality is
MeLIME:
LINEX :
MeLIME:
0.543 0.381 0.296 0.278 0.430 0.356 0.314 0.271
LINEX :
an ugly pointless stupid movie MeLIME: stupid pointless ugly movie an stupid pointless ugly movie an MeLIME: 0.543 0.499 0.385 0.365 0.276 0.446 0.411 0.373 0.360 0.350
LINEX :
LINEX :
so devoid of pleasure or sensuality that it cannot even be dubbed hedonistic MeLIME: devoid even be dubbed of
LINEX devoid dubbed be cannot
MeLIME: 0. 666 416 0.413 0.372 0.344 LINEX 400 0.392 0.387 0.380 0.368
neither revelatory nor truly edgy--merely crassly flamboyant and comedically labored
MeLIME:
edgy neither nor labored revelatory
LINEX :
edgy neither nor labored truly 1.256 0.338 0.277 0.204 0.021 0.4 0.398 0.398 0.369 0.349
MeLIME:
LINEX :
occasionally funny , sometimes inspiring often boring MeLIME: boring occasionally inspiring sometimes often LINEX boring occasionally sometimes often inspiring
MeLIME: 0.669 0.242 0.218 0.210 0.182
LINEX 0.377 0.266 0.266 0.250 0.236
cumbersome and cliche-ridden movie greased with every emotional device known to man MeLIME: cliche every device movie with
20

---

Under review as a conference paper at ICLR 2022
LINEX cliche every man cumbersome emotional MeLIME: 0. 695 0.449 0.327 0.280 0.268 LINEX 0.385 0.361 0.354 0.349 0.309
ponderous plodding soap opera disguised MeLIME: plodding soap ponderous opera disguised
feature film
LINEX : plodding soap film ponderous feature
MeLIME: 0.579 0.522 0.421 0.408 0.382
LINEX 0.442 0.440 0.418 0.406 0.377
kitschy flashy overlong soap opera MeLIME: soap flashy opera overlong kitschy soap flashy opera overlong kitschy MeLIME: 0.499 0.397 0.391 0.358 0.230 389 0.362 0.360 0.346 0.300
LINEX :
LINEX :
[a] poorly executed comedy MeLIME: poorly comedy executed poorly comedy executed MeLIME: 0. 653 0.348 0.257 5.502 0.335 0.309
LINEX :
LINEX :
bad movie that happened to good actors MeLIME: bad happened movie to that
LINEX bad happened to movie actors
MeLIME: 0. 692 0.396 0.371 0.367 0.242
LINEX 0.442 0.384 0.367 0.361 0.344
complete waste of time MeLIME: waste complete time of waste complete time
LINEX :
MeLIME: 0.614 0.425 0.313 0.247 480 3.1 0.348 0.278
LINEX :
don't t waste your money MeLIME: waste money don your waste money don your
LINEX :
MeLIME: 0.592 0.497 0.408 0.309 0.483 0.450 0.411 0.337
LINEX :
witless and utterly pointless
MeLIME: pointless witless and utterly pointless witless utterly and
LINEX :
MeLIME: 0. 652 0.491 0.263 0.245
LINEX :
0.506 0.444 0.311 0.269
E EXAMPLE FEATURE ATTRIBUTIONS IN IMAGE DATA: MELIME VS LINEX
We show feature attributions for individual example images with MeLIME and LINEX with MeLIME perturbations in Figure 19 In Figure 20 we show class-wise mean feature attributions along with mean images. Clearly, LINEX explanations provide more meaningful feature attributions.
F RESULTS FOR ALL METHODS INCLUDING SHAP
In Table 4 provide the results for SHAP along with all methods for easy comparison. Note that SHAP does not have standard errors since it is computed only once per test point. The INFD values for SHAP miniscule since SHAP values add up the predictions by definition. In order to
21

---

Under review as a conference paper at ICLR 2022
Original
[2a] MeLIMEY (26)
(2c) Original
25
27-
(3a) ELIIEE 0.259 (3b) LINEX:r 0.711
(3c) Original
(4a) MeLIME/ 0.292 (4b) LINEX:r - 0.633
(4c) Original
13-
18-
16
26
20
(5a)
0.271
(5b)
Original
(6a)
(6b)
(6c) Original
20
(7a) McLIME::= 0.482 (7b) LINEX = 0.820
(7c) Original
(Ba) McLIME: 0.177 (8b) LINE X = 0.788
(Bc) Original
15-
10-
(9a) 0.247 (9b)
(9c) Original
(10a) MeLIME 0.381(1bb) LINEXI 0 0.792
(10c) Original
35
29
28
20
23
Figure 19: Results using individual samples for realistic perturbations for FMNIST dataset for all classes:1-10 (T-shirt/top. Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag and Ankle boot). (a) MeLIME feature attributions for image. (b) LINEX feature attributions for an image. (c) Original image the class. The values show Pearson's correlation between feature attributions and the original image from the respective class. We observe that LINEX attributionn/eepllaaatton exhibit significantly higher correlation with the original image belonging to particular class (i.c.
high CAC).
compute GI, CL, Y. CAC, we convert the SHAP values SHAP attributions (Amparore et al. 2021 equation 4) first and follow the same approach used by other explanation methods.
G ERROR ANALYSIS OF LINEX
We perform error analysis for LINEX gain better understanding about the method. We choose FMNIST dataset for doing this since: (a) This is the highest dimensional dataset (784 dimensions) that we have. (b) LINEX/real under performs McLIME in terms of the INFD measure here (see T more heavily compared other datasets. (c) It is easier to visualize the explanations for this dataset.
We start by observing that even though LINEX/real underperforms in the INFD metric, the gap is not so great in the GI metric, which suggests that MeLIME may be overfitting explanations here. We also note that in terms of CI, T, and CAC metrics, LINEX/real clearly outperforms
22

---

Under review as a conference paper at ICLR 2022
(1a) (1b)
Original
(2a)
(2c) Original
25
27-
25
(3a) -...M
[3b] LINEX//0.612
(3c) Original
(4a) MeLIME//-0A41
(4b) LINEX:/=0.860
(4c) Original
13
18-
16
20
25
20
20
(5a) (5b)
Original
(6a)
(6c) Original
20
20
(7a) D0.
(7b)
(7c) Original
(Ba) MeLIME:r=0.443
(Bb) LINEX:=0..a
(Bc) Original
19-
10-
11-
24
(9a) (9b) LINEX>=0.751
(9c) Original
(10a) MeLIME:r=0 0.14444111000 (10b)
(10c) Original
23
23
28
22
23
Figure 20: Results using realistic perturbations for FMNIST dataset with mean feature importances for all classes:1-10 (T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag and Ankle boot). (a) Mean feature attributions of all images in the class using MeLIME. (b) Mean feature attributions of images in the class using LINEX. (c) Mean all images show Pearson's 's correlation between average feature attributions and mean of the original images from the respective classes. We observe that LINEX explanatiooss/atribbtiin exhibit significantly higher correlation with the original images belonging to a particular class (i.c. high CAC).
class. The r values
We now choose a sample of images from the dataset where LINEX/real has highest instance- level infidelity numbers and display them in Figure 21 Just looking at the explanations and the corresponding original images visually, it is evident that LINEX/real highlights the prominent features like stripes the t-shirt, handles of the bags, outlines of the boots/shoes, even though the infidelity values are high. However, MeLIME misses out on some of these prominent features and focuses only on optimizing the fit. The fact that LINEX zeroes in on important features also provides additional evidence for the closeness of GI metries between the two methods, and the better performance of LINEX/real with CI, and CAC metrics.
23

---

Under review as a conference paper at ICLR 2022
Table 4: Comparing the different methods using metrics infidelity (INFD), generalized infidelity (GD), coefficient inconsistency (CT), class attribution consistency (CAC) and unidirectionality (T). indicates higher value for the metric is better, and L indicates lower is better. Results better by 2 1 % are bolded. LINEX is better than baselines 18 out of 35 cases, and worse only in 5 cases. We neighborhood sizes 10 (IRIS), 500 (MEPS), 100 (FMNIST-random), 500 (FMNIST-realliste
100 (Rotten tomatoes) for generating local explanations. We also use 3, 10, 10, as exemplar neighborhood sizes compute GI, CI and Y metries for the four datasets respectively.
Dataset
Method
INFD 4
G
C
Y1
CAC1 1
LIME
0.015 1 0.011
0.132 1.32:00013
0.319 0.31910.133 0.132
0.646 19661000 0.040
0.667 1 0.167
S-LIME
0.015 1 0.010
0.077 0.011
0.143 0.045
1.704
0.878 1 0.034
LINEX/rand
0.013
0.052 1 0.008
0.044 1 0.013
0.802 0.0043
0.921 1 0.042
McLIME
0.008 0.003
6.049:#0011
0.100
0.629
0.464 0.100
IRIS
LINEX/real
0.009 + 0.003
0.029 - 0.003
0.024 + 0.002
0.744 - 0.044
0.942 = 0.023
MAPLE
0.009 + 0.001
0.038 = 0.004
0.261 = 0.033
0.458 = 0.032
0.586
0.035
LINEX/mpl
0.013 0.000
0.000 0.020
0.026 + 0.002
0.694
0.008
0.929
0.004
SHAP
0.007
0.197
0.248
0.664
0.524
LIME
0.158
0.066
0.214
0.041
0.005 1 0.001
0.981
0.006
S-LIME
0.158
0.066
0.214 1 0.042
0.005 1 0.001
0.974 1 0.008
NA
LINEX/rand
0.130 1 0.052
0.164 1 0.021
0.003 1 0.001
0.979 1 0.006
MEPS
MAPLE
0.063 E 0.000
0.067
0.000
0.007 = 0.000
0.957 0.000
NA
LINEX/mpl
0.098 0.001
0.001 0.094
0.007 = 0.000
0.950 0.000
SHAP
0.000
0.091
0.009
0.940
NA
LIME 0.162 0.003
S-LIME
0.142 + 0.008
NA
NA
NA
NA
FMNIST
LINEX/rand
0.149 1 0.002
McLIME
0.001 0.000
0.277
0.000
0.007 1 0.000
0.769
1 0.000
0.327
0.000
LINEX/real
0.100 0.002
0.304
0.001
0.002 + 0.000
0.780
0.000
0.649
0.001
SHAP
0.000
1.962
0.589
0.551
0.038
LIME
0.079 0.036
Rotten
S-LIME
0.075 + 0.035
NA
NA
NA
NA
Tomatoes
LINEX/rand
0.069 + 0.032
McLIME
0.029 0.001
0.391
0.000
0.000 = 0.000
0.999
0.000
0.909
0.000
LINEX/real
0.053 0.000
0.361
0.000
0.000 + 0.000
1.000
0.000
0.953
0.001
SHAP
0.000
0.384
0.008
0.999
0.015
24

---

Under review as a conference paper at ICLR 2022
Table 5: Comparing the different methods using metrics infidelity (INFD), generalized infidelity (GI), coefficient inconsistency (CI), class attribution consistency (CAC) and unidirectionality (Y). I indicates higher value for the metric better, and indicates lower is better. Results that are statistically significantly better based on paired t-test are bolded. LINEX is better than baselines in 19 out of 35 cases, and worse only in cases. Plots showing behavior of these metrics with varying neighborhood size, number of environments and kernel width are in Appendix C
Dataset
Method
INFD 1
G
C1
T1
CAC f
LIME
0.015 0.011
0.132 1 0.042
0.319 1 0.132
0.646 1 0.040
0.667 1 0.167
S-LIME
0.015 + 0.010
0.077 =
0.143 = 0.045
0.704
0.878 + 0.034
LINEX/rand
0.013
0.052 0.008
0.044 0.003
0.802 0.043
0.921 = 0.042
IRIS
McLIME
0.008 0.003
= 0.018
0.219 0.108
0.464 0.100
LINEX/real
0.009 0.003
0.029 0.003
0.024 + 0.002
0.744 0.044
0.942
0.023
MAPLE
0.009 + 0.001
0.038 + 0.004
0.261 = 0.033
0.458 0.032
0.586
= 0.035
LINEX/mpl
0.013 + 0.000
0.020 + 0.000
0.026 + 0.002
0.694 + 0.008
0.929 0.004
LIME
0.158 1 0.066
0.214 T 0.041
0.005 1 0.001
0.981 1: 0.006
S-LIME
0.158 1 0.066
0.214 + 0.042
0.005 1 0.001
0.974 1: 0.008
NA
MEPS
LINEX/rand
0.130 0.052
0.164 = 0.021
0.003 = 0.001
0.979 0.006
MAPLE
0.063 0.000
0.067 = 0.000
0.007 = 0.000
0.957 = 0.000
NA
LINEX/mpl
0.098 + 0.001
0.094 + 0.001
0.007 = 0.000
0.950 = 0.000
LIME
0.162 + 0.003
S-LIME
0.142 + 0.003
NA
NA
NA
NA
FMNIST
LINEX/rand
0.149 0.002
McLIME
0.001 0.000
0.277 0.000
0.007 1 0.000
0.769 0.000
0.327 0.000
LINEX/real
0.100 + 0.002
0.304 + 0.001
0.002 + 0.000
0.780 + 0.000
0.649
0.001
LIME
0.079 0.036
S-LIME
0.075 1 0.035
NA
NA
NA
NA
Rotten
Lrdd
0.069 + 0.032
Tomatoes
McLIME
0.029 0.001
0.391 0.000
0.000 = 0.000
0.999 0.000
0.909 0.000
LrneX/rcal
0.053 + 0.000
0.361 - 0.000
0.000 = 0.000
1.000 - 0.000
0.953 - 0.001
25

---

Under review as a conference paper at ICLR 2022
MeLime: r=0.121, INFD=0.001 LINEXreal:: r=0.334, INFD=0.169
Original
INFD-
Original
MeLime: r=0.112, INFD=0.001 LINEXreal: 0=5.9,,, INFD=0.201
Original
r-0.114, r=0.412, INFD-Q.189
Original
MeLime: r-0.393, INFD=0.00 LINEX/rea:: r=0,713, INFD=0.180
Original
MeLime: r=0.162, INFD 0.001 LINEX/real: r-0.528 INFD=G.161
Original
Figure 21: Error analysis for chosen set of examples in FMNIST using MeLIME and LINEX/real methods. The three columns are the MeL.ME feature attributions, LINEX/real feature attributions, and the original images. The rows correspond to different examples. We show the Pearson's correlation coefficient between feature attributions and mean of the original images from the respective classes (r) and instance-level infidelity (INFD) measures. LINEX seems highlight important features like stripes the t-shirt, handles the bags, outlines of the boots/shoes more prominently, while MeLIME seems to overfit to the data while missing out highlighting some key features prominently.
26


Reviews:
[{"review_id": "kVvtKWPm6Av", "reviewer": "Reviewer_H918", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe introduction presents a good motivation and explains well the gap in current research. Although you do a great job defining locality later in the paper, a sentence defining what is meant by local decisions and locality would make the intro easier to read. Similar for the terms faithful, stable, and unidirectional, which are technical terms that are used before definition in the intro.\n\nThe related work does a great job of summarizing global vs local explainability, exemplar vs feature based methods, as well as robust and causal approaches.\n\nThe preliminaries offers a faithful explanation of IRM and does well to distinguish local explainability from IRM.\n\nThe description of Nash equilibria and definition of a pure strategy Nash equilibrium as well as the Desirable properties are very well written, but I wonder if this shouldn't be part of the preliminaries? With the exception of unidirectionality, which may be considered a contribution?\n\nThe method description well describes and justifies the main contributions of the paper.\n\nThe experiments section does well to describe the datasets and metrics used.\n\n**Weaknesses**\nThe main issues I have are with the experiments section.\n\n1. I'm not sure that Figure 2. demonstrates that the explanations for LINEX are better than MeLIME. They. do look smoother, but does that make them better explanations? I don't like the use of the correlation coefficient here to imply that the LINEX explanations are better as I could just put the original image as the explanation and have a perfect score. For the shoe, why is the LINEX explanation better than MeLIME?\n\n2. There doesn't seem to be a great deal of difference in the top attributed works for the sentiment analysis experiment. Is it possible to look at the top attributed bi-grams rather than just the top attributed words?\n\n3. I appreciate the effort to report standard errors, but I'm not sure it makes sense to take these over different kernel sizes. How is this a sensible approach? Assuming that this is ok, why highlight 1% improvements rather than statistical significance?", "summary_of_the_paper": "This paper proposes an extension of Locally interpretable model agnostic explanations (LIME) called Locally invariant explanations (LINEX).\nLINEX is inspired by invariant risk minimization and aims to provide high-fidelity, stable, black-box invariant, unidirectional local explanations.\nLINEX is empirically evaluated using tabular, image, and text data and the authors show improvement over LIME and other baselines over several metrics and datasets.", "summary_of_the_review": "I think this is an interesting paper that has technical novelty, however there are issues with the experimental results that need to be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZW9Y1HSGeC0", "reviewer": "Reviewer_388N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n+ LINEX is a robust form of LIME, without losing much fidelity\n+ The motivation for choosing the IRM framework is laid out well\n+ The experiments are over a wide range of datasets and evaluation metrics.\n\nConcerns:\n- LINEX doesn’t seem to be significantly better than MeLime or MAPLE, especially for non-tabular data. MAPLE is competitive on the MEPS dataset, while MeLime shows competitive performance on the FMNIST and Rotten Tomatoes datasets. When considering that LINEX takes 2.5x the training time as LIME (as mentioned in Appendix A), did other methods have the same benefit of additional training time/iterations?\n- It is not clear why the paper states ‘Causal’ in the title (and also elsewhere in the paper), this seems unnecessary. The paper uses the word ‘causal’ loosely, and there is no support to the claim that LINEX explanations are causal.\n- The paper states that ‘unidirectionality’ is a new property. However, there have been papers in recent years on robust explanations and attributional robustness (see [3],[4],[5],[6] references below). How is this different from these efforts? \n- The novelty of the work seems limited, as the algorithm and theoretical results in the paper are largely based on [1],[2].\n\n[1] Ahuja, Kartik et al. “Invariant Risk Minimization Games.” ArXiv abs/2002.04692, 2020\n[2] Ahuja, Kartik et al. “Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions.” ArXiv abs/2010.15234, 2021\n[3] Lakkaraju, Himabindu et al. “Robust and Stable Black Box Explanations.” ArXiv abs/2011.06169, 2020\n[4] Zhao, Xingyu et al. “BayLIME: Bayesian Local Interpretable Model-Agnostic Explanations.” ArXiv abs/2012.03058, 2020\n[5] Chen, et al, Robust Attributional Regularization,  NeurIPS 2019\n[6] Sarkar, Anindya et al, Enhanced Regularizers for Attributional Robustness, AAAI’21\n", "summary_of_the_paper": "The paper proposes an algorithm LINEX for learning a robust LIME-like explanation for a black box model. The algorithm is based on the Invariant Risk Minimization Games framework proposed by [1],[2]. LIME is notorious for being very sensitive to its hyperparameters, and the paper posits that a robust variant of LIME is one which is invariant to these hyperparameters. The primary contribution of the paper is to formulate the perturbation-neighbourhoods of the point to be explained as ‘environments’ in an IRM setup. The paper also notes that characteristics of such invariant predictors (Theorem 2 of [2]) make them well-suited for explanations.", "summary_of_the_review": "Utilizing invariances for robust explanations is an interesting direction, but the paper relies heavily on the algorithms and results presented in [1],[2]. The results, though competitive, are not striking enough to warrant the usage of LINEX over existing methods (and also over other formulations for robust explanations such as [3],[4] in the above references). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YgN-6yONNUy", "reviewer": "Reviewer_tbk2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- I think this paper is well motivated and offers good contributions.\n- Experimental results appear to strongly support the efficacy of the proposed method. The proposed method outperforms LIME, S-LIME and MeLime in over half the experiments, while achieving comparable performance on most that remain. However, results on FMNIST and Rotten Tomatoes appear to not be as convincing as the results on tabular data. The paper is also missing discussion on this aspect.\n- I didn't see an error analysis of the proposed method, especially for cases where the performance is worse than the baselines. That's something that I think could add to the paper quite a bit.\n- Overall the paper is well written and the presentation is quite clear. I do have some minor comments that I have listed below.\n\n------------------------------------------------------------------------\nAdditional comments and questions:\n- Page 3 local explainability setup vs IRM setup: \"we want to highlight features in our explanation that may be spurious from the domain perspective, but nonetheless the black-box model uses them to make decisions\". I think this is a wrong assertion that the underlying model \"uses them to make decisions\". These are post-hoc explanations and per my understanding, nowhere in the paper do you actually establish that the model indeed only relies on the explanations you've identified. Am I missing something?\n- Section 4.3, Assumption 1: Is this independence across dimensions in the same environment or the same dimension across environments? Current phrasing doesn't disambiguate but I'm assuming it's the latter? Please rephrase.\n- Section 4.3 Assumption 2: I'm not quite sure I follow when you say that Assumption 2 ensures \"that we closely analyze the role of the l_\\inf penalty. Please elaborate.\n\n-------------------------------------------------------\nPresentation and typos:\n- I'm not quite sure the title (\"Towards Causal Explanations\") is appropriate. To the best of my understanding, I could not identify any causal estimand being estimated in this paper.\n- Introduction Line 6: \"based on a image scan\" -> \"based on an image scan\"\n- Introduction Page 2 first paragraph: \"There have been variants suggested to overcome...\" Please expand upon this sentence and give some additional background to the reader.\n- Section 4.2.1: \"optimized subject two constraints\" -> \"optimized subject to two constraints\"\n- Section 4.2.1: \"In other words, for features where there is massive disagreement in even the direction of their impact are eliminated by our method\" There is an agreement error here. Rephrase it as \"In other words, features that have a massive disagreement in even the direction of their impact are eliminated by our method.\" Further, what does \"massive\" even mean here? Better to use grounded language or not use adjectives at all and just say disagreement.\n- Missing related work: Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. \"Causal mediation analysis for interpreting neural nlp: The case of gender bias.\" NeurIPS 2020.", "summary_of_the_paper": "Update:\nIn our back and forth, the authors have addressed or committed to address the concerns I had with this work. As a result, I have updated my score\n\n---------------------------------------------\n\nThis paper introduces a new model agnostic local explanation method based on a game-theoretic formulation to ensure that resulting explanations are high fidelity and unidirectional across nearby examples. The idea is to first form several local environments via random perturbations or using creating realistic neighbors using generative or retrieval methods. Then, using the proposed algorithm, we iteratively learn a constrained least squares predictor for each environment and the final predictor is then the sum of these individual predictors. Experiments on three data modalities (tabular, vision, and text) show that the proposed method leads to better explanations than existing methods such as LIME.", "summary_of_the_review": "Overall I think the paper provides a nice and sound contribution. The results are convincing on the tabular data but I would like some discussion about other modalities where results are somewhat comparable across several metrics. In addition, I have a few points I'd like some clarity on.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9RIkBXf9og", "reviewer": "Reviewer_9WKN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One concern about creation of the local environments. Faithfulness of the local predictor to the original model has commonly be a concern in local explanation methods. Creating “environments” around a data unit further exacerbates this concern. How can one assure the faithfulness of the predictors in each environment to the original model? Even if one can empirically show that such approach works, without a theoretical guarantee, one cannot rely on LINEX. On the other hand, one other related concern is the answer to the following question: Hoe exactly to create environments? The authors themselves acknowledge that this is important and propose two ways of doing so: Random perturbation, and realistic generation according to the literature that they cite. IMO, one major concern is solving this issue. Pointing to ongoing work about this problem does not suffice. This paper is not clearly stating whether they are proposing a solid solution on how to create local environments.\n\nThe title of the paper suggests that this paper provides causal explanations. The main text attempts at slightly maneuvering over the terminology of causality. The abstract claims that the work ascertains explanatory features without using a causal graph. All of these evidence point toward using causal inference and/or explaining the black box model with causal features. However, the paper falls short of providing such explanations. In fact, to my understanding, the authors do not extract causal features. It would be great if the authors clarify on when/where they extract causal features, or alternatively, modify their writing not to reflect causal inference.\n\n The constraint in the optimization of the proposed algorithm does not necessarily formulate the situation of a Nash equilibrium. I am not clear about the theoretical properties of the Nash equilibrium in this problem. If one wants to interpret the results using concepts of this equilibrium, the whole problem needs to be formally formulated and axiomatically translated.\n\n The authors have shown their results on more than one environment (<= 5) in the Appendix. I am wondering about the behavior on even more environments? How would the authors recommend choosing the number of environments? I have an understanding that this is, at the moment, determined only empirically, and am not clear if there is any theoretical analysis over the number of environments.\n\nOn the application side, the paper is showing interesting results.\n", "summary_of_the_paper": "This paper approaches the problem of locally explaining predictions of black box models. They attempt to do so by creating local environments around a data unit. They propose their method, LINEX to overcome the shortcomings of the method, LIME. LINEX learns locally invariant predictors and aggregates their explanation for locally explaining the original model. The authors show some theoretical properties as well as empirical evidence.", "summary_of_the_review": "The paper lacks in novelty, contribution, and supporting theory of the proposed methodology. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are largely in agreement that this proposal would benefit from more clarity and comparison to key papers/findings in this space. While one reviewer is leaning towards acceptance, and their points were considered by the other reviewers, there wasn't a consensus towards aligning towardsa an acceptance. Thus, I recommend that the authors take advantage of the reviewers' comments to further improve their manuscript.
Decision: Reject