Can I Trust the Explainer? Verifying Post-Hoc Explanatory Methods
Authors: Oana-Maria Camburu*
Eleonora Giunchiglia*
Jakob Foerster
Thomas Lukasiewicz
Phil Blunsom
Venue:  (2020)
Keywords: explainability
neural networks

Abstract: For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks. In this work, we identify two issues of current explanatory methods. First, we show that two prevalent perspectives on explanations—feature-additivity and feature-selection—lead to fundamentally different instance-wise explanations. In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals. The second issue is that current post-hoc explainers have only been thoroughly validated on simple models, such as linear regression, and, when applied to real-world neural networks, explainers are commonly evaluated under the assumption that the learned models behave reasonably. However, neural networks often rely on unreasonable correlations, even when producing correct decisions. We introduce a verification framework for explanatory methods under the feature-selection perspective. Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able to provide guarantees on its inner workings. We validate the efficacy of our evaluation by showing the failure modes of current explainers. We aim for this framework to provide a publicly available,1 off-the-shelf evaluation when the feature-selection perspective on explanations is needed.

Under review as a conference paper at ICLR 2020
CAN I TRUST THE EXPLAINER? VERIFYING POST-HOC EXPLANATORY METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
For Al systems to garner widespread public acceptance, we must develop meth- ods capable of explaining the decisions of black-box models such neural net- works. In this work, we identify two issues of current explanatory methods. First, we show that two prevalent perspectives on to fundamentally different instance-wise explanations. In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals. The second issue is that cur- rent post-hoc explainers are either validated under simplistic scenarios (on simple models such linear regression, when applied to real-world neural networks, explainers are commonly validated under the assumption that the learned models behave reasonably. However, neural networks often rely on unreasonable correlations, even when producing correct decisions. We introduce a verification framework for explanatory methods under the feature-selection perspective. Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able pro- vide guarantees on its inner workings. We validate the efficacy of our evaluation by showing the failure modes of current explainers. We aim for this framework provide publicly available off shellf evaluation when the feature-selection perspective on explanations needed.
and
on models trained syntactic datasets), or,
INTRODUCTION
A large number of post-hoc explanatory methods have recently been developed with the goal of shedding light on highly accurate, yet black- box machine learning models (Ribeiro al.] 2016a
Lundberg & Lee 2017 Arras et al. 2017 [Shrikumar et al., 2017: Ribeiro et al. [01 2018
Plumb et al., 2018: Chen et al., 2018). Among these methods, there are currently at least two
widely used perspectives on explanations: feature-addiiivvty (Ribeiro et al., 20166a Lundberg &
Lee 2017: Shrikumar et al.) 2017 Arras et al.) (017)) and feature-selectie
(Chen et al. 2018 Ribeiro et al.) 2018 Carter et al.J 2018), which we describe in detail in the sections below. While both shed light on the overall behavior of model, we show that, when comes to explaining the prediction on single input in isolation, i.e., instance-wise explanations, the two perspectives lead to fundamentally different explanations. In practice, explanatory methods adhering to different perspectives are being directly compared. For example. Chen et al. (2018) and Yoon al. (2019) compare L2X, fcature-selection explainer, with LIME (Ribeiro et al. 2016aa and SHAP (Lundberg & Lee 2017). two feature-additivity explainers. We draw attention the fact that these comparisons may not be coherent, given the fundamentally different explanation targets, and we discuss the strengths and limitations of the two perspectives.
Secondly, while current explanatory methods are successful in pointing out catastrophic biases, such as relying on headers to discriminate between pieces of text about Christianity and atheism (Ribeiro) et al. 20), it is an open question to what extent they are reliable when the model that they aim to explain (which we call the targel model) has a less dramatic bias. This a difficult task, precisely because the ground-truth -truth decision-making process of neural networks is not known. Consequently, when applied to complex neural networks trained on real- wolld datasets, a prevalent way to evaluate the explainers is assume that the target models behave reasonably, i.e., that they did not rely
Code, generated datasets, and trained models will be released.

---

Under review as a conference paper at ICLR 2020
on irrelevant correlations. For example, in their morphosyntactic agreement paradigm, Pörner et al. (2018) assume that a model that predicts if a verb should be singular or plural given the tokens before the verb, must be doing so by focusing on a noun that the model had identified as the subject. Such assumptions may be poor, since recent works show a series of surprising spurious correlations in human-annotated datasets, on which neural networks learn to heavily rely (Gururangan al. 2018 Glockner et al. 20188) Carmona et al. 2018). Therefore, it is not reliable to penalize an explainer for pointing to tokens that just do not appear significant us.
We address the above issue by proposing a framework capable of generating evaluation tests for the explanatory methods under the feature-selection perspective. Our tests consist of pairs of (tar- get model, dataset). Given a pair, for each instance in the dataset, the specific architecture of our model allows us to identify a subset of tokens that have zero contribution to the model's prediction on the instance. We further identify a subset of tokens clearly relevant to the prediction. Hence, we test explainers rank zero-contribution tokens higher than relevant tokens. We instantiated our framework on three pairs of (target model, dataset) on the task of multi-aspect sentiment analysis. Each pair corresponds to an aspect and the three models (of same architecture) have been trained independently. We highlight that our test is not a sufficient test for concluding the power of explain- ers in full generality, since we do not know the whole ground-truth behaviour of the target models. Indeed, we do not introduce an explanation generation framework but framework for generating evaluation tests for which we provide certain guarantees on the behaviour of the target model. Un- der these guarantces we are able to test the explainers for critical failures. Our framework therefore generates necessary evaluation tests, and our metrics penalize explainers only when we are able to guarantee that they produced an error. To our knowledge, we are the first to introduce an automatic and non-trivial evaluation test that does not rely on speculations on the behavior of the target model.
Finally, we evaluate L2X (Chen et al. 2018), a feature-selection explainer, under our test. Even though our test is specifically designed for seature-selection explanatory methods, since, in prac- tice, the two types of explainers are being compared, and, since LIME (Ribeiro et al.) 2016a) and SHAP (Lundberg & Lee, 2017) are two very popular explainers, we were interested in how the latter perform on our test, even though they adhere to the fcature-additivity perspective. Interestingly, we find that, most of the time, LIME and SHAP perform better than L2X. We will detail in Section 5 the reasons why we believe this is the case. We provide the error rates of these explanatory meth- ods to raise awareness of their possible modes of failure under the crature-selection perspective of explanations. For example, our findings show that, in certain cases, the explainers predict the most relevant token to be among the tokens with zero contribution. We will release our test, which can be used off-the-shelf, and encourage the community to use it for testing future work on explanatory methods under the feature-selection perspective. We also note that our methodology for creating this evaluation is generic and can be instantiated on other tasks or areas of research.
2 RELATED WORK
The most common instance- wise explanatory methods are feature-based, i.e., they explain a predic- tion in terms of the input unit-features (e.g.. tokens for text and super-pixels for images). Among the feature-based explainers, there are two major types of explanations: (i) feature-additive: provide
signed weights for each input feature, proportional to the contributions of the features to the model's
prediction (Ribeiro et al. 2016aa Lundberg & Lee| 2017 [Shiikuarr et al. 2017 Arras al. 2017)
and (ii) selectiv:: provide a (potentially ranked) subset of features responsible for the pre- diction (Chen et al. 2018 Ribeiro al.) 2018 Carter et al.) 2018). We discuss these explanatory methods in more detail in Section 3 Other types of explanations are (iii) example-based (Koh & Liang] 2017): identify the most relevant instances in the training set that influenced the model's prediction on the current input, and (iv) human--evee explanations (Camburu et al. [018]] Park al.) 2018 Kim et al.) 20188 Bekele et al., (2018): explanations that are similar to what humans provide in real-world, both in terms of arguments cauuan-bisses, and form (full-sentence natural language). In this work, we focus on verifying feature-based explainers, since they represent the majority of current works.
While many explainers have been proposed, it is still an open question how to thoroughly validate their faithfulness to the target model. There are four types of evaluations commonly performed:
2

---

Under review as a conference paper at ICLR 2020
1. Interpretable target models. Typically, explainers tested on linear regression and decision trees (c.g., LIME (Ribeiro et al., 2016ab) or support vector representations (e.g., MAPLE (Plumb et al., 2018)). While this evaluation accurately assesses the faithfulness of the explainer to the target model, these very simple models may not be representative for the large and intricate neural networks used in practice.
2. Synthetic setups. Another popular evaluation setup is to create synthetic tasks where the set of important features is controlled. For example, L2X (Chen et al. [2018) was evaluated on four synthetic tasks: 2-dim XOR, orange skin, nonlinear additive model, and switch fea- ture. While there is no limit on the complexity of the target models trained on these setups, their synthetic nature may still prompt the target models to learn simpler functions than the ones needed for real-world applications. This, in turn, may ease the job for the explainers.
3. Assuming reasonable behavior. In this setup, one identifies certain intuitive heuristics that a high-performing target model is assumed to follow. For example, in sentiment analy- sis, the model is supposed to rely adjectives and adverbs in agreement with the predicted sentiment. Crowd-sourcing evaluation is often performed to assert if the features produced by the explainer are in agreement with the model's prediction (Lundberg & Leel 2017; Chen et al. 2018). However, neural networks may discover surprising artifacts (Gururan- gan et al. 2018) to rely on, even when they obtain high accuracy. Hence, this evaluation is not reliable for assessing the faithfulness of the explainer to the target model.
4. Are explanations helping humans to predict the model's behaviour? In this evaluation, humans are presented with a series of predictions of model and explanations from differ- ent explainers, and are asked to infer the predictions (outputs) that the model will make on a separate set of examples. One concludes that an explainer E1 is better than an explainer F2 if humans are consistently better at predicting the output of the model after seeing ex- planations from El than after seeing explanations from E2 (Ribeiro et al. 2018). While this framework is a good proxy for evaluating the real-world usage of explanations, it ex- pensive and requires considerable human effort if it is to be applied on complex real-world neural network models.
In contrast to the above, our evaluation is fully automatic, the target model is a non-trivial neural network trained on a real-world task and for which we provide guarantees on its inner workings.. Our framework is similar in scope with the sanity check introduced by [Adebayo et al. (2018). However, their test filters for the basic requirement that an explainer should provide different explanations for a model trained on real data than when the data and/or model are randomized. Our test is therefore more challenging and requires a stronger fidelity of the explainer to the target model.
INSTANCE- EXPLANATIONS
As mentioned before, current explanatory methods adhere to two major perspectives of explanations:
Perspective I (Feature-addiiivity) For a model I and an instance v, the explanation of the predic- tion f(xx consists of set of contributions {wf (f)); for each feature of a such that the sum of the contributions of the features in approximates f(x),, i.e E, (f) f(x).
Many explanatory methods adhere to this perspective (Arras et al. 2017 Shrikumar et al. 2017 Ribeiro et al. 2016a). For example, LIME (Ribeiro et al.) 2016a) learns the weights via linear regression on the neighborhood (explained below) of the instance. Lundberg & Lee (2017) unified contributions that verify three
this class of methods by showing that the only set of feature- desired constraints (local accuracy, missingness, and consistency- we refer to their paper for details) given by the Shapley values from game theory:
|x'|
1)!
wf( = Σ
f(a'))
(I)
x
x' 2'E2(())
where the sum enumerates over all subsets x' of features in x that do not include the feature i, and |- denotes the number of features of its argument.
Thus, the contribution of each feature in the instance x is an average of its contributions over a neighborhood of the instance. Usually, this neighborhood consists of all the perturbations given by
3

---

Under review as a conference paper at ICLR 2020
M: "very good" input: RETURN 0.9; IF "nice" IN input: RETURN 0.7;
IF "good" IN inpur: RETURN 0.6;
RETURN O.
21: "The movie was good, was actually nice."
22: "The movie was nice, in fact, it was very good."
((1) 0.7
(x2)) 0.9
Feature-additivity
Feature-selection
Feature-additivity
Feature-selection
nice: 0.4
{nice)
good: 0,417
{good, very)
good: 0.3
nice: 0.367
rest of tokens: 0
very: 0.116
rest of tokens: 0
Figure Examples on which the two perspectives give different instance-wiss explanations.
masking out combinations of features in z; see, e.g.. (Ribeiro et al. 2016a Lundberg & Lee 2017).
However, [Laugel al. (2018) show that the choice of the neighborhood is critical, and it is open question what neighborhood is best to use in practice.
Perspective 2 (Feature-seleccionn): For model f and an instance the explanation of ((xr consists of sufficient (ideally small) subset (xx) of (potentially ranked) features that alone lead (almost) the same prediction as the original one, i.e., f(S(z)) sz f(x)..
Chen al. (2018); Carter et al. (0118) and Ribeiro al. (2018) adhere this perspective. For example, L2X (Chen et al. 2018 learns S(z) by maximizing the mutual information between S(x:) and the prediction. However, assumes that the number of important features per instance, i.c., [s()).. is known, which is usually not the case in practice. A downside of this perspective is that may not always be true that the model relied only on a (small) subset of features, as opposed to using all the features. However, this can be the case for certain tasks, such as sentiment analysis.
To better understand the differences between the two perspectives, in Figure I we provide the instance-wise explanations that each perspective aims to provide for hypothetical sentiment anal- ysis regression model, where is the most negative and the most positive score. We note that our hypothetical model is not far, in behaviour, from what real-world neural networks learn, especially given the notorious biases in the datasets. For example, Gururangan ct al. (2018) show that natural language inference neural networks trained on SNLI (Bowman et al. 2015) may heavily rely the presence of a few specific tokens the input, which should not even be, in general, indicators for the correct target class, e.g., "outdoors" for the entailment class, "tall" for the neutral class, and "sleeping" for the contradiction class.
In our examples in Figure [ clearly the differences between the perspectives. For the instance 21, the fcature-additive explanation tells us that "nice" was the most relevant feature, with weight of 0.4, but also that "good" had a significant contribution of 0.3. While for this instance alone, our model relied only on "nice" to provide a positive score of 0.7, it is also true that, if "nice" was not present, the model would have relied on "good" to provide a score of 0.6. Thus, we see that the Ceature-additiv perspective aims to provide average explanation of the model on neighborhood of the instance, while the feature-selective perspective aims to tell us the pointwise features used by the model on the instance in isolation, such as "nice" for instance æ1.
An even more pronounced difference between the two perspectives is visible on instance 22, where the ranking of features is now different. The feature-selective explanation ranks "good" and "nice" as the two most important features, while on the instance 22 in isolation, the model relied on the tokens "very" and "good", that the feature-selection perspective would aim to provide.
Therefore, we see that, while both perspectives of explanations give insights into the model's behav- ior, one perspective might be preferred over the other in different real-world use-cases. In the rest of the paper, we propose a verification framework for the feature feature-selection selection perspective of mastance-wise explanations.
4

---

Under review as a conference paper at ICLR 2020
4 OUR VERIFICATION FRAMEWORK
Our proposed verification framework leverages the architecture of the RCNN model introduced by Lei et al. 2016). We further prune the original dataset on which the RCNN had been trained to ensure that, for each datapoint z, there exists set of tokens that have zero contribution (irrelevant features) and set of tokens that have a significant contribution (clearly relevant features) RCNN's prediction on v. We further introduce a set of metrics that measure how explainers fail to rank the irrelevant tokens lower than the clearly revelant ones. We describe each of these steps detail below.
The RCNN. The RCNN (Lei al. 2016) consists of two modules: a generator followed by an encoder, both instantiated with recurrent convolutional neural networks (Lei et al. 2013). The generator bidirectional network that takes as input piece of text æ and, for each of its tokens, outputs the parameter of a Bernoulli distribution. According to this distribution, the RCNN selects a subset of tokens from x, called S₂ = generator(zr). and passes it to the encoder, which makes the final prediction solely as function of S,.. Thus:
RCNN(x) encoder ((neodorgeeeerato((()) = encoder(S,))
(2)
There is no direct supervision on the subset selection, and the generator and encoder were trained jointly, with supervision only on the final prediction. The authors also used two regularizers: one to encourage the generator to select short sub-phrase, rather than disconnected tokens, and a second to encourage the selection of fewer tokens. At training time, to circumvent the con-differentiability introduced by the intermediate sampling, the gradients for the generator were estimated using REINFORCE-style procedure (Williams 1992).
This intermediate hard selection facili- tates the existence of tokens that not have any contribution to the final pre- diction. While Lei et al. (2016) aimed for S₂ to be the sufficient rationals for each prediction, the model might have learned internal (emergent) commu- nication protocol (Foerster et al. (016)) that encodes information from the non-selected via the selected tokens, which call handshake. For example, the RCNN could learn handshake such as the one in Figure 2 where the feature "good" was important in all three cases, but not selected in the first two.
IF "very good" IN input: SELECT "very" & RETURN 1; IF "not good" IN input: SELECT "not" & RETURN 0.1; IF "good" IN impur: SELECT "good" & RETURN 0.8; ELSE SELECT @ & RETURN 0.5.
Figure 2: Example of handshake.
Eliminating handshakes. Our goal is to gather dataset D such that for all E D, the set of non-selected tokens, which we denote N2 = x SSy. has zero contribution to the RCNN's prediction on Equivalently, we want eliminate instances that contain handshakes. We show that:
Ss. S.
no handshake in Ø
The proof is in Appendix B On our example in Figure 2 on the instance "The movie was very good.", the model selects "very" and predicts a score of However, if we input the instance con- sisting of just "very", the model will not select anything and would return score of 0.5. Thus, Equation
/ iindeed captures the handshake in this example. From now on, we refer to non-selected tokens as irrelevant cero-contribution interchangeably.
On the other hand, we note that Ss, * S2 does not necessarily imply that there was a handshake. There might be tokens (e.g., the at the ends of the selection sequence(s)) that might have been selected in the original instance x and that become non-selected in the instance formed by S, without significantly changing the actual prediction. However, since it would be diffieult to differentiate between such a case and an actual handshake, we simply prune the dataset by retaining only the instances for which Sg.- Sr.
At least one clearly relevant feature. With our pruming above, we ensured that the non-selected tokens have no contribution to the prediction. However, we are yet not sure that all the non-selected
Our experiments show that the RCNN is capable of not selecting anything and providing its "hias" score as prediction. For example, this happened when inputted sentences completely irrelevant the task.
5

---

Under review as a conference paper at ICLR 2020
tokens relevant the prediction. In fact, is possible that some tokens (such "the" "a") actually noise, but have been selected only to ensure that the selection is contiguous sequence (as mentioned, the RCNN was penalized during training for selecting disconnected tokens). Since we do not want to penalize explainers for not differentiating between noise and zero-contributton features, further prune the dataset such that there exists least selected token which is, without any doubt, clearly relevant for the prediction. To ensure that a given selected token is clearly relevant, we check that, when removing the token s, the absolute change prediction with respect to the original prediction is higher than significant threshold т. Precisely, if for the selected encoder((.)) 2 т, then the selected token is clearly
token s E Sg, we have that (encoder(S,, relevant for the prediction.
Thus, we have further partitioned S, into S SR U SDK where SR. are the clearly relevant tokens, and SDK,, are the rest of the selected tokens for which we do know if they are relevant or noise (SDK stands for "selected don't know"). We see dia- gram of this partition Figure 3 We highlight that simply because selected token alone did not make change in prediction higher than a threshold does not mean that this token is not relevant, as may be essential in combination with other tokens. Our proce- dure only ensures that the tokens that change the prediction by a given (high) threshold are indeed important and should therefore be ranked higher than any of the non-selected tokens, which have zero contribution. We thus further prune the dataset retain only the datapoints for which [SR=z] 2 1, i.e., there is at least one clearly relevant token per instance.
SR (cloarly relevant features)
(rrolevant features)
SDK (we don't know relevant noise)
Figure 3: Partition of the fea- tures in our instances.
Evaluation metrics. First, we note that our procedure does not provide an explainer in itself, since we do not give an actual ranking, nor any contribution weights, and it is possible for some of the tokens in SDK,, to be even more important than tokens in SRR.. However, we guarantee the following two properties:
Pl: All tokens in N₂ have to be ranked lower than any token in SR
P2: The first most important token has be Sz.
We evaluate explainers that provide ranking over the features. We denote by the ranking (in decreasing order of importance) given by an explainer on the n = |ar features in the instance æ. Under our two properties above, we define the following error metries:
ri (x),, r2(x))....,
(A) Percentage of instances for which the most important token provided by the explainer is among the non-selected tokens:
% first
Σ ((1()))N)))
P.I
ZED=
where is the indicator function.
(B) Percentage of instances for which at least one non-selected token is ranked higher than a clearly relevant token:
mimismk
Σ 1/3i< such (x)))N, and (x)ESR..))
(5)
Da
ZED,
(C) Average number of non-selected tokens ranked higher than any clearly relevant to- ken:
avg Pall ZED, i<lansi
Σ 1(x()))Na))
(6)
where last_si argmax, (ry(x) SRR} is the lowest rank of the clearly relevant tokens.
Metric (A) shows the most dramatic failure: the percentage of times when the explainer tells us that the most relevant token is of the contribution ones. Metrie (B) shows the percentage of instances for which there at least an error in the explanation. Finally, metric (C) quantifies the number of zeno-contribution features that ranked higher than any clearly relevant feature.

---

Under review as a conference paper at ICLR 2020
Table 1: Error rates of the explainers. The lower the values, the better the explainer. Best results in bold. In parenthesis are the standard deviations when averages are reported.
APPEARANCE
AROMA
PALATE
Model
fi.ristt
%.misrnn
avg muisrnk
%..first
S.misrnk
avg avg.misrrk
% .o.first
K_mismnk
avg_mismnk
LIME
4.24
24.39
7,02 (24.12)
14.79
32.08
12.74 (33.54)
2.92
13.93
3,48 (17.38)
SHAP
4.74
16.81
1.16 (7.75)
4.24
13.53
0.83 (7.10)
2.65
9.20
9.25 (9.70)
L2X
6.58
28.85
3.54 (12.66)
12.95
31.61
4.41 (16.25)
12.77
29.83
3.70 (13.05)
RESULTS AND DISCUSSION
In this work, we instantiate our framework on the RCNN model trained the BeerAdvocate cor- pus@on on which the RCNN was initially evaluated (Lei et al. 2016). BeerAdvocate consists of a total of ~ .100K cuman-gennrated multi-aspect beer reviews, where the three considered aspects are appearance, aroma, and palate. The reviews are accompanied with fractional ratings originally between 0 and 5 for each aspect independently. The RCNN is a regression model with the goal to predict the rating, rescaled between O and 1 for simplicity. Three separate RCNNs are trained, one for each aspect independently, with the same default settings
With the above procedure, we gathered three datasets Da. one for each aspect a. For each dataset, we know that for each instance E De, the set of non-selected tokens N₂ has zero contribution to the prediction of the model. For obtaining the clearly relevant tokens, we chose a threshold of T = 0.1, since the scores are in [0, 1], and the ground-truth ratings correspond to {0, 0.1, O. 1}. Therefore, change in prediction of 0.1 is to be considered clearly significant for this task.
We provide several statistics of our datasets in Appendix A For example, we provide the average lengths of the reviews, of the selected tokens per review, of the clearly relevant tokens among the selected, and of the non-selected tokens. We note that we usually obtained or 2 clearly relevant tokens per datapoints, showing that our threshold of 0.1 is likely very strict. However, be more conservative in order to ensure high guarantees on our evaluation test. We also provide the percentages of datapoints eliminated in order ensure the no-handshake condition (Equation
prefer to
Evaluating explainers. We test three popular explainers: LIME (Ribeiro et al., 2016a), SHAP (Lundberg & Lee, 2017). and L2X (Chen al. 2018). We used the code of the explainers as pro- vided in the original repositories with their default settings for text explanations, with the exception that, for 1.2X, we set the dimension of the word embeddings to 200 (the same as in the RCNN), and also allowed training for a maximum of 30 epochs instead of 5.
As mentioned Section 3 LIME and SHAP adhere to the feature-additivity perspective, hence our evaluation is not directly targeting these explainers. However, we see in Table I that, in practice, LIME and SHAP outperformed L2X on the majority of the metrics, even though L2X is a feature- selection explainer. We hypothesize that major limitation of L2X is the requirement to know the number of important features per instance. Indeed, 1.2X learns a distribution over the set of features by maximizing the mutual information between subsets of K features and the response variable, where K is assumed to be known. In practice, one usually does not know how many features per instance model relied on. To test L2X under real-world circumstances, we used K the average number of tokens highlighted by human annotators on the subset manually annotated by MccAllyy al. 2012). We obtained an average K of 23, 18, and for the three aspects, respectively.
In Table we see that, on metric (A) all explainers are prone to stating that the most relevant feature a token with zero contribution, as much as 14.79% of the time for LIME and 12.95% of the time for I.2X in the aroma aspect. We consider this the most dramatic form of failure. Metric (B) shows that both explainers can rank at least one zero-contribution token higher than a clearly relevant feature, i.e., there is at least one mistake in the predicted ranking. Finally, metric (C) shows
http://wwwpp...aaa......ccc/l/// httpp://gihuu..coo//oool/////// hhtps://githh..cmm//// http://ggttt..oooommoo/////e//ee http://ghhhaaccm//ummmber/gaggg
7

---

Under review as a conference paper at ICLR 2020
that, in average, SHAP only places one zero-contribution token ahead of clearly relevant token for the first two aspects and around 9 tokens for the third aspect, while L2X places around 3-4 zero-contribution tokens ahead of clearly relevant one for all three aspects.
LIME: shared at kahns with easher, 2009 vintage poured black and beautiful
1.
with a chewing tobbaco brown colored head smelled and tasted like roasty
2. mouthfeel
malts and barley.. slight coffee slight bitter chocolate. barely any bouroon
3. lacing
present in the smell and Easte... great mouthheee and gorgeous
1. great
lacing in 's solid beer however was expecting more)
5. and
SHAP: shared kahhus with ansher, 2009 viatage pouzed black and beautiful
1. mouthieed
with a chowing tohbaca hrown colored head. smelled and tasted like roasty malts and barley... , slight coffce, , slight. bister chocolate.. barely any bourhon
2. lacing
3. great
present in the smmell laste. great mouthlee
gorgcous
4. gorgesuus
lacing
solid beer. however was expecting more...
5. and
L2X: shared at kahns with nasher 2008 vintage poured black and beautiful
1. mouthfeel
with a chewing tubbacu brown colored head. smelled tasted like roady
2. lacing
malts and barley slight coffee slight bitter chocolate barely any
3. and
bourbon present in the smell and taste.. great mourthoee and gorgeons
4, and
Lacing it 's a solid beer. i however was expecting more
5.
Figure 4: Explainers' rankings (with top 5 features on the right-hand side) on an instance from the palate aspect in our evaluation dataset.
Qualitative Analysis. In Figure 6 we present an example from our dataset of the palate aspect. More examples in Appendix C The heatmap corresponds to the ranking determined by each ex- plainer, and the intensity of the color decreases linearly with the ranking of the tokens 6 We only show in the heatmap the first K 10 ranked tokens, for visibility reasons. Tokens in Sg are in bold, and the clearly relevant tokens from SR2 are additionally underlined. The first selected by the explainer is marked wth a rectangular. Additionally the ranks tokens by each explainer are on the right-hand side. Firstly, we notice that both explainers are prone to attributing importance to non- selected tokens, with LIME and SHAP even ranking the tokens "mouthfeel" and "lacing" belonging to N₂r as first two (most important). Further, "gorgeous", the only relevant word used by the model, did not even make it top 13 tokens for L2X. Instead, L2X gives "taste", "great", "mouthfeel" and "lacing" as most important tokens. We note that if the explainer evaluated by humans assuming that the RCNN behaves reasonably, then this choice could have well been considered correct.
6 CONCLUSIONS AND FUTURE WORK
In this work, we first shed light on an important distinction between two widely used perspectives of explanations. Secondly, we introduced an off-the-shelf evaluation test for post-hoc explanatory methods under the feature-selection perspective. To our knowledge, this is the first automatic veri- fication framework offering guarantees on the behaviour of a non-trivial real-world neural network. We presented the error rates on different metrics for three popular explanatory methods to raise awareness of the types of failures that these explainers can produce, such as incorrectly predicting even the most relevant token. While instantiated on a natural language processing task, our method- ology is generic and can be adapted to other tasks and other areas. For example, in computer vision, one could train a neural network that first makes a hard selection of super-pixels to retain, and sub- sequently makes a prediction based on the image where the non-selected super-pixels have been blurred. The same procedure of checking for zero contribution of non- seleseeeceed super super-pixels would then apply. We also point out that the core algorithm in the majority of the current post-hoc explain- ers are also domain- aggnoiiag Therefore, we expect our evaluation to provide a representative view of the fundamental limitations of the explainers.
Whiie for LIME and SHAP we could have used the actual weights, for consistency, and since evaluate the explainers only their rankings, we keep the ranking-like heatmap.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. CoRR,
2018. URL http:://axxi...../////
abs/ 1010/1110.0 03292
Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojeiech Samek. Explaining recurrent neural network predictions in sentiment analysis. In Proceedings of the 8th Workshop Com- putational Approaches to Subjectiviry, Sentiment and Social Media Analysis, pp. 169.168.11 2017.
doi: 10.18653/11/117-5222 URL http:// web . org/ anthol ogy/ 5222
Esube Bekele, Wallace E. Lawson, Zachary Horne, and Sangeet Khemlani. Implementing robust explanatory bias person re-identification network. In The IEEE Conference on Computer Vision and Puttern Recognition ( (CVPR) Workshops, June 2018.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning, A large an- notated corpus for learning natural language inference. CoRR, abs/15008055 053326, 2015. URL http://arxiv/ www.:oggg/aa.o....ooooo...o.... abtp//b//1100 /1088 05326
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-SNLI Nat- ural language inference with natural language explanations. In Advances in Neural Informa- tion Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), pp. 9560-9572, 2018.
Vicente Iván Sánchez Carmona, Jeff Mitchell, and Sebastian Riedel. Behavior analysis of NLI models: Uncovering the influence of three factors on robustness. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2018, Volume / (Long Papers), pp. 1975-1985, 2018. URLAattpss//aclaathoooooo info wwtp::/oooooppooeeaoooaaemm 1111://////11111111111111111
Brandon Carter, Jonas Mueller, Siddhartha Jain, and David K. Gifford. What made you do this? Understanding black box decisions with sufficient input subsets. CoRR, abs/ abs/1810.03805, 2018. URL /att:://arxte org/ abs/1810 03805
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-thhoreiic perspective on model interpretation. In Proceedings of the 35th Interna- tional Conference Machine Learning, volume 80 of Proceedings of Machine Learning Re- search, pp. 983-892, 10-15 Jul 2018. URL httt::/pproceednngg mir. mlr www.s/peessssc.s. chen18j.htm.
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to com- municate with deep multi multi-agent reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2145-2153, USA, 2016. Curran Associates Inc. ISBN 978-1-5108-3881--1 URL Utthttp://cl acm. wwtpoooggttaioooooo.
0i11i--3157996.317333
Max Glockner, Vered Shwartz, and Yoav Goldberg. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), Volume 2: Short Papers, pp. 650-655, 2018. URL 211221101ppp1 2103
https::/acllathollooo wwwphhhwwwoo...o.ooooooooocooooe wtw.:oow..nnnooaaan/rr.... pttppppppppp//11111///pp....
Suchin Gururangan, Swabha Swayamdipta. Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume (Short Papers), pp. 107-112, 2018.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John F. Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. CoRR, tbs/1807.1 1546, 2018. URL http:h::///xxwi..ro/////////
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pp. 1885-1894, 2017. URL httl::t/ppooceeiiiiiii
mlr.pre www.peass/v//kee 7a html

---

Under review as a conference paper at ICLR 2020
Thibault Laugel, Xavier Renard, Marie-Jeanne Lesot, Christophe Marsala, and Marcin Detyniecki. Defining locality for surrogates in post-hoe interpretablity. CoRR, abs/1806.07798, 2018. URL
http::/arxiv..om org/ http//bbbs........//. 07498
Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Molding CNNs for text: Non-linear, non-
consecutive convolutions. CoRR, (bs//508.04111 12, 2015. URL
04112
Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. abs/1606.04155 2016.
Rationalizing neural predictions. CoRR,
Scott M Lundberg and Su-In Lee. In Advances in Neural Information Processing Sys- tems 30, pp. 4765-4774. 2017.
A unified approach
interpreting
model predictions.
URL http://papers. www..inps........mm....m cctppper/
7062 a wwwuiiiieee approchddaapprcch Coniccch-t-innsee to--interreetinne peappprrettng-meeeepprprrpeteeee
pdf
Julian J. McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi- aspect reviews. CoRR, abs/1210.3326,, 2012. URL http:::/aax.....rg////ooo//o
3926
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. CoRR, abs/1802.08 129, 2018. htl/aattpp/arxxi.ogppp/
08129.
Gregory Plumb, Denali Molitor, and Ameet S. Talwalkar. Model agnostic super- vised local explanations.
In Advances in Neural Information Processing Sys- tems 31: Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), pp. 2520-2529, 2018. URL http:://papess..etee// nips. cc/paper/ 7518-mode wtt.:/pppptttt..uur..eeee-oooooo
Nina Pörner, Hinrich Schütze, and Benjamin Roth. Evaluating neural network explanation meth- ods using hybrid documents and morphological prediction. CoRR, abs/1801.06422, 2018. URL
http:://rxx:
hbargg/bb/180011 06422
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Ex- plaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, 2016a. doi:
10.1145293966222933773 URL 1BLL///tss///oo.oorr///111/////1
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. Nothing else matters: Model-agnostic explanations by identifying prediction invariance. CoRR, abs/1611.0055 2016b. URL, httt:t
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High- precisirciiion model-agnostic explanations. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through
propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning (ICML 2017), pp. 3145-3153, 2017. URL Tenss//70/shriiunn
html
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforrmmmnnniinnneeeceeee learning. Machine Learning, 8:229-256. 1992. doi: 10. 10.10007/BF00992266 URL https:// doi. 0tcg/100.1007/B00099555
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. INVASE: Instance- wise variable selection using neural networks. In Proceedings the 7th International Conference Learning Represen- tations (ICLR 2019), 2019. http::ttt///peeeeeeee
10

---

Under review as a conference paper at ICLR 2020
A STATISTICS OF OUR GATHERED DATASETS
We provide the statistics our dataset in Table 2 is the number of instances that we retain with our procedure, |x| is the average length of the reviews, and [S SR and [W are the average numbers of selected tokens, selected tokens that give an absolute difference of prediction of least 0.1 when eliminated individually, and non-selected tokens, respectively. In parenthesis are the standard deviations. The column %(S,, * Sz) provides the percentage of instances eliminated from the original BeerAdvocate dataset due to a potential handshake. Finally. %[ 0) shows the percentage of datapoints (out of the non-handshake ones) further eliminated due to the absence of selected token with absolute effect of at least 0.1 on the prediction.
Table 2: Statistics of our datasets Da for each aspect.
Aspect
N
S
SR.
[
%(Ss. # S.))
% = 0)
APPEARANCE
20508
145 (79)
16.9 (8.4)
1.33 (0.70)
121 (56)
15.9
73.2
AROMA
7621
139 (74)
11.15 (6.48)
1.16 (0.49)
123 (57)
72.0
58.0
PALATE
16494
153 (76)
9.14 (5.38)
1.21 (0.55)
137 (59)
39.2
66.5
B PROOF FOR NO HANDSHAKE CONDITION
We show that:
Ss. = Proof: We note that if there is a handshake in the instance i.c., at least one non-selected token Jy E Nr is actually influencing the final prediction via an internal encoding of its information into the selected tokens, then the model should have different prediction when is eliminated from the instance, i.c., RCNN(x) き RCNN( - Ix). Equivalently. RCNN Ix = RCNN(z). then Ia could not have been part of handshake. Thus, if the RCNN gives the same prediction when eliminating all the non-selected tokens, means that there was no handshake for the instance x, and hence the tokens in N₂ have indeed zero contribution. Hence, have that:
handshake in
RCNN( - -N+)= = RCNN(x) =
no handshake in x.
(8)
Since æ - N2 Sp. Equation 8 rewrites as:
RCNN(S,) RCNN(x) = no handshake in x
From Equation we further rewrite Equation 9 ass
(rccodergenerator(((
pacoder(generator/e
no handshake in
(10)
Since, by definition, generator(2 (x) = S,, we have that:
encoder(Ss, = encoder(S,)) = no handshake in x
(11)
Hence, is sufficient have Ss, S, order to satisfy the right-hand-see condition of Equa- tion [1 which finishes our proof.
11

---

Under review as a conference paper at ICLR 2020
C
MORE EXAMPLES FROM OUR EVALUATION
LIME:
nice brown groisch like bottle ( good for re-use pours
dark
1. laces
yellow color with a lot of head in the beginning which laces well
yellow
very fizzy smells like fruit maybe some apple and blueberry mouthfeel
3. lot
w whatsover. besides being wet and a small initial alcohol could n't feel
4. dark
anything tastes of fruit and not much alcohol, but can start to feel slight warming finish off the bottle better than most american lagers but very smooth think would normally drink this too fast
5. of
SHAPP:
grolsch " like bottle ( good for re-use) ) pours dark nice brown
1. fizzy
ycllow color with a lot of head in the beginming which laces well very fizzy
2. laces
smells like fruit maybe some apple and blueberry mouthfeel
3. head
whatsover
besides being wet and small initial alcohol i could n't feel
4. nice
anything tastes of fruit and not much alcohol but can start to feel a slight warming as finish off the bottle. better than most american lagers, but very smooth. i think would normalll drink this fast
yellow
1,2X
nice brown grolsch like bottle ( good for re-use ) pours a dark
1. laces
yellow color with lot of head in the beginming which Isces well
brown
very fizzy smells like fruit maybe some apple and blueberry no mouthfoel
3. lot
whatsover besides being and small initial alcohol, i could n't feel anything. tastes of fruit and not much alcohol, but can start to fool a slight warming finish off the bottle better than most american lagers, but very smooth i think would normally drink this fast.
whatsover
nice
Figure 5: Explainers' rankings (with the top features on the right-hand side) on an instance from the appearance aspect our evaluation.
12

---

Under review as a conference paper at ICLR 2020
LIME: pouns out very dark amber-red almost no head, which dissappears quickly.
1. and
maltiness some toast, maybe bit cherry or some berry (i do n't often eat berries of any type, makes it hard to descern) )
thick " aroma rich
maltiness
3.
alcohol is pretty easy to pick up though. very rich malty flavour, little sweet first: i get hit choclate malt maybe very slightly fruity alcohol quite prominent sweet malt flavour falls off at finish and leaves mouth a little dry mouthfeel quite thick, , and more carbon dioxide in beer than is suggested by lack of head. all in all give it a pretty ok its definately strong flavour and aroma got ta give it little while for drinking, ca n't down this quick { not that i 'd ever want to )
4. flavour
malt
SIIAP: pours out very dark amber-red almost no head, which dissappears quickly
1. aroma
thick
aroma
rich maltiness some toast maybe bit of cherry jor
aroma
some herry do n't often eat berries any type, makes it hard to descern )
rich
alcohol is pretty easy to pick up though. very rich malty flavour, , little sweet first: i get hit of choclate malt maybe very slightly fruity alcohol quite prominent. sweet malt flavour falls off at finish and leaves mouth little dry- mouthfeel quite thick, and more carbon dioxide in beer than is suggested by lack of head. all in all give pretty ok its definately strong flavour and aroma got ta give it little while for drinking ca n't down this quick not that i 'd ever want to )
4. some
L2X: pours out very dark amber-rod. almost no head, which dissappears quickly
aroma
thick aroma rich maltiness some toast, maybe a bit of cherry or some berry do n't often eat berries of any type. makes it hard to descern alcohol is pretty easy to pick up though. very rich malty flavour a little
2. rich
little
which
sweet at first i get hit of choclate malt maybe very slightly fruity alcohol
5.
quite promineut sweet malt flavour falls off at finish and leaves mouth little dry mouthfeel quite thick. and more carbon dioxide in beer than suggested by lack of head. all in all give it pretty ok its definately strong flavour and aroma. got ta give a little while for drinking n't down this quick not that ever want to )
Figure 6: Explainers' rankings (with the top features on the right-hand side) on an instance from the aroma aspect in our evaluation.
13


Reviews:
[{"review_id": "rJxy1Mc0YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-------------------- AFTER\nThe original rating of \"Weak Reject\" still holds as the authors failed to provide proper justification for the raise concerns and support their claims through additional experiments.\n\n\"We do not introduce an explanation generation framework, as explainers do. \" - The proposed evaluation requires the explainer of the NLP model to agree with the RCNN in-terms of relevant or irrelevant words, to be considered a good explainer. The RCNN model which is defining the relevant and irrelevant tokens for a prediction task is in fact stating that we can explain the decision of an NLP model in terms of relevant and irrelevant tokens. Hence, the proposed RCNN can also be considered as an explainer. The evaluation task is demonstrating if the other explainers are providing explanations consistent with this new explainer based on RCNN.\n\n\n\"The RCNN is not meant to explain any other models except itself.\" - Unclear\n\n\"Regarding the request for more experiments:\" - The authors don't provide enough justification to \"why they didn't perform more experiments?\"\n\n\" Hence, with our current instantiations, any domain-agnostic explainer can be evaluated\" - The experiment to validate this claim are missing.\n\n\"The novelty of our paper consists in the fact that, to our knowledge, it is the first to (1) shed light over a fundamental difference\" - This is not a technical novelty. This is an exploratory analysis based observation\n\n\"and (2) propose a methodology for evaluating explainers that ...and without human intervention (unlike evaluation type 4).\" -  In Section 5 Qualitative Analysis, the authors are also doing human evaluation like other methods in evaluation type-4 of their related works. Also, doing human evaluation is a strong way to justify an explainer. Though expensive, whenever possible it should be done and is in no way a limitation of current evaluation metrics.\n\nYour model needs labelled data for training RCNN. This adds a constraint on the usability and scalability of your proposed evaluation method. Since RCNN is also black-box, one will required another explainer to explain the RCNN. \n\nIn the worst-case scenario, if RCNN is trained with data such that it considers all relevant words as irrelevant, the evaluation made by RCNN will be incorrect. Hence, \" Success depends on the ability of the RCNN to extract correct subsets of tokens.\"\n\n\n\n-------------------  BEFORE\nThe paper proposed a verification framework to evaluate the performance of different explanatory methods in interpreting a given target model. Specifically, the authors evaluated three explanatory methods namely, LIME, SHAP and L2X for a target model trained to perform sentiment analysis on text data. Authors assume for each input text, there is a subset of tokens that are most relevant and that are completely irrelevant to the final prediction task.  The proposed framework uses a recurrent convolutional neural network (RCNN) to find these subsets. The performance of an explainer is evaluated in terms of overlap between the RCNN most relevant tokens and the most relevant tokens provided by the explainer as an explanation. \n\nMajor\n•\tThe paper lack technical novelty.\n•\tThe proposed architecture uses a RCNN to find the most relevant subset of tokens. Firstly, RCNN is also a black box that provides no intuition behind its selection decision. Secondly, in the absence of the ground truth labels for true relevance and irrelevance of a token in input sentence, this explainer method can also suffer from “assuming a reasonable behavior” assumption. The method assumes that the RCNN is performing reasonably in identifying relevant subsets.\n•\tThe success of the method depends on the ability of the RCNN to extract correct subsets of tokens. The data used for training the RCNN, might have some underlying bias. In that case, the evaluation is not accurate.\n•\tIn related work, for “Interpretable target models” the authors mentioned LIME as an example of explainer functions that explains target models that are “very simple models may not be representative for the large and intricate neural networks used in practice”. LIME locally explains the decision of a complex function for a given data point using simpler models like linear regression. But LIME itself can be used for generating explanation for prediction of complex neural network like Inception Net. \n•\tThe example used to explain the difference between feature additive and feature selection-based explainer methods, is confusing. Its not clear how in health diagnostics, one will prefer feature-selection perspective. Although the most relevant features used for the instance are important to understand the decision, but in clinical settings sometimes low rank features can also be useful to understand the target model.\n•\tFor text, the relevant features are the individual tokens of the input sentence. Similarly, for images relevance can be important regions of the image. The authors did not have any experiments on images or tabular data.\n•\tIn the experiment section, the comparison is made with only 3 explainer models and for just one task. The experiments are inadequate.\n•\tIn Figure 4, the colormap is not readable.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byg_L_yEqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview/Contribution:\n====================\nThe authors present a explanation generation framework that help validate post-hoc explanations when the explanations are generated based on feature selection. They claim to demonstrate their method by showing failure modes of exiting explanation generation methods.\n\nOverall, the paper is not ready to be accepted to the conference and I describe my rational with the following strengths and weaknesses.\n\nStrength:\n========\n+ Explanations make models more transparent and easy to understand for end users of the decision made by complex models such as deep neural networks [1]. In that respect, having a verification mechanism for post-hoc explanations is interesting and useful.\n+ The paper is easy to read and follow.\nWeakness:\n===========\n- evaluating explanations generated for an opaque model with another opaque model (RCNN) is cyclical.\n- Just like many literature in this nascent space, interpretation (which is measuring the contribution of features or subsets of features towards predicted output) is confused as explanation. Human level explanations don’t necessarily depend on the direct interaction or contribution of model derived features. Rather they describe ‘why’ the model come up with the decision produced.\n- Explanation generation is gaining traction in the deep learning community especially for critical applications such as healthcare and security. However, the authors claim that post-hoc explanations currently are only evaluated for only simple non-neural model. That is misleading given the recent attention toward generating explanations for various deep learning models.\n- As a generalized pos-hoc explanation generators verification framework, the experiments are seriously lacking and are not well designed to illicit broad applicability.\n\n1) Bekele, E., Lawson, W. E., Horne, Z., & Khemlani, S. (2018). Implementing a Robust Explanatory Bias in a Person Re-identification Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 2165-2172).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgnv8LCtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary - \n\nThe paper proposes a verification method for instance wise feature explanations. The verification framework uses an RCNN to identify two types of tokens a) the tokens that are not predictive of outcome b) the subset of clearly relevant tokens for prediction. The data used for RCNN is a pruned version of the data used to train the  black-box. The pruning eliminates data points to ensure that the tokens not selected by RCNN have no contribution to the outcome and that the model does not exhibit suffer from learning \"handshakes\". A handshake is defined as the set of tokens that may be spuriously missed because their information is encoded in another relevant token. This proof to identify such data points is shown and the RNN is therefore expected to be able to reliably identify 3 kinds of tokens a) Those that have zero contribution to the outcome. b) Those that definitely have some contribution to the outcome and c) those that could be relevant or noisy. Three instance-wise feature selection methods are compared. Results are provided on 3 metrics. a) % instances for which the most important tokes provided by the explainer is among the non-selected tokens, b) % of instances for which at least one non-selected token is ranked higher than a relevant token, and c) Average number of non-selected tokens ranked higher than any obviously relevant tokens. \n\nClarifications and concerns:\n1. For the dataset considered here, I would like to see the distribution of the irrelevant, clearly relevant and unsure if they are relevant tokens as detected by the RCNN. How does this change if I further prune the dataset after ensuring that handshake and other issues have been eliminated. The main concern I have is the idea of verifying other explanations using a neural network itself. I can train the RCNN neural network with half the data (and satisfy the properties the authors mention) and my evaluation would change significantly. From the appendix I see that most of the tokens could be in the set $SDR_x$. \n\n2. What if the set of tokens don't overlap between the RCNN and the black-box to be verified. That said, I think the assumptions of the framework should be much more explicitly mentioned.\n\n3. The std deviations in the experiments are very high. Can the authors justify this and how it is still okay to use this framework for evaluating feature importance based explanations.\n\nMinor:\n1. You have cited the \"Anchors\" paper twice?\n2. Page 3 - typo - \"....explainer should provide different explanations for the trained model on real data than when the data...\"\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------\nUpdate -\n\nI have read the authors response. \nIf the pruned dataset is created using an RCNN, then it is not clear if the RCNN is used to just explain itself or all other methods as well. Like I said, if we just train the model on a slightly different distribution of labels, or half the data randomly sampled irrespective of labels, the explanations will change because the pruned dataset i.e. the ground truth may significantly change. I am still not convinced how this makes for a good verification framework to asses other explainers.\n\nIt is also unclear how generalizable this verification process is to other domains. I will therefore not be updating my score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for generating evaluation tests for feature-based explainers. The framework provides guarantees on the behaviors of each trained model in that non-selected tokens are irrelevant for each prediction,  and for each instance in the pruned dataset, one subset of clearly relevant tokens is selected. 

After reading the paper, I think there are a few issues with the current version of the paper: 

(1) the writing can be significantly improved: the motivation is unclear, which makes it difficult for readers to fully appreciate the work. It seems that each part of the paper is written by different persons, so the transition between different parts seems abrupt and the consistency of the texts is poor. For example, the framework is targeted at NLP applications, but in the introduction the texts are more focused on general purpose explainers. The transition from the RCNN approach to the proposed framework is not well thought-out, which makes the readers confused about what exactly is the proposed framework and what is the novelty.

(2) the claimed properties of the proposed framework are rather straightforward derivations. The technical novelty is not as high as claimed in the paper.

(3) The experiment results are not fully convincing. 

All the reviewers have read the authors' feedback and responded. It is agreed that the current version of the paper is not ready for publication. 

Decision: Reject