Evaluation of Similarity-based Explanations
Authors: Kazuaki Hanawa
Sho Yokoi
Satoshi Hara
Kentaro Inui
Venue: ICLR 2021 Poster (2021)
Keywords: Interpretability
Explainability

Abstract: Explaining the predictions made by complex machine learning models helps users to understand and accept the predicted outputs with confidence. One promising way is to use similarity-based explanation that provides similar instances as evidence to support model predictions. Several relevance metrics are used for this purpose. In this study, we investigated relevance metrics that can provide reasonable explanations to users. Specifically, we adopted three tests to evaluate whether the relevance metrics satisfy the minimal requirements for similarity-based explanation. Our experiments revealed that the cosine similarity of the gradients of the loss performs best, which would be a recommended choice in practice. In addition, we showed that some metrics perform poorly in our tests and analyzed the reasons of their failure. We expect our insights to help practitioners in selecting appropriate relevance metrics and also aid further researches for designing better relevance metrics for explanations.

Published as a conference paper at ICLR 2021
EVALUATION OF SIMILARITY-BASEI EXPLANATIONS
Kazuaki Hanawa' 122 Sho Yokoi Satoshi Hara 8, Kentaro Inui
RIKEN Center for Advanced Intelligence Project', Tohoku University", Osaka University? kazuaki www.aaaawwwaaeeeeee jp, yokoi@ece wtw.eeeeeeee..uuu..e.eeeeoo ac ac.jp,
satoharaear, sanken. osaka-u. ac. jp, inui@ecei. twwoku.accc...o.oo jp
ABSTRACT
Explaining the predictions made by complex machine learning models helps users understand and accept the predicted outputs with confidence. One promising way is to use similarity-basee explanation that provides similar instances as evidence to support model predictions. Several relevance metrics are used for this purpose. In this study, we investigated relevance metrics that can provide reasonable ex- planations to users. Specifically, we adopted three tests to evaluate whether the relevance metrics satisfy the minimal requirements for similarity based explanation. Our experiments revealed that the cosine similarity the gradients of the loss performs best, which would be a recommended choice in practice. In addition, we showed that some metrics perform poorly in our tests and analyzed the reasons their failure. We expect insights to help practitioners in selecting appropriate relevance metrics and also aid further researches for designing better relevance metrics for explanations.
INTRODUCTION
Explaining the predictions made by complex machine learning models helps users understand and accept the predicted outputs with confidence (Ribeiro et al. 2016 Lundberg & Lee 2017 Guidotti et al. 2018 Adadi Berrada 2018 Molnar 2020). Instance- explanations are popular type of explanation that achieve this goal by presenting one several training instances that support the predictions of a model. Several types of restance-based explanations have been proposed, such explaining with instances similar to the instance of interest (i.e., the test instance in Barshan et al. 2020): harmful instances that degrade the performance
question) (Charpiat et al. 2019
of models (Koh & Liang 2017
Khanna al. 2019); counter-examples that contrast how a prediction
can be changed (Wachter et al.
2018); and irregular instances (Kim et al. 2016).
Among these, we focus on the first one, the type of explanation that gives one or several training instances that are similar to the test instance in question and corresponding model predictions. We refer this type of instance-based based explanation as similarity-be samieedity-based explanation. similarity-brä based explanation is of the form "I (the model) think this image is cat because similar images I saw in the past were also cat." This type of explanation is analogous to the way humans make decisions by referring to their prior experiences (Klein & Calderwood, 1988) Klein 1989 Read & Cesa 1991).
Hence, it tends to be easy to understand even to users with little expertise about machine learning. A report stated that with this type of explanation, users tend to have higher confidence in model predictions compared to explanations that presents contributing features (Cunningham et al. 2003).
In the nastacce-bssed explanation paradigm, including similarity baased explanation, relevance metric R(2, x') E IR is typically used to quantify the relationship between two instances, 2 = (xc y)
and x' = (x', (x'.)')
Definition (Instance-based Explanation Using Relevance Metric). Let D
{
(3)
(6) V D be a set of training instances and Ptest be a test input of interest whose pre- dicted output is given by Trest = f((testt with a predictive model f. An instance based explanation method gives the most relevant training instance ž E 'D to the test instance Z = (Teet Vrest) by arg max a ED R(Etest> 2(rain) using a relevance metric Ri test Ptrain).
(rainn
Previously proposed relevance metrics include similarity (Caruana et al. et al. 2016, Khanna al. 2019), and Coflueencefunction function Koh & Liang
1999,, kernet
Kim
2017

---

Published as a conference paper at ICLR 2021
The relevance metrics and their evaluation results. For the model randomization test, the results that passed the test are colored. For the identical class test and identical subclass test, the results with the five highest average evaluation scores are colored. The details of the relevance metrics, the evaluation criteria, and the evaluation procedures can be found in Sections 1.2 and respectively.
Table I:
Evaluation Criteria
Relevance Metries
Abbrv.
Model Randomization Test
Tdentical Class Test
Identical Subclass Test
6(z)
Failed
0.615 0,261
0.644 1 0.264
&
6(2) let
Passed
= 0,106
0,631 1 0.237
0(2) hill
2
Failed
= 0.128
0.691 + 0.211
6(z)
cos
Failed
0.609 0.248
0.621 + 0.242
Cosine
6(2) Jast
last
Passed
0.888 0.098
0.636 + 0.234
((z) hall
cog
Failed
0.871 = 0.110
0.738 +:: 0.166
((2) =
dot
Falled
0.336 = 0.187
0.346 = 0.201
Dot
((=) h
dog last
Falled
0.579 0.344
0,284 + 0.122
((2) h
dotal
Falled
0.630
0.488 + 0.267
Influence Function
IF
Passed
0l.772
0.309 + 0.174
Relative
RIF
Passed
0.779 1 0.309
0.659 1 0,266
Gradient
Fisher Kernel
FK
Passed
0.226 = 0.103
0,180 +: 0.076
Grad-Dot
GD
Passed
0.701 + 0.287
0.403 + 0.131
Grad-Cos
GC
Passed
0.996 0.009
0.753 1 0.196
An immediate critical question is which relevance metric is appropriate for which type of instance- based explanations. There is no doubt that different types of explanations require different metrics. Despite its potential importance, however, little has been explored on this question. Given this background, in this study, we focused on similarity- based explanation and investigated its appropriate relevance metries through comprehensive experiments!!
Contributions We provide the first answer to the question about which relevance metrics have desirable properties for similarity-baser explanation. For this purpose, we propose to use three minimal requirement tests to evaluate various relevance metrics in terms of their appropriateness. The first test the model randomization test originally proposed by Adebayo et al. (2018) for evaluating saliency-based methods, and the other tests, the identical class test and identical subclass test, are newly designed in this study. As summarized in Table our experiments revealed that (i) the cosine similarity of gradients performs best, which is probably recommended choice for similarity- based explanation in practice, and (ii) some relevance metrics demonstrated poor performances on the identical class and identical subclass tests, indicating that their should be deprecated for similarity-bassee explanation. We also analyzed the reasons behind the success and failure of metrics. We expect these insights to help practitioners selecting appropriate relevance metries.
1.1 PRELIMINARIES
Notations For vectors a, b E RP, we denote the dot product by (a, b) == Σi a;b; the &g norm by ||a|| == ((, a), and the cosine similarity by cos(a, (,, b) == (a.b)/[|a|l]0))
Classification Problem We consider standard classification problem as the evaluation bench- mark, which is the most actively explored application of instance-based explanations. The model the conditional probability p(y x;; 0) with parameter 0. Let
trained parameter
((2(6) 2 0), where the loss function is the cross entropy ((2; 0) poy x; 0) for an input-output pair z = (z,y). The model classifies a test input Etest by assigning the class with the highest probability Prest = arg max, ply Drest: 0).
arg ming Lirain = N D E
1.2 RELEVANCE METRICS
We present an overview of the two types of relevance metrics considered in this study, namely similarity metrics and gradient-based metrics. To the best of our knowledge, all major relevance
Our implementation is available instance_based Laaaaace__aaeeeeeeeaaattiiio
2

---

Published as a conference paper at ICLR 2021
metrics proposed thus far can be classified under these two types. Table Preseenss a list of metrics and their abbreviations.
Similarity Metries
We consider the following popular similarity metrics with feature map ((z).
E Metric:
Re, (z, z') = || [()) (z²)) which a typical choice for nearest neighbor
methods (Hastie et al. 2009 Abu Alfeilat al. 2019).
Cosine Metric: Reos (z, =²) == cos (6( 2 (b (2())))) ))) which is commonly used in natural language
processing tasks (Mikolov et al. 2013 Arora et al.
2017 Conneau et al. (017).
Dot Metric: Riton 2, z') (d(z). <b (=))), which is kernel function used in kernel models
such as SVM (Schölkopf et al. 2002 Fan et al. 2005 Bien & Tibshirani 2011).
As the feature map ((z), we consider (i) identity map Ø(z) 2;; (ii) the last hidden layer 6(z) = h last, which is the latent representation of input æ, one layer before the output in deep neural network; and, (iii) all hidden layers (==) hall, where hall [h h² ..... h last is the concatenation of all latent representations in the network. Note that the metrics with the identity map merely measure the similarity of inputs without model information. We adopt these metrics as naive baselines to contrast with other ad advanced metrics that utilize model information.
Gradient-based Metries Gradient-based metrics use a gradient g := Vol(z; 0) measure the
relevance. We consider five metries: Influence Function (IF) (Koh & Liang 2017). Relative IF
(RIF) (Barshan et al. 2020), Fisher Kernel (FK) (Khanna al., 2019). Grad-Dot (GD) YYh et al.
2018 Charpiat et al. 2019), and Grad-Cos (GC) (Perronnin et al. 2010 Appendix A for further detail.
Charpiat et al.
2019). See
IF: RIF(z, 2/) = 5
GD: (Z. 2') (g g
RIF: RRIF(Z, 2z) = cos
H
GC: Roc(z, 2/) == cos (g g
FK: REK 22)= == 5 I
g
where H and I are the Hessian and Fisher information matrices of the loss Cerain respectively.
RELATED WORK
Model-specific Explanation Aside of the relevance metrics, there is another approach for similarity- based explanation that uses specific models that can provide explanations by their design (Kim et al.. 2014 Plötz& & Roth 2018 Chen et al. 2019). We set aside these specific models and focus on generic relevance metrics because of their applicability to wide range of problems.
Evaluation of Metrics for Improving Classification Accuracy In several machine learning prob- lems, the metrics between instances play essential role. For example, the distance between instances is essential for distance-based methods such as nearest neighbor methods (Hastic et al. 2009). Another example is kernel models where the kernel function represents the relationship between two instances (Schölkopf et al.) 2002). Several studies have evaluated the desirable metrics for specific tasks (Hussain al. 2001 I Hu et al. 2016 Li Li 2018 Abu Alfeilat al. (019). These studies aimed to find metrics that could improve the classification accuracy. Different from these evaluations based on accuracy, our goal in this study to evaluate the validity of relevance metrics for similarity- based explanation; thus, the findings in these previous studies are not directly applicable to our goal.
Evaluation of Explanations There are a variety of desiderata argued as requirements for explana-
Lakkaraju et al. 2019 Jacovi Goldberg 2020). Strout et al. (019), robustness (Alvarez-Melis & Jaakkola 2018), and readability (Wang& & Rudin 2015 Yang et al. 2017 Angelino et al. 2017)). It is important to evaluate the existing explanation methods considering these requirements. However, there is no standard test established for evaluating these requirements, and designing such tests still remains open problem (Doshi- Velez & Kim 2017 Jacovi & Goldberg 2020). In this study, the first empirical study for evaluating the existing relevance metrics for similarity based explanation, we take an alternative approach by designing minimal requirement tests for two primary requirements,
tions, such as faithfulness (Adebayo et al. 2018
plausibility (Lei et al.) [016]] Lage et al., 2019.
3

---

Published as a conference paper at ICLR 2021
namely faithfulness and plausibility. With this alternative approach, we can avoid the difficulty of directly evaluating these primary requirements.
EVALUATION CRITERIA FOR SIMILARITY-BASED EXPLANATION
This study aims to investigate the relevance metrics with desirable properties for similarity- bbasd explanation. In this section, we propose three tests to evaluate whether the relevance metrics satisfy the minimal requirements for similarity- tests, conclude that the metric does not meet the minimal requirements: thus, its use would be deprecated. The first test (model randomization test) assesses whether each relevance metric satisfies the minimal requirements for the faithfilneess of explanation, which requires that explanation to a model prediction must reflect the underlying inference process (Adebayo et al.) 2018 Lakkaraju et al. 2019; Jacovi & Goldberg) 2020). The latter two tests (identical class and identical subelass tests) are designed to assess relevance metrics in terms of the plausibility of the explanations they produce (Lei al. 2016 Lage et al. 2019 Strout et al. 2019), which requires explanations to be sufficiently convincing to users.
explanation. If a relevance metric fails one of the
3.1 MODEL RANDOMIZATION TEST
Explanations that are irrelevant model should be avoided because such fake explanations can mislead users. Thus, any valid relevance metric should be model-dependent, which constitutes the first requirement.
We use the model randomization test of Adebayo et al. (2018) to assess whether given relevance metric satisfies a minimal requirement for faithfulness. If relevance metric produces almost same explanations for the same inputs on two models with different inference processes, it is likely to ignore the underlying model, i.e., the metric is independent the model. Thus, we can evaluate whether the metric is model-dependent by comparing explanations from two different models. In the test, a typical choice of the models a well-trained model that can predict the output well and randomly initialized model that make only poor prediction. These two models have different inference processes; hence, their explanations should be different.
Definition (Model Randomization Test). Let R denote the relevance metric of interest. Let f and frand be well-trained model and randomly initialized model, respectively. For given R. f. and test instance Ztest (Fee Frest), let o be a permutation the indices of the training instances based
(m)( 2 R( 2train
(2)(
on the degree of relevance to the given test instance, i.e., R[ Ztrain
(N)) We also define frond accordingly. Then, we require T and π frand to
R( test Zerain ensure small rank correlation.
If relevance metric R is independent of the model, it produces the same permutation for both f and frand. and their rank correlation becomes one. If the rank correlation is significantly smaller than one and close zero, we can confirm that the relevance metric is model-dependent.
3.2 IDENTICAL CLASS TEST
The second minimal requirement is that the raised similar instance should belong to the same class the test instance, shown in Figure The violation of this requirement leads to nonsensical explanations such as "I think this image is cat because similar image saw in the past was dog." " in Figure T When predictions and ignore the predictions even the underlying model is valid. This observation leads to the identical class below.
encounter such explanations, they might question the validity of model
Definition 3 (Identical Class Test). We require that the most similar (relevant) instance of a test instance Zceet = (Feeet: Freat) a training instance of the same class as the given test instance.
arg max R (RZeest: 2) (z. 7)
Prest
EDD
Although this test may look trivial, some relevance metrics do not satisfy this minimal requirement, demonstrated in Section 4.2
4

---

Published as a conference paper at ICLR 2021
is cat because
is cat because
similar
is cat.
similar
is dog.
Figure 1: Valid (V) and invalid (X) examples for the identical class test.
cat
cat
animal because
is animal because frog is animal.
cat is animal.
a similar
a similar
Figure 2: Valid (v) and invalid (X) examples for the identical subclass test.
3.3 IDENTICAL SUBCLASS TEST
The third minimal requirement is that the raised similar instance should belong the same subclass as that of the test instance when the the classes consist latent subclasses, as shown Figure 2 For example, consider a problem of classifying images of CIFAR 10 into two classes, i.e., animal and vehicle. The animal class consists of images from subclasses such as cat and frog, while the vehicle class consists of images from subclasses such as airplane and automobile. Under the presence of subclasses, the violation of this requirement leads to nonsensical explanations such as "I think this image (cat) is animal because similar image (frog) saw in the past was also animal." in Figure 2 This observation leads to the identical subclass test below.
Definition (Identical Subclass Test). Let s(2) be a subelass for class y of an instance 2 (ax, y)- We require that the most similar (relevant) instance of test instance Ztest (Ørest Vesst is the training instance the same subclass as the test instance, under the assumption that the prediction of the test instance is correct Veest
Ylest
arg max R( (R(zestt2) = - s(barz) = =((ccctt).
(2)
ZED
In the experiments, we used modified datasets: we split the dataset into two new classes (A and B) by randomly assigning the existing classes to either classes. The new two classes now contain the original data classes subclasses that are mutually exclusive and collectively exhaustive, which can be used for the identical subclass test.
3.4 DISCUSSIONS ON VALIDITY OF CRITERIA
Here, we discuss the validity of the new criteria, i.e., the identical class and identical subclass tests.
Why do relevance metrics that cannot pass these tests matter? Dietvorst et al. (2015) revealed bias in humans, called algorithm aversion, which states that people tend ignore an algorithm if it makes errors. It should be noted that the explanations that do not satisfy the identical class test or identical subclass test appear be logically broken, shown in Figures [ and Given such logically broken explanations, users will consider that the models are making errors, even if they making accurate predictions. Eventually, the users will start to ignore models.
Is the identical subclass test necessary? This is an essential requirement for ensuring that the explanations are plausible to any users. Some users may not consider the explanations that violate the identical subclass test to be logically broken. For example, some users may find frog to be an appropriate explanation for a cat being animal by inferring taxonomy of the classes (c.g., both have eyes). However, cannot hope all users infer the same taxonomy. Therefore, if there is
We require correct predictions this because the subclass does match for incorrect

---

Published as a conference paper at ICLR 2021
discrepancy between the explanation and the taxonomy inferred by user, the user will consider the explanation to be implausible. To make explanations plausible to any user, instances of the same subclass need to be provided.
Is random class assignment in the identical subelass test appropriate? We adopted random assignment to evaluate the performance of each metric independent from the underlying taxonomy. specific taxonomy was considered for the evaluations, a metric that performed well with will be highly valued. Random assignment eliminates such effects, and can purely measure the performance of the metrics themselves.
Do classification models actually recognize subclasses? Is the identical subelass test suitable to evaluate the explanations of predictions made by practical models? It is true that if model ignores subclasses in its training and inference processes, any explanation will fail the test. We conducted simple preliminary experiments and confirmed that the practical classification models used this study capture the subclasses. See Appendix E for further detail.
4 EVALUATION RESULTS
Here, we examine the validity of relevance metrics with respect to the three minimal requirements. For this evaluation, we used two image datasets (MNIST (LeCun al. 1998). CIFARIO (Krizhevsky) 2009)), two text datasets (TREC (Li & Roth 2002). AGNews (Zhang et al. 2015)) and two table datasets (Vehicle (Dua & Graff 2017). Segment (Dua & Graff (017)). As benehmarks, we employed logistic regression and deep neural networks trained on these datasets. Details of the datasets, models, and computing infrastructure used this study is provided in Appendix B
Procedure We repeated the following procedure 10 times for each evaluation test.
Train a model using subset of training instances Then, randomly sample 500 test instances from the test s
2. For each test instance, compute the relevance score for all instances used for training.
3. (a) For the model randomization test, compute the Spearman rank correlation coefficients be- tween the relevance scores from the trained model and relevance scores from the randomized model.
(b) For the identical class and identical subclass tests, compute the success rate, which the ratio of test instances that passed the test.
In this section, we mainly present the results for CIFARIO with CNN and AGNews with Bi-LSTM. The other results were similar, and can be found in Appendix F
Result Summary We summarize the main results before discussing individual results.
flast, coslast, and gradient-based metrics scored low correlation in the model randomization test for all datasets and models, indicating that they are model-dependent.
GC performed the best in most of the identical class and identical subclass tests; thus, GC would be the recommended choice in practice.
Dot metrics well as IF, FK, and GD performed poorly the identical class test and identical subclass test.
In Section 5 we analyze why some relevance metries succeed or fail in the identical class and identical subclass tests.
4.1 RESULT OF MODEL RANDOMIZATION TEST
Figure shows the Spearman rank correlation coefficients for the model randomization test. The similarities with the identity feature map f cos". and dot are irrelevant the model and their correlations are trivially one. In the figures, the other metrics scored correlations close to zero,
randomly sampled 10% of MNIST and CIFAR10; 50% of TREC, Vehicle and Segment; and 5% of
AGNews
Forr the identical subclass test, we sampled instances with correct predictions only.

---

Published as a conference paper at ICLR 2021
Average Correlation + Std.
Average Correlation Std.
0.5
0.5
0
Ball
cos"
cos lass
cosau
dot
dot
dotal
IF
RIF
FK
GD
GC
P2
cos
cos luet
cas all
dot
dot lost
dot all
IF
RIF
FK
GD
GC
82
Cosine Dot
Gradient
82
Cosine Dot
Gradient
(a) CIFARIO CNN
(b) AGNews Bi-LSTM
Figure 3: Result of the model randomization test. Correlations close to zero are ideal.
Average Success Rate i Std.
Averuge Success Rate + Std.
Identical Class Test
Identical Class Test
0.5
0.5
0.8
0.8
Identical Subclass Test
Identical Subclass Test
0.6
0.6
0.4
0.4
0.2
0.2
cos
cos all
dot
dotlant
dotalll
IF
RIF
FK
GD
GC
cos costart
dot dot dot all
cos
Dos
Cosine
Dot
Gradient
Cosine
Gradient
(a) CIFAR10 with CNN
(b) AGNews with Bi-LSTM
Figure 4: Results of the identical class test and identical subclass test.
indicating they will be model-dependent. However, the correlation of mill cos dot last was observed to be more than 0.7 on the MNIST and Vehicle datasets (see Appendix [F). Therefore, we conclude that these relevance metrics failed the model randomization test because they can raise instances irrelevant to the model for some datasets.
4.2 RESULTS IDENTICAL CLASS AND IDENTICAL SUBCLASS TESTS
Figure 4 depiets the success rates for the identical class and identical subclass tests. We also summarized the average success rates of our experiments in Table [
noteworthy that GC performed consistently well the identical class and identical subclass tests for all the datasets and models used in the experiment (see Appendix [F). In contrast, some relevance metrics such as the dot metrics as well as IF, FK, and GD performed poorly both tests. The reasons for their failure discussed in the next section.
To conclude, the results of our evaluations indicate that only GC performed well all tests. That is, only GC seems to meet the minimal requirements; thus, it would be a recommended choice for similarity- explanation.
5 WHY SOME METRICS ARE SUCCESSFUL AND WHY SOME ARE NOT
We observed that the dot metrics and gradient-based metrics such IF, FK, and GD failed the identical class and identical subclass tests, in comparison to GC that exhibited remarkable performance. Here, analyze the reasons why the aforementioned metrics failed while GC performed well. In Appendix [ we also discuss a way to repair IF, FK, and GD to improve their performance based on the findings this section.
7

---

Published as a conference paper at ICLR 2021
Training Instances
Selected Instances
Frequency
0.8
0.6
(a) IF
(b) FK
(c) GD
(d) GC
0.4
0.2
3
7,000
10 Norm of feature map Norm of feature map
140
140
Norm of feature map Norm feature map
Figure 5: Distributions of norms of the feature maps of all training instances (colored) and the instances selected by the identical class test (meshed) on CIFARIO with CNN.
Test Instances
Found
FK
Test Instances
Found
airplane
frog
bird
truck
cat
ship
bird
ship
COs( #sest: Fcrain)
[
Co8 Co8(Ztest; (zest, Ztrain
(((()))))))
.00008
.00010
.00007
3,585
.021
.020
.019
345,292,727
GD
Test Instances
Found
GC
Test Instances
Found
cal
bird
horse
truck
truck
truck
truck
truck
COS Zenat Fernin)
[(
cos Zieut: Strain)
[
.385
.291
.329
112.8
.754
.754
.752
.0008
Figure 6: Training instances frequently selected in the identical class test with multiple test instances on CIFARIO with CNN, the cosine between them, and the norm of training instances.
Failure of Dot Metrics and Gradient-based Metrics To understand the failure, we reformulate IF, FK, and GD dot metries of the form Raot (Zusst Ztrain) (6((tost) ), di ((tzzai))) to ensure that the following discussion is valid for any relevance metric of this form. is evident that IF, FK, and GD can be expressed in this form by defining the feature maps by (bz) H g=1;/g((2;))
<(z) g=1 0), and d(z)= (2; 0). respectively.
Given a criterion, let (4)
train a desirable instance for a test instance 2 The failures of dot _()) metrics indicate the existence of an undesirable instance Strain such that (Ø(ztest), ((=(1) (5)
(Ø( (o(z ((2())))) The following sufficient condition for *train is useful to understand the failure.
| (0(2()) (1) < (=(()))))) (os( Φ( (1) )).
The condition implies that any instance with an extremely large norm and cosine slightly larger _() than zero can be the candidate of *train In our experiments, we observed that the condition on the norm is especially crucial. As shown in Figure even though instances with significanty large norms were scarce, only such extreme instances were selected as relevant instances by IF, FK, and GD. This indicates that these these metrics tend to consider such extreme instances as relevant. contrast, GC was not attracted by large norms because it completely cancels the norm through normalization.
Figure shows some training instances frequently selected in the identical class test on CIFAR10 with CNN. When using IF, FK, and GD, these training instances were frequently selected inre- spective of their classes because the training instances had large norms. these metrics, the term cos(Ø (ttttt) (( seems to have negligible effects. In contrast, GC successfully selected the instances of the same class and ignored those with large norms.
Success of GC We now analyze why GC performed well, specifically in the identical class test. To simplify the discussion, we consider linear logistic regression whose conditional distribution ply ax; 0) is given by the y-th entry of a (War), where is the softmax function, W E RCxd
8

---

Published as a conference paper at ICLR 2021
and C' and d denote the number of classes and dimensionality of æ, respectively. With some algebra, we obtain RGc(z, x') cos(r", ,r"") cos(æ, x²) for (æ, y) and x' = (a', g'), where r = (W>x) - Ey is the residual for the prediction on z and Cy is a vector whose y-th entry is one, and zero, otherwise. See Appendix C the derivation. Here, the term cos(r", r plays an essential role in GC. By definition, r 0f y and r 0, otherwise. Thus, cos(r=, r= always holds true when y - y'. while cos(r*, r* can be negative for y + y' Hence, the chance of RGc(z, 22 x²) being positive can be larger for the instances from the same class compared to those from different class.
Figure 7 shows that (eerr,r=) is essential also for deep neural networks. Here, for each test instance Zesst on CIFARIO with CNN, we randomly sampled two training instances Ztrain (one with the same class and the other with different class), and computed Rec(Zust: p
(Zust: S and
We also note that cos(r r alone was not helpful for the identical subclass test, whose success rate was around the chance level. We thus conjecture that while
same class
different class
Frequency
3 2
cos(r Rerwt)) is particularly helpful for the identical class test, the use of the entire gradient is still essential for GC to work ef- fectively.
1
0
Rec (Fceztest: Strain) cos(r Fteat y F
Figure 7: Distributions of Rac(Zrestr usinn and
cos( (onmruu xisaai))) for training instances with the same different classes on CIFAR10 with CNN.
CONCLUSION
We investigated and determined relevance metrics that are effective for similarity- explanation. For this purpose, we evaluated whether the metrics satisfied the minimal requirements for similarity- based explanation. In this study, we conducted three tests, namely, the model randomization test of Adebayo et al. (2018) to evaluate whether the metrics are model-dependent, and two newly designed tests, the identical class and identical subclass tests, to evaluate whether the metrics can provide plausible explanations. Quantitative evaluations based on these tests revealed that the cosine similarity of gradients performs best, which would be a recommended choice in practice. We also observed that some relevance metrics do not meet the requirements; thus, the use of such metrics would not be appropriate for similarity -baed explanation. We expect our insights to help practitioners in selecting appropriate relevance metrics, and also to help further researches for designing better relevance metrics for instance- based explanations.
Finally, we present two future direction for this study. First, the proposed criteria only evaluated limited aspects of the faithfulness and plausibility of relevance metrics. Thus, it is important to investigate further criteria for more detailed evaluations. Second, in addition to similarity- based explanation, it is necessary to consider the evaluation of other explanation methods, such counter- examples. We expect this study to be the first step toward the rigorous evaluation of several instance- based explanation methods.
ACKNOWLEDGMENTS
We thank Dr. Ryo Karakida and Dr. Takanori Machara for their helpful advice. We also thank Overfit Summer Seminar for an opportunity that inspired this research. Additionally, we are grateful to our supported by JST, ACT-X Grant
laboratory members for their helpful comments. Sho Yokoi Number JPMIAX200S. Japan. Satoshi Hara was supported by JSPS KAKENHI Grant Number 20K 19860, and JST, PRESTO Grant Number JPMJPR20C8, Japan.
REFERENCES
Haneen Arafat Abu Alfeilat, Ahmad B.A. Hassanat, Omar Lasassmeh, Ahmad S. Tarawneh, Mah- moud Bashir Alhasanat, Hamzeh S. Eyal Salman, and V.B. Surya Prasath. Effects of Distance
https://sites: wow.less.oooleeeoo http:///aooc/ao///a//////a////aa

---

Published as a conference paper at ICLR 2021
Measure Choice K-Nearest Neighbor Classifier Performance: A Review. Big Data, 7(4): 221-248, 2019.
Amina Adadi and Mohammed Berrada. Peeking Inside the Black-box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6:52138 522600, 2018.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems 31, pp. 9505-9515. 2018.
David Mlvirez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv: 1806.0849,, 2018.
Elaine Angelino, Nicholas Larus-Stone Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists for categorical data. The Journal of Machine Learning Research, 18 (1):8753-8830, 2017.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A Simple but Tough-to-Beat Baseline for Sentence Embeddings. Proceedings of the 5th International Conference on Learning Representations, 2017.
Elnaz Barshan, Marc-Fitenne Brunet, and Gintare Karolina Dziugaite. RelatlF: Identifying Explana- tory Training Samples via Relative Influence. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, pp. 1899-1909, 2020.
Jacob Bien and Robert Tibshirani. Prototype Selection for Interpretable Classification. Annals of
Applied Statistics, 5(4):2403-2424, 2011.
Rich Caruana, Hooshang Kangarloo. John David N. Dionisio. Usha Sinha, and David Johnson. Case-Based Explanation of Non-Case-Basd Based Learning Methods. In Proceedings of the AMIA Symposium, pp. 212-215, 1999.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input Similarity from the Neural Network Perspective. Advances Neural Information Processing Systems 32, pp- 5342-5351, 2019.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. In Advances in Neural Information Processing Systems 32, pp.
2019.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680, 2017.
Pádraig Cunningham, Dónal Doyle, and John Loughrey. An Evaluation of the Usefulness of Case- Based Explanation. International Conference on Case-Based Reasoning, pp. 122-130. Springer, 2003.
Berkeley Dietvorst, Joseph Simmons, and Cade Massey, Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1): 114-8830, 2015.
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint 1702::170.00666 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://accchiai ics. uci edu/m1
Rong-En En Fan, Pai-Hsuee Chen, and Chih-Jen Lin. Working Set Selection Using Second Order Information for Training Support Vector Machines. Journal of Machine Learning Research, 1819-1918, 2005.
10

---

Published as a conference paper at ICLR 2021
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys, 51 15):1-42,40 2018.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2009.
Li Yu Hu, Min Wei Huang, Shih Wen Ke, and Chih Fong Tsai. The Distance Function Effect on k-Nearest Neighbor Classification for Medical Datasets. SpringerPlus, 5(1):1304, 2016.
Muhammad Hussain, Summrina Kanwal Wajid, Ali Elzaart, and Mohammed Berbar. A Comparison of SVM Kernel Functions for Breast Cancer Detection. In Proceedings of the 8th International Conference on Computer Graphics, Imaging and Visualization, pp. 150, 2011.
Alon Jacovi and Yoav Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198-4205, 2020.
Rajiv Khanna, Been Kim, Joydeep Ghosh, and Sanmi Koyejo. Interpreting Black Box Predictions using Fisher Kernels. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89, pp. 3382 -390, 2019.
Been Kim, Cynthia Rudin, and Julie Shah. The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification. In Advances in Neural Information Processing Systems 27, pp. 1952-1960, 2014.
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples Are Not Enough, Learn to Criticize! Criticism for Interpretability. Advances in neural information processing systems 29, pp. 2280-2288, 2016.
Gary A Klein. Strategies of Decision Making. Technical report, 1989.
Gary A Klein and Roberta Calderwood. How Do People Use Analogues to Make Decisions? In
Proceedings of the DARPA Workshop on Base-Based Reasoning, 1988, pp. 209-223, 1988.
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In
Proceedings of the 34th International Conference Machine Learning, pp. 1885-1881888 2017.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
Isaac Lage, Emily Chen, Jeffrey Hc, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Veles An Evaluation the Human- Intrrpretttrrr of Explanation. 2019. URL http
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 131-138, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):227 (122)))22233333 1998.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing Neural Predictions. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 107-117, 2016. 2016.
Xin Li and Dan Roth. Learning Question Classifiers. In Proceedings of the 19th International Conference on Computational Linguistics, 2002.
Zhou Li and Chunxiang Li. Selection of Kernel Function for Least Squares Support Vector Machines in Downburst Wind Speed Forecasting. In Proceedings of the 1 1th International Symposium on Computational Intelligence and Design, volume pp. 337-341, 2018.
Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, pp. 4765-4774, 2017.
11

---

Published as a conference paper at ICLR 2021
Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint:1802.034220 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality, In Advances in Neural Information Processing Systems 26, pp. 3111-3119. 2013.
Christoph Molnar. Interpretable Machine Learning. Lulu. com, 2020.
Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-Scale Image Retrieval With Compressed Fisher Vectors. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.
2010.
Tobias Plötz and Stefan Roth. Neural Nearest Neighbors Networks. In Advances in Neural Information Processing Systems 31, pp. 1087-1098. 2018.
Stephen Read and lan L Cesa. This Reminds Me of the Time When Expectation Failures in Reminding and Explanation. Journal of Experimental Social Psychology, 27(1): 1991.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You" Explaining the Predictions of Any Classifier. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1135- 144, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bileNetV2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 4510-4520, 2018.
Bernhard Schölkopf, Alexander J Smola, and Francis Bach. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT press, 2002.
Julia Strout, Ye Zhang, and Raymond Mooney. Do Human Rationales Improve Machine Expla- nations? In Proceedings of the 2019 ACL Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pp. 56-62, 2019.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology. 31(2): 841-887, 2018.
Fulton Wang and Cynthia Rudin. Falling rule lists. In Artificial Intelligence and Statistics, pp. 1013-1022, 2015.
Hongyu Yang, Cynthia Rudin, and Margo Seltzer. Scalable bayesian rule lists. In International Conference Machine Learning, pp.
2017.
Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep Ravikumar. Representer Point Selection for Explaining Deep Neural Networks. In Advances in Neural Information Processing Systems 31, pp. 9291 190019901111 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28, pp. 649-657, 2015.
12

---

Published as a conference paper at ICLR 2021
A GRADIENT-BASED METRICS
In gradient-basee metrics, we consider a model with parameter 0, its loss {(=; 0), and its gradient Vol(2; 0) to measure relevance, where 2 = (2. is an input-output pair.
Inftuence Function (Koh & Liang, 2017) Koh & Liang (017) proposed to measure relevance according to "how largely the test loss will increase if the training instance is omitted from the training set." Here, the model parameter trained using all of the training set denoted by 0, and the parameter trained using all of the training set except the i-th instance ttrain is denoted by 0_y. The relevance metric proposed by Koh Liang (2017) is then defined as the difference between the test loss under parameters and 0_; as follows:
Rzr zuraia) == ((ttest; Q-:) E(Zrest; 0).
(4)
Here, greater value indicates that the loss on the test instance increases drastically by removing the i-th training instance from the training set. Thus, the i-th training instance is essential relative to predicting the test instance; therefore, it is highly relevant.
In practice, the following approximation is used to avoid computing 0_, explicitly.
Rir (2test> ttrain)
(o 0), H T--H-VVel
Vol( (Zurain (2 0)))
where H is the Hessian matrix of the loss Ltrain-
Relative IF (Barshan et al., 2020) Barshan et al. (2020) proposed to measure relevance according
to "how largely the test loss will increase the training instance omitted from the training set under the constraint that the expected squared change in loss is sufficiently small which is the
modified version the influence function. Relative is computed as the cosine similarity of
<(z) = V
RRr reett: strain) = cos H 1/2
0). H 1/2
(( 0))-
(6)
Fisher Kernel (Khanna et al. 2019
Khanna et al. (019 proposed to measure the relevance of
instances using the Fisher kernel follows:
RFK (Zeest: train) == (Vol(Z 0), I 'Ve((2 Praan 0)),
where I is the Fisher information matrix of the loss Ltrain-
Grad-Dot, Grad-Cos (Perronnin et al. 2010 Yeh al., 2018 Charpiat al. 2019) Charpiat
et al. (2019) proposed to measure relevance according to "how largely the loss will decrease when small update is added to the model using the training instance." This can be computed as the dot product of the loss gradients, which we refer to as Grad-Dot.
RGD(Ztest: F = (Vol(Zrest) 0), Vol(z θ))-
(8)
Note that a similar metric is studied by Yeh et al. (2018) as the representer point value.
As modification of Grad-Dot, Charpiat et (2019) also proposed the following cosine version. which we refer to as Grad-Cos.
Rec(21est Ztrain) cos( V Vol(z 2test 0), Vol(2 0)).
Note that the use of the cosine between the gradients is also proposed by Perronnin et 2010
B EXPERIMENTAL SETUP
B. DATASETS AND MODELS
MNIST (LeCun et al. 1998) The MNIST dataset is used for handwritten digit image classification tasks. Here, input is an image of handwritten digit, and the output y consists of 10 classes ("0"
This metric called &-RelatFF by Barshan
2020
13

---

Published as a conference paper at ICLR 2021
to "9"). We adopted logistic regression and CNN the classification models. The CNN has six convolutional layers, and max- pooling layers for each two convolutional layers. The features obtained by these layers are fed into the global average pooling layer followed by single linear layer. The number of the output channels of all the convolutional layers is set to 16. We trained the models using the Adam optimizer with learning rate of 0.001. We used randomly sampled 5,500 training instances to train the models.
CIFARIO (Krizhevsky, 2009) The CIFARIO dataset is used for object recognition tasks. Here, input æ is image containing certain object, and output y consists of 10 classes, e.g., "bird" or "airplane." Note that we used the same models as for the MNIST dataset. In addition, we adopted MobileNetV2 (Sandler et al.) 2018) as model with a higher performance than the previous model. We trained the models using the Adam optimizer with learning rate of 0,001. In the experiments, we first pre-trained the models using all the training instances of CIFAR1O, and then trained the models using randomly sampled 5,000 training instances. Without the pre-training, the classification performance of the models dropped significantly.
Note that we did not examine IF and FK on MobileNetV2 because the matrix inverse in these metrics required too much time to calculate even with the conjugate gradient approximation proposed by Koh & Liang 2017).
TREC (Li & Roth 2002 The TREC dataset is used for question classification tasks. Here, input a a question sentence, and output y is a question category consisting six classes, e.g., "LOC" and "NUM." " We used bag-of-words logistic regression and two-layer Bi-LSTM as the classification models. In the Bi-LSTM, the last state is fed into one linear layer. The word embedding dimension is set to 16, and the dimension of the LSTM is set to 16 also. We trained the models using the Adam optimizer with a learning rate of 0.001. We used randomly sampled 2,726 training instances to train the models.
AGNews (Zhang al. 2015) The AGNews dataset is used for news article classification tasks. Here, input æ is sentence, and output y is category comprising four classes, e.g., "business" and "sports." " We used the same models TREC. We trained the models using the Adam optimizer with learning rate of 0.001. We used randomly sampled 6,000 training instances to train the models.
Vehicle (Dua & Graff, 2017) The vehicle dataset is used for vehicle type classification tasks. Here, the input æ consists of 18 features, and the output y type of vehicle comprising four classes, e.g., "bus" and "van." We used logistic regression and a three-layer MLP as the classification models. We trained the models using the Adam optimizer with learning rate of 0.001. We used randomly sampled 423 training instances to train the models.
The segment dataset is used for image classification tasks. Here, the input æ consists of 19 features, and the output y consists of seven classes, e.g., "sky" and "window." We used the same models as Vehicle. We trained the models using the Adam optimizer with learning rate 0.001. We used randomly sampled 924 training instances train the models.
Segment (Dua & Graff, 2017)
B.2 COMPUTING INFRASTRUCTURE
In our experiments, training of the models was run NVIDIA GTX 1080 GPU with Intel Xeon Silver 4112 CPU and 64GB RAM. Testing and computing relevance metrics were run on Xeon E5-2680 v2 CPU with 256GB RAM.
C DERIVATION OF GC FOR LINEAR LOGISTIC REGRESSION
We consider linear logistic regression whose conditional distribution p(y x; 0) is given by the y-th entry of o ( x), where a the softmax function, 0 = W Cxxd and C and d are the number of classes and the dimensionality of æ, respectively. Recall that the cross entropy loss for linear logistic
14

---

Published as a conference paper at ICLR 2021
regression is given
c ((200) Σ ye (ee 2) + log E exp((We,,))
(10)
where W [wi, 22..... wel. Let Ey be a vector whose y-th entry is one and zero otherwise. Then, the gradient of the loss with respect to we can be expressed as
Vm ((; 0) = (o(Wa) x) (ey)a)a = )a2,
(11)
where p² = [[W>) - e the residual for the prediction on z. Hence, we have
(Vol(=;0), vo)(2'))) = Σ V 00 (z; 0), V (2):0))
(12)
)rr)))))))) (e x²)
(13)
(r² r (ar, a²),
(14)
which yields
(r+ p² (xx x²) | | |||| 2 ||||0||
Roo x²)
(15)
cos (r* y= cos( xx, x¹).
(16)
D REPAIRING GRADIENT-BASED METRICS
As described in Section 5 we found that training instances with extremely large norms were selected as relevant by IF, FK, and GD. Thus, to repair these metrics, we need to design metrics that can ignore instances with large norms. A simple yet effective way of repairing the metrics is to use lz cosine instead of the dot product. As Figure 4 shows, the lg and cosine metries performed better than the dot metrics. Indeed, the lz metrics do not favor instances with large norms that lead to large f2-distance, and, through normalization, the cosine metrics completely ignore the effect of the norms
We name the repaired metrics of IF, FK, and GD based on the l2 metric as C 4FK and egD, respectively, and the repaired metries based the cosine metric as cos and FRs and costnn respectively We observed that these repaired metrics attained higher success rates on several evaluation criteria. The details of the results can be found in Appendix F
E
DO THE MODELS CAPTURE SUBCLASSES?
The identical subclass test requires the model to obtain internal representations that can distinguish subclasses. Here, we confirm that this condition is satisfied for all the datasets and models we used in the experiments. We consider that the model captures the subclasses if the latent representation hall has cluster structures. Figure 9 visualizes hall for each dataset and model using UMAP (McInnes) al. 2018). The figures show that the instances from different subclasses are not mixed completely random. MNIST and TREC have relatively clear cluster structures, while CIFAR 10 and AGNews have vague clusters without explicit boundaries. These figures imply that the models capture subclases (although it may not be perfect).
F COMPLETE EVALUATION RESULTS
F.1 FULL RESULTS
We show the complete results of the model randonization test in Table 2 the identical class test in Table 3 and the identical subclass test in Table The results we present here are consistent with our observations in Section
Note that cos is the as RIF and cos GD the GC.
15

---

Published as a conference paper at ICLR 2021
Table 2: Average Spearman rank correlation coefficients + std. of each similarity function for model randomization test. The metrics prefixed with & are the ones we have repaired. The results with the average score the 95% confidence interval of the null distribution that the correlation is zero, which is [-0.088, 0.088J, are colored.
MNIST
CIFARIO
TREC
Model
CNN
logreg
MohilenerV2
CNN
Ingreg
Bi-L.TMM
logreg
Parameter size
12K
8K
2.2M
12K
31K
20K
7K
Accuracy
0.98 1 0.00
0.92 1 0.00
0.89 0.01
0.72
0.02
0.35 1. 0.01
0.86 1: 0.01
0.81 1. 0.02
1.00 + .00
1.00 .00
1.00 +00
1.00 + .00
1.00 .00
1.00 +00
1.00 + .00
Faat
.15 f: .01
.07 00
.05 1:
.19 .01
22
.79 1:
.02 011
.13 1 001
.25 -
cos
1.00 .00
1.00 .00
1.00 .00
1.00 .00
1.00
1.00 + .00
coglast
.24 +
.07
.02
.78 1:
dot"
+
1.00 .00
.00
1.00 .00
1.00
1.00 .00
1.00 +
hust
.39 + .00
.05 + .01
.04 01
.25 = .01
dot
.80 = .00
.00 + .01
.12 = 001
.26 + .03
IF
.05 1 .00
.00 .00
.05 .10
004 .01
.04
.01 .01
.06 1 001
.00 1:
.00
.01 02
00
.05
10 .00
.13 1
.02 + .00
.05 .00
.04+
.03+
-03 + .01
-01 + +01
.03 + .01
FK
.02
.02
01
-.03
.01 .00
.03 =
-.10 +00+.00
15=+00
-06
-.12
.03
-.14
.15
PK
1.00
005+.00
-05
-003
-00
.02
.03
.08
01:1.00
.03
.02
0.07 +
-003
-00
-005
-.04 +04+00
.01
grad
.00 + -.13 .01
-.09
-0 + 00
-06 .01
00 .02
-.10 + .02
AGNews
Vehicle
Segment
Model
Bi-ISTM
logreg
ML.P
logreg
ML.P
Ingreg
Parameter size
27K
9K
IK
76
IK
140
Accuracy
0.80 +: 0.02
0.80 + 0.01
0.77 +: 0.02
0.77 +: 0.01
0.98
0.01
0.97 1:: 0.00
Fg
1.00 + .00
1.00 .00
1.00 .00
1.00 :: .0
1.00 :: .00
1.00 :: .00
Alst
.07 1 .00
.16 1. .04
.62 1. .15
2
.17 1
.00 1 .10
.78 .08
0o8
1.00
1.00 .00
1.00 .00
1.00 .0
1.00 .00
1.00
logglast
.07
.18
.09
olgall
.12 1
..3
.06
dot"
1.00 +
1.00 .00
1.00 = .00
1.00 00)
1.00 .00
1.00 +:
dor lust
.07 + .00
.85 .33
.61 = 22
dor
.20 + .01
.97 .03
.72 = .16
IF
.03 1
1 001
01 .03
.01 002
.00 .01
.01 1: .02
dF
.04 +
.131 .00
2.24
.28
.13
.10 :::
IP
.02 + .01
.03 +
-01 .03
-01 .05
.04 + .10
.01 + .05
FK
.04 +
.03 .00
.01 .06
..000
OF
-.17
.14+
-001
.07
-02
PK
.08 +.13
.01
.02
.01
.04
.02 -13
.06
.01
? egrad
-01 + .02
-.14
-.13
.11 +.23
.02 + .12
-.09
16

---

Published as a conference paper at ICLR 2021
125
75
50
25
00
75
10.0
(a) MNIST with CNN. y
(b) MNIST with CNN. y
ship
airplane
frog
dog
deer bird
(c) CIFARIO with MobileNetV2. A.
(d) CIFAR1O with MobileNetV2. B.
Car
ship
frag
truck
sutomobile
dog
deer
(e) CIFARIO with CNN. A.
(I) CIFARIO with CNN. B.
MUM
LOC ABBR
ENTY
14
10
-20
-20
Figure 8: TREC with LSTM. y = A.
(a) TREC with L.STM. B.
Business
Sci/Tech
10
Sports
16
(b) AGNews with L.STM y A.
(c) AGNews with LSTM. y B.
Figure 9: visualization of hall in each dataset and model using UMAP.
17

---

Published as a conference paper at ICLR 2021
Table 3: Average success rate + std. of each relevancy metric for identical class test. The metrics prefixed with < are the ones we have repaired. The results with the average success rate over 0.5 are colored.
MNIST
CIFARIO
TREC
Model
CNN
logreg
MohilenerV2
CNN
Ingreg
Bi-LSTM
logreg
Parameter
12K
SK
2.2M
31K
20K
Accuracy
0.98 1: 0.00
0.92 0.00
0.89 1 0.01
0.72 0.02
0.35 1: 0.01
0.86 1 0.01
0.81 0.02
.93
001
.02
.66 -02
.24
.00
.75 +
Aast
.99
.00
.00
.94 =
.01
.30 .03
.29 002
.20 =
= .00
.76 +
hust
.99 =
1.00 00
.78 .02
.00
Coyall
.88 1 .0
.97 .01
.71 11 .02
.0 .00
dot"
69 + 02
.68 + 02
.09 + .02
.10 + 001
.11 =1 00
.3 .00
.34 + .00
dorfua boot
.67 + .02
1.00 + 00
.20 = 002
.93 .00
dot
.96 +
.96 +
.01
.93 .00
IF
.09 +
.02
.10 .01
.09
.29 .00
.86 +
.72
.14
.14
.82
_02
.12 -01
.00
.96
FK
.10
.02
.20 .02
.20 =
.2 .00
.24 =
KFK
.77
.93 .01
.01
.98 =
.99 .00
.96 + .00
PK
.92 .01
.97
.93 .01
.99
1.00 .00
.96 .00
GD
.0 1 010
.87 1 .01
.26 00
.71 002
1.00 .00
.49 .00
1.00 1 .00
1.00 .00
00
1.00
1.00 1
94 + 01
.99 + .00
.97 + .01
.99 + .00
1.00 = .00
1.00 + .00
1.00 + .00
AGNews
Vehicle
Segment
Model
Bi-LSTM
logreg
MLP
logreg
MLP
logreg
Parameter
27K
9K
76
1K
Accuracy
0.80 1: 0.02
0.80 + 0.01
0.77 0.02
0.77 1: 0.01
0.98 0.01
0.97 1:: 0.00
:39
.02
.65
.02
.01
::
Aasst
.72
.72
011
.47 +
.02
.66 = .02
00
.01
.90 +
bust
=
.74 = .04
.01
84 1 001
.73 1 .04
.96 1 .01
dot"
.88 + 022
.47 + 02
25 .00
.27 +: .01
.37 + .01
.37 4: .01
dor laxt
.8 + .01
.26 = .02
.17 = .06
dor
.90 +
27
.13 .01
IF
.24
002
.39 .16
.78 .08
.15
.52 +
AF IP
.99
.88
.01
.80
.96
+
FK
.32 +
.03
.31 .18
.26 .17
.04
.17 +
KFK
.94 .01
.68 .02
.93 .04
.94 .03
.86 + .06
.95 + .02
.95 + .01
.02
.99 .01
.99
.97 + .02
.99 + .01
GD
.76 1 001
1.00 .000
.90 1.10
.98 .02
.30 1 .14
.55 1 .1
1.00 =
1.00
1.00 .00
1.00 .00
1 .02
1,00
1.00 + .0
1.00 + -00
.99 .01
1.00+.00
.90 + .05
.99 +9++001 .01
18

---

Published as a conference paper at ICLR 2021
Table 4: Average success rate + std. of each relevancy metric for identical subclass test. The metrics prefixed with < are the ones we have repaired. The results with the average success rate over 0.5 are colored.
MNIST
CIFARIO
TREC
Model
CNN
logreg
MohilenerV2
CNN
Ingreg
Bi-LSTM
logreg
Parameter
12K
8K
2.2M
31K
20K
Accuracy
0.99 1: 0.00
0.88 0.01
0.92 0.01
0.84 0.03
0.71 1: 0.03
0.86 0.01
0.81 0.02
.93
.02
.02
004
.31
.03
.78 +
Aast
.02
.49
.03
.95 =
.02
.29 003
= 04
.31
.02
.81 + .02
hust
=
.32 .03
.33
.02
Coyall
.88 1 .00
.71 1: .04
.50 1 .03
.77 .02
dot"
.7 + 03
.75 + 00
.00 + .02
.11 = 033
.09 = 00
.33 .03
.34 + .03
dor!ua boot
24 = .04
.22 = 02
.20 = 001
.40 = .03
dot
.94 =
03
.25 .03
.03
IF
.12
.05
.06 .02
.08
.02
.49
d
.62
.17 -02
.12
+88+.0 .02
.79
.70
.02
.02
.09
_01
.75
FK
.19
+ .02
= .01
.11
.30 .02
.16 = .02
81 +
.76
.03
.24
.73 .03
.78 + .02
PK
.91 +
.85 .02
.37
.03
.23
.02
.79 + .01
GD
42 1 .05
.48 .03
.20 1 02
.24 03
.1 .04
.45 .02
.60 1 .02
.97
.01
1 03
.30
.01
.87 1
91 + .20
.95 + .01
.28 + .03
.3 + 003
334
= .03
.78 + .02
.88 + .02
AGNews
Vehicle
Segment
Model
Bi-LSTM
logreg
MLP
logreg
MLP
logreg
Parameter
27K
9K
38
1K
Accuracy
0.80 1: 0.02
0.80 + 0.01
0.73 0.02
0.73 1: 0.01
0.94 0.01
0.90 1:: 0.01
.40 1:
011
.67
002
.01
.95 ::
Aasst
.53
.66
011
.49 =
.02
.68 = .04
00
.92 .01
.93 +
bust
.54 =
.68 =
.01
.50 1 022
.67 1 .03
.94 1 .01
dot"
.28 + 002
.48 + 02
26 = .03
.26 + .03
.38 + .01
.41 :: .02
dor laxt
.52 = .02
.27 = .03
.15 = .03
dot
.54 +
28
.02
IF
.25 +
001
34 .12
+ .09
.16
.49
AF cP
.56
.76 -14
.06
.10
.48
.56
.00
.86
+
FK
.28 +
.02
.16 .08
.20 .05
.07
.10 +
.56 +
.63 .02
.73 .13
.62 .09
.73 + .13
.93 +
.61 +
.02
.80 .06
.67 +.10
+ .10
.96 +
GD
.50 1 02
.4.1 002
.77 .09
.3 1. .0
.34 .08
.37 1. .08
.72 00
.2 .06
4 .07
.10
.96
.61 + .02
.73 + -03
.72 - .10
.75 +09
.75 .10
.90 300+-00 .03
19

---

Published as a conference paper at ICLR 2021
F.2 ADDITIONAL RESULTS
The identical class test require the most relevant instance to be of the same class as the test instance. In practice, users be more confident about model's output if several instances provided as evidence. In other words, we expect that the most relevant and first few relevant instances will be of the same class. This observation leads to the additional criterion, which is generalization of the identical class test.
Definition (Top-k Identical Class Test). For *test = (Fess Treat): let 2 = (æ), D" be a training instance with the j-th largest relevance score. Then, require y' Frest for any (1,2..... k).
This observation also applies to identical subclass test, which leads to the following criterion
Definition 6 (Top-k Identical Subclass Test), For Fust = (F Brest). let æ = ( F be training instance with the j-th largest relevance score. Then, we require s(z)) s 2( Vj (k...
We show the results of the top-10 10 identical class test in Table 3 and the top-10 identical subclass test in Table
G EXAMPLES OF EACH EXPLANATION METHOD
We show some examples of the relevant instances using several relevance metrics CIFARIO with CNN in Figure 10 and Figure 11 and on AGNews with LSTM in Table 7 and Table g We show examples of both correct (in Figure T0 and Table 7 and incorrect (in Figure TT and Table predictions. As mentioned in Section 3 the relevance metrics based on the dot product of the gradient, such IF, FK, and GD, tend select instances with large norms, and therefore we can see that non-typical instances have been selected.
20

---

Published as a conference paper at ICLR 2021
Table 5: Average success rate + std. of each relevancy metric for top- identical class test. The metrics prefixed with & are the ones we have repaired. The results with the average success rate over 0.5 are colored.
MNIST
CIFARIO
TREC
Model
CNN
logreg
MohilenerV2
CNN
Ingreg
Bi-LSTM
logreg
Parameter
12K
SK
2.2M
31K
20K
Accuracy
0.98 1: 0.00
0.92 + 0.00
0.89 1 0.01
0.72
0.02
0.35 1: 0.01
0.86 1 0.01
0.81 0.02
.63
.02
.0
.00
.00
.00
.23 +
Aast
.95
.00
011
=
.02
.0 .00
.00
:24 .00
.24 +
hust
.97 =
.33 000
.00
Coyall
92 1 .00
.84 1: .03
.23 1 .02
.68 .00
dot"
.99 + .01
.20 + 002
.00 100::: 000
.00 = .00
.00 = .0
.03 .00
.05 + .00
dor boot
42 = .03
.98 + .01
.04 = 001
.75 .00
dot
.88 =
03
.01
84 .00
IF
.00
.00 .00
.00 .00
.00
.00 .00
.24
d
.25
.10
.00
.00
.83+
.47
.59 +
-17++000
.00
.00
.00
.65
FK
.00 =
.02
.00 .00
.06 =
.01 .00
.00 .00
KFK
.23 +
.65 .02
.02
.87 =
.9 .00
.71 + .00
PK
.59 + .01
.02
.02
.93 .01
.95 + .00
.77 + .00
GD
.00 1 .0
.1 .02
.00 1. 00
.15 1 001
1.00 .00
.11 .00
.99 1 .00
.95 1
02
1.00
.961
1.00 1
.57 + .20
.95 + .01
.78 1.4000 .03
.0 + .01
.99 .00
.94 + .00
1.00 + .00
AGNews
Vehicle
Segment
Model
Bi-LSTM
logreg
MLP
logreg
MLP
logreg
Parameter
27K
76
1K
Accuracy
0.80 1: 0.02
0.80 0.01
0.77 0.02
0.77 1: 0.01
0.98 0.01
0.97 1:: 0.00
.00 ::
.00
.09
.09 .02
.60 000
.60
Aasst
1
.16 -
.74 .03
.01 =
.01
.10 .02
.10
.02
.44 +
bust
.51 =
.22 =
.03
48 1 .02
.17 .06
.72 1 .04
dot"
.01 + 00
.01 + .00
115 .12
.16 .13
.23 = .02
.23 + .02
dor laxt
.64 + .03
111
.06 = .06
dot
.66 +
.15 111
.01
IF
.00
001
000
.10 .03
.10 +
AF cP
_02
25 +13
.47
.15
.48
+
FK
.00 +
.00 .00
.05 -11
.01
.03 +
.61
.06+
.55 .19
.64
.32 + .17
.60 +
.71 +
.01
.85 .06
.85 .08
.78 .08
.92 + .03
GD
.55 1. 022
.98 01
.56 .9
.70 .05
.00 1 .08
.37 1 .05
1.00 =
1.00
.95 ..4
1.00 .00
.84 .08
.97 +
.99 + .01
.98 00
.1 .09
.95 +5++003
.43 .0
.84 + .06
21

---

Published as a conference paper at ICLR 2021
Table 6: Average success rate + std. of each relevancy metric for top- identical subclass test. The metrics prefixed with < are the ones we have repaired. The results with the average success rate over 0.5 are colored.
MNIST
CIFARIO
TREC
Model
CNN
logreg
MohilenerV2
CNN
Ingreg
Bi-LSTM
logreg
Parameter
12K
8K
2.2M
31K
20K
Accuracy
0.99 1: 0.00
0.88 0.01
0.92 1 0.01
0.84 0.03
0.71 1: 0.03
0.86 0.01
0.81 0.02
64
.03
+ .0
.00 .00
.00
.05
.25 +
Aast
.54
.02
.02
.00
.02
+
.03
= 01
.00
.04
.27 + .02
hust
.57 +
.00 00
.00 .00
.30 .02
Coyall
80 1 .22
.16 1: .02
.02 1 011
.34 14.11 .02
dot"
21 + 022
.23 00
.00 00
.00 +: .00
.00 = .0
.05 .01
.05 + .01
dor boot
.08 = .02
.00 + 00
.00 = .00
.14 .01
dot
.79 =
.13 02
.01
.17 .02
IF
.01
.00
.00 .00
.00
.00 .00
.01 + .01
.14
.02
.00
.00
+11
.24
.37 +
.04
.00
.03
.25
FK
.00
+ .00
.00 .00
.00 .00
.00 .00
.00 = .00
KFK
.22 =
.30 .03
.00 +00=+00 .00
.00 = .00
28
.26 + .02
PK
.58 +
.04
.00 .00
.00
.03
.25 + .02
GD
.00 1 .0
.1 1 .01
.01 01
.00 .00
.00 .00
.10 .22
.01 .00
1
.00
02
01
.01 =
.37 .02
.0 + 03
.69 + .04
002 .01
.00 + .00
.00 = .00
.24 + .03
.34 .02
AGNews
Vehicle
Segment
Model
Bi-LSTM
logreg
MLP
logreg
MLP
logreg
Parameter
27K
38
1K
Accuracy
0.80 1: 0.02
0.80 + 0.01
0.73 0.02
0.73 4: 0.01
0.94
0.01
0.90 1:: 0.01
.00 1:
.00
.10
.02
.64 ::
Aasst
001
.05
.01
.01
.10 .00
.00
.02
.48 +
bust
.01 =
.06 = .00
.07
All
00 1 001
.10 1 .00
.62 1 .08
dor"
.01 + 00
.02 + 011
.00 = .00
.00 + 00
.24 + .02
.25 +: .02
dor laxt
.01 = 00
.00 = .0
.01 = 002
dor
.02 =
.00
.05
IF
.00
.00
.02
.10 .00
.00
.17
AF
.01
_01
.20+
.01
.00
.00
.14
.66 +
FK
.00 +
.00 .00
.02 = .00
.00 .00
.00 .00
.00 +
.01
.02+
.18 .00
.00+000
.15 .14
.61 + .09
FK
.01 +
.01
.10 .00
.01 +00
.53 .09
.75 +
GD
01 1 00
06.1..0 01101
.9 .00
.39 39 .00
.00
1 .05
.13 1. .08
.011 011
1.0
.30 .00
.52 1 .09
.02 + .01
.08 0 .01
37 000
.00+.00
.18 + .11
.51 + .08
22

---

Published as a conference paper at ICLR 2021
Gold: frog
Predict: frog
deer
frog
frog
deer
frog
frog
Test Input
2
E2
62
cos
cos
cos all
airplane
automobile
truck
decr
frog
frog
dot
dot Jast
dot all
IF
cos
frog
frog
bird
frog
frog
FK
ePK
cos
GD
GC
grrad
Gold: airplane
Predict: airplane
bird
airplane
airplane
bird
airplane
airplane
Test Input
glass
fall
cos x
cos last
cos all
airplane
truck
airplane
ship
horse
dot
dot lass
dot
IF
JF
cos IF
frog
airplane
airplane
airplane
airplane
airplane
FK
cog
PK
GD
GC
pgrad
Figure 10: Relevant instances selected for random test inputs with correct prediction using several relevance metrics CIFAR10 with CNN.
23

---

Published as a conference paper at ICLR 2021
Gold: dog
Predict: deer
dog
deer
deer
dog
deer
deer
Test Input
glast
$2
COs
cos
Jast
cos
all
airplane
automobile
automobile
automobile
deer
deer
dot
dut Jast
dot all
IF
ALF
cos
deer
deer
deer
deer
FK
EPk
cos
GD
GC
grrad
Gold: ship
Predict: automobile
truck
automobile
ship
ship
automobile
ship
Test Input
plast
pill
cos
cos
cos
airplane
automobile
automobile
deer
automobile
deer
d
dot Jast
all
IF
FF
cos"
frog
automobile
automobile
automobile
automobile
automobile
cus
GD
GC
ggewl
Figure 11: Relevant instances selected for random test inputs with incorrect prediction using several relevance metrics on CIFAR10 with CNN.
24

---

Published as a conference paper at ICLR 2021
Table 7: Relevant instances selected for random test inputs with correct predictions using several relevance metrics AGNews with LSTM. Out-of-vocabulary words are followed by [unk].
Sentence
Class
Gold: World
Test Input
kerry widens lead in california, poll finds reuters)
Predict: World
E
in brief
Sci/Tech
clact
strong hurricane approaches(unk, [unk bahamas[unk] florida (reuters)
Sei/Tech
eall
strong hurricane approaches[unk] bahamas[unk] florida reuters reuters poll : bush holds two -point lead over kerry ( (reuters) ) strong hurricane approaches(unk. bahamas[unk]. florida reuters strong hurricane approaches(unk, bahamas[um [unk] florida (eeueers) reuters poll : bush holds two point lead over kerry (reuters) eurozone finance ministers debate action on oil prices surge afp
Sci/Tech
cos"
World
cos castast
Sei/Tech
cosall
Sci/Tech
dot
World
dot lost
World
dot IF
business cash for bush campaign lawyers[unk] for kerry (reuters) greek [uno dies in hospital after balcony[unk) suicide leap[unk]
World
Sports
ez
world front
World
cos IF
arafat family bickers[unk] over medical[unk] records of palestinian Icader
World
FK
linux # 39;s latest moneymaker[unk]
Business
e
china launches zy-2[unk] resourcefunk] satellite
Sci/Tech
cos FK
china launches zy-2[unk] resource[unk] satellite
Sci/Tech
GD
judge adjournsfunk ba[unk] # 39:asyi[[unk] 39;s trial until nov.
World
GC (grad
reuters poll bush holds two point lead over kerry (reuters)
World
reuters poll bush holds two point lead over kerry (reuters)
World
Sentence
Class
Gold: Sci/Tech
Test Input
some people not eligible/unk] get in on google ipo
Predict: Sci/Tech
E
insiders[unk] get rich[unk] through google ipo european judge probes microsoft antitrust case insiders[unk] get rich[unk] through google ipo insiders[unk] get rich[unk] through google ipo breakthrough in hyyrrgee[unkk [unk] fuel research insiders [unk] get rich[unk) through google ipo italians[unk], canadians[une gather[unk] to honour[unk] living legend[unk] ve[unk] winner smoky [unk] smith[unk] (canadian press)
Sei/Tech
Sci/Tech
gall
Sci/Tech
Sci/Tech
loglas
Sci/Tech
cogall
Sci/Tech
dot."
World
last
carnings alert: novell sees weaknessfung | spending siemens backs new wireless technology matchinglunk| wits[unk] on politics insiders [unk] get rich[unk] through google ipo congress probes fda in vioxx case
Sei/Tech
al
Sci/Tech
IF
Sports
gr
Sci/Tech
cos IP
Business
FK
bin laden tape urges oil attack
Business
epk
[aaiders[unk]] get rich[unk] through google ipo (nsiders[unk] get rich[unk] through google ipo issue 65 news hound[unk]: this week in gaming
Sci/Tech
cos
Sci/Tech
GD
Sei/Tech
GC
google responds [unk]] google news china controversy/unk]
Sci/Tech
ggrad
(nsiders[unk] get rich[unk] through google ipo
Sci/Tech
25

---

Published as a conference paper at ICLR 2021
Table Relevant instances selected for random test inputs with incorrect predictions using several relevance metrics AGNews with LSTM. Out-of-vocabulary words are followed by [unk].
Sentence
Class
Gold:Sci/Tech
Test Input
ibm to hire even[unk] more new workers
Predict:Busi ness
E
athletes[unk] to watch[unk]
Sports
clact
tech stocks lumble(unk] after chip makers warn microsoft foe[unk] wins settlement
Business
eall
Sci/Tech
cos"
volkswagenjunk, workers stage new stoppages[unk] tech stocks tumblc[unk] after chip makers warn microsoft revenue tops forecast
Business
last
Business
cosall
Business
dot*
italians[unk]] canadians(ume [unk] gatherfunk] to honour[unk] living
World
legend[unk]: ve[unk] winner smoky[unk] smith[unk] canadian press
dot Just
google up in market debut after bumpy [umpy[unk] ipo (eeuter)) ) google up in market debut after bumpy [unk] ipo ( (reuttrs) greek judoka[unk] dies in hospital after balcony[unk] suicide leap[unk]
Business
dot
Business
IF
Sports
ey
ibm # 39;s third quarter carnings and revenue up
Business
IP TP
arafat family bickers[unk] over medical[unk] records of palestinian leader
World
FK
great white sharks[unk] given new protection
World
grk
ibm # 39;s third quarter earnings and revenue ibm to buy danish[unk] firms
Business
FK
Business
GD
some question speed of intel chief bill ( ap
World
GC
ibm shrugs [unk] off industry blues[unk] in q3
Business
ggrad
ibm # 39;s third quarter earnings and revenue up
Business
Sentence
Class
Gold: Sports
Test Input
lougher[unk] rules n't softenfunk law 's game
Predict: Sci/Tech
profitinglunki from moore[unk] 's law
Business
devil[unk] rays[unk] stuck[unk] in florida hours[unk] before game devil[unk] rays[unk] stuck[unk] in florida hours[unk] before game profiting(unk] from moore/unk; 's law
Sports
fall
Sports
Business
logl
devi][unk] rays[unk] stuck[unk] in florida hours[unk] before game devil[unk] rays[unk] stuck[und (unc[[un]]]] in florida hours before game [unk]] canadians [unk] gather[unk] to honour[unk] living legend[unk]: ve[unk] winner smoky(unds smith(unk: canadian press)
Sports
cogall
Sports
dot."
World
last
world 's top game players battle for cash ( ap sportsnetwork((nkk] game preview top gradesfunt rising again for geses[unk] calif. oks toughest[unk] auto amissions(unk) rules envoy headed to darfur yankee[unk] batters[unk] hit wall
Sci/Tech
dlt
Sports
IF
World
gi
World
cos IP
World
FK
Sports
epk
flat panel does n't always[unk] compute(unk] flat panel does n't always[unk] compute[unk] issue 65 news hound[unk] this week in gaming
Sei/Tech
cos
Sci/Tech
GD
Sei/Tech
GC
atari[unk] announces first 64-bitfunke game
Sci/Tech
ggrad
atari[unk] announces first 64-bit(unk game
Sci/Tech
26


Reviews:
[{"review_id": "bX5DMTb2TaT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n \nThis paper is on the evaluation of XAI for supervised models. XAI is a trending topic since several years now but evaluating the quality of an explanation is still challenging. This works focuses on similarity-based explanations, which means providing similar training examples along with the predicted class. Another assumption is that the chosen examples must be of the same class.\n\nThe main contribution is to benchmark several similarity measures against various datasets, using three tests. One test is taken from the literature and the two others are proposed in the paper. The experiments show that some similarity measures are better than others. I really like that the authors find an explanation for those results, based on geometrical properties of the measured objects. Besides, they use this finding to correct existing measures.\n\n\n##########################################################################\n\nReasons for score: \n\nAs a whole, I like the main aim of the paper for I think XAI deserves better evaluation metrics. However, I have some concerns that explains my vote.\n\nFirst of all, I am not fully convinced that the assumption of restricting to similar examples with the *same* class is really good for giving a relevant explanation. I believe that a good explanation will need both positive and negative examples, in particular near the decision frontier. Therefore this tempers the interest of the proposal made in this paper.\n\nSecond, I had a hard time figuring out how and when the third test (with subclasses) can be used, and how it has been implemented in the presented experiments. From my understanding, this test is useful when you access a class taxonomy (e.g., animal -> mammal -> cat). But in Section 4.2 the authors explain that they randomly assign existing classes to on of the two (subclass) A and B. What does it mean? Besides I'm not sure to be convinced by the argument raised in 3.3 (\"...the violation of this third requirement again leads to nonsense explanation\"). We can find similarities between two animals (here, car and frog) because they share common properties (e.g., breathing), in particular if you compare them to a truck.\n\nThird, the randomization test relies on a unique random model that is confronted to the learned model. However it is possible that the random model is close to the learned model and I expected the random model to be generated several times in order to get more robust results.\n\nFinally, I was puzzled by Fig.6. Why don't the authors use the same examples for the different measures? It seems unfair as the examples given to GC looks easier to classify/explain.\n\n \n##########################################################################Pros: \n\n \n1. address the difficult task of evaluating explanations in XAI\n2. paper easy to read (except for the third test)\n3. really interesting discussion on the failure of (some) existing measures\n \n##########################################################################\n\nCons: \n\n \n1. restricted to a narrow form of XAI\n2. subclass test not well explained \n3. Fig.6 is misleading (see above)\n\n \n##########################################################################\n\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above.\n\n \n#########################################################################\n\nAdditionnal comments:\n\n\"two models with different reasoning process\": this expression is misleading, it's not really a \"reasoning process\". I would write \"inference process\" or \"inductive process\".\n\nThe accuracy of the learned models is never taken into account. I think evaluating the explanations should be done at the light of classification ability. The authors may discuss this point.\n\nSome typos: \n\n\"two instance\"\n\"the the\" (twice)\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "a6oHf3PMlO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\n This work provides an empirical evaluation of similarity metrics used in example-based explanations methods, where the goal is to provide decision support examples in the training set for a black-box model's prediction. The paper evaluates gradient based metrics popular in the literature such as Influence functions, fisher kernels but also simpler naive approaches that relies on l2, cosine distances and dot product on different embedding spaces. The authors introduce 2 new tasks for assessing the reliability of the different methods: an identical class test and an identical subclass test.\n\n#### Strengths\n - The paper is clearly written and easy to follow. The idea to empirically evaluate different similarity metrics for example-based explanation methods is interesting and could be useful to practitioners. The motivation for the work is clear and well introduced. \n\n#### Weaknesses\n- Identical class test: the requirement that the returned explanatory examples in the training set should belong to the same class as the example is ambiguous and raises some concerns.  1) Is is not clear how this applies in case where the test example is misclassified? Why does the identity requirement applied only on the predicted label and not the true label as well? 2) Training examples that are either predicted differently or belong to a different true class as the test example can be useful in understanding failure modes of the model, misclassified examples, or inhibitory examples. This validity of this test need to be discussed and justified.\n- Same applies for the Identity subclass test. The subclass test has even stronger requirements than the identity class test. The validity of this test should be justified beyond a simple definition.\n\n-Strong statements are made on the 'usefulness' of the explanations but these are mainly intuitive assessment of individual cases.  For example, in section 3.2, the authors state \"The violation of this requirement leads to nonsense explanations such as “I think this image is cat because a similar image I saw in the past was dog.” which do not make sense to the users at all. When faced with such an explanation, the users will find the validity of the model’s prediction questionable, and will ignore the prediction even if the underlying model is valid\". \nThe usefulness of the explanations provided by these methods to human users should be assessed through a human evaluation.\n\n#### Recommendation with reasons\nWhile I like to motivation behind this work and it's potential usefulness, the validity of the proposed tests is not clearly addressed. For that reason, I can not recommend acceptance for this work. \n  \n#### Questions\n- Justify the correctness of the identity and subclass identity test\n\n#### Additional feedback\nAdditional reference: Yeh, C. K., Kim, J., Yen, I. E. H., & Ravikumar, P. K. (2018). Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems (pp. 9291-9301)", "extra_scores": {}, "extra_text": {}}, {"review_id": "TBj9uhHfBfv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The presented study tackled the problem of evaluating different metrics for generating similarity-based explanations. To this end, the authors used three tests to evaluate four types of metrics on several different datasets. The experiment results revealed that the cosine similarity of the gradients of the loss performed the best. Overall, the methods adopted by the study are technically sound and the paper is well written (though with a few typo errors). In particular, the presented experiments are extensive and the results are somewhat interesting. Still, there are a few comments that can be considered to further improve the paper:\n\n1. It would be good to provide a more comprehensive description of relevant studies in generating explanations (e.g., what are those application scenarios, what datasets were used).\n2. Why only logistic regression and deep neural networks like CNN and Bi-LSTM were used for experiments? Any justification for this choice?\n3. It would be good to provide some examples (especially those by using textual data) to provide a more in-depth explanation the actual value behind those reported numbers, e.g., what specific explanations can be produced by applying the best-performing metric vs. the worst-performing metric?\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6fXmxnb2wCH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate which relevance metrics are desirable for explanations that are based on extracting similar instances as evidence to support a model prediction. They establish three minimal requirements for those metrics:  one adapted from the state-of the art: (1) model randomization to ensure the measure is model-specific, and two proposed requirements,  (2) identical class test (instance pointed out should be of the same class as the test instance) and (3) identical subclass test (instance pointed out should be of the same subclass as test instance). Several metrics are compared and the authors conclude that many of the state-of-the-art metrics do not even satisfy the first requirement and that the cosine similarity of the gradients of the  loss perform best. \nThe authors conduct an empirical evaluation on two image datasets, two text datasets and one tabular dataset. \nThe paper is well-written, the method and the two requirements straightforward and the empirical comparative approach acceptable. \nIt is not clear from the paper how the noise in the labels is handled (what if the nearest neighbor used in the second requirement is mislabeled?). More extensive experiments on tabular data would have been good. \nFurthermore, how to ensure these three requirements are sufficient?  While the approach presented and the two additional principles are quite simple, it is nice to see a comparative study on explanation metrics. The work could benefit from deeper insights on the evaluation metrics from a theoretical perspective.\n\nMinor comments:\n- Figure 1 and 2 are not necessary in my opinion\n- Figure 3: sub-captions. missing. \n- concrete examples on the vehicle data would have been desirable. Why this data specifically? there are many benchmark datasets from the UCI repository and others.\n- \"do\" --> does in section 3\n- \"the the\" in section 3.1\n- \"do\" -> does on section 3.2\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper performs an empirical comparison of similarity-based attribution methods, which aim to "explain" model predictions via training samples. To this end, the authors propose a handful of metrics intended to measure the acceptability of such methods. While one reviewer took issue with the proposed criteria, the general consensus amongst reviewers is that this provides at least a start for measuring and comparing instance-attribution methods. 

In sum, this is a worthwhile contribution to the interpretability literature that provides measures for comparing and contrasting explanation-by-training-example methods. 
Decision: Accept (Poster)