On Evaluating Explainability Algorithms
Authors: Gokula Krishnan Santhanam
Ali Alami-Idrissi
Nuno Mota
Anika Schumann
Ioana Giurgiu
Venue:  (2020)
Keywords: interpretability
Deep Learning

Abstract: A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artificial Intelligence (XAI) community. Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial. In this work, we propose a suite of multifaceted metrics that enables us to objectively compare explainers based on the correctness, consistency, as well as the confidence of the generated explanations. These metrics are computationally inexpensive, do not require model-retraining and can be used across different data modalities. We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients. Our experiments show that the proposed metrics reflect qualitative observations reported in earlier works.

Under review as a conference paper at ICLR 2020
ON EVALUATING EXPLAINABILITY ALGORITHMS
Anonymous authors Paper under double-blind review
ABSTRACT
A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artificial Intelligence (XAI) community. Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial. tifaceted metries that enables to objectively compare explainers based the correctnes,, consistency, as well as the confidence of the generated explanations. These metrics computationally inexpensive, do not require model-retraining and can be used across different data modalities. We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients. Our experiments show that the proposed metrics reflect qualitative observations re- ported in earlier works.
this work, we propose a suite of mul-
INTRODUCTION
Over the past few years, deep learning has made sigmificant progress, outperforming the state-of- the-art in many tasks like image classification (Mahajan et al. 2018), semantic segmentation (Zhul et al., 2018), machine translation (Kalchbrenner et al.) (016) and even surpassing humans in the games of Chess and Go (Silver et al. 2016). As these models are deployed in more mission-critical standard metrics, they are frag- ile (Szegedy et al. [0133; Goodfellow et al. 2014) and can be casily fooled by small perturbations to the inputs (Engstrom al. 2017). Further research has also exposed that these models are bi- ased in undesirable ways exacerbating gender and racial biases (Howard et al.) 2017 Escudé Font & Costa-Jussál 2019). These issues have amplified the need for making these black-box models interpretable. Consequently, the XAI community has proposed a variety of algorithms that aim to explain predictions of these models (Ribeiro al., 2016 2018 Lundberg & Lee 2017 Shrikumar
systems, notice that despite their incredible performance
it al. 2017, Smilkov et al. 2017 /eevaraju et al. 2016 Sundararajan et al. 2017)
With such an explosion of interpretability methods (hereon referred as explainers), evaluating them has become non-trivial. This is due to the lack of widely accepted metric to quantitatively compare them. There have been several attempts to propose such metrics. Unfortunately, they tend to suffer from major drawbacks like computational cost (Hooker et al. 2018)., inability to be extended to non-image domains (Kindermans et al. 2017a), or simply focusing only one desirable attribute of good explainer. (Yeh et al. 2019))
In this paper, we propose suite of metrics that attempt to alleviate these drawbacks and can be ap- plied across multiple data modalities. Unlike the vast majority of prior work, we not only consider the correctness of an explainer, but also the consistency and confidence of the generated explana- tions. We use these metrics to evaluate and compare widely used explainers such as LIME (Ribeiro) jet al. (2166), Brad-CAM (Selvaraju et al.) 2016). SmoothGrad (Smilkov et 2017) and Integrated Gradients (Sundararajan et al. (2017) on the ImageNet dataset (ILSVR20112) (Deng et al. 2009), in an objective manner (i.e., without
Inception-V3 (Szegedy al. 2015)) model pretrained
the need of human- loop). Moreover, our proposed metrics are general and computationally inexpensive. Our main contributions are:
1. Identifying and formulating the properties of good explainer.
2. Proposing a generic, computationally inexpensive suite of metrics to evaluate explainers.
3. Comparing common explainers and discussing pros and cons of each.

---

Under review as a conference paper at ICLR 2020
We find that while Grad-CAMe CA- seems perform best overall, it does suffer from drawbacks not reported in prior works. On the other hand, LIME consistently underperforms in comparison to the other models.
2 RELATED WORKS
The field of XAI has become an active area of research (Doshi-Vele' & Kim 2017 Lipton 2016) with significant efforts being made to explain Al models, either by generating local (Ribeiro et al.
2016 Shrikumar et al.) 2017 Sundararajan et al. 2017 Selvaraju et al. 2016 Smilkov et al. 2017)
or global (Lundberg & Lee 2017 Ribeiro et al.
2018 explanations.
Simultaneously, there growing research efforts into methods to formally evaluate and compare explainers [Mohseni et al. 2018 Gunning, 2019) Wolf 2019) Gilpin et al. 2019). Notably, Mur- doch et al. (2019) introduced framework with three desiderata for evaluation, viz. predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human. In con- trast, Hall al. (2019) compiled set of desired characteristics around effectiveness, versatility, constraints (i.c., privacy, computation cost, information collection effort) and the type of generated explanations, which do not need human evaluation, and therefore are objective. However, they focus very little on aspects such as correctness. Recently, DeConvNet (Noh et al. (015), Guided BackProp (Springenberg et al. 2013) and LRP (Bach et al., 2015) have been shown to not produce theoreti- cally correct explanations of linear models (Kindermans et al. 2017b). As result, two explanation techniques, PatternNet and Pattern Attiibuntr,,utionn, that are theoretically sound for linear models were proposed. Other efforts focus
evaluating saliency methods (Kindermans et al. 201 Adebayo et al.] 2018) and show that they are unreliable for tasks that are sensitive to either data or model. Samek al. (017)) and its variations (Hooker al., 2018:) Fong & Vedaldi, 2017: Ancona al.) 2018) infer whether a feature attribution is correct by measuring performance degradation when highly attributed features are removed. For instance, Hooker et al. (2018) shows that commonly used interpretability methods are less accurate or are on-par with random designation of feature importance, whereas ensemble approaches such as SmoothGrad (Smilkov al. 2017)
superior.
Yang & Kim (2019) proposed three complementary metrics to evaluate explainers: model contrast score - comparing two models trained consider opposite concepts as important, input dependence score comparing one model with two inputs of different concepts, and input dependence rate comparing one model with two functionally identical inputs. These metries aim to specifically cover aspects of false-positives. Alvarez-Melis & Jaakkola (2018) define alternative set of metrics, around explicitness intelligibility of explanations, faithfulness feature relevance, and stability consistency of explanations for similar or neighboring samples. Finally, Yeh et al. (201)) define and evaluate fidelity of explanations, namely quantifying the degree to which an explanation captures how the underlying model itself changes in response to significant perturbations.
Similar to previous work, we focus on objective metrics to evaluate and compare explainers. How- ever, not only consider correctness, but also consistency and confidence (as defined next).
3 WHAT MAKES AN EXPLAINER GOOD?
3.1 PRELIMINARIES
In the following discussions, r E I2 be an arbitrary data point and y be the corresponding ground truth label from the dataset D {( v.), < i < M). Let f be the classifier realized the set of transformations under which semantic information in the input remains unchanged. If is an arbitrary transform from this set, let t-1 be its inverse transform. For example, if = Rot-90", then Vx E D.
as a neural network parameterized by weights
Let
Rot + 90". In general,
t (t(x))
Let Ey be any explainer that generates explanations for predictions made by classifier
the subscripty used only when its absence causes ambiguity.
2

---

Under review as a conference paper at ICLR 2020
Finally, let d(,) be function that computes the difference between the generated explanation For example, if the explainer E generates saliency maps (e.g. GradCAM and SmoothGrad), d could be simple Ep norm.
Additionally, in order to ensure that we minimize the impact that pathologies of the underlying classifier have the properties we are interested in, we assume that the classifier has acceptable test-set performance. Furthermore, we also assume that the classifier performance does not degrade significantly under the transformations we perform (described in Sec. (.2.2p. If the classifier does not satisfy these conditions, it is prudent to improve its performance to acceptable levels prior to attempting explain its outputs. One cannot extract reliable explanations from underperforming underlying models (Ghorbani et al. 20199; Samek et al. (017))
3.2 DESIDERATA FOR A GOOD EXPLAINER
Inspired by earlier works on important aspects of an explainer's quality (Yang Kim 2019 Alvarez- Melis & Jaakkola [2018) Yeh et al. 2019), our proposed evaluation framework consists of the following components:
Correctness
Consistency
Confidence
We elaborate on these components as well as methods to compute them in the image classification scenario. Even though these are evaluated independently, they can be combined together to give single scalar value to compare explainers component depends heavily on the use case and end-user preference. This is beyond the scope of the current work and thus is not discussed further. Further, since we elaborate on the image classification scenario, we use inputs and images interchangeably with the understanding that the described methods or equivalents can be trivially adapted in other modalities.
straightforward way. However, the weight for each
3.2.1 CORRECTNESS
Correctness (sensitivity fidelity in literature) refers to the ability of explainer correctly iden- tify components of the input that contribute most to the prediction of the classifier. Most metrics proposed far focus solely correctness and attempt to compute it in different ways, often requir- ing retraining of the underlying classifier. Moreover, they do not capture all aspects of correctness nor do they generalize to other data modalities.
We propose novel into consideration both that the explainer identifies most of the relevant components and does not incorrectly select non-imponn con-impottant components as important.
method that addresses these drawbacks. It takes
If the explainers are performing as expected, simple masking of the input image with the associated explanation should provide better accuracy as the network is unlikely to be confused by the non- important pixels. However, we do not observe this practice, as we show empirically that vanilla masking results in severe performance deterioration (see and 10 for results). We hypothesize that this because of the following reasons:
The masked image has a large proportion of empty pixel and thus does not belong to the data distribution (Pdata)
Extracted pixels are important in the context of the background pixels, and as such remov- ing context makes the masking meaningless. Additionally, Convolutions have the inductive bias that the neighbouring pixels are highly correlated that helps perform well on visual tasks (LeCun et al. 1998). Simple masking breaks this correlation.
Based on the above observations, we conclude that it is crucial to have realistic background for the extracted patches to properly evaluate them. We propose the following procedure to provide background such that the resulting image is closer the data distribution
We do not require d(,) ) to he a distance metric in the strictest sense.
using the first convolution layer bias values for the blank pixels does help either
3

---

Under review as a conference paper at ICLR 2020
For each class in the dataset, select the top k and bottom & images, sorted in decreasing order based on the probability assigned by the classifier to the ground truth class. We then randomly pair each of the top images with of the bottom images. For each pair, extract important regions identified by the explainer from the top image (i.e high confidence images) and overlap them over the corresponding bottom image (i.e low confidence images). We use the bottom images for this task as we know that they are uninformative for the classifier as evidenced by the assigned probability. We thus obtain dataset of masked images with important regions from the most important images along with relevant yet non- non-informative backgrounds for each class (see Fig. for an example). Formally, the masking operation can be represented as:
M (h(E & + Th (1 E))H))
Where M is the new masked image, threshold function, H the high confidence image, L the low confidence image and @theelement -
We then measure the accuracy on this masked dataset and compare it with the accuracy on the bottom k images subset.
Note that the above mentioned process only evaluates the explainer is capturing important pixels. In order to verify that the explainer does not select non-important pixels, we repeat the same process but instead use the inverted saliency map we expect the accuracy to deteriorate. Formally, the inverse masking process can be defined as follow:
2 and recompute accuracy on this dataset. In this scenario,
M = Th ()(H)) H TH(E;(H))
Inputs
LIME
GradCAM
Integ. Grads
SmoothGrad
FG Explanation
BG Masked images
FG Explanation
BG Masked images
FG Explanation
BG Masked images
Figure 1: Examples of the proposed algorithm for correctness
Interestingly, these masked accuracies are similar to the precision and recall metrics used in infor- mation retrieval (Manning et al. 2009). This provides motivation to combine these differences into
#1. saliency map we pre-process the saliency map values lie in [0, 1] using min-max scaling

---

Under review as a conference paper at ICLR 2020
Pseudo-F) F1 score by computing the harmonic mean of accuracy normal masked images and 1 accuracy on inverse masked images. Formally this can be computed as:
2
Pseudo Fl = (1 acc inveers)
(3)
-1 + (acc@l)-1
3.2.2 CONSISTENCY
We define consistency as the ability of the explainer to capture the same relevant components under various transformations to the input. More specifically, if the classifier predicts the same class for both the original and transformed inputs. Then, consistency measures whether the generated explanation for the transformed input (after applying inverse transform) is similar to the one generated for the original input.
For example, if we apply vertical flip as semantically invariant transformation, we flip the gen- crated heatmap from the transformed image before comparing with the heatmap generated for the original image.
Formally, this can be represented as
S s. f(x) [((()), LET
(4)
Semantically Invariant Transforms We focus on subset of all potential transformations which does not change the semantic information contained in the input. We call this subset Semantically Invariant Transforms. Most work so far has considered only noising as a method of transforming the input. By constraining the magnitude of the added noise, we can control the size of the neighbour- hood in which we perturb the images. In this work, we consider not only simple transformations that perturb the image in small neighbourhood but also those that move the data point vastly different regions in the input space while still retaining the semantic information contained within. This allows space.
verify whether the explainer works as expected across larger regions of the input
For example, in the of images, the family of transformations T include affine transformations (translations and rotations), horizontal and vertical flips, noising (white, blue etc.), scaling etc.
In the image domain, d could be realized the & (Euclidean) distance between explanations of the ground truth and inverted the transformed images (according to Eq. 4.. However, Wang et al.) (2005) and Zhao & Itti (2016) have shown that lg is not robust for images and may result in larger distances between the pairs of mostly similar explanations. This is attributed to the fact that &2 is only a summation of the pixel-wiss wise intensity differences and, as a result, small deformations may results in large distances. Even when the images are normalized before hand, lg is still not a suitable distance for images.
Therefore, we instead use Dynamic Time Warping (DTW) (Sakoe & Chiba 1978) which allows for computing distances between two time series, even when misaligned or out of phase. [lbrahim & Valli (2008) has shown that DTW is effective for images well, not only for temporal data as originally intended. Due to DTW's high computational cost (quadratic time and space complexity). use FastDTW (Salvador & Chan 2007), approximation of DTW that has linear complexity in order to compute the distance between pairs of explanations.
3.2.3 CONFIDENCE
Finally, confidence concerned with whether the generated explanation and the masked input result in high confidence predictions. This is a desirable property to enable explanations to be useful for downstream processes including human inspection (Hall et al., 2019). So far, our method for computing correctness sheds light only the average case and is not particularly useful for individual explanations.
Generating high-confidence predictions is related to the well researched field of max-margin classi- fiers (Gong & Xu (007). A large margin in classifiers is widely accepted as a desirable property. Here, extend this notion to explainers and propose that explainers generating explanations that result in high confidence predictions are desirable to those that do not. In addition to the desirable statistical properties that this enforces, high confidence predictions also vital for building trust
5

---

Under review as a conference paper at ICLR 2020
with human users of the explainer they the average (Narayanan et al. 2018 Ross et al. (017).
interested in the per-instance performance than
Concretely, use the same procedure as in Sec. 3.2.1 Instead of computing the increase in accu- racy, we compute instead the difference in probability assigned to the ground-truth class, as well as the difference in entropy of the softmax distributions of the original and masked images. We report this for both normal and inverted saliency maps. We expect to observe a positive probability differ- ence and negative entropy difference under normal masking and inverted behavior under inverse masking owing to similar reasons discussed in Sec. 3.2.1
However, explainers that generate coarse explanations can easily fool this metric. An extreme case is when the explainer considers the entire input useful. Such am explainer is useless but will have the theoretically highest change in confidence and entropy. To combat this and establish how sparse the generated explanations are, we also report the average number of pixels in the explanations, normalized by the total number of pixels in the image.
We do not combine these numbers into one as different situations have different preferences. For example, in computational biology domains, sparsity is not as important as increase confidence. The right weighting again depends the use case and user preference.
4 EXPERIMENTS
We use an Inception v3 (Szegedy al. 2015) architecture pretrained on ImageNet (ISLVC-2012)
5
We compare LIME, Grad-CAM, SmoothGrad and Integrated Gradients and measure how they perform on the metrics described previously. All experiments and explainers (except LIME) were implemented in Py'oocel (Paszke et al. 2017). Wherever possible, we reused the official implemen- tation or kept our re-implementation as close to the official codebase as possible. The correctness and confidence metrics for every explainer are computed over 5 runs and mean values are reported. We use a fixed constant threshold to binarize explanation masks. We conducted further experiments by thresholding the percentiles instead(as done in (Smilkov al. 2017 These results have been reported in tables [ndd]] 7
We found that the choice did not affect the relative trends observed.
We consider the following semantically invariant transforms: translations (x: +0.2, y +0.2), rotations ( (-115 15 10" -5, 50, 10", 15"), flips (horizontal and vertical). To establish that these do produce too many out-of-distribttinn samples (causing decrease in classifier performance), compute the accuracy of the underlying classifier under these transformations. Table shows that, indeed, drops in accuracy are not significant.
Even though noising is semantically invariant in the image domain, we do not consider in our experiments as some explainers like Smoothgrad would be unfairly favoured.
Table I: Baseline accuracies under transformations
Transformation
Param
Acc 1
Acc@ 5
No Transformation
77.21
93.53
Horizontal Flip
77.19
93.45
Vertical Flip
50.72
75.33
= 0.2
75.46
92.41
X -0.2
75.21
92.40
Translation
y 0.2
74.90
92.00
y = -0.2
74.87
92.04
15"
68.29
87.49
I0"
71.19
89.35
-
74.62
91.90
Rotation
5"
74.76
92.08
10a
71.35
89.40
15"
67.99
87.18
"The weights were taken from pytorch-hub PPercentiles computed on individual explanation map pixels

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy the bottom images after masking, using normal inverse masking, when
normal accuracy of the bottom images with no masking is acc 1=11.42 and acc @ 5=53.8
LIME
Grad-CAM
Integ. Grad.
Smoothgrad
acc I
53.03
97.44
61.6
76.22
acc inverse
42.89
15.4
10.64
9.74
Pseudo-F1
0.56
0.91
0.73
0.83
acc @5
80.34
99.59
80.72
89.72
acc@25 inverse
70.08
51.4
28.42
30.95
Pseudo-F1 0.44
0.65
0.76
0.78
4.1 EVALUATING CORRECTNESS
We perform the procedure described Sec. 3.2.1 and report the results in Table 2 The baseline acc@1 and @c were 11.42% and 53.8% respectively. We hypothesized that good explainer's accuracy increases with the normal masking and decreases with inverse masking. We see the ex- pected increases in accuracies across all the explainers with Grad-CAM obtaining the highest in- crease at 97.44%%
However, for the inverse masking, we see that both LIME and Grad-CAM show results contrary to our hypothesis. This can be explained by observing examples of maps generated in Figs. and [ We see that, on average, Grad-CAM generates much larger explanations than all other explainers (can be seen in Table 33 as well). This implies that Grad-CAM misidentifies several non-important pixels as important and thus when we compute the inverse masks, we remove non-important pixels that could confuse the classifier.
In the case LIME, we again see from Table 3 and Figs. I and 4 that LIME generates the smallest explanations. We further see from Table 2 that LIME has the smallest accuracy gain (in both acc @ and acc These indicate that LIME fails to select important pixels that were selected by all other explainers. Thus, pixels as well and thus would cause increase in accuracy as observed.
conclude that the inverse masks in case of LIME would contain important
4.1.1 EFFECT OF NUMBER OF IMAGES
As detailed previously, our methodology for computing correctness involves choosing a number k of top and bottom images to be used for masking. We evaluate how sensitive the measured correctness of explainers are to the value of k. We report the changes in accuracy with respect to the unmasked bottom images for 1= 15,20,55) in Fig. 2. The actual accuracy numbers are also reported in Tables 4 and 5.
We that for both @cc@ I and @ the change in accuracy for normal masking decreases increase k. This is as expected since the average confidence gap between the top-k and the bottom-k images decreases k increases. This means that the important pixels in the background images are masked with non-important pixels from the foreground images. On the contrary, LIME shows smaller decrease accuracy (both @cc@ and acc@5). This can be explained by the fact that LIME does not capture all important pixels, and therefore all important pixels from the background are not replaced by res-iinformativ pixels.
Similarly, for acc @ and 55 for inverse masking, we see that LIME, Smoothgrad and Integrated Gradients behave as expected, i.c., the drop accuracy is diminished with increased as we are retaining the informative parts from the new background images. Interestingly, the drop in accuracy for Grad-CAI is stable and close to zero. To understand this, we refer again to Table 3 and note that Grad-CAM produces the smallest inverse maps average. This implies that when perform the inverse masking, we retain much of the informative pixels of the background image and thus do not significant drops in accuracy relative the unmasked bottom-k image dataset.
7

---

Under review as a conference paper at ICLR 2020
Figure 2: Effect of k correctness with normal and inverse masking
4.2 EVALUATING CONSISTENCY
Next, evaluate consistency by computing the distance with FastDDTW between the saliency maps generated on the original images and those generated when transformations are applied to the input image (Following Eq. Fig. 3 and Table g report the normalized distances relative to each transformation (i.e., heatmaps sum to 1).
First, transformations become more drastic relative to the original saliency maps, the distances also increase. This is the desired behavior one would expect, thus motivating our choice for using FastDTW. Second, Grad-CAM outperforms all other explainers, as reflected by the fact that its corresponding distances are always smallest. It is followed by Smoothgrad, Integrated Gradients and LIME. This is expected given the grainy saliency maps obtained with Integrated Gradients and Smoothgrad, as well as the patchy heatmaps generated with LIME.
Net
34
41
20
Figure 3: Normalized DTW distances across invariant transforms
4.3 EVALUATING CONFIDENCE
Measuring confidence quantifies the performance of the explainers on a per-instance case and not only in the average. As described in Sec. [32.3] we compute the change in probability assigned the ground-truth class (A conf) as well as the change in entropy of the softmax distribution (A entropy) proxies for estimating the confidence of explanations. Additionally, we report the proportions of pixels in the heatmaps to the total number of pixel averaged across the top-k dataset.
We see that for confidence, the trends mimic the ones observed in Table This implies that masking with extracted heatmaps not only increases accuracy but also results in high-confidence predictions across explainers. More specifically, we see that Grad-CAM again outperforms the other explainers (both A conf and A entropy) in the normal heatmaps by large margins. In the case of inverse masking, confidence and entropy for LIME show behaviours contrary to our expectations. This can be attributed to the "patchiness" of the explanations generated by LIME which was discussed in the previous sections.
789401 in the standard ImageNet preprocessing
preprcesinn pipeline for Inception-v3
8

---

Under review as a conference paper at ICLR 2020
Table 3: Confidence of explanation (inverse refers masking with the inverse saliency map)
LIME
Grad-CAM
Integ. Grad.
Smoothgrad
A conf
0.34
0.83
0.4
0.54
A conf inverse
0.23
0.03
-0.01
-0.01
À entropy
-0.17
-2.41
-1.04
-1.62
A entropy inverse
-0.23
0.19
0.65
0.46
avg. map size
0.22
0.62
0.47
0.57
CONCLUSIONS AND FUTURE WORK
In this paper, formulated desired properties of good explainer and proposed generic, com- putationally inexpensive suite of metrics correctness, consistency and confidence to objectively evaluate and compare explainers. We compared well-known explainers, such LIME, Grad-CAM, Integrated Gradients and SmoothGrad, on a pretrained Inception- -V model on the ImageNet dataset. Our experiments show that the metrics proposed capture various pros and cons of each explainer al- lowing users to make an informed choice about which explainer to use for their use case.
Specifically, we observe that Grad-CAM often performs better than the other explainers but suffers from trawbacks when inverse masking situations are considered. On the other hand, LIME performs poorly in all situations we consider.
Moreover, we also point out the pitfalls of trying to combine results from multiple metrics as they tend to hide anomalous behaviours of the underlying metries (as seen from Pseudo-F1 from Table We recommend that users sanity-check explainers by looking at individual metrics before making decision based on the combined metric.
Furthermore, we urge the XAI community to resist the temptation to propose an one-size-fitss metries as have shown that such metrics tend to hide nuanced trade-offs that practitioners need to be aware of.
Going forward, we invite the research community test our metrics on other explainers, datasets, underlying classifiers and data modalities. Additionally, since the metrics proposed are differen- tiable, we believe exciting new liens of research would be to develop explainers that directly op- timize for these metrics, well as self-explaining models that incorporate such metrics into their learning regiment.

---

Under review as a conference paper at ICLR 2020
REFERENCES
J. Adebayo, J. Gilmer, M. Muelly, I. J. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. NeurIPS, 2018.
David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks. NewrIPS, 2018.
Marco B. Ancona, Enea Ceolini, Cengiz Oztireli, and Markus H. Gross. Towards better understand- ing of gradient-based attribution methods for deep neural networks. ICLR, 2018.
Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus- Robert Miler, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS, 10(7), 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, Icee, 2009.
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: / 7702.08608 2017.
Logan Engstrom, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and translation suffice: Fooling cnns with simple transformations. CoRR, abs/1711.02779, 2017. URL http:tttp///ii...tt/// org abs 02779
Joel Escudé Font and Marta Costa-Jussà. Equalizing gender biases in neural machine translation with word embeddings techniques. arXiv preprint arXiv: 19X1v:/901.000 /6,06/661111061111116631 2019.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. CoRR, 2017.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. AAAI, 2019.
Leilani H. Gilpin, David Bau, Ben Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Ex- plaining explanations: An overview of interpretability of machine learning. arXiv:/806.0000 2019.
Yihong Gong and Wei Xu. Machine learning for multimedia content analysis, volume 30. Springer Science Business Media, 2007.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
David Gunning. Darpa's explainable artificial intelligence (xai) program. IUI, 2019.
Mark Hall, Daniel Harborne, Richard Tomsett, Vedran Galetic, Santiago Quintana- Amate, Alistair Nottle, and Alun Preece. A systematic method to understand requirements for explainable ai (xai) systems. IJCAI XAI Workshop. 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. CoRR, abs/1806. 10758, 2018. URL http:://////:.......mmm wrwp/ii.orrraaee wwwp:/o///////./.. /100s//16..... 10758
Ayanna Howard, Cha Zhang, and Eric Horvitz. Addressing bias in machine learning algorithms: A pilot study emotion recognition for intelligent systems. In IEEE Workshop on Advanced Robotics and Its Social Impacts (ARSO), pp. 1-7. IEFE, 2017.
Ahmed Ibrahim and Craig Valli. Image similarity using dynamic time warping of fractal features. 3th Australian Digital Forensics Conference, 2008
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aäron den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation linear time. CoRR, abs/1610.10099, 2016. URL
http://araa org hwwpgg///////// hwwp://abs///....../ 10099
10

---

Under review as a conference paper at ICLR 2020
Pieter-Jan Kindermans, Sara Hooker, and Julius Adebayo. The (un)reliability of saliency methods. arXiv:1711.00867, 2017a.
Pieter-Jan Kindermans, Kristof T. Schutt, Maximilian Alber, Klau-Robert Muller, Dumitru Erhan, Been Kim, and Sven Dahne. Learning how to explain neural networks: Patternnet and patternat- tribution. arXiv: 705.05598, 2017b.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-222) 1998.
Zachary Chase Lipton. The mythos of model interpretability. CoRR, abs/1606.0344 2016. URL http://arxiv. org/ hatp//bs//iee abs/1606. 03490
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp. 4765-4774. Curran Associates, Inc., 2017. URL http: /ttp:e//pppprsaaee ni wwws.cc/paper.......ccccppppppp. 1062-a-unified apwroeec-pppchcttooo wwt.ttoooitteeetinniee mode predictions....o .dd.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. CoRR, ftss/1805.00933 2018. URL http: /arxxv wwwrxiiv..g//e..
00932
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schtze. An introduction to information retrieval. Cambridge University Press, 2009,
Sina Mohseni, Nillofar Zarei, and Eric D. Ragan. A survey evaluation methods and measures for interpretable machine learning, arxXiv:1811.1188 2018
W. James Murdoch, Chandan Singh, Karl Kumbler, Reza Abbasi-Asl, and Bin Yu. Interpretable machine learning: Definitions, methods, and applications. arXiv:1901.0459) 2019.
Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Dosli-Velez. How do humans understand explanations from machine learning systems? an evaluation of the human- Humann-nttepreetabilee of explanation. arX2v:/802.0000 2018.
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman- tic segmentation. arXiv:1505.04366 2015.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In SIGKDD, pp. 1135-1144, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: figh-precision model-agnostic explanations. AAAI, 2018.
Andrew Ross, Michael C. Hughes, and Finale Velez. Right for the right rasons: Training differentiable models by constraining their explanations. arXiv:1703.03717,20 2017.
H. Sakoe and S. Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions Acoustics, Speech and Signal Processing. 26(1):43 49, 1978.
Stan Salvador and Philip Chan. Fastdtw: Toward accurate dynamic time warping in linear time and space. Journal Intelligent Data Analysis, 1165):561-580, 2007.
Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Muller. Evaluating the visualization of what a deep neural network has learned. IEEE Transac- fions Neurol Networks and Learning Systems, 28, 2017.

---

Under review as a conference paper at ICLR 2020
Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR, abs/1610.02391, 2016. httttt/t//aaxx http://axxxvoo../ab///
1610.0233
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad:
Removing noise by adding noise. arXiv preprint arXiv: www. 7tt.::7765..25... 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv:1412.6806 2015.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR, 01365
abs/1703.01365, 2017. URL htt.:::pp//araatooor///a///////s
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint
2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567,, 2015. URL
org/abs http:://axxv..org//
00567
Liwei Wang. Yan Zhang, and Jufu Feng. On the euclidean distance of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 2005.
Christine T. Wolf. Explainability scenarios: Towards scenario-based xai design. IU1, pp. 252-257, 2019.
Mengjiao Yang and Been Kim. Bim: Towards quantitative evaluation of interpretability methods with ground truth. arXiv preprint arXiv:190..09701, 2019.
Chih-Kuan Yeh, Cheng-Yu Yu Hsieh, Arun Sai Suggala, David Inouye, and Pradeep Ravikumar. the (in)fidelity and sensitivity for explanations. arXiv preprint 1rXiv:1901.000 2019.
Jiaping Zhao and Laurent Itti. shapedtw: shape dynamic time warping. arXiv:1606.01601, 2016.
Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn D. Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. CoRR, abs/1812.01199,, 2018. URL //axxii org/ abs/ 01593
12

---

Under review as a conference paper at ICLR 2020
A
ADDITIONAL EXPERIMENTAL RESULTS
A.1
VISUAL MASKING RESULTS
Inputs
LIME
GradCAM
Integ Grads
SmoothGrad
FG Explenation
Masked images
Inverse Masked images
FG Explanation
BG Masked images
Inverse Masked images
FG Explamation
BG Masked images
Inverse Masked images
Figure 4: Examples of the proposed algorithm for correctness
13

---

Under review as a conference paper at ICLR 2020
A.2
EFFECT OF ON CORRECTNESS FOR NORMAL AND INVERSE MASKING
Table 4: Effect on correctness with normal masking
Original
LIME
Grad-CAM
Integ.
Grad.
Smoothgrad
@
acc
acc
acc
acc
acc@5
acc
acc
ace I
acc
11.42
53.8
55.03
80.34
97.44
99.59
61.6
80.72
76.22
89.72
10
25.39
70.01
60.8
84.5
97.44
99.63
63.5
82.14
76.39
90.13
15
37.91
78.88
64.37
86.64
96.87
99.63
63.49
82.3
75.55
90.19
20
48. 18
83.9
66.95
87.92
96.08
99.59
63.44
82.35
74.68
89.83
25
56.23
87.06
69.0
89.15
94.86
99.44
62.5
81.77
73.4
89.22
Table 5: Effect on correctness with inverse masking
Original
LIME
Grad-CAM
CAM
Integ.
Grad.
Smoothgrad
@
acc
acc
acc@5
acc
acc
acc
acc 5
acc 1
acc
11.42
53.8
42.89
70.08
15.4
51.4
10.64
28.42
9.74
30.95
10
25.39
70.01
45.29
72.87
27.09
64.02
14.83
34.59
16.14
39.51
15
37.91
78,88
47.04
74.1
36.52
71,63
18.16
38,85
21.56
45.8
20
48.18
83.9
48.05
74.95
44.14
76.06
21.42
42.52
25.94
50.01
25
56.23
87.06
48.64
75.51
50.26
79.63
23.78
45.17
30.06
53.81
A.3
EFFECT OF GRADIENT MAP THRESHOLLD VALUE ON CORRECTNESS AND CONFIDENCE
Table Accuracy on the top/bottom 5 images after percentile threshold masking, using normal or inverse masking, when normal accuracy of the bottom 5 images with no masking is acc 1=11.42 and aco@5=53.8
Threshold Percentile
acc@)
@cc inverse
acc@5
acc inverse
Grad-CAM
99.48
14.00
99.92
52.89
20th Percentile
Integrated Gradient
9T.15
12.37
96.928
41.29
Smoothgrad
91.51
11.95
97.17
41.0
Grad-CAM
96.81
17.39
99.38
53.93
40th Percentile
Integrated Gradient
75,45
11.12
88.02
33.13
Smoothgrad
75.18
10.73
88.12
32.90
Grad-CAM
88.92
25.91
97.06
59.66
60th Percentile
Integrated Gradient
52.00
12.91
70.55
30.94
Smoothgrad
49.38
11.7
68.43
29.72
Grad-CAM
63.85
49.62
87.3
76.72
80th Percentile
Integrated Gradient
25.10
27.79
48.56
49.17
Smoothgrad
23.68
26.37
46.65
47.73
14

---

Under review as a conference paper at ICLR 2020
Table 7: Confidence of explanation using different thresholds
Threshold
Å conf
Å conf inverse
Å entropy
A entropy inverse
avg. map
Grad-CAM
0.869
0.018
-2.69
0.13
0.8
20th percentile
Integ. Grad.
0.737
0.007
-2.36
0.28
0.8
Smoothgrad
0.738
0.005
-2.30
0.27
0.8
Grad-CAM
0.816
0.043
-2.46
0.21
0.6
40th percentile
Integ. Grad.
0.5533
-0.004
-1.76
0.41
0.6
Smoothgrad
0.5488
-0.005
-1.81
0.39
0.6
Grad-CAM
0.693
0.102
-1.67
0.23
0.4
60th percentile
Integ. Grad. 0.321
0.004
-0.72
0.54
0.4
Smoothgrad
0.304
-0.003
-0.74
0,46
0.4
Grad-CAM
0.42
0.281
-0.42
-0.20
0.2
80th percentile
Integ. Grad.
0.09
0.I11
0.39
-0.11
0.2
Smoothgrad
0.088
0.103
0.45
-0.05
0.2
A.4
NORMALIZED DTW ON SALIENCY MAPS UNDER INVARIANT TRANSFORMS
Table 8: Normalized DTW computed on the generated saliency maps under transformations
Parameter Horizontal Flip
Parameter
LIME
Grad-CAM
Integ. Grad.
Smoothgrad
0.67981
0.1006316
0.52763
0.468309
Vertical Flip
0.7894171
0.2697139
0.62144
0.558899
= 0.2
0.789417
0.2428162
0.60073
0.546869
-0.2
0.793548
0.2452753
0.60145
0.54723
Translation
= 0.2
0,812584
0.2688328
0,68498
0.633491
y = -0.2
0.81300
0.2847767
0.6845
0.622429
- 15
0.8662869
0.2046104
0.61906
0.564211
-10-
0.844136
0.1830615
0.60026
0,545514
5"
0.7984
0.1449023
0.56954
0.515016
Rotation
5
0.806391
O. 0.1452136
0.56956
0.514659
10
0.84784
0.1827558
0.60129
0.545972
15c
0.86896
0.2038526
0.61984
0,564713
A.5 EFFECT OF USING ZERO AND GREY MASKING FOR CORRECTNESS
Table ace@ and @cc@5 obtained after using zero masking
Original
LIME
Grad-CAM
Integ. Grad.
Smoothgrad
acc @
77.21
48.3
72.64
25.03
28.62
acc @
93.53
66.86
89.69
42.95
49.03
Table 10: @ 1 and acc @5 obtained after using grey masking (i.e multiplying images with gradient maps and filling the background with the bias of the first convolutional filter in the model)
Original
LIME
-radd-CAM
Integ. Grad.
Smoothgrad
acc @
77.21
48.36
72.54
24.86
28.53
acc @5
93.53
66.88
89.59
42.72
48.81
15


Reviews:
[{"review_id": "SklqelnnKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "See post-rebuttal updates below!\n\nSummary\n---\n\n(motivation)\nThere are lots of heat map/saliency/visual explanation approaches that try to deep image classifiers more interpretable.\nIt's hard to tell which ones are good, so we need better ways of evaluating explanations.\nThis paper proposes 3 such explanation evaluation metrics, correctness, consistency, and confidence.\n\n(approach - correctness)\nAn explanation is correct if it highlights enough of an image for a classifier to tell the correct class with only the highlight parts of the image.\nThe default way to evaluate on only highlighted portions is to set the non-highlighted bckground to black/grey.\nInstead, this method finds images with the same ground truth class which the classifier scored the lowest of all such images, forming a low-confidence baseline.\nIt copies the background from one of these images instead of using a black/grey background\nto try and put the masked image back into the distribution of images from the ground truth class.\nThis style of masking is used to compute correctness.\n\n(approach - consistency)\nAn explanation is consistent if it is invariance w.r.t. a number of mostly semantically invariant transformations.\nThese include small affine transformations, horizontal flips, vertical flips, and adding noise.\n\n(approach - confidence)\nAn explanation is confident if the masked images it produces still have high condidence under the classifier.\nMasked images are produced as for correctness, by copying a distractor from the same class into the background.\n\n(experiments)\nThe experiments compare existing explanations (LIME, Grad-CAM, Integrated Gradients, SmoothGrad) using the proposed metrics.\n1. Correctness: Classifiers have higher accuracy on explanation-masked images than on images they were least confident on (the ones used to fill in the background).\n2. Grad-CAM is most correct, followed by SmoothGrad, Integrated Gradients, and LIME.\n3. Consistency: Grad-CAM explanations are most resilient to the proposed transformations with Integrated Gradients, SmoothGrad, and LIME being successively less invariant.\n4. Confidence: Explanation-masked images have higher scores for their ground truth class than the low-confidence baseline images.\n5. Hyperparameter variations in the correcness/confidence metrics mostly preserve the ranking of methods, though the absolute values of performance do change substantially.\n\n(conclusion)\nThe paper concludes that Grad-CAM is usually the best of the methods tested according to the new metrics and that LIME is the worst.\n\n\nStrengths\n---\n\nI really like the related work section. It could be a valuable resource going forward.\n\nI like the research direction of this paper very much. I think that enumerating a suite of complementary benchmarks is a good way to measure explanation quality because we can only come up with benchmarks that capture a small part of what we want so far.\n\n\nWeaknesses\n---\n\n\nI see some major conceptual flaws with these metrics:\n\n* In section 3.1 it seems like the first reasons that normal masking failed is not solved by the proposed approach. The generated images are still out of distribution because the \"foreground\" and the \"background\" don't match.\n\n* I'm concerned about the low-confidence distractor images used in the background. They are from the same ground truth class as the high confidence images they are pasted into the background of, correct? The correctness metric is supposed to capture whether or not an explanation highlights all the class-relevant content in an image and no more. However, information that the explanation did not highlight (the background) can inform the classifier of the ground truth class because the background came from an image of that class (even if a low confidence one). This is especially true because the relevant objects might be in differrent positions in the two images. Thus it could be that the explanation did not highlight informative content but the classifier still gets the corresponding masked image correct because of the background. How often does this happen?\n\n* Consistency is supposed to measure \"the ability of the explainer to capture the relevant components\" under semantically invariant transformations.\nThe reported metric is mimized when the explanation is the same before and after a variety of transformations.\nIf this were the case then at least one of them must be wrong in the sense that it would not have captured some relevant components\n(unless perhaps it just highlighted everything and was thus useless).\nBecause of the transformation (e.g. 15 degree rotation) the relevant components would have been at a different position, but the best explanation according\nto this metric would have been at the same position. Thus this metric seems to reward explanations for not capturing relevant components.\n\n\nParts I Didn't Understand:\n\n* In section 3.1, I don't understand the second reason that masking failed. In what sense is masking made meaningless? How is that sense different from the out of distribution concern from the first point?\n\n\nMissing Details / Presentation Weaknesses:\n\n* Missing reference to [1] which provides more metrics.\n\n* The meaning of confidence is different than it normally is and this may be confusing.\nNeural networks should be well calibrated, not necessarily confident (in the commonly used sense of [3]).\n\n\nMinor flaws:\n\n* Masking by replacing the background with grey (i.e., the bias of the first conv layer) rather than black is more common (e.g., [2] and Grad-CAM). A grey background negates the bias. It's not clear that the background should cancel the bias, but it would be nice to compare to both grey and black masking in Table 7.\n\n\n[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).\n[2]: Zeiler, Matthew D. and Rob Fergus. “Visualizing and Understanding Convolutional Networks.” ECCV (2013).\n[3]: Guo, Chuan et al. “On Calibration of Modern Neural Networks.” ICML (2017).\n\n\nFinal Evaluation\n---\n\nThis paper relies solely on theoretical arguments to show its metrics capture meaningful information. Empirically, it only shows that the proposed metrics can differentiate between some popular explanations. It does not empirically show that the differentiation is meaningful (e.g., by measuring agreement with human judgement). This by itself isn't a problem. However, above I detailed significant flaws in the theoretical justification for the metrics, so I can't recommend these metrics (this paper) on either a theoretical or an empirical basis.\n\nQuality: Per above, I do not think the arguments/evidence in the paper support its conclusions.\nClarity: The paper could be clearer, but can be understood without too much effort.\nOriginality: These metrics are new enough, being novel variations on prior approaches.\nSignificance: If I was convinced the metrics made sense then I would guess this paper would be very impactful. As is, I don't think it will have much impact.\n\nThe quality of the paper is my reason for the low rating. I'm interested to see whether what others think to make sure I've understood the paper correctly and analyzed it accurately. If my understanding is incorrect I could definitely raise my rating.\n\nPost-Rebuttal Evaluation\n---\nAfter reading the other reviews and the author responses and taking a brief look at the updated paper I still think this paper should be rejected.\n\nThe authors' response to my comments clarified my understanding of the consistency metric. Now I understand it and think it is a useful metric.\n\nHowever, I did not find clarification about the confidence or correctness metrics, though I agree they are not redundant. They still don't really quite make sense to me. This puts me in about the same position as R3, who also doubts those metrics. In the end, this leaves my initial evaluation essentially unchanged. I still recommend rejection because the paper relies on a theoretical understanding of what makes confidence and correctness metrics useful and that understanding is not provided.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJx1EXq0YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--------- AFTER rebuttal\n\n1) \"We identify issues with current masking procedures as proposed in other papers\"\n\nOne of the major issues with the current masking procedure is that the resulting image is out of the data distribution. Even though your method achieved high accuracy in Table 2 for correctness, the generates images is still out of the data distribution. \n\n2) \"We propose a cost-effective masking technique that doesn’t require retraining of the underlying classifier\"\n\nThe authors compared against zero and gray masking for correctness. None of those masking methods require retraining of the underlying classifier. It is not clear, which previous masking technique required retraining of the underlying classifier?\n\n3) We also further show that when performing a comprehensive evaluation, there is no one clearly better explainer and thus practitioners need to be careful about which explainer they choose.\n\nThis is an observation made upon through exploratory analysis and is not a technical novelty.\n\n4) \" Confidence on the other hand, also informs us about the per-instance behavior.\"\n\nThe confidence measures the change in probability assigned to the ground truth class. Table 3 should also show the variance in the confidence to understand the instance-level behavior.\n\nThe experiments given in the paper, it looks like confidence and correctness are positively correlated. An example of the model where they are not positively correlated will help the reader understand the importance of each of these terms.\n\n5) \" Effect of Thresholding on results\"\nThank you for the explanation and new experimental results.\n\n\n\n\n------------------------- BEFORE rebuttal\nThe paper proposed different metrics for comparing explainers based on their correctness (ability to find most relevant features in an input, used in prediction), consistency (ability to capture the relevant components while input is transformed), and the confidence of the generated explanations. To evaluate correctness, the authors proposed to study the change in the classification accuracy of the target model, under a perturbed dataset where the most relevant regions (as given by explainer) of the image is preserved and the remaining content is replaced with non-informative backgrounds for the target class. For consistency evaluation, the authors proposed to apply transformations like rotation, translation and flip that doesn’t semantically change the input image. For confidence evaluation, they compared the prediction performance on the original image, masked image (only salient regions) and inverted masked image (only non-salient regions). \n\nMajor\n•\tThe paper lack technical novelty.\n•\tThe confidence component looks redundant and can be incorporated in the correctness component.\n•\tThe inverse saliency map idea is already proposed in “Evaluating the visualization of what a deep neural network has learned” for evaluating saliency maps. There the authors gradually replace the most salient regions with random noise and observe a decrease in prediction accuracy.\n•\tMost of the saliency maps producing methods, generate continuous maps. For making, we need to convert the continuous map to binary by using a threshold. An analysis of choosing different values as threshold is missing. By choosing an appropriate threshold, the size of the most salient region can be controlled. Thus, although Grad-Cam spread saliency over large area, we can use a higher threshold to define the binary mask.\n•\tGrad-CAM, integrated grad and smooth grad are all gradient-based saliency maps. There are perturbation-based saliency maps, which aims to find most salient regions such that removing those regions produce a maximum drop in prediction accuracy. Example “Interpretable Explanations of Black Boxes by Meaningful Perturbation.”, “Object detectors emerge in deep scene cnns” .  An evaluation of such methods is missing.\n\nMinor:\n•\tThe text in the figures has very small font size and is not readable.\n•\t\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJxYPItUYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the interesting question of comparing the deep network visualization algorithms quantitatively. Several metrics are proposed, including correctness, consistency and confidence. \n\nI like the notion of consistency, where an explainer should produce the same explanation under transformations of the image that does not change its “semantic content”. \n\nHowever, I am confused or unconvinced by several arguments made in the paper, and if the authors can clarify them I am willing to increase my review. I think the major issue is that most metrics are justified with flimsy arguments, not compared with prior work, and do not lead to consistent ranking of the models. \n\nCorrectness: I am not convinced by the correctness evaluation for several reasons\n\n1. The combined image is still out of distribution, and it is unclear why this is better compared to e.g. using a white background. \n2. Does it favor visualization methods with a blob-shaped saliency map vs. scattered dots shaped saliency map? Does it favor methods with a larger salient region? For example, just from visual appeal, I do not think smoothgrad is worse than gradCAM, but the number says otherwise. I think the arbitrariness of this metric makes the numbers hard to believe. \n3. If the original image is already incorrectly classified (since they are the ones where the classifier assigns the lowest probability) it is hard to imagine that adding random background can make the performance worse.\n\nTherefore, it is also unclear what to make of the numbers in e.g. Table 2. There are so many metrics, precision, recall, F1, and none of them seem particularly well justified. They also do not rank the model in the same way. Which result should a practitioner believe? \n\nConfidence: I am not sure the confidence vs. number of pixels comparison are useful. Across all methods, it seems to be more pixels -> increased confidence, which is unsurprising. I think the results are only useful if one method pareto dominates another, which is not what is observed in the experiments. \n\nI do not understand the difference between confidence and correctness. It seems like both measure how well a model can predict the correct class given only the salient region. For example, if method A has higher confidence and lower correctness compared to method B, what does that mean? Under which situation should one choose method A over method B? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes metrics for comparing explainability metrics.

Both reviewers and authors have engaged in a thorough discussion of the paper and feedback. The reviewers, although appreciating aspects of the paper, all see major issues with the paper. 

All reviewers recommend reject.  
Decision: Reject