Aggregating explanation methods for neural networks stabilizes explanations
Authors: Laura Rieger
Lars Kai Hansen
Venue:  (2020)
Keywords: explainability
deep learning
interpretability
XAI

Abstract: 	Despite a growing literature on explaining neural networks, no consensus has been reached on how to explain a neural network decision or how to evaluate an explanation.
	Our contributions in this paper are twofold. First, we investigate schemes to combine explanation methods and reduce model uncertainty to obtain a single aggregated explanation. The aggregation is more robust and aligns better with the neural network than any single explanation method..
	Second, we propose a new approach to evaluating explanation methods that circumvents the need for manual evaluation and is not reliant on the alignment of neural networks and humans decision processes. 

Under review as a conference paper at ICLR 2020
AGGREGATING EXPLANATION METHODS FOR NEURAL NETWORKS STABILIZES EXPLANATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
Despite a growing literature explaining neural networks, no consensus has been reached on how to explain a neural network decision or how to evaluate an explanation. Our contributions in this paper are twofold. First, we investigate schemes to combine explanation methods and reduce model uncertainty to obtain a single aggregated explanation. The aggregation is more robust and aligns better with the neural network than any single explanation method. Secondly, we propose new approach to evaluating explanation methods that circumvents the need for manual evaluation and not reliant on the alignment of neural networks and humans decision processes.
INTRODUCTION
Despite the great success of neural networks especially in classic visual recognition problems, explaining the networks' decisions remains an open research problem Samek et al. (2019). This is due in part to the complexity of the visual recognition problem and in part to the basic 'ill-posedness' of the explanation task. This challenge is amplified by the fact that there is no agreement what sufficient explanation is and how to evaluate an explanation method.
Many different explanation strategies and methods have been proposed (Simonyan et al. 2013 Zeiler & Fergus 2014] Bach et al. 2015 Selvaraju et al. 2017; Smilkov et al.) 2017: Sundararajan 2017). Focusing visual explanations for individual decisions, most methods either backpropagation approach or aim to construct a simpler linear model with an intuitive explanation. The plethora of explanation approaches a signature of the high-level epistemic uncertainty of the explanation task.
This paper is motivated by key insight in machine learning: Ensemble models reduce both bias and variance compared to applying single model. A related approach was pursued for functional visualization in neuroimaging (Hansen et al. 2001). Here for the first time explore the potential of aggregating explanations of individual visual decisions in reducing epistemic uncertainty for neural networks.
We test the hypothesis that ensembles of multiple explanation methods are more robust than any single method. This idea is analyzed theoretically and evaluated empirically. We discuss the properties of the aggregate explanations and provide visual evidence that they combine features, hence are more complete and less biased than individual schemes. Based on this insight, we propose two ways to aggregate explanation methods, AGG-Mean and AGG-Va Var. In experiments on Imagenet, MNIST, and FashionMNiST, the aggregates identify relevant parts of the image more accurately than any single method.
Second, we introduce IROF (Iterative Removal Of Features) as a new approach to quantitatively evaluate explanation methods without relying human evaluation. We eircumvent the problems of high correlation between neighbor pixels as well as the human bias that are present in current evaluation methods.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
2.1 EXPLANATION METHODS
The open problem of explainability is reflected in a lot of recent work KKindermans et al.
2017 7
Selvaraju et al.] 2017 Bach et al. [015 Zhang et al. [0018] Zhou et al., 2016
Ancona et al.
2018
Ribeiro et al.] 2016;; Rieger et al. 2018: Kim et al. 2018 Lundberg & Lee
2017: Zintgral et al.
2017; Simonyan an et al. 2013] Zeiler & Fergus| 2014 Selvaraju et al. 2017
Smilkov et al. 2017
Sundararajan et al. 2017; Shrikumar et al. 201 Montavon et al.) 2017; | et |0.8 We
focus on generating visual explanations for single samples. The first work in this direction was Simonyan et al. (2013) with Saliency Maps (SM) that proposed backpropagating the output onto the input to gain an understanding of a neural network decision. The relevance for each input dimension is extracted by taking the gradient of the output w. t. to the input. This idea was extended by Springenberg et al. (2014) into Guided Backpropagation (GM) by applying ReLU non-linearities after each layer during the backpropagation. Compared Saliency, this removes visual noise in the explanation. Grad-CAM (GC) from Selvaraju et al. (017)) is an explanation method, developed for use with convolutional neural networks. By backpropagating relevance through the dense layers and up-sampling the evidence for the convolutional part of the network, the method obtains coarse heatmaps that highlight relevant parts of the input image. Integrated Gradients (IG) |Sundararajan et al. (2017) sums up the gradients from linearly interpolated pictures between a baseline, e.g. a black image, and the actual image. SmoothGrad (SG) filters out noise from basic saliency map by creating many samples of the original input with Gaussian noise Smilkov et al. (017)) The final saliency map is the average over all samples.
Finally, also consider LIME Ribeiro al. (2016). In contrast to the other methods, LIME is not based on backpropagation. Instead, it approximates the neural network with a linear model locally around the input to be explained. The coefficients of the linear model for the respective input dimensions give the importance of each dimension. Compared to the other methods this is much more computationally expensive it requires many passes through the neural network.
2.2 EVALUATION OF EXPLANATION METHODS
The evaluation of explanation methods is a relatively recent topic with few systematic approaches (Bau et al.] 2017; Ancona et al. 2018 Hooker et al., al. 2018 AAdebayo et al., 2018 Fong & Vedaldi 2017]. To our know phowledge,, |Bach et al. (2015) proposed the first quantitative approach to evaluate an explanation method by flipping pixels to their opposite and comparing the decrease in output with the relevance attributed to the pixel by the explanation method. As the authors note, this only works for www-dimensional input. This work was followed up upon in Samek et al. (0016). By dividing high-dimensional images into squares, they make the method feasible for high- gigh-dimensional inputs. Squares with high relevance (as measured by the explanation method) consecutively get replaced with noise sampled from the uniform distribution. The difference between the original output and the output for the degraded images indicates the quality of the explanation method.
Hooker et al. (2018) proposes another quantitative approach to evaluate explanation methods called ROAR. For each explanation method they extract the relevance maps over the entire training set. They degrade the training set by setting different percentages of the pixels with the highest relevance to the mean and retrain the network. Each retrained network is evaluated on the test set. The accuracy on the test set decreases dependent on the percentage of pixels set to the mean. This requires retraining the same architecture multiple times for each explanation method at high computational cost.
Ancona et al. 2018)) proposed different approach to evaluate explanation methods, called Sensitivity- n. based on the notion that the decrease in output when a number of inputs are canceled out should be equal to the sum of their relevances.
For range of n (between and the total number of inputs) they sample hundred subsets of the input. For each n, the Pearson Correlation Coefficient (PCC) between the decrease in output, when the subset of features is removed, and the sum of their relevances is reported. The result is curve of the PCC dependent on the percentage of the input being removed. For a good explanation method, the PCC will decrease slowly.
2

---

Under review as a conference paper at ICLR 2020
3 METHODS
3.1
AGGREGATING EXPLANATION METHODS TO REDUCE VARIANCE
All currently available explanation methods have weaknesses that are inherent to the approach and include significant uncertainty in the resulting heatmap (Kindermans et al., 2017 Adebayo et al. 2018: Smilkov et al. 2017). A natural way to mitigate this issue and reduce noise to combine multiple explanation methods. Ensemble methods have been used for long time to reduce the variance and bias of machine learning models. We apply the same idea to explanation methods and build an ensemble of explanation methods.
We assume neural network {e,)/=1 with Ej y, F + E with E E _=xm We write Ejnn for the explanation obtained for Xx with method e, and denote the mean aggregate explanation as E with E, = Σx= E While we assume the imput to be image Rm this method is generalizable inputs of other dimensionalities as well.
X + y with X E mR and set explanation methods
To get theoretical understanding of the benefit of aggregation, hypothesize the existence of 'true' explanation É,. This allows to quantify the error of explanation method as the mean squared difference between the 'true' explanation and an explanation procured by an explanation method, i.e. the MSE.
For clarity we subsequently omit the notation for the neural network. We write the error of explanation
method on image X₂, as emr = |EEnn - Ex
11 with
MSE(E,) Σ erry.n
and MSE(E) Σn E, E1 is the MSE of the aggregate. The typical error of an explanation method is the mean error over all explanation methods
1 MSE J Σ MSE(E,).
With these definitions we can do standard bias-variance decomposition (Geman et al. 1992). Accordingly we can show error of the aggregate will be less that the typical error of explanation methods,
MSE
= E.
E Ez,nl?
# En 1 En11 + N Em. ||En - E2
hence,
MSE
+MSE(E) 2 MSE(E).
J
3
epistemic unzertainty
detailed calculation given in appendix A.1 The error of the aggregate MSE(E) () is less than the typical error of the participating methods. The difference 'variance' term represents the epistemic uncertainty and only vanishes if all methods produce identical maps. By taking the average over all available explanation methods, reduce the variance of the explanation compared using a single method. To obtain this average, we normalize all input heatmaps such that the relevance over all pixels sum up one. This reflects our initial assumption that all individual explanation methods are equally good estimators. We refer to this approach as AGG-Mean.
EAgg. Mean.n
Σ E j
1
This estimator however does not take into account the estimate of the local epistemic uncertainty, Lee. the disagreement between methods. A way incorporate this information is to form 'effect
3

---

Under review as a conference paper at ICLR 2020
size' map by dividing the mean aggregate locally with its standard deviation (iiuurdsson et al. 2004). Intuitively, this will assign less relevance to segments with high disagreement between methods.
For stability, divide not directly by the local variance but add constant to the estimate of the local variance. This can be interpreted as a smoothing regularizer or a priori information regarding epistemic and aleatoric uncertainties. We refer to this approach as AGG-Var
EAGG Var, J Σ
Eij.n
j=1 (BBeJn +
where a (EjeJ,n) the point-wise standard deviation over all explanations j E J for X,,
In section we will evaluate and compare AGG-Mean and AGG-Var Var against basic explanation methods.
3.2 EVALUATING EXPLANATION METHODS QUANTITATIVELY WITH IROF
overlaid
by imaaa Segmencaiion
relevant seg regment
with segment replaced
Repeat consecutively
coucal
dragontry
ruddy tumstonn
Figure 1: Quantitative evaluation with IROF: Relevant segments identified by explanation method get consecutively replaced by the mean colour over the entire dataset. The IROF score of explanation method is the integrated decrease in the class score the number of removed segments.
Quantitative evaluation recurring problem with explainability methods. This especially true for high-diman sigh-dimensional input, such as images, where important features are made up of locally highly correlated pixels. If the information in one pixel is lost, this will not change the overall feature and should therefore not result in changed output score. The relevance values of single pixels are not indicative of the feature's importance as a whole. We circumvent this problem by utilizing conventional image segmentation, well-explored area. By first dividing the image into coherent segments, we avoid the interdependency between the inputs.
Methodology We assume neural network F X y with X Rmxm and of explanation methods {e with X. E with E E Rmxx.
Furthermore we partition each image X,, into set of segments (8)) using given segmentation method with " 1 indicating that pixel Tnii. belongs to segment Computing the mean || ll of each segment according a given explanation method j. two segments can importance [|s] I
be directly compared against each other. By sorting the segments in decreasing order of importance according to the explanation method, get ranking of how relevant each segment of the image is.
4

---

Under review as a conference paper at ICLR 2020
We use X to indicate X, with the I segments with highest mean relevance replaced with the mean value. Taking F(X)), repeatedly with increasing / E 0, .... I. results in curve of the class score dependent on how many segments of the image are removed. Dividing this curve by F(X,) y normalizes the scores to be within [0, 1] and makes curves comparable between input samples and networks.
If an explanation method works well, it will attribute high relevance to segments important for classification. As segments with high relevance are removed first, the score for the target class will go down faster. By computing the area over the curve (AOC) for the class score curve and averaging over number of input samples, we can identify the method that identifies relevant areas of the image more reliably. For good explanation method, the AOC will be higher. We refer this evaluation method as the iterative removal of features (IROF). The IROF for given explanation method ej expressed as:
N Σ AOC 1=1
F(X))) & F(X.))
IROF(e,)
1=0
This approach is a quantitative comparison of two or more explainability methods that does not rely human evaluation alignment between human and neural network reasoning. For each explanation method the workflow produces a single value, enabling convenient comparison between two or more explanation methods. If the AOC is higher, the explanation method captures more information about the neural network classification.
IROF dependent on having meaningful segments in the input, as natural images do. Dividing up text or non- non-natural images such as EEG into meaningful and independent segments does not have natural solution and is left for future research.
4 EXPERIMENTS
We first present empirical validation of our proposed evaluation technique IROF in section 4.2 Subsequently we evaluate the aggregation of explanation techniques against the vanilla techniques with IROF, Sensitivity-n and qualitative evaluation. In appendix A.6.1 we compare aggregated methods on a dataset human-annotated heatmaps.
4.1 EXPERIMENTAL DETAILS
We tested our method on five neural network architectures that were pre-trained on ImageNet: VGG19, Xception. Inception. ResNet50 and ResNet101 (Deng al. 2009: Simonyan & Zisserman 2014 He ctt al. 2016 Chollet 2017 Szegedy et al. 2016 Additionally, we ran experiments on CNN trained
on the MNIST and MASSI dataset LeCun un & Cortes (2100) Xiao et al. (2017).
We compared the aggregation methods against Saliency (SM), Guided Backpropagation (GB), SmoothGrad (SG), Grad-CAM (GC) and Integrated Gradients (IG) to have selection of attribution- based methods. Additionally we compared against LIME as a method that is not based on attribution but rather on local approximation Ribeiro al. (2016). The aggregations are based all attribution- based methods.
Some of the methods result in positive and negative evidence. We only considered positive evidence for the ImageNet tasks to compare methods against each other. To check that this does not corrupt the methods, we compared the methods that do contain negative results against their filtered version and found negligible difference between the two versions of method in the used metrics.
For Agg- Mean we introduced an additional parameter, to the divisor. our experiments we set to be ten times the mean o over the entire dataset.
Models retrieved from http:://gghhuuu..ooo/ccaaaaaaaaa
5

---

Under review as a conference paper at ICLR 2020
4.2 EVALUATING IROF FOR VALIDITY AS AN EVALUATION METHOD
A good evaluation method should be able to reject the null hypothesis (a given explanation method is no better than random choice) with high confidence. We use this as motivation to evaluate and compare IROF by calculating the paired t-test of an explanation method versus random guessing. This is done with multiple explanation methods and networks, reduce the impact of the explanation method.
We compare IROF and pixel removal with mean value and black as a replacement value respectively. Additionally we compare against [Samek against[Samek et al. (2016) as explained in section 2 For IROF and [aame et al.] (2016) we set the 10% most relevant segments to the mean value over the dataset. For pixel removal, we set the equivalent number of pixels to the mean value. The percentage of segments or pixels being removed was chosen ad hoc. the difference in degradation between random choice and the explanation method is high, the explanation method reports meaningful information. Since compare the same explanation method on the same neural network with different evaluation methods, the p-values only contain information about how meaningful the evaluation method is.
We computed IROF and pixel removal with black or mean replacement values and compared the p-value changes dependent on the number of samples. Results are shown in fig. 2 (extended in appendix A.6). In table we provide results for forty images in tabular form for an easier overview (other methods in appendix (. On forty images, all evaluation methods produce p-values below 0.05. Thus, all evaluation methods can distinguish between random guessing and an explanation method.
However, IROF can reject the null hypothesis (the explanation method does not contain any informa- tion), with much higher confidence with the same number of samples for any configuration. We can conclude that IROF is more sensitive to the explanation method than pixel removal or (aaee et al. (2016). making it the better choice to quantitatively evaluate an explanation method.
VGG19| Saliency
VG619 GradCAM
VGG19 | Guidedllackprpp
10a
10
10-3
10-3
10-11
10
10
10-41
10
-9
10-31
10-90
10-42
10-41
45
10
Evatuation method
IROF (Mean)
10-48
10-90
Pizel removal (Mean)
10-41
(Black)
10
10-91
10-11
Pixel removal (Black)
10
10
Samek (2016)
10-07
0
100
200
300
400
500
600
700
100
200
300
400
500
600
700
0
100
200
300
400
300
600
700
samples
ssamples
# ssamples
Figure 2: P-values for the rejection of the random removal null- null-hypothesis. Lower is better. P-values are on a logarithmic scale. IROF performs best in all scenarios.
Table 1: t-test: p-values of Random choice vs Saliency Mapping on forty images. All p-values < 0.05.
EVALUATION METHOD
T-STAT
P-VAL
IROF (BLACK)
1.58
1.22E-01
IROF (MEAN)
5.44
3.60E-06
PIXEL FLIPPING (BLACK)
1.92
6.22E-02
PIXEL FLIPPING (MRAN)
2.10
4.25E-02
SAMEK (2016)
2.77
8.65R-03
4.3 EVALUATING EXPLANATION METHODS WITH IROF
To quantitatively compare the quality of the explanation methods on a more challenging dataset, we use IROF number of neural network architectures trained on ImageNet. In table 2 we report

---

Under review as a conference paper at ICLR 2020
Table 2: IROF scores methods and architectures. AGG-Mean and AGG- Var surpass all methods in all scenarios, All std < 0.05.
METHOD
INCEPTION
RESNETI01
RESNET5O
VGG19
XCEPTION
RANDOM
60.3
61.0
63.5
68.4
50.1
SOBEL
71.5
74.5
75.6
79.6
68.3
LIME
74.0
67.6
66.7
73.4
74.0
SM
76.2
75.6
76.2
81.4
72.6
GB
73.7
77.6
80.9
84.3
74.8
IG
76.0
76.0
76.2
79.8
73.3
SG
77.9
77.7
77.5
83.9
75.0
GC
78.7
77.8
78.5
73.5
AGG-MEAN
79.9
81.1
80.9
86.6
76.7
AGG-VAR
80.4
81.2
81.0
86.7
76.7
We include non-informative baselines. Random randomly chooses segments to remove. Sobel is a sobel edge detector. Neither of them contain information about the neural network.
the IROF described in section 3.2
All explanation methods have lower IROF than the random baseline on all architectures tested, indicating that all methods contain information about the image classification. Except for LIME, all methods also surpass the stronger baseline, SOBEL. The ranking of unaggregated methods varies con- siderably between architectures. This variance indicates that the accuracy of explanation method depends on the complexity and structure of the neural network architecture. For all architectures AGG-Mean and AGG- Var have lower IROF score than any non-aggregated method. For ResNet101 the difference between the best unaggregated method and the aggregated methods is especially large. We hypothesize, that the benefit of aggregating explanation methods increases for more complex neural network with large epistemic uncertainty the explanation.
We can empirically confirm that aggregating methods improves over unaggregated methods and more reliably identifies parts of the images that relevant for classification.
4.4 QUALITATIVE VISUAL EVALUATION
We show heatmaps for individual examination for each of the methods in fig. 3 and compare qualitatively (large version of fig. 3 in (pp..dii(A.6.2).. While visual evaluation explanations for neural networks can be misleading, there no better way available of checking whether any given explanation method agrees with intuitive human understanding (Adebayo et al. 2018). Additionally, compute alignment between the human benchmark for evaluation introduced in Mohseni & Ragan (2018).
images and the explanation methods in ?? using
Aggregated methods
Steffrrchire
Erriunity
Oystercaatower
Figure Example images from Imagenet and the heatmaps produced by different methods on VGG19, Aggregated methods combine features from all methods. Too heavy focus on one feature by SmoothGrad is smoothed away by aggregation.

---

Under review as a conference paper at ICLR 2020
AGG- Var combines features of the aggregrated methods by attributing relevance the classified object as a whole, but considering smaller details such as the face of an animal as more relevant. It is a combination of the detail- and shape-oriented methods. Compared to SmoothGrad, which concentrates on one isolated feature, the relevance more evenly distributed and aligned with human intuition that classification takes context into account and does not rely e.g. only the beak of a bird. We can conclude that combining explainability methods provides a meaningful visual improvement over single methods.
4.5 EVALUATION WITH SENSITIVITY-N ON LOW-DIMENSIONAN INPUT
To quantitatively compare explanation methods on a low-dimensional input we use Sensitivity-n (Ancona et al., 2018). The exact procedure described in section 2 We compare on MNIST (LeCun
& Cortes 20TO) and FashionMNIST (Xiao et al. 2017). two low-dimensional dataset with a basic CNN (architecture in appendix). We follow the procedure suggested in Ancona et al. (2018) and test on hundred randomly sampled subsets for 1000 randomly sampled test images. The number of pixels in the set n is chosen fifteen points logarithmically spaced between and 780 pixels.
As described in section 2 for range of (between and the total number of inputs) a hundred subsets of the input features are removed. For each n, the average Pearson Correlation Coefficient (PCC) between the decrease in output and the relevance of the removed output features is reported. The result is a curve the PCC dependent on the removed percentage.
We show results in fig. AGG-Mean and AGG-Var perform in range of the best methods. For the CNN trained on FashionMNIST,, AGG-Mean and AGG- Var perform better than unaggregated methods. For the CNN trained on MNIST, Guided Backpropagation and AGG-Mean perform best. For both networks (trained on MNSH and MNIST respectively), SmoothGrad and GradCAM perform considerably worse than the other methods.
In summary, aggregation seems to not be as beneficial when applied to a low-dimensional, "easier" tasks such MNIST is for ImageNet, performing range of the best unaggregated method. We hypothesize that this is because there is less epistemic uncertainty in explanations for less complex tasks and network architectures.
MNIST
FashionMNIST
1.0
Method
0.6
GB
GC
Agg Mean
02
Agg-Var
00
10a
10a
10a
10a
pixels
pixels
Figure 4: Sensitivity-n for explanation methods. Higher is better. The proposed methods, AGG-Mean and AGG- Var perform better or equally good as all other methods.
5 CONCLUSION
In this work we gave simple proof that aggregating explanation methods will perform at least as good as the typical individual method. In practice, we found evidence that aggregating methods outperforms any single method. We found this evidence substantiated across quantitative metrics. While our results show that different vanilla explanation methods perform best on different network architectures, an aggregation supersedes all them on any given architecture.
Model and code retrieved from htw.h//ww..ccccooeeeee/eeeeeeeee
8

---

Under review as a conference paper at ICLR 2020
Additionally proposed novel way of evaluation for explanation methods that circumvents the problem of high correlation between pixels and does not rely on visual inspection by humans, an inherently misleading metric.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, (4(11):2274 22227722222, 2012.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow. Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9525-9536, 2018.
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In 6th International Conference on Learning Representations (ICLR 2018), 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by fayer-wise relevance propagation. PloS one, 10(7):e0130100, 2015.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3319-3327. IEEE, 2017.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image
classifiers by counterfactual generation. 2018.
Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1800-1807. IEEE, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009, CVPR 2009. IEEE Conference on, pp. 248-255 leee, 2009.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference Computer Vision, pp. 24297437, 2017.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
Lars Kai Hansen, Finn Arup Nielsen. Stephen C Strother, and Nicholas Lange. Consensus inference in neuroimaging. Neurolmage, 1216)11112-1218, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. arXiv preprint arXiv: 10758, 2018.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp. 2673-2682, 2018.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Google Brain, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un)reliability of saliency methods. In Proceedings Workshop on Interpreting. Explaining and Visualizing Deep Learning (at NIPS), 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.
Sina Mohseni and Eric Ragan. A human- evaluation benchmark for local explanations of machine learning. arXiv preprint 1801 2018.
10

---

Under review as a conference paper at ICLR 2020
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robes Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211-2241 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144. ACM, 2016.
Laura Rieger, Pattarawat Chormai, Grégoire Montavon, Lars Kai Hansen, and Klaus-Robert Müller. Structuring Neural Networks for More Explainable Predictions. pp. 115-131. Springer, Cham, 2018.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions neural networks and learning systems,
2016.
Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Muller. Explainable Al: Interpreting. Explaining and Visualizing Deep Learning. Springer Nature, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee based localiza- tion. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626. IEEE, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145- 3153, 2017.
Sigurdur Sigurdsson, Peter Alshede Philipsen, Lars Kai Hansen, Jan Larsen, Monika Gniadecka, and HanssChhistinn Wulf. Detection of skin cancer by classification of raman spectra. IEEE transactions on biomedical engineering, 51(10):1784-1793 2004.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1405.156,, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. dec 2013. http://///aaxxi
org/abs 1112. 6034
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. 06 2017.
JT Springenberg, A Dosovitskiy, T Brox, and M Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, pp. 3319-3328, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Andrea Vedaldi and Stefano Soatto. Quick shift and kernel methods for mode seeking. Lecture Notes in Computer Science (including subseriey Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 5305 LNCS, pp. 705-718, Berlin, Heidelberg, oct 2008. Springer Berlin Heidelberg, ISBN 3540886923. doi: 10. 111007/978-3-5404 88693-8-52.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Bishion-MNISTT Novel Image Dataset for Bench- marking Machine Learning Algorithms. aug 2017. URL atttp:://arxta org/ abs / 1708
77747
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 122.551,, 2012.
11

---

Under review as a conference paper at ICLR 2020
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pp. 318-833. Springer, 2014.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827-8836, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2921-2929, 2016.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In ICLR, pp. 3, 2017.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 AGGREGATING EXPLANATION METHODS TO REDUCE VARIANCE DETAILED DERIVATION
All currently available explanation methods have weaknesses that inherent to the approach and include significant noise in the heatmap Kindermans et al., 2017] Adebayo et al., 2018: Smilkoy al. (017). A natural way to mitigate this issue and reduce noise to combine multiple explanation methods. Ensemble methods have been used for long time to reduce the variance and bias of machine learning models. We apply the same idea explanation methods and build ensemble of explanation methods.
We assume a neural network F X + y with X R.aa and set of explanation methods {ey) with ej x, F E' with E E Ruzm. We write Σz,, for the explanation obtained for Xn, with method Ej and denote the mean aggregate explanation as with E,, J Σラ= Ej While assume the input to be an image E Rmmm this method is generalizable to inputs of other dimensions as well.
We define the error of explanation method as the mean squared difference between hypothetical 'true' explanation and an explanation procured by the explanation method, i.e. the MSE. For this definition we assume the existence of the hypothetical 'true' explanation Ey for image X,,
For clarity subsequently omit the notation for the neural network.
We write the error of explanation method j on image Xn as errj.n ||Ej, En. with
MSE(E,) = N
1
Σ errjm
and MSE(E) Σn E,, - 11 is the MSE of the aggregate. The typical error of an explanation method is represented by the mean
MSE
N Σ J
Σ
En -Ejnnl?
12
Σ
|| |
Ejn + En - E22
NJ
1 NJ
Σ
(É - En) + +En - -Ejn)l
ny
1 NJ
Σ C1 - Em + +1 - E + 2Tr [EE, En) - Egn))
1 N
1
1 Σ
( EsSn
Σ
|,n E1
Σ1l,, - 12 +
Th
(Ên En)
NJ
N
n.j
1 N
EE 1.
1 Σ1E, E3
+2
1
Σ
Tr
(Én En) j > (En Ey,n)
1
NJ
N
0
1 N
Σ
1in
En
Σ
|| Ejal1?
NJ
12
n.
hence,
MSE MSB(F)
E1E - Eg 2 MSE(F)
NJ nJ
epistemic uncertainty
The error of the aggregate MSE(E) is less than the typical error of the participating methods. The difference a 'variance' term represents the epistemic uncertainty and only vanishes if all methods produce identical maps.
13

---

Under review as a conference paper at ICLR 2020
A.2 COMPARING AGGREGATE OF TWO METHODS
In section 3.1 we showed theoretically that the average MSE of two more explanation methods will always higher than the error of the averaged those methods. Empirically, we test this for IROF with combinations of any two methods for ResNet101 and show the results in fig. 5 For any two methods, the matrix shows the ratio between the aggregate method IROF and the average IROF of the aggregated methods. The aggregate IROF is always lower, confirming theoretical results.
X6
0.9
0.91
0.91
0.97
0.98
8
0.9
1
0.92
0.95
0.89
8
0.91
0.9
0.95
0.9
0.94
8
0.97
0.89
0.9
0.89
1
0.90
SM
G8
5G
GC
1G
Figure 5: Ratios between the aggregate IROF and the averaged IROF of two methods. Aggregation always improves on the results, as all values outside the diagonal are below one.
A.3 IROF EXTENDED
A.4 CHOOSING EXPLANATION METHODS
An important part of method the choice of explanation methods to be included in the aggregation. In our work we focus on backpropagation-bant methods, since they tend to be computationally cheap. In opinion this makes for more realistic case, not only for human but also for machine post processing.
In contrast, locality based methods such LIME the method by Fong & Vedaldi (2017) require many forward passes, since their methods essentially "learn" what parts of the input are relevant. We included LIME in our experiments have bakoappppoootttiionn based method included.
A.5
EXPERIMENTAL SETUP
A.5.1 GENERAL
We SLIC for image segmentation due to availability and quick run time (Achanta et al. 2012). Preliminary experiments with Quickshift showed similar results (Vedaldi & Soatto| 2008). SLIC was chosen over Quickshift due the quicker time. The number of segments was set to 300 ad hoc. fig. shows the same procedure with 100 segments for the sake of clarity.
For AGG-Var, we add a constant to the denominator. We set this constant 10 times the mean std, value chosen empirically after trying values in the range of [1, 10, 100) times the mean.
Evaluations were run with a random seed for reproducibility. Stddev were reported either for each individual result or if they were non-significant in the caption avoid cluttering the results.
Since our work does not include computationally heavy training, we did not record the exact computing infrastructure.
A.5.2 MNISTS
The training for both models equivalent. The architecture as follows: (input)-(convv32,333) (coo,,,3)) (maxPool(2.2)) -fully connected( (drrrroooot.5)) (10)
14

---

Under review as a conference paper at ICLR 2020
ReLU was used as non-linearity for both. All networks were trained with Adadelta and early stopping on the validation set (patience of three epochs) Zeiler (2012). The final accuracy for MNIST was 99.21%.: The final accuracy on FashionMNIST was 92.46%.
A.5.3 IMAGENET
We tested our method on five network architectures that were pre-trained on ImageNet: VGG19, Xception, Inception, ResNet50 and ResNet101 (Deng et al. 2009 Simonyan & Zisserman 2014 He et al. 2016 Chollet 2017 Szegedy et al.
601 We used the pre-trained networks VGG19,
Xception and Inception, obtained from the keras library and did not change the networks in any way.
(Deng al. 2009 Szegedy et al. 2016 Chollet 2017: Simonyan & Zisserman 2014).
We downloaded the data from the ImageNet Large Scale Visual Recognition Challenge website and used the validation only. No images were excluded. The images preprocessed to be within [-1,1] 1] unless a custom range was used for training (indicated by the preprocess function of keras).
15

---

Under review as a conference paper at ICLR 2020
A.6 EVALUATING THE EVALUATION
We report p-values for evaluating with 50 images on ResNet101 in the manner described in section 4.2 in tabular form provide clear overview.
Table 3: 1-test p-values of explanation methods for SmoothGrad.
EVALUATION METHOD
T STATISTIC
P-VALUE
IROF (BLACK)
3.97
3.22E-04
IROF (MEAN)
5.99
6.42E-07
PIXEL FLIPPING (BLACK)
0.55
5.86E-01
PIXEL FLIPPING (MEAN)
5.00
1.39E-05
SAMEK (2016)
2.97
5.17E-03
Table 4: t-test p-values of explanation methods for Guided backpropagation.
EVALUATION METHOD
T STATISTIC
P-VALUE
IROF (BLACK)
3.79
5.42E-04
IROF (MEAN)
5.64
1.93E-06
PIXEL FLIPPING (BLACK)
1.16
2.52E-01
PIXEL FLIPPING (MEAN)
0.49
6.30E-01
SAMEK (2016)
3.02
4.54E-03
Table 5: t-test p-values of explanation methods for Saliency.
EVALUATION METHOD
T-STAT
P-VAL
IROF (BLACK)
1.58
1.22E-01
IROF (MEAN)
5.44
3.60E-06
PIXEL FLIPPING (BLACK)
1.92
6.22n-02
PIXEL FLIPPING (MEAN)
2.10
4.25m-02
SAMEK (2016)
2.77
8.65E-03
Table t-test p-values of explanation methods for GradCAM.
EVALUATION METHOD
T STATISTIC
P-VALUE
IROF (BLACK)
4.73
3.23F-05
IROF (MEAN)
7.81
2.42F-09
PIXEL FLIPPING (BLACK)
-1.75
8.85f.02
PIXEL FLIPPING (MEAN)
3.26
2.38E-03
SAMEK (2016)
3.32
2.04E-03
16

---

Under review as a conference paper at ICLR 2020
We provide an extended version of fig. 2
VGGG1| Saliency
VGG19| GradCAM
VCG19 | GuidedSackprop
10a
10-
10-3
10-1
10-20
10-16
10
10-27
10-25
10
10-37
10-54
10
30
10-45
10 15
30-37
Exaluction mithed
00
10
ISOF (Mean)
10%
Piael removal (Mean)
10
10
ROF (Black)
10-17
removal
10
10-94
Samek (2016)
10-97
RecNet50 Saaiency
ResNet50| | GradCAM
ResNet50 | Guidedfaccprop
10F
10
10-1
10 44
10-31
10
10-16
10-30
30-21
Pvolue
10-24
30
22
90
10
10-10
10-10
10 43
10 44
10-11
10-56
10-92
10
10-45
10
ResNel101 | Seliency
ResNet101 I GradCAM
ResNet101 Guidedlleckrop
10-1
10-1
10
10-4
30-38
10
10-14
10
10-13
pvalae
10
10
10-41
10-14
10 11
4/
10-38
44
41
10-33
10-11
10
10-31
0
100
200
300
400 $samples
600
700
100
200
400
500
600
700
0
100
200
300
500
600
700
$samples
#samples
Figure 6: P-values for the rejection of the random removal null-hypothesis. Extended graphs
17

---

Under review as a conference paper at ICLR 2020
Figure Sensitivity-n for explanation methods. Higher belter. The proposed methods, AGG- Me-a and AGG Var perform better equally good as all other methods. (large version of fig. 4
Agg- Agg-Mean
Agg-Var
Method
LIME
SM
GB
SG
GC
IG
103
102
FashionMNIST
# pixels
101
100
103
102
# pixels
MNIST
101
100
1.0
0.8
0.6
0.4
0.2
0.0
PCC
81

---

Under review as a conference paper at ICLR 2020
A.6.1 ALIGNMENT BETWEEN HUMAN ATTRIBUTION AND EXPLANATION METHODS
We want quantify whether an explanation method agrees with human judgement which parts of an image should be important. While human annotation is expensive, there exists a benchmark for human evaluation introduced in Mohseni & Ragan (2018). The benchmark includes ninety images of
Figure 8: Example images from Mohseni & Ragan 2018 with numan-annotated overlays.
categories in the ImageNet Challenge (ten images were excluded due to the category being the ImageNet challenge) and provides annotations of relevant segments that ten human test objects found important. Example images are shown in fig. g
While human evaluation is not precise measure, we still expect some correlation between neural network and human judgement.
To test the alignment, we calculate the cosine similarity,
ΣN
An
similarity(e,)
E E?
between the human annotation and the explanations produced by the respective explanation methods. Ag is the human annotation what is important for image Xn
Since the images in this dataset are 224x224 pixel large, only compute the cosine similarity for the network architectures where pretrained networks with this input size were available.
We see that AGG-Mean and AGG-Var perform on-par with the best methods (SmoothGrad and GradCAM). While the aggregated methods perform better than the average explanation method, they do not surpass the best method.
When we combine the two best pess-performing single methods, SmoothGrad and GradCAM, we surpass each individual method. We hypothesize that this because the epistemic uncertainty is reduced by the aggregate.
Table 7: Cosine similarity between heatmap and human annotated benchmark. All std below 0.05
RESNETI01
RESNET50
VGG19
METHOD
AGG-MEAN
0.63
0.66
0.64
AGG-VAR
0.66
0.68
0.67
GB
0.42
0.49
0.47
GC
0.60
0.62
0.60
IG
0.45
0.45
0.47
MEAN(SG+GC)
0.69
0.70
0.65
SG
0.63
0.65
0.59
SM
0.45
0.45
0.47
19

---

Under review as a conference paper at ICLR 2020
A.6.2 EXAMPLE HEATMAPS
Aorg Var
Agg
LIME
18
GB
SM
Image
Figure 9: Example images from Imagenet and the heatmaps produced by different methods on VGG19. Aggregated methods combine features from all methods. Too heavy focus on feature by SmoothGrad (f.e. the beak in lower row) is smoothed away by the aggregation.
20


Reviews:
[{"review_id": "S1eXG9ujFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper has two main messages: 1- Averaging over the explanation (saliency map in the case of image data) of different methods results in a smaller error than an expected error of a single explanation method. 2- Introducing a new saliency map evaluation method by seeking to mitigate the effect of high spatial correlation in image data through grouping pixels into coherent segments. The paper then reports experimental results of the methods introduced in the first message being superior to existing saliency map methods using the second message (and an additional saliency map evaluation method in the literature). They also seek to magnify the capability of the 2nd message's evaluation method by showing its better capability at distinguishing between a random explanation and an explanation method with a signal in it.\n\n\nI vote for rejecting this paper for two main reasons: the contributions are not enough for this veneue, and the paper's introduced methods are not backed by convincing motivations. The first message of the paper is trivial and cannot be considered as a novel contribution: the ''proof'' is basically the error of the mean is smaller than the mean of the errors. Additionally, this could have been useful if the case was that there was a need for playing the safe-card: that is, all of the existing methods have equal saliency map error and averaging will decrease the risk. Not only authors do not provide any evidence but also both the experimental results of the paper itself (results in Table 2 and Fig 4 are disproving this assumption) and the existing literature disprove it. Even considering this assumption to be correct, the contribution is minimal to the field and benefits of averaging saliency maps have been known since the SmoothGrad paper. The second contribution is an extension of existing evaluation methods (e.g. SDC) where instead of removing (replacing by mean) individual pixels, the first segment the image and remove the segments. The method, apart from being very similar to what is already there in the literature, is not introduced in a well-motivated manner. The authors claim that their evaluation method is able to circumvent the problem with removing individual pixels (which is the removed information of one pixel is mitigated by the spatial correlations in the image and therefore will not result in a proportional loss of prediction power) by removing ''features'' instead. Their definition of a feature, though, are segments generated by simple segmentation methods. There is a long line of literature showing the incorrectness of this assumption; i.e. a group of coherent nearby pixels does not necessarily constitute a feature seen by the network and does not necessarily remove the mentioned problem of the high correlation of pixels. This method does not remove \"the interdependency of inputs\" for the saliency evalatuion metric. Even assuming the correctness of this assumption, the contribution over what already exists in the literature is not enough for this venue.\n\nA few suggestions:\n\n* The authors talk about a ''true explanation''. This concept needs to be discussed more clearly and extensively. What does it mean to be a true evaluation? It is also important to prove that the introduced evaluation metric of IROF would assign perfect score for a given true explanation.\n\n* The mentioned problem of pixel correlations that IROF seeks to mitigate is also existing in other modalities of data and the authors do not talk about how IROF could potentially be extended.\n\n* The qualitative results in the text and the appendix do not show an advantage. It would be more crips if the authors could run simple tests on human subjects following the methods in the previous literature.\n\n* There are many many grammatical and spelling errors in the paper. The font size for Figures is very small and unreadable unless by zooming in.\n\n* Many of the introduced heuristics are not backed by evidence or arguments. One example is normalizing individual saliency maps between 0-1 which can naturally be harmful; e.g. aggregating a noisy low-variance method with almost equal importance everywhere (plus additive noise) and a high-variance one which does a good job at distinguishing important pixels - AGG-VAR will not mitigate this issue. \n\nOne question:\n\nThe authors introduced aggregation as a method for a ''better explanation''. It has been known that another problem with saliency maps is robustness: one can generate adversarial examples against saliency maps. It would be an interesting question to see whether aggregation would improve robustness rather than how good the map itself is.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bye0ht1B9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper, inspired by the established technique of model ensembling, proposes two methods (AGG-Mean and AGG-Var) for aggregating different model explanations into a single unified explanation. The authors mathematically prove that the derived explanation is guaranteed to be more truthful than the average performance of the constituent explanations. In practice, the aggregation consistently outperforms *all* individual explanations, not just their aggregated performance. Additionally, the paper introduces a new quantitative evaluation metric for explanations, free of human intervention: IROF (Incremental Removal of Features) incrementally grays out the segments deemed as relevant by an explanation method and observes how quickly the end-task performance is degraded (good explanations will cause fast degradation). Solid validation confirms that the IROF metric is sound.\n\nI support paper acceptance. The experimental section is particularly strong, and makes a convincing argument for both the aggregation methods and the IROF metric. Even though I am not very familiar with the explainability literature and I would not be able to point out an omitted baseline for instance, the wide range of model architectures and aggregated explanation techniques makes a solid case. I appreciate the experiments on low-dimensional input, where the authors are deliberately showing a scenario in which their method does not score huge gains; this brings even more credibility to the paper. The presentation itself is clear, and there are no language or formatting issues.\n\nThe only obvious downside of AGG-Mean and AGG-Var is that one would have to implement and run all constituent evaluation methods, which is expensive. Just as an idea for future work: given N explanation methods, one could ablate away one method at a time, thus getting an idea of whether any of the N explanations are redundant in the presence of others. Recommending a minimal set of useful explanation methods to the NLP community would then decrease the overall complexity of replicating the end-to-end explanation system.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkejrz1otS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a study on explanation methods, proposing an interesting way to aggregate their results and providing empirical evidence that aggregation can improve the quality of the explanations.\n\nThe paper considered only methods using CNN for classifying images, leaving other applications for future investigation.\n\nThe results show aggregation of different explanation methods leads to better explanations, and is therefore exploitable, for high dimensional images while degrades with low-dimensional ones. In the latter case, it happens that the aggregated explanation explains a bit worse than the best non-aggregated explanation (from the graph it seems a very small difference though). This is odd because I would have assumed to see an improvement (at least a small one) using information from more than one system.\n\nThe paper also presents a score for evaluating explanation methods, which shows good results.\n\nThe paper is interesting and well written, the experimental campaign extensive enough and the methods are presented clearly.\n\nThere are some minor problems to be solved:\n- In the abstract, there are two periods before the last sentence.\n- When defining the size of matrices I suggest using \\times instead of x to improve readability\n- The caption in Figure 1 says \"The decrease of the class score over the number of removed segments is reported.\", but I cannot find where this decrease is reported.\n- On page 5, in the parentheses it is reported that a given explanation method is no better than random choice. I would say that it is better (or not worse) than random choice, otherwise the methods would not provide any useful information.\n- On page 6, forty images are considered in the text while fifty images are considered in the caption of Table 1. Please align the number with the correct one.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new method for explaining the predictions of a CNN on a particular image. The method is based on aggregating the explanations of several methods.   They also describe a new method of evaluating explanation methods which avoids manual evaluation of the explanations.

However, the most critical reviewer questions the contribution of the proposed method, which is simple.  Simple isn't always a bad thing, but I think here the reviewer has a point.  The new method for evaluating explanation methods is interesting, but the sample images given are also very simple -- how does the method work when the image is cluttered?   How about when the prediction is uncertain or wrong?
Decision: Reject