Title: TOWARDS A UNIFIED EVALUATION OF EXPLANATION METHODS WITHOUT GROUND TRUTH Anonymous authors Paper under double-blind review

Abstract: This paper proposes a set of criteria to evaluate the objectiveness of explanation methods of neural networks, which is crucial for the development of explainable AI, but it also presents significant challenges. The core challenge is that people usually cannot obtain ground-truth explanations of the neural network. To this end, we design four metrics to evaluate the explanation result without ground-truth explanations. Our metrics can be broadly applied to nine benchmark methods of interpreting neural networks, which provides new insights of explanation methods.

1. INTRODUCTION:

Nowadays, many methods are proposed to explain the logic of a deep neural network (DNN) in a post-hoc manner. In this research, we limit our attention to existing methods of estimating the importance/attribution/saliency of input pixels or intermediate-layer neural units w.r.t. the network output (Shrikumar et al., 2016; Lundberg & Lee, 2017; Ribeiro et al., 2016; Binder et al., 2016) , which present the mainstream of explaining neural networks. To avoid ambiguity, the estimated importance/saliency/attribution maps are all termed "attribution maps" in this paper.

However, some methods usually pursue attribution maps which look reasonable from the perspective of human users, instead of objectively reflecting the true logic of information processing in the DNN. A trustworthy evaluation of the objectiveness of attribution maps is crucial for the development of deep learning and proposes significant challenges to state-of-the-art algorithms.

Existing metrics (Cui et al., 2019; Arras et al., 2019; Vu et al., 2019; Yang & Kim, 2019; Kim et al., 2017; Adebayo et al., 2018; Ghorbani et al., 2019; Alvarez-Melis & Jaakkola, 2018) of evaluating explanation methods have certain shortcomings.

Issue 1, evaluation of the accuracy of a DNN = evaluation of the objectiveness of attribution maps: Some methods only evaluate whether a DNN encodes a correct logic, instead of examining whether an attribution map objectively reflects the true logic of a DNN. (Cui et al., 2019) used human cognition to evaluate the explanation result. (Yang & Kim, 2019; Kim et al., 2017) aimed to construct a specific dataset with ground-truth explanations for evaluation. For example, they added an irrelevant object into the image. Pixels from the irrelevant object are expected to be assigned with zero attributions. However, strictly speaking, it is impossible to religiously annotate ground-truth explanations for a DNN. Currently, the ground-truth explanation is constructed under the assumption that a DNN cannot learn irrelevant objects for classification with the purpose of evaluating the logic of the DNN, instead of examining whether an explanation method mistakenly generates attribution maps with seemingly correct logic for an incorrectly learned DNN.

Issue 2, broad applicability: We aim to design an evaluation metric that can be broadly applied to various tasks. In aforementioned methods (Yang & Kim, 2019; Kim et al., 2017) , the requirement for constructing a new testing dataset limits the applicability of the evaluation.

Issue 3, quantification of the objectiveness: Some methods quantitatively evaluate the accuracy and robustness of attribution maps. However, there is no strict mechanism to ensure the objectiveness of each numerical value in the attribution map. I.e., if the attribution value of a pixel is twice of that of another pixel, then the first pixel is supposed to contribute twice numerical values to the prediction w.r.t. the second pixel.

Table 1: Review of explanation methods:



Method:

What to explain Quantitative evaluation of limitations in application CAM (Zhou et al., 2016) Attribution distribution at intermediate layer 1. Requirement for global average pooling.

2.. Usually explain features at high layers:

Grad CAM (Selvaraju et al., 2017) Attribution distribution at intermediate layer Usually explain features at high layers Grad (Simonyan et al., 2013) Pixel-wise attribution -GI (Shrikumar et al., 2016) Pixel-wise attribution -GB (Springenberg et al., 2014) Pixel-wise attribution Requirement for using ReLU as non-linear layers Shapley Value (Shapley, 1953) Pixel-wise attribution NP-complete problem DeepSHAP (Lundberg & Lee, 2017) Pixel-wise attribution Similar to LRP, DeepLIFT (Shrikumar et al., 2016) with a designed backward rule LIME (Ribeiro et al., 2016) Pixel-wise attribution Attribution maps at the super-pixel level, rather than at the pixel level LRP (Binder et al., 2016) Pixel-wise attribution Relevance propagation rules of every layer should be defined Pert (Fong & Vedaldi, 2017) Pixel-wise attribution -Except for the objectiveness, previous studies mainly conducted the evaluation from other perspectives. (Arras et al., 2019; Vu et al., 2019) evaluated attribution maps from the perspective of adversarial attacks by adding random noise. (Adebayo et al., 2018; Ghorbani et al., 2019; Alvarez-Melis & Jaakkola, 2018) proposed methods to evaluate the robustness of explanation methods w.r.t. the perturbation. (Adebayo et al., 2018) randomized the layer of DNN from the top to the bottom and visualized the change of attribution maps.

Note that in most applications, people cannot faithfully obtain the ground-truth logic of a DNN. Therefore, considering the above three issues, in this study, we aim to fairly evaluate the objectiveness and robustness of attribution maps from the following four perspectives without ground-truth explanations.

Perspective 1, bias of the attribution map at the pixel level: In order to evaluate the bias of the attribution map, we first need to propose a standard metric to evaluate the accuracy of explanation methods. The Shapley value is the unique solution to model the attribution value of each pixel that satisfies desirable properties including efficiency, symmetry and monotonicity (Lundberg & Lee, 2017). However, the computation of the Shapley value is an NP-complete problem, and previous studies (Lundberg & Lee, 2017) showed that the accurate estimation of the Shapley value is still a significant challenge. To this end, we extend the theory of Shapley sampling (Castro et al., 2009) and design a new evaluation metric, which achieves high accuracy without significantly boosting the computational cost.

We use the new evaluation metric to quantify the bias of the attribution map. Note that this evaluation has no partiality to the Shapley-value-based explanation methods. For example, experimental results showed that LRP (Binder et al., 2016) exhibited significant lower bias than DeepSHAP (Lundberg & Lee, 2017).

Perspective 2, quantification of unexplainable feature components: Given an input image and its attribution map, we revise the input image to generate a new image that reflects the logic of the attribution map. We then compare the intermediate-layer feature of the original image with that of the generated image, so as to disentangle feature components that can and cannot be explained by the attribution map.

Perspective 3, robustness of the explanation: Robustness of the explanation means whether the attribution map is robust to spatial masking of the input image. When we randomly mask a certain region of the input image, we admit that spatial masking destroys global contexts and affects pixelwise attribution value to some extent. The quantification of the robustness of the explanation is an important perspective of evaluating an explanation method.

Perspective 4, mutual verification: The mutual verification means whether different explanation methods can verify each other. Methods generating similar attribution maps are usually believed more reliable.

In this paper, we used our metrics to evaluate nine widely used explanation methods listed in Table 1 .

We conducted experiments using the LeNet, VGG and ResNet on different benchmark datasets including the CIFAR-10 (Krizhevsky & Hinton, 2009) dataset and the Pascal VOC 2012 (Everingham et al., 2010) dataset. Our experimental results proved the effectiveness of the proposed evaluation methods and provided an insightful understanding of various explanation methods.

The contribution of our work can be concluded as follows.

1. In this study, we invent a set of standard metrics to evaluate the objectiveness and the robustness of the attribution map without knowing ground-truth explanations.

2. The metric of evaluating the pixel-wise bias of the attribution map can be estimated with a relatively low computational cost, which avoids falling into the computational bottleneck of estimating accurate pixel-wise attributions.

3. Since our metrics do not need any annotations of ground-truth explanations, our metrics can be applied to different neural networks trained on different datasets.

2. RELATED WORK:

Explainable AI is an emerging direction in artificial intelligence, and different explanation methods have been proposed. In this section, we briefly review the Shapley value and limit our discussions to existing methods of evaluating methods of extracting attribution/importance/saliency maps to simplify the story. Appendix C discusses other research directions of explainable AI.

The Shapley value: The Shapley value (Shapley, 1953) was proposed to compute the attribution distribution over all players in a particular cooperative game. However, it is an NP-complete problem to compute the accurate Shapley value. The Shapley value approximated by sampling strategy could be very inaccurate due to the high variance. We extend the theory of the Shapley value to obtain an evaluation metric with a high accuracy but a low computational cost.

Qualitative evaluation: Some studies used a qualitative criterion for evaluation. (Cui et al., 2019) qualitatively defined basic concepts in the evaluation of explanation results, including the complexity of the explanation, the correlation, and the completeness. In contrast, this paper aims to evaluate the methods quantitatively, which makes our metrics more objective and reliable.

Accuracy evaluation: To evaluate the accuracy of attribution maps, (Arras et al., 2019; Vu et al., 2019) used the noise/occlusion to perturb the original image according to the attribution value. There was no mechanism to ensure the prediction result objectively reflected the logic of a DNN. (Yang & Kim, 2019; Kim et al., 2017) built a dataset to help them generate ground-truth explanations.

Essentially, these methods tried to obtain the "correct" logic for an input image. However, a rigorous study should not assume that the DNN encodes the correct logic. As a result, this paper proposes to evaluate the objectiveness of explanation results without knowing ground-truth explanations.

Stability evaluation: (Adebayo et al., 2018; Ghorbani et al., 2019; Alvarez-Melis & Jaakkola, 2018) mainly paid attention to the attribution map change when the model input was perturbed. (Adebayo et al., 2018) visualized the change in the attribution map when the weights of the model were destroyed from the top to the bottom. (Ghorbani et al., 2019; Alvarez-Melis & Jaakkola, 2018) used the adversarial image to alter the attribution map. In comparison, we propose a metric to evaluate the robustness to spatial masking.

3. ALGORITHM:



3.1. PRELIMINARIES: THE SHAPLEY VALUE:

The Shapley value measures the instancewise feature importance ranking problem. Let Ω be the set of all pixels of an image I. I P denotes an image that replaces all pixels in set Ω \ P with average pixel value over images. F (I P ) denotes the scalar output of a DNN based on a subset of pixels P ⊂ Ω. To compute the Shapley value of the i-th feature, (Shapley, 1953 ) considered all subsets of Ω not containing the i-th feature and defined the Shapley value A * i as follows:

A * i = P ⊂Ω\{i} |P |!(|Ω| -|P | -1)! |P |! F (I P ∪{i} ) -F (IP ) (1)

It is the unique solution that satisfies desirable properties to assign attribution value to each feature dimension in the input (Chen et al., 2018) . Appendix A shows properties of the Shapley value.

3.2. EVALUATING THE BIAS OF THE ATTRIBUTION MAP AT THE PIXEL LEVEL:

In this section, we design a metric to accurately evaluate the objectiveness of the attribution map.

Given an image I ∈ I, let us consider the DNN F with a single scalar output y = F (I). For DNNs with multiple outputs, existing methods usually explain each individual output dimension independently. Let {a i } denote the pixel-wise attribution map estimated by a specific explanation method. We aim to evaluate the bias of {a i }. People usually formulate the network output as the sum of pixel-wise attribution value, i.e. the output y can be decomposed as follows.

y = b + i∈Ω Ai, s.t. Ai = λai (2)

b denotes the bias; i denotes the index of each pixel in the input image; Ω denotes the set of all pixels in the image. Aforementioned {A * i } can be considered as the ground-truth of {A i } (Lundberg & Lee, 2017). Since many explanation methods (Selvaraju et al., 2017; Simonyan et al., 2013) mainly compute relative values of attributions {a i }, instead of a strict attribution map {A i }. We use λ to bridge {A i } and {a i }. λ is a constant for normalization, which can be eliminated during the implementation of the evaluation.

The estimated attribution of each pixel can be assumed to follow a Gaussian distribution Ai ∼ N (µi, σ 2 i ) (Castro et al., 2009) . Attribution distributions of different pixels can be further assumed to share a unified variance, i.e.

σ 2 1 ≈ σ 2 2 ≈ ... ≈ σ 2 n .

The evaluation of the attribution distribution {a i } has two aspects, i.e. 1. the sampling of pixels whose attributions that are more likely to have large deviations; 2. the evaluation of the bias of the sampled attributions.

First, for the sampling of attributions of interest, we sample the set of pixels S with top-ranked high (or low) attributions. Attribution values of pixels in S are sampled as those with the highest (or the lowest) values, and these pixels are supposed to be more likely to be significantly biased towards high (or low) attribution values. Meanwhile, from another perspective, the distribution of the sampled attribution values is close to the Gumbel distribution.

Second, although the Shapley value can be considered as a standard formulation of the pixel-wise attribution, it usually cannot be accurately computed because of its high computational cost. In order to accurately evaluate the sampled attribution values without significantly increasing the computational cost, we applied the Shapley value approximated by the sampling method. Just like the target attribution distribution A i , the approximated Shapley value

A shap i is assumed to follow a normal distribution N (A * i , (σ shap ) 2 ). A shap i

is an unbiased approximation of the true Shapley value A * i , Thus, the average value over different pixels in

S satisfies i∈S A shap i |S| ∼ N ( i∈S A * i |S| , (σ shap ) 2 |S| ).

We can prove that the measurement of the average attribution among all sampled pixels i∈S A shap i |S| is of much higher accuracy than the raw Shapley value with the same computational cost. The difference between the highest (or lowest) values and its true values

i∈S A shap i |S| A shap -i∈S A i |S| A

can reflect the system bias, as follows.

Mpixel = EI i∈S A shap i |S| A shap -i∈S Ai |S| A = EI 1 |S| i∈S A shap i A shap -i∈S ai a s.t. ∀i ∈ S, j ∈ Ω \ S, ai ≥ aj or ∀i ∈ S, j ∈ Ω \ S, ai ≤ aj (3)

where Ω is the set of all pixels in an image. A shap and a are used for normalization. A small value of M pixel indicates the low bias of the attribution map.

Analysis of the high computational efficiency: Suppose that the computational complexity of processing one sample is O(N ), then the computational complexity of sampling m times is O(mN ).

If the raw Shapley value needs to obtain the same accuracy, it needs significantly more samples, and the computational complexity is O(|S|mN ). Please see Appendix B for the theoretical analysis of the save of computational cost.

In addition, the proposed metric can also be used to evaluate the attribution of neural activations in the intermediate layer, such as those generated by Grad-CAM. In this case, we can regard the target intermediate-layer feature as the input image to compute attributions, so as to implement the evaluation. For each image, we need to sample multiple times to increase the accuracy. We compute the average performance over different images for evaluation. We need to sample multiple times with different images to increase the accuracy of the evaluation. Note that although the metric is designed based on the Shapley value, experimental results showed that LRP outperforms DeepSHAP.

3.3. QUANTIFICATION OF UNEXPLAINABLE FEATURE COMPONENTS:

We propose another metric to quantify unexplainable feature components. Given an image I and its attribution map {a i }, we generate a new image, which reflects the logic of the attribution map. In this way, we can consider the feature of the newly generated image f as feature components that can be explained. Let f denote the feature of the original image I. Then, f -f corresponds to the unexplainable feature components.

To generate the new image, we mask specific pixels in the original image I, which have the lowest attributions. We select and mask a set of pixels S with the lowest absolute attributions, and the number of the selected points is determined subjects to i∈S |a i | = 0.1 i∈Ω |a i | to generate the new image I. The metric is formulated as

EQUATION

where α =

1 E I [ f I -E I [f I ]

] is used for normalization. A small value of M feature indicates most feature components in f are explainable.

3.4. EVALUATING THE ROBUSTNESS OF EXPLANATION:

This metric is used to measure the robustness of explanation methods to the spatial masking. We believe that the method, which is robust to spatial masking, can be considered more convincing. The robustness is an important perspective of evaluating explanation methods.

Given an input image I ∈ I and its attribution map {a i } w.r.t a DNN, we use a mask M to cover specific parts of the image to get a masked image Î. For each input image I, we can generate four masked images by masking the right, left, top, and bottom half of the image, respectively. For each masked image Î, the explanation method estimates the attribution âi for each pixel. We compare pixel-level attributions of the unmask pixels between original images and masked images, as follows.

Mnon-robust = EI   1 a i∈I\I mask (ai -âi) 2   (5)

We used a for normalization, and a large value of M non-robust indicates a high non-robustness.

3.5. EVALUATING THE MUTUAL VERIFICATION:

This metric aims to quantitatively measure the mutual verification between different explanation methods. It is usually believed that two explanation methods are more reliable if they can verify each other. Given a DNN F and an image I ∈ I, two different explanation methods α and β produce attribution maps a α and a β , respectively. We measure their difference as follows.

Mmutual = EI aα aα - a β a β (6)

Attribution maps from different explanation methods are normalized by their L2-norm. A lower value of M mutual indicates a more convincing mutual verification between explanation methods α and β. 

4.1. BASELINE:

In our experiments, we mainly evaluated the following explanation methods. Figure 1 shows attribution maps yielded by these explanation methods. Appendix D provides more attribution maps.

Grad: Given an input, (Simonyan et al., 2013) quantified the attribution value with the gradient of the input. We termed this algorithm as Grad. For RGB images with multiple channels, Grad selected the maximum magnitude across all channels for each pixel. Mask values ranging between 0 and 1 indicated the saliency of each pixel. We termed this method Pert.

CAM: CAM computed attribution map over the feature from the last convolutional layer (Zhou et al., 2016) . It required the special structure with a global average pooling layer and a fully connected layer at the end of the DNN.

Grad CAM: Grad CAM was similar to CAM (Selvaraju et al., 2017) . Grad CAM used gradients over the feature map, instead of the parameters of the fully connected layer.

4.2. IMPLEMENTATION DETAILS:

Bias of the attribution map at the pixel level: To approximate the Shapley value for each image, we sampled 1000 times for each image in the CIFAR-10 dataset and sampled 100 times for each image in the Pascal VOC 2012 dataset. We sampled the top-10%, 30%, 50%, 70%, 90% pixels with the highest/lowest values. Quantification of unexplainable feature components: Given an image, we masked the pixels with the lowest absolute attribution value. The number of the masked pixels was determined to ensure that the sum of masked absolute attribution value took 10% of the total absolute attribution value. On average, around 30% pixels were masked. The masked pixels were assigned with the average pixel value over images. We used features of the last convolutional layer to compute M f eature .

4.3. EXPERIMENT RESULT AND ANALYSIS:

Bias of the attribution map at the pixel level: Figure 2 shows curves of evaluation results on different models learned using different datasets. According to these curves, GI and LIME provided the least biased attribution maps for ResNet at the pixel level. For AlexNet, VGG-16/19 and LeNet, LRP outperformed other methods. Detailed numbers corresponding curves in Figure 2 are listed in the Appendix E.

Some methods could not be evaluated using the bias at the pixel level. For example, Pert computed an importance map without negative values instead of an attribution map for each image. The code of CAM (Zhou et al., 2016) projected attribution values to the range between 0 and 1. Grad CAM and LRP were not used on residual networks, because there was only one fully connected layer behind the last convolutional layer in residual networks. In this case, Grad CAM could not diagnose the logic contained in the cascaded non-liner layers of the DNN. For LRP, the relevance propagation rules of some structures in ResNet were not defined to the best of our knowledge.

Quantification of unexplainable feature components: Table 2 compares the amount of unexplainable feature components between explanation methods. We found that LIME, GB and Pert explained more feature components than other methods. We noticed that the quantification of unexplainable feature components of most explanation methods were considerable larger than expected. It was be- cause the attribution maps from some methods contained relatively larger noise. Thus, the masked pixels were almost uniformly distributed over images, which destroyed the context information and led to worse results.

We did not evaluate CAM and Grad CAM, because they calculated attribution maps at the feature level, which were not comparable with attribution values at the pixel level.

Robustness of explanation: Table 3 shows the quantitative results of M non-robust on different models trained using different datasets. We found that GB and Grad CAM exhibited a lower non-robustness to spatial masking. For more results, Appendix F shows examples of attribution maps of masked images.

Mutual verification: Figure 3 visualizes the mutual verification M mutual between different explanation methods, which indicates a high level mutual verification between LRP, GI and DeepSHAP.

Appendix G provides more detailed results. Note that we did not compare CAM and Grad CAM with other methods. It was because they computed attribution maps on intermediate-layer features.

5. CONCLUSION:

In this paper, we have proposed four metrics to evaluate explanation methods from four different perspectives. The proposed evaluation metrics are computed without requirements for ground-truth explanations. Our metrics can be broadly applied to different methods, w.r.t. DNNs learned using different datasets. These metrics evaluate the bias of the attribution map at the pixel level, quantify the unexplainable feature components, the robustness of the explanation and the mutual verification.

In experiments, we used our metrics to evaluate nine widely used explanation methods. Experimental results showed that attribution maps from LRP, GI and LIME exhibited lower bias at the pixel level. LIME and GB explained more feature components than other methods. Regarding the robustness, GB, CAM and Grad CAM were more robust to spatial masking than other explanation methods. DeepSHAP, GI and LRP can better verified each other.

A PROPERTIES OF THE SHAPLEY VALUE:

Let I denote the input image; let Ω denote the set of all pixels in I. We can use I ∅ to denote a baseline image, i.e. all pixels in I ∅ equal to the average value over all images. For a subset S ⊂ Ω, I S denotes an image that satisfies

EQUATION

)

where i is the index of the pixel in I and I Ω is the same image as I. Let F and G denote two models with scalar output. The Shapley value of the i-th pixel is represented by A * i , and they have the following properties.

Efficiency: The sum of Shapley values i∈Ω A *:

i is equal to F (I Ω ) -F (I ∅ ). Symmetry: The features that are treated equally by the model are treated equally by the Shapley value. If F (I S∪{i} ) = F (I S∪{j} ) for all subsets S, then A * i = A * j . Additivity: For any two models F and G, if they are combined into one model F + G, the Shapley value must be added pixel by pixel:

(A * ) F +G i = (A * ) F i + (A * ) G i .

Monotonicity: For any two models F and G, if for all subsets S we have F (I S∪{i} ) -F (I S ) ≥ G(I S∪{i} ) -G(I S ) for all subsets S, then we have

(A * ) F i ≥ (A * ) G i .

B ANALYSIS OF THE COMPUTATIONAL COST:

In this section, we continue using the notation in Section 3.1 and Section 3.2. Suppose that we sample m times to approximate the Shapley value. The variance of A shap i is σ 2 m where σ 2 satisfies (Castro et al., 2009 )

EQUATION

So we have (σ shap ) 2 = σ 2 /m. For the set of sampled pixels S, the variance of their average Shapley

value is |S|(σ shap ) 2 |S| 2 = (σ shap) 2 |S| = σ 2 m|S| .

Apparently, if we want to get the same accuracy for a single pixel as the set of pixels, we need to sample m|S| times, which needs much more computational cost than our metric, especially when the number of sampled pixels is large.

C STUDIES OF EXPLANATION METHODS BESIDES THE ESTIMATION OF ATTRIBUTION MAPS.:

Network visualization: The visualization of feature representations inside a neural network is the most direct way of opening the black-box of the neural network. Related techniques include gradient-based visualization (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015; Yosinski et al., 2015) and up-convolutional nets (Dosovitskiy & Brox, 2016) to invert feature maps of conv-layers into images.

Network diagnosis: Some studies diagnose feature representations inside a neural network. (Yosinski et al., 2014) measured features transferability in intermediate layers of a neural network. (Aubry & Russell, 2015) visualized feature distributions of different categories in the feature space. (Kindermans et al., 2018) extracted rough pixel-level correlations between network inputs and outputs, i.e. estimating image regions that directly contribute the network output. Network-attack methods (Koh & Liang, 2017; Szegedy et al., 2014) computed adversarial samples to diagnose a CNN. (Lakkaraju et al., 2017) discovered knowledge blind spots of a CNN in a weakly-supervised manner. However, above methods usually analyzed a neural network at the pixel level and did not summarize the network knowledge into clear visual concepts. (Bau et al., 2017) defined six types of semantics for CNN filters, i.e. objects, parts, scenes, textures, materials, and colors. Then, (Zhou et al., 2015) proposed a method to compute the image-resolution receptive field of neural activations in a feature map. Fong and Vedaldi (Fong & Vedaldi, 2018) analyzed how multiple filters jointly represented a certain semantic concept. Other studies retrieved intermediate-layer features from CNNs representing clear concepts. Simon & Rodner (2015) retrieved features to describe objects from feature maps, respectively. (Zhou et al., 2015) selected neural units to describe scenes.

Learning interpretable representations::

A new trend in the scope of network interpretability is to learn interpretable feature representations in neural networks Hu et al. (2016) ; Stone et al. (2017) ; Liao et al. (2016) in an un-/weakly-supervised manner. Capsule nets Sabour et al. (2017) and interpretable RCNN Wu et al. (2017b) Explaining neural networks via knowledge distillation: Distilling knowledge from a black-box model into an explainable model is an emerging direction in recent years. (Choi et al., 2017) learned an explainable additive model, and (Vaughan et al., 2018) distilled knowledge of a network into an additive model. In order to disentangle feature representations of object parts from intermediate layers of a CNN, (Zhang et al., 2018) distilled the CNN's knowledge into an explainer network with interpretable conv-layers, in which each filter represented a specific object part. (Frosst & Hinton, 2017; Tan et al., 2018; Che et al., 2016; Wu et al., 2017a) distilled representations of neural networks into tree structures. These methods did not explain the network knowledge using humaninterpretable semantic concepts. 

D MORE EXAMPLES OF ATTRIBUTION MAPS:



:



Figures and Tables:

Figure 1: Example of attribution maps of different methods.4 EXPERIMENTTo evaluate explanation methods, we conducted experiments on the CIFAR-10 (Krizhevsky & Hinton, 2009) dataset and the Pascal VOC 2012(Everingham et al., 2010) dataset. For images in the Pascal VOC 2012 dataset, we cropped objects using their bounding boxes and used the cropped objects as inputs to train DNNs for object classification. We trained and explainedLeNet (LeCun et al.,  1998), ResNet-20/32/44/56(He et al., 2016) using the CIFAR-10 dataset. AlexNet(Krizhevsky et al., 2012), VGG-16/19(Simonyan & Zisserman, 2015), ResNet-50/101(He et al., 2016) were trained using the Pascal VOC 2012 dataset.

GI:(Shrikumar et al., 2016) proposed a method, namely GI, which used the pixel-wise product of the input and its gradient as attribution value. Attribution values for RGB channels were summed to get the final attribution value. GB: Guided Back-propagation, namely GB, corresponded to Grad where the back-propagation rule at ReLU units was redefined(Springenberg et al., 2014). LRP: Layer-wise relevance propagation (LRP)(Binder et al., 2016) redefined back-propagation rules for each layer to decompose the output of a DNN over the input. We used LRP-and set the parameter = 1 in experiments. DeepSHAP: DeepSHAP adapted DeepLIFT(Shrikumar et al., 2016) to approximate pixel-wise Shapley values for the input image (Lundberg & Lee, 2017). We applied the code released by (Lundberg & Lee, 2017). LIME: LIME(Ribeiro et al., 2016) trained an interpretable model to compute the attribute value for each super-pixel. We used the code released by(Ribeiro et al., 2016). Pert:(Fong & Vedaldi, 2017) explained a prediction by training a mask to perturb the input image.

Figure 2: Results of the bias of the attribution map at the pixel level. Row (I) and row (II) used trained LeNet, ResNet20, ResNet32, ResNet44, ResNet56 on the CIFAR-10 dataset from left to right; row (III) and row (IV) used trained ResNet50, ResNet101, VGG16, VGG19, AlexNet on the Pascal VOC 2012 dataset from left to right. Row (I) and row (III) sampled pixels with the highest attribution values; row (II) and row (IV) sampled the pixels with the lowest attribution values.

Figure 3: Heat maps of mutual verification. A low value of M mutual between two methods indicates a more convincing mutual verification between them.

learned interpretable features in intermediate layers. InfoGAN Chen et al. (2016) and β-VAE Higgins et al. (2017) learned well-disentangled codes for generative networks.

Figure 5: Example of attribution maps after spatial masking.

Quantification of unexplainable feature components.

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td><td>LRP</td><td>Pert</td></tr><tr><td>CIFAR10-LeNet</td><td colspan="3">0.96821 1.10399 0.91546</td><td>1.10371</td><td colspan="3">0.67032 1.08469 0.70177</td></tr><tr><td>CIFAR10-ResNet20</td><td colspan="3">1.22212 1.28065 1.14596</td><td>1.26002</td><td>1.05286</td><td>-</td><td>1.13032</td></tr><tr><td>CIFAR10-ResNet32</td><td colspan="3">1.22803 1.29324 1.16028</td><td>1.27681</td><td>1.04412</td><td>-</td><td>1.14261</td></tr><tr><td>CIFAR10-ResNet44</td><td colspan="3">1.22382 1.26434 1.10411</td><td>1.24961</td><td>1.02039</td><td>-</td><td>1.10772</td></tr><tr><td>CIFAR10-ResNet56</td><td colspan="3">1.22487 1.24123 1.11306</td><td>1.24146</td><td>1.00916</td><td>-</td><td>1.10122</td></tr><tr><td>VOC2012-AlexNet</td><td colspan="3">1.02425 1.09375 1.04981</td><td>1.0809</td><td colspan="3">1.01942 1.11513 1.04048</td></tr><tr><td>VOC2012-VGG16</td><td colspan="3">1.18959 1.22715 1.22724</td><td>1.32364</td><td colspan="3">1.12339 1.32674 1.12878</td></tr><tr><td>VOC2012-VGG19</td><td>1.08084</td><td>1.1226</td><td>1.10411</td><td>1.17121</td><td colspan="3">1.08387 1.23784 1.13043</td></tr><tr><td>VOC2012-ResNet50</td><td colspan="2">1.23345 1.24123</td><td>1.2156</td><td>1.23441</td><td>1.22411</td><td>-</td><td>1.25732</td></tr><tr><td colspan="4">VOC2012-ResNet101 1.08592 1.09363 1.07853</td><td>1.07856</td><td>1.11289</td><td>-</td><td>1.23279</td></tr></table>

Non-robustness of explanation methods with different datasets/models.

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td><td>LRP</td><td>Pert</td><td>CAM Grad CAM</td></tr><tr><td>CIFAR10-LeNet</td><td colspan="3">0.510 0.610 0.548</td><td>0.843</td><td>-</td><td colspan="2">0.932 0.369</td><td>-</td><td>1.039</td></tr><tr><td>CIFAR10-ResNet20</td><td colspan="3">0.515 0.599 0.330</td><td>0.541</td><td>-</td><td>-</td><td colspan="2">0.466 0.443</td><td>0.386</td></tr><tr><td>CIFAR10-ResNet32</td><td colspan="3">0.762 0.832 0.321</td><td>0.645</td><td>-</td><td>-</td><td colspan="2">0.469 0.351</td><td>0.318</td></tr><tr><td>CIFAR10-ResNet44</td><td colspan="3">0.957 1.010 0.299</td><td>0.513</td><td>-</td><td>-</td><td colspan="2">0.470 0.321</td><td>0.287</td></tr><tr><td>CIFAR10-ResNet56</td><td colspan="3">0.911 0.959 0.298</td><td>0.567</td><td>-</td><td>-</td><td colspan="2">0.475 0.341</td><td>0.321</td></tr><tr><td>VOC2012-AlexNet</td><td colspan="3">0.519 0.603 0.421</td><td>0.448</td><td colspan="3">3.994 0.719 0.602</td><td>-</td><td>0.489</td></tr><tr><td>VOC2012-VGG16</td><td colspan="3">0.475 0.500 0.271</td><td>0.388</td><td colspan="3">2.175 0.447 0.635</td><td>-</td><td>0.381</td></tr><tr><td>VOC2012-VGG19</td><td colspan="3">0.505 0.524 0.286</td><td>0.356</td><td colspan="3">2.172 0.444 0.666</td><td>-</td><td>0.381</td></tr><tr><td>VOC2012-ResNet50</td><td colspan="3">0.605 0.663 0.280</td><td>0.936</td><td>2.354</td><td>-</td><td colspan="2">0.552 0.367</td><td>0.309</td></tr><tr><td colspan="4">VOC2012-ResNet101 0.553 0.593 0.269</td><td>0.847</td><td>4.667</td><td>-</td><td colspan="2">0.565 0.347</td><td>0.266</td></tr></table>

Bias of the attribution map at the pixel level on CIFAR-10-LeNet

Table content: <table><tr><td>Grad_CAM</td><td>Grad</td><td>GI</td><td>GB</td><td>LIME</td><td>LRP</td><td>Pert</td><td>DeepSHAP</td><td>Origin</td></tr></table>

Bias of the attribution map at the pixel level on CIFAR-10-ResNet20

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td></tr><tr><td>top-10%</td><td colspan="3">0.04692 0.03962 0.04801</td><td>0.04702</td><td>0.02716</td></tr><tr><td>top-30%</td><td colspan="3">0.02552 0.01522 0.01848</td><td>0.02175</td><td>0.02706</td></tr><tr><td>top-50%</td><td colspan="3">0.01616 0.00795 0.00943</td><td>0.01241</td><td>0.02666</td></tr><tr><td>top-70%</td><td colspan="3">0.00762 0.00416 0.00425</td><td>0.00643</td><td>0.02456</td></tr><tr><td>top-90%</td><td colspan="3">0.00288 0.00283 0.00336</td><td>0.00270</td><td>0.02185</td></tr><tr><td colspan="4">bottom-10% 0.06610 0.06449 0.06726</td><td>0.06866</td><td>0.01790</td></tr><tr><td colspan="4">bottom-30% 0.04272 0.03437 0.03833</td><td>0.03962</td><td>0.01765</td></tr><tr><td colspan="4">bottom-50% 0.03117 0.02296 0.02704</td><td>0.02722</td><td>0.01836</td></tr><tr><td colspan="4">bottom-70% 0.02168 0.01726 0.02053</td><td>0.01992</td><td>0.01897</td></tr><tr><td colspan="4">bottom-90% 0.01357 0.01275 0.01513</td><td>0.01346</td><td>0.02005</td></tr></table>

Bias of the attribution map at the pixel level on CIFAR-10-ResNet32

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td></tr><tr><td>top-10%</td><td colspan="3">0.04836 0.04126 0.03984</td><td>0.04773</td><td>0.02699</td></tr><tr><td>top-30%</td><td colspan="3">0.02600 0.01641 0.01728</td><td>0.02247</td><td>0.02702</td></tr><tr><td>top-50%</td><td colspan="3">0.01639 0.00874 0.00972</td><td>0.01289</td><td>0.02630</td></tr><tr><td>top-70%</td><td colspan="3">0.00793 0.00459 0.00565</td><td>0.00683</td><td>0.02465</td></tr><tr><td>top-90%</td><td colspan="3">0.00268 0.00264 0.00242</td><td>0.00260</td><td>0.02267</td></tr><tr><td colspan="4">bottom-10% 0.06535 0.06389 0.07271</td><td>0.06787</td><td>0.02059</td></tr><tr><td colspan="4">bottom-30% 0.04217 0.03433 0.03612</td><td>0.03917</td><td>0.02053</td></tr><tr><td colspan="4">bottom-50% 0.03064 0.02304 0.02359</td><td>0.02686</td><td>0.02065</td></tr><tr><td colspan="4">bottom-70% 0.02135 0.01726 0.01733</td><td>0.01963</td><td>0.02103</td></tr><tr><td colspan="4">bottom-90% 0.01331 0.01254 0.01214</td><td>0.01307</td><td>0.02145</td></tr></table>

Bias of the attribution map at the pixel level on CIFAR-10-ResNet44

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td></tr><tr><td>top-10%</td><td colspan="3">0.04795 0.04155 0.04055</td><td>0.04751</td><td>0.02712</td></tr><tr><td>top-30%</td><td colspan="3">0.02576 0.01626 0.01698</td><td>0.02254</td><td>0.02709</td></tr><tr><td>top-50%</td><td colspan="3">0.01609 0.00848 0.00935</td><td>0.01307</td><td>0.02637</td></tr><tr><td>top-70%</td><td colspan="3">0.00748 0.00433 0.00502</td><td>0.00686</td><td>0.02485</td></tr><tr><td>top-90%</td><td colspan="3">0.00248 0.00228 0.00230</td><td>0.00226</td><td>0.02271</td></tr><tr><td colspan="4">bottom-10% 0.06510 0.06452 0.07143</td><td>0.06802</td><td>0.01924</td></tr><tr><td colspan="4">bottom-30% 0.04196 0.03449 0.03663</td><td>0.03977</td><td>0.01879</td></tr><tr><td colspan="4">bottom-50% 0.03082 0.02323 0.02448</td><td>0.02736</td><td>0.01967</td></tr><tr><td colspan="4">bottom-70% 0.02159 0.01752 0.01811</td><td>0.01989</td><td>0.02064</td></tr><tr><td colspan="4">bottom-90% 0.01353 0.01282 0.01293</td><td>0.01323</td><td>0.02137</td></tr></table>

Bias of the attribution map at the pixel level on CIFAR-10-ResNet56

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td></tr><tr><td>top-10%</td><td colspan="3">0.04716 0.04129 0.04780</td><td>0.04601</td><td>0.02690</td></tr><tr><td>top-30%</td><td colspan="3">0.02537 0.01659 0.01855</td><td>0.02089</td><td>0.02714</td></tr><tr><td>top-50%</td><td colspan="3">0.01607 0.00900 0.01022</td><td>0.01215</td><td>0.02679</td></tr><tr><td>top-70%</td><td colspan="3">0.00791 0.00482 0.00569</td><td>0.00652</td><td>0.02509</td></tr><tr><td>top-90%</td><td colspan="3">0.00212 0.00194 0.00214</td><td>0.00234</td><td>0.02252</td></tr><tr><td colspan="4">bottom-10% 0.06625 0.06551 0.06065</td><td>0.06727</td><td>0.01828</td></tr><tr><td colspan="4">bottom-30% 0.04164 0.03449 0.03158</td><td>0.03779</td><td>0.01812</td></tr><tr><td colspan="4">bottom-50% 0.03001 0.02299 0.02129</td><td>0.02574</td><td>0.01911</td></tr><tr><td colspan="4">bottom-70% 0.02086 0.01712 0.01587</td><td>0.01868</td><td>0.02023</td></tr><tr><td colspan="4">bottom-90% 0.01300 0.01237 0.01146</td><td>0.01267</td><td>0.02088</td></tr></table>

Bias of the attribution map at the pixel level on VOC2012-ResNet50

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td></tr><tr><td>top-10%</td><td colspan="3">0.00766 0.00728 0.00633</td><td>0.00738</td><td>0.00876</td></tr><tr><td>top-30%</td><td colspan="3">0.00437 0.00345 0.00276</td><td>0.00359</td><td>0.00606</td></tr><tr><td>top-50%</td><td colspan="3">0.00297 0.00213 0.00170</td><td>0.00223</td><td>0.00451</td></tr><tr><td>top-70%</td><td colspan="3">0.00173 0.00138 0.00113</td><td>0.00144</td><td>0.00335</td></tr><tr><td>top-90%</td><td colspan="3">0.00060 0.00060 0.00053</td><td>0.00062</td><td>0.00235</td></tr><tr><td colspan="4">bottom-10% 0.00831 0.00824 0.00850</td><td>0.00839</td><td>0.00430</td></tr><tr><td colspan="4">bottom-30% 0.00500 0.00417 0.00386</td><td>0.00432</td><td>0.00240</td></tr><tr><td colspan="4">bottom-50% 0.00354 0.00269 0.00244</td><td>0.00280</td><td>0.00151</td></tr><tr><td colspan="4">bottom-70% 0.00228 0.00189 0.00171</td><td>0.00194</td><td>0.00113</td></tr><tr><td colspan="4">bottom-90% 0.00117 0.00112 0.00111</td><td>0.00114</td><td>0.00128</td></tr></table>

Bias of the attribution map at the pixel level on VOC2012-ResNet101

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td></tr><tr><td>top-10%</td><td colspan="3">0.00790 0.00678 0.00610</td><td>0.00685</td><td>0.00827</td></tr><tr><td>top-30%</td><td colspan="3">0.00438 0.00307 0.00260</td><td>0.00318</td><td>0.00608</td></tr><tr><td>top-50%</td><td colspan="3">0.00291 0.00185 0.00158</td><td>0.00194</td><td>0.00473</td></tr><tr><td>top-70%</td><td colspan="3">0.00175 0.00121 0.00104</td><td>0.00127</td><td>0.00371</td></tr><tr><td>top-90%</td><td colspan="3">0.00067 0.00055 0.00049</td><td>0.00056</td><td>0.00278</td></tr><tr><td colspan="4">bottom-10% 0.00861 0.00754 0.00839</td><td>0.00767</td><td>0.00387</td></tr><tr><td colspan="4">bottom-30% 0.00496 0.00369 0.00377</td><td>0.00381</td><td>0.00222</td></tr><tr><td colspan="4">bottom-50% 0.00343 0.00237 0.00238</td><td>0.00245</td><td>0.00142</td></tr><tr><td colspan="4">bottom-70% 0.00225 0.00169 0.00169</td><td>0.00173</td><td>0.00124</td></tr><tr><td colspan="4">bottom-90% 0.00117 0.00104 0.00112</td><td>0.00105</td><td>0.00174</td></tr></table>

Bias of the attribution map at the pixel level on VOC2012-AlexNet

Table content: <table><tr><td>Method</td><td>Grad CAM</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td><td>LRP</td></tr><tr><td>top-10%</td><td>0.05403</td><td colspan="3">0.00744 0.00505 0.00646</td><td>0.00845</td><td colspan="2">0.00854 0.00453</td></tr><tr><td>top-30%</td><td>0.03467</td><td colspan="3">0.00434 0.00243 0.00296</td><td>0.00447</td><td colspan="2">0.00598 0.00197</td></tr><tr><td>top-50%</td><td>0.02459</td><td colspan="3">0.00298 0.00149 0.00182</td><td>0.00295</td><td colspan="2">0.00443 0.00119</td></tr><tr><td>top-70%</td><td>0.01711</td><td colspan="3">0.00176 0.00094 0.00120</td><td>0.00209</td><td colspan="2">0.00329 0.00080</td></tr><tr><td>top-90%</td><td>0.01115</td><td colspan="3">0.00060 0.00035 0.00058</td><td>0.00140</td><td colspan="2">0.00222 0.00043</td></tr><tr><td>bottom-10%</td><td>0.01026</td><td colspan="3">0.00906 0.00638 0.00876</td><td>0.00438</td><td colspan="2">0.00466 0.00430</td></tr><tr><td>bottom-30%</td><td>0.00978</td><td colspan="3">0.00534 0.00327 0.00400</td><td>0.00215</td><td colspan="2">0.00267 0.00202</td></tr><tr><td>bottom-50%</td><td>0.00803</td><td colspan="3">0.00372 0.00215 0.00253</td><td>0.00131</td><td colspan="2">0.00165 0.00129</td></tr><tr><td>bottom-70%</td><td>0.00610</td><td colspan="3">0.00239 0.00151 0.00178</td><td>0.00075</td><td colspan="2">0.00103 0.00091</td></tr><tr><td>bottom-90%</td><td>0.00690</td><td colspan="3">0.00124 0.00093 0.00111</td><td>0.00030</td><td colspan="2">0.00104 0.00056</td></tr></table>

Bias of the attribution map at the pixel level on VOC2012-VGG16

Table content: <table><tr><td>Method</td><td>Grad CAM</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td><td>LRP</td></tr><tr><td>top-10%</td><td>0.02984</td><td colspan="3">0.00764 0.00656 0.00563</td><td>0.00768</td><td colspan="2">0.00864 0.00461</td></tr><tr><td>top-30%</td><td>0.01876</td><td colspan="3">0.00405 0.00292 0.00253</td><td>0.00310</td><td colspan="2">0.00554 0.00159</td></tr><tr><td>top-50%</td><td>0.01365</td><td colspan="3">0.00267 0.00176 0.00154</td><td>0.00187</td><td colspan="2">0.00387 0.00092</td></tr><tr><td>top-70%</td><td>0.00947</td><td colspan="3">0.00162 0.00115 0.00098</td><td>0.00132</td><td colspan="2">0.00272 0.00065</td></tr><tr><td>top-90%</td><td>0.00650</td><td colspan="3">0.00061 0.00051 0.00038</td><td>0.00089</td><td colspan="2">0.00172 0.00044</td></tr><tr><td>bottom-10%</td><td>0.00767</td><td colspan="3">0.00869 0.00768 0.00865</td><td>0.00396</td><td colspan="2">0.00580 0.00288</td></tr><tr><td>bottom-30%</td><td>0.00708</td><td colspan="3">0.00483 0.00372 0.00406</td><td>0.00173</td><td colspan="2">0.00344 0.00117</td></tr><tr><td>bottom-50%</td><td>0.00654</td><td colspan="3">0.00330 0.00238 0.00261</td><td>0.00106</td><td colspan="2">0.00223 0.00072</td></tr><tr><td>bottom-70%</td><td>0.00540</td><td colspan="3">0.00219 0.00170 0.00185</td><td>0.00075</td><td colspan="2">0.00153 0.00054</td></tr><tr><td>bottom-90%</td><td>0.00509</td><td colspan="3">0.00120 0.00107 0.00122</td><td>0.00041</td><td colspan="2">0.00114 0.00040</td></tr></table>

Bias of the attribution map at the pixel level on VOC2012-VGG19

Table content: <table><tr><td>Method</td><td>Grad CAM</td><td>Grad</td><td>GI</td><td>GB</td><td>DeepSHAP</td><td>LIME</td><td>LRP</td></tr><tr><td>top-10%</td><td>0.02564</td><td colspan="3">0.00761 0.00672 0.00580</td><td>0.00802</td><td colspan="2">0.00851 0.00468</td></tr><tr><td>top-30%</td><td>0.01991</td><td colspan="3">0.00403 0.00301 0.00261</td><td>0.00337</td><td colspan="2">0.00565 0.00157</td></tr><tr><td>top-50%</td><td>0.01517</td><td colspan="3">0.00263 0.00181 0.00159</td><td>0.00205</td><td colspan="2">0.00402 0.00091</td></tr><tr><td>top-70%</td><td>0.01139</td><td colspan="3">0.00161 0.00118 0.00102</td><td>0.00143</td><td colspan="2">0.00282 0.00064</td></tr><tr><td>top-90%</td><td>0.00799</td><td colspan="3">0.00062 0.00053 0.00043</td><td>0.00096</td><td colspan="2">0.00183 0.00045</td></tr><tr><td>bottom-10%</td><td>0.00932</td><td colspan="3">0.00867 0.00782 0.00871</td><td>0.00442</td><td colspan="2">0.00587 0.00236</td></tr><tr><td>bottom-30%</td><td>0.00764</td><td colspan="3">0.00480 0.00378 0.00406</td><td>0.00193</td><td colspan="2">0.00352 0.00095</td></tr><tr><td>bottom-50%</td><td>0.00683</td><td colspan="3">0.00326 0.00243 0.00260</td><td>0.00120</td><td colspan="2">0.00231 0.00059</td></tr><tr><td>bottom-70%</td><td>0.00630</td><td colspan="3">0.00217 0.00173 0.00184</td><td>0.00084</td><td colspan="2">0.00140 0.00046</td></tr><tr><td>bottom-90%</td><td>0.00649</td><td colspan="3">0.00119 0.00109 0.00120</td><td>0.00043</td><td colspan="2">0.00107 0.00036</td></tr><tr><td colspan="5">F ATTRIBUTION MAPS OF MASKED IMAGES</td><td/><td/><td/></tr><tr><td colspan="6">G DETAILED RESULTS OF THE MUTUAL VERIFICATION</td><td/><td/></tr></table>

Mutual verification on VOC2012-VGG19

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td><td>LRP</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.6661 1.4054</td><td>1.4177</td><td colspan="2">1.4142 1.4375</td></tr><tr><td>GI</td><td colspan="3">1.6661 0.0000 1.4159</td><td>1.4008</td><td colspan="2">1.4140 1.3437</td></tr><tr><td>GB</td><td colspan="3">1.4054 1.4159 0.0000</td><td>1.5834</td><td colspan="2">1.4244 1.5242</td></tr><tr><td colspan="4">DeepSHAP 1.4177 1.4009 1.5835</td><td>0.0000</td><td colspan="2">1.3691 1.1106</td></tr><tr><td>LIME</td><td colspan="3">1.4142 1.4141 1.4245</td><td>1.3691</td><td colspan="2">0.0000 1.3874</td></tr><tr><td>LRP</td><td colspan="3">1.4375 1.3437 1.5243</td><td>1.1106</td><td colspan="2">1.3874 0.0000</td></tr></table>

Mutual verification on VOC2012-VGG16

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td><td>LRP</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.6647 1.3992</td><td>1.4302</td><td colspan="2">1.4142 1.4621</td></tr><tr><td>GI</td><td colspan="3">1.6647 0.0000 1.4191</td><td>1.3711</td><td colspan="2">1.4139 1.2649</td></tr><tr><td>GB</td><td colspan="3">1.3992 1.4191 0.0000</td><td>1.5839</td><td colspan="2">1.4260 1.5320</td></tr><tr><td colspan="4">DeepSHAP 1.4302 1.3712 1.5840</td><td>0.0000</td><td colspan="2">1.3824 1.0636</td></tr><tr><td>LIME</td><td colspan="3">1.4142 1.4140 1.4261</td><td>1.3825</td><td colspan="2">0.0000 1.3960</td></tr><tr><td>LRP</td><td colspan="3">1.4621 1.2649 1.5321</td><td>1.0636</td><td colspan="2">1.3960 0.0000</td></tr></table>

Mutual verification on VOC2012-AlexNet

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="3">DeepSHAP LIME</td><td>LRP</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.5540 1.3477</td><td/><td>1.4348</td><td>1.4148 1.4851</td></tr><tr><td>GI</td><td colspan="3">1.5541 0.0000 1.4344</td><td/><td>1.3224</td><td>1.4113 0.8341</td></tr><tr><td>GB</td><td colspan="3">1.3477 1.4344 0.0000</td><td/><td>1.4477</td><td>1.4146 1.4573</td></tr><tr><td colspan="4">DeepSHAP 1.4349 1.3225 1.4478</td><td/><td>0.0000</td><td>1.3221 1.2251</td></tr><tr><td>LIME</td><td colspan="3">1.4149 1.4115 1.4147</td><td/><td>1.3222</td><td>0.0000 1.3916</td></tr><tr><td>LRP</td><td colspan="3">1.4851 0.8341 1.4573</td><td/><td>1.2250</td><td>1.3915 0.0000</td></tr><tr><td colspan="7">Table 17: Mutual verification on VOC2012-ResNet50</td></tr><tr><td>Method</td><td>Grad</td><td>GI</td><td colspan="2">GB</td><td colspan="2">DeepSHAP LIME</td></tr><tr><td>Grad</td><td colspan="4">0.0000 1.6285 1.4143</td><td>1.6242</td><td>1.4143</td></tr><tr><td>GI</td><td colspan="4">1.6285 0.0000 1.4137</td><td>0.6537</td><td>1.4139</td></tr><tr><td>GB</td><td colspan="4">1.4143 1.4136 0.0000</td><td>1.4137</td><td>1.4238</td></tr><tr><td colspan="5">DeepSHAP 1.6242 0.6537 1.4137</td><td>0.0000</td><td>1.4139</td></tr><tr><td>LIME</td><td colspan="4">1.4144 1.4141 1.4240</td><td>1.4140</td><td>0.0000</td></tr></table>

Mutual verification on VOC2012-ResNet101

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.6877 1.4136</td><td>1.6428</td><td>1.4145</td></tr><tr><td>GI</td><td colspan="3">1.6877 0.0000 1.4140</td><td>0.6778</td><td>1.4139</td></tr><tr><td>GB</td><td colspan="3">1.4136 1.4140 0.0000</td><td>1.4140</td><td>1.4275</td></tr><tr><td colspan="4">DeepSHAP 1.6428 0.6778 1.4140</td><td>0.0000</td><td>1.4138</td></tr><tr><td>LIME</td><td colspan="3">1.4146 1.4140 1.4277</td><td>1.4139</td><td>0.0000</td></tr><tr><td colspan="6">Table 19: Mutual verification on CIFAR10-ResNet56</td></tr><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.4183 1.4107</td><td>1.4135</td><td>1.4142</td></tr><tr><td>GI</td><td colspan="3">1.4183 0.0000 1.4140</td><td>1.4104</td><td>1.4140</td></tr><tr><td>GB</td><td colspan="3">1.4107 1.4140 0.0000</td><td>1.4128</td><td>1.3879</td></tr><tr><td colspan="4">DeepSHAP 1.4135 1.4104 1.4128</td><td>0.0000</td><td>1.4097</td></tr><tr><td>LIME</td><td colspan="3">1.4142 1.4140 1.3879</td><td>1.4098</td><td>0.0000</td></tr></table>

Mutual verification on CIFAR10-ResNet44

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.3999 1.4128</td><td>1.4147</td><td>1.4141</td></tr><tr><td>GI</td><td colspan="3">1.3999 0.0000 1.4125</td><td>1.4114</td><td>1.4143</td></tr><tr><td>GB</td><td colspan="3">1.4128 1.4125 0.0000</td><td>1.4152</td><td>1.4175</td></tr><tr><td colspan="4">DeepSHAP 1.4147 1.4114 1.4152</td><td>0.0000</td><td>1.4097</td></tr><tr><td>LIME</td><td colspan="3">1.4142 1.4144 1.4152</td><td>1.4175</td><td>0.0000</td></tr><tr><td colspan="6">Table 21: Mutual verification on CIFAR10-ResNet32</td></tr><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="2">DeepSHAP LIME</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.4259 1.4129</td><td>1.4117</td><td>1.4143</td></tr><tr><td>GI</td><td colspan="3">1.4259 0.0000 1.4164</td><td>1.4113</td><td>1.4145</td></tr><tr><td>GB</td><td colspan="3">1.4129 1.4164 0.0000</td><td>1.4144</td><td>1.4111</td></tr><tr><td colspan="4">DeepSHAP 1.4117 1.4113 1.4144</td><td>0.0000</td><td>1.4117</td></tr><tr><td>LIME</td><td colspan="3">1.4144 1.4145 1.4112</td><td>1.4117</td><td>0.0000</td></tr></table>

Mutual verification on CIFAR10-ResNet20

Table content: <table><tr><td>Method</td><td>Grad</td><td>GI</td><td colspan="2">GB</td><td colspan="2">DeepSHAP LIME</td></tr><tr><td>Grad</td><td colspan="4">0.0000 1.3895 1.4010</td><td>1.4134</td><td>1.4143</td></tr><tr><td>GI</td><td colspan="4">1.3895 0.0000 1.4140</td><td>1.4133</td><td>1.4145</td></tr><tr><td>GB</td><td colspan="4">1.4010 1.4140 0.0000</td><td>1.4164</td><td>1.4382</td></tr><tr><td colspan="5">DeepSHAP 1.4134 1.4133 1.4164</td><td>0.0000</td><td>1.4118</td></tr><tr><td>LIME</td><td colspan="4">1.4144 1.4145 1.4382</td><td>1.4118</td><td>0.0000</td></tr><tr><td colspan="7">Table 23: Mutual verification on CIFAR10-LeNet</td></tr><tr><td>Method</td><td>Grad</td><td>GI</td><td>GB</td><td colspan="3">DeepSHAP LIME</td><td>LRP</td></tr><tr><td>Grad</td><td colspan="3">0.0000 1.5079 1.1955</td><td/><td>1.4352</td><td>1.4130 1.4904</td></tr><tr><td>GI</td><td colspan="3">1.5079 0.0000 1.4714</td><td/><td>1.3025</td><td>1.4003 0.7179</td></tr><tr><td>GB</td><td colspan="3">1.1955 1.4714 0.0000</td><td/><td>1.4329</td><td>1.4249 1.4881</td></tr><tr><td colspan="4">DeepSHAP 1.4352 1.3025 1.4329</td><td/><td>0.0000</td><td>1.3958 1.3063</td></tr><tr><td>LIME</td><td colspan="3">1.4131 1.4004 1.4249</td><td/><td>1.3959</td><td>0.0000 1.3805</td></tr><tr><td>LRP</td><td colspan="3">1.4904 0.7179 1.4881</td><td/><td>1.3063</td><td>1.3804 0.0000</td></tr></table>

Peer review (iclr2020):
The manuscript proposes an evaluation methodology to assess objectively methods for model explanation without the need of ground truth. This methodology is based on four metrics evaluating different aspects of the generated explanation. These four metrics evaluated the bias of the explanation map at the pixel level, quantify the unexplainable features components, robustness of the explanation to spatial masking, and mutual verification of different explanation methods. Experiments on the CIFAR10 an Pascal VOC 2012 considering a good set of representative explanation methods show the strenghts and weaknesses of the proposed method. Given the recent proposal of a significant number of works aiming a model explanation, the manuscript touches the critical point of assessing objectively the performance of these methods. Overall the manuscript has a good presentation, and its content, to great extent, is clear and easy to follow and good formal preentation of the proposed method is given. I appreciate that the experiments section covers two datasets and various landmark methods for model explanation. My main concerns with the manuscript are as follows: In several parts of the manuscript they were sentences refering to some explanation methods as ables to reflect/diagnose the *logic of a CNN/model/network*. I could not help but all the time that I encountered the expression *logic* ideas related to reasoning or internal decision-making process came to my mind. This really affected the flow of the content, and with all due respect, none of the explanations generated by existent methods is able to faithfully reflect the decision-making process of the models being explained. The majority are oriented towards generating approximations and indicating relevant regions of the input data. Along the direction of objective evaluation of explanation methods, Oramas et al. ICLR*19 and more recently Yang and Kim, arXiv:1907.09701 proposed methodologies aimed at quantitative evaluations of explanation methods. Given the common objectives that these works have with the manuscript, it would be strenghten the manuscript to positition itself wrt. these methods in the related work section. In Sec.3.3 the proposed method quantifies the amount of unexplainable regions, i.e. regions with low absolute attribution, of different methods. However, unless I missed something, it seems that these unexplainable regions are native of the explanation methods but native by the masking step introduced by the proposed metric. In addition, this metric relies on a pre-defined threshold which determines the number of pixels to be considered. Could you motivate how this threshold value was selected? what would be the effect of selecting different values? The metric proposed in Sec. 3.4 works under the assumption that a explanation method which is robust to spatial masking, can be considered more convincing. More specifically, it assumes that explanations from occluded versions of the same input should be similar. Do this metric takes into account the prediction produced by the model to be explained? if these predictions differ, assumming that the explanations should be similar might not be correct. Could you comment on this? The multual verification metric proposed in Sec. 3.5: assumes that methods for explanation are more reliable if they produce similar heatmaps. I think this assumption might be incorrect. For instance, Adebayo et al., NIPS*18 showed that several methods were able to produce very similar explanations. Yet, all these explanations were wrong since they were all biases towards edge-like structures of the input data and were not faithful to the decision-making process of the model being explained. Under the proposed mutual verification metric, these biased methods would have been flagged as objectively reliable. In the evaluation, when conducting experiments on PascalVOC 2012, could you motivate why cropping the images is necessary? Isn*t this removing some background clutter that could pose an interesting scenario on which the performance of explanation methods could be analyzed? The manuscript had a good detailed and motivated beginning. In comparison to its beginning its conclusion was somewhat abrupt. Discussions presented in the evaluation section related to the four proposed metrics were very shallow and most of the time pointed towards the supplementary material. In this regard, it is not clear how much of an insight we got from the evaluated explanation methods. Moreover, having this relevant part in the supplementary, gives the impression that the manuscript is not self-contained. Perhaps some re-organization of the content might be needed. Finally, the manuscript claims the proposed method to be sufficiently general to be applied to different method. However, as was evidenced in the experiments, there were some methods, e.g. CAM, grad-CAM, Pert, where some of the proposed metrics were not applicable. Therefore the claims related to the generality of the proposed method should be relaxed.