An evaluation of quality and robustness of smoothed explanations
Authors: Ahmad Ajalloeian
Seyed-Mohsen Moosavi-Dezfooli
Michalis Vlachos
Pascal Frossard
Venue: ICLR 2022 Submitted (2022)
Keywords: Explanation methods
Interpretability
Robustness
Adversarial attacks

Abstract: Explanation methods play a crucial role in helping to understand the decisions of deep neural networks (DNNs) to develop trust that is critical for the adoption of predictive models. However, explanation methods are easily manipulated through visually imperceptible perturbations that generate misleading explanations. The geometry of the decision surface of the DNNs has been identified as the main cause of this phenomenon and several \emph{smoothing} approaches have been proposed to build more robust explanations.
In this work, we provide a thorough evaluation of the quality and robustness of the explanations derived by smoothing approaches. Their different properties are evaluated with extensive experiments, which reveal the settings where the smoothed explanations are better, and also worse than the explanations derived by the common Gradient method. By making the connection with the literature on adversarial attacks, we further show that such smoothed explanations are robust primarily against additive $\ell_p$-norm attacks. However, a combination of additive and non-additive attacks can still manipulate these explanations, which reveals shortcomings in their robustness properties.

Under review as a conference paper at ICLR 2022
AN EVALUATION OF QUALITY AND ROBUSTNESS OF SMOOTHED EXPLANATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Explanation methods play crucial role in helping to understand the decisions of deep neural networks (DNNs) develop trust that is critical for the adoption of predictive models. However, explanation methods are easily manipulated through visually imperceptible perturbations that generate misleading explanations. The geometry of the decision surface of the DNNs has been identified as the main cause of this phenomenon and several smoothing approaches have been proposed build more robust explanations. In this work, we provide thorough eval- uation of the quality and robustness of the explanations derived by smoothing approaches. Their different properties which reveal the settings where the smoothed explanations are better, and also worse than the explanations derived by the common Gradient method. By mak- ing the connection with the literature on adversarial attacks, we further show that such smoothed explanations are robust primarily against additive fp-norm attacks. However, a combination of additive and non-additive attacks can still manipulate these explanations, which reveals shortcomings in their robustness properties.
evaluated with extensive experiments,
INTRODUCTION
Explanation methods attribute numerical value each data feature in order quantify its relative importance towards the model's prediction. Such attributions help to better understand and trust complex models like deep neural networks (DNNs). safety-critical tasks, such an understanding is prerequisite to the deployment of DNNs, because domain expert will never make important decisions based on a model's prediction unless that model is trustworthy. Moreover, explanations can help to understand the reasons behind the decision of model, and when it comes to model debugging, they can reveal the presence of any spurious data correlations that may lead to faulty predictions during inference (Ribeiro et al. 2016).
In the context of image classification with deep neural networks, several explanation methods have been proposed based the gradient with respect to input, also called cradient-based explana- tions (Bachrens et al.] 22010) Bach et al.] [001 Selvaraju et al., 2017;; Sundararajan et al., 2017] Springenberg et al.| 2015). The explanation generated by these methods, saliency map, highlights the parts of the image that contributed to the prediction. Recent work has shown that gradient-based explanations of neural networks can be fragile and can be easily manipulated via adversarially per-
turbed inputs (Ghorbani et al., [2019] Dombrowski et al. 2201 Heo et al. 2019 Viering et al.) 2019) Kindermans et al. 2019). That one can find a small-norm perturbation be added an input ( often imperceptible), such that the focus of the explanation changes towards irrelevant features while the model's output remains unchanged. This, in turn, can make explanations inappropriate to help end-users gain trust in a model's 's prediction.
The large curvature of the decision surface of neural networks has been identified as one of the causes of fragility for gradient-based explanations (Ghorbani et al. 2019 Dombrowski et al.. 2019) Wang et al., 20200). To make explanations more robust, a class of approaches proposed smooth- ing the explanation or making the decision surface of neural networks more smooth (Wang et al.) 2020 Dombrowski et al. 2019 Ivankay et al., 2020). We refer to these approaches as smoothing approaches. It is worth mentioning that similar methods have been proposed in the context of ad- versarial robustness, with the aim of flattening the decision surface neural networks in order to
reach more robust predictions (Moosavi-Deztooli et al. 2019 Qin et al. 2019).

---

Under review as a conference paper at ICLR 2022
Here, we provide thorough investigation of the explanations derived by smoothing approaches in terms of explanation quality and robustness. We employ various tests to assess the quality of these explanations. Each test evaluates a desirable property for explanations, such as: sensitivity to changes in the model, fidelity to the predictor function, etc. In terms of robustness, we show that explanations derived by smoothing approaches only provide robustness against additive Ep norm at- tacks. Specifically, in this work, we show that compared to additive attacks, attacks based on the combination of spatial transformation (Xiao al., 2018) and/or color transformation (Laidlaw & Feizi 2019) together with additive perturbations are more effective in manipulating these explana- tions. Our contributions can be summarized as follows:
We study the effectiveness of smoothing approaches to achieve robust explanations. We present results on evaluating both the quality and robustness properties of smoothed expla- nations.
We assess the quality of smoothed explanations via presenting the results of various quality tests. Our results demonstrate the pros and cons of smoothed explanations with respect to the following quality aspects: sensitivity to model parameters, class discriminativeness, Infidelity, and sparseness.
We present results for different combination of additive and non-additive attacks, and show that they are able manipulate explanations derived by smoothing approaches more suc- cessfully. Combining different types of perturbations to achieve stronger attacks has been topic of investigation in the context of adversarial examples (Jordan et al.) 2019). To the best of our knowledge, this is the first time such attacks have been used in the context of explanations.
works mostly focused on either modifying the explanation method itself or modifying the predictor Related works. There have been several works aiming to make explanations more robust. These
model to achieve robust explanations. WWng et al. (2020) introduced Uniform Gradient, which is similar to Smooth Gradient unless it uses Uniform noise, and showed that it can hardly be manipu- lated by additive attacks. Dombrowski al. (2199) proved that network with soft-plus activations has a more robust Gradient explanation compared to a ReLU network, given that the parameter 3 of the soft-plus function chosen to be sufficiently small. Consequently, they proposed the 3-smoothing approach in which they substitute the ReLU activations of a trained network by soft- plus functions with a small /3 parameter, Wang et al. (020) introduced a regularization term called Smooth Surface Regularization (SSR) to the training objective of a DNN. This training objective penalizes the large curvature of a DNN by regularizing the eigenvalue of the input hessian with the maximum absolute value. Moreover, they showed that adversarial training (Madry et al.) (018) also leads to more robust explanations. This fact can also be deduced from the results of (Moosavi- Dezfooli et al. (219)) they showed that adversarial training leads to significant decrease in the curvature of the loss surface with respect to inputs. Anders et al. (2020) proposed an attack in which they adversarially manipulate the model instead of the input in order to manipulate the explanation. Then they propose a modification to the existing explanation methods to make them more robust against such manipulated models. Lakkaraju et al. (2020) proposed a framework for generating ro- bust and stable black box explanations based on adversarial training. Chen et al. (2019) introduced a regularization term to the training objective of neural networks achieve robust Integrated Gradient explanations. Finally, Dombrowski et al. (2020) developed a theoretical framework to derive bounds on the maximum maniuubality of explanations and proposed three different techniques boost the robustness of explanations. In this work, we show that the robustness of smoothed explanations can be affected by employing a combination of additive and non-additive attacks. Furthermore, we present through evaluation of the differrent quality aspects of smoothed explanations.
2 BACKGROUND
First, provide the definition of explanation map and then briefly describe the explanation methods we used in this paper. Then we continue with introducing the attacks to explanations and the smoothing approaches we going study in this paper.
Consider a model f : Id + RK which classifies an input x e Rd into one of the K classes. An explanation map, denoted by hy(x) : RRd Rd, associates score each feature of the input
2

---

Under review as a conference paper at ICLR 2022
indicating the relevance of that feature towards the model's prediction. For instance, in the context of image classification, saliency maps associate a score to each pixel of the imput image resulting in heatmap that highlights important regions of the image leading the model prediction. In this work, we focus on the gradient-based explanations and mainly on the Gradient method. Given a model and an input x, the Gradient explanation is defined (x)(x). Since other gradient-based explanation methods make use of the gradients with respect to input, we argue that our results could be extended those explanation methods as well. We will also consider two smoothed variants, namely Smooth (Smilkov et al. 2017) and Uniform Gradient (Wang et al. 2020) methods.
2.1 ATTACKS TO MANIPULATE EXPLANATIONS
Similarly to common adversarial attacks (Goodfellow et al. 2015 Moosavi-Dezfooli et al. 2016 Szegedy et al., 2014), recent work has shown that explanations can also be manipulated by adding a small and almost imperceptible perturbation the input (Ghorbani et al. 2019: Dombrowski al.) 2019). We refer to this class of attacks as explanation altacks. There have been various formulations for explanation attacks (Ghorbani et al. 2019 Dombrowski et al., 2019). In this work, we will use the formulation introduced by Dombrowski et al. (2019). In this attack, the attacker tries to find a perturbed input for which the explanation is manipulated to very similar to given target explanation map while the output of the model remains approximately unchanged. Note that the target map could be any heatmap in general: however, we used the explanation of a target image as target map in this work. Below, we will give formal definition of this attack.
Definition 1 (Targeted manipulation attack). An explanation hy(x) for model f(x) vulnerable to attack at input x if there exist perturbed input Xedv, such that (Xade) is similar to given target map he but the model's ' output remains unchanged. An attucker finds Xado by minimizing the following objective function: finction:
L = =l - 1011 (Xfd)) -
-f f(x)]] + +22Freg Xado)
(1)
The first term in 4 ensures the similarity of the manipulated explanation to the target map, the sec- ond term ensures the similarity between the model output for the original and perturbed inputs, and the third term regularizes the perturbation to ensure perceptual similarity between the original and perturbed images. Note that Lyeg is defined by the attacker according to the type of the perturbation. The relative weighting of the terms in is controlled by the hyper-parameted 7/1 and 72-
2.2 TOWARDS ROBUST EXPLANATIONS
Recent works have tried to define the robustness of explanations in terms of the sensitivity of input gradients to changes in the input data (Wang et al., 2020 Dombrowski et al., 2019).. Wang et al. (2020) define the robustness of explanations by the Lipschitz continuity coefficient of the input gradients; a smaller coefficient means that the explanation is less sensitive to the changes in the input and hence more robust. In this regard, class of approaches to generate robust explanations have been proposed in the recent works, which are either based on smoothing out the explanation maps or flattening the decision boundary of the model itself. Broadly, these approaches can be classified (1) Post-hoc approaches do not require retraining of the network and can be applicd as a post- posst-sccssiigg step. (2) Ad-hoc approaches to robust explanations require retraining of the network and hence are more cosily.
into two categories:
In this work, we consider Smooth Gradient (Smilkov et al. 2017), Uniform Gradient (Wang et al, 2020), and 3-smoothing (Dombrowski et al. (219) as post-hoc approaches. The first two methods involve smoothing the explanation map, while the third one smooths the decision surface of the model. All three approaches act on pre-trained models, and hence characterized as post-hoc. Among the ad-hoc methods, we study the explanations generated by adversarially trained networks, and networks trained with curvature regularization (CURE) (Moosavi-Dezfoolli al. 2019). which is a similar approach to SSR (Wang et al. 2020
We experiment only with CURF, because with the publicly available code of SSR were not able to reproduce the results in (Wang et al. 2020)
3

---

Under review as a conference paper at ICLR 2022
3 EVALUATING POST-HOC APPROACHES
Here, we begin by evaluating the quality of explanations derived by post-hoc approaches that do require retraining of the network. Then, we evaluate the robustness of these explanations by presenting results on effective non-additive attacks to manipulate them. For all of the experiments in this section, we used VGG-16 network trained on ImageNet (Russakovsky al. 2015), and for generating the explanation maps we used the Captum (Kokhlikyan et (2020) package. Moreover, for the B B-sooothing approach we always set 0.8 as suggested in (Dombrowski al. 2019).
3.1 QUALITY OF EXPLANATIONS OF POST-HIC APPROACHES
To evaluate and compare the quality of the explanations, the literature. In general, assessing the quality of an explanation is a challenging task and each quality test only evaluates specific quality aspect of an explanation. Therefore the assemblage of quality tests helps to understand which quality aspects of the explanations are improved and which deteriorated by the smoothing approaches.
various quality tests presented in
ONP estanatn
Cascade randomization from top bottom layurs
10 Dogiol
Grddeett
Smooth
Unioomm Sondiett
Sammothing
Figure I: Cascade randomization of the VGG-16 (ImageNet) layers. The first column shows the original explanation for the image "Jay" bird. Each subsequent column shows the effect of random- ization of the parameters of the network up to that layer (inclusive) on the explanations.
Cascade randomization of model parameters. Adebayo et al. (0018) argued that it is desired for explanation be sensitive to the changes in the model parameters. They proposed model parameter randomization test to assess this sensitivity. In this test, the parameters of model are progressively randomized from the top layer (logits) to the bottom layers. In each step of ran- domization, the explanation from the resulting model is compared against the explanation from the original model. Randomizing the model parameters means losing what the model has learned from the data during training. Therefore, we expect "good" explanation to be destroyed in this process. However, an explanation is insensitive to the randomization the model parameters, then it not deemed appropriate for debugging the model under erroneous predictions.
The visual results of this test for Gradient explanation and post-hoc approaches are shown in Figure More examples of this test can be found in the Appendix. One can observe that the explanations derived from post-hoc approaches show less sensitivity to the randomization of model parameters than compared to the Gradient method. This also be verified by the Spearman rank correla- tion between the original and randomized explanations shown in Figure [ We observe that for the smoothed explanation methods, the original and randomized explanations have high rank corre- lation after the randomization of the top layers of the network. These results highlight that using Smooth Gradient, Uniform Gradient, and B-smoothing to achieve a more robust explanation can come at the expense of having explanations that are less sensitive to model parameters.
Class sensitivity explanations. A good visual explanation should be able to localize the image regions relevant to the target category, i.e., it should be class discriminative (Selvaraju et al. 2017 This is particularly significant when dealing with images containing more than one object. To as- sess the class discriminativeness of an explanation game (Zhang et al.) 2016). We sampled images from the MS COCO dataset (Lin et al., 2014), con- taining two objects that are also present among the ImageNet class labels. For this test only keep
used quality test equivalent to the pointing
4

---

Under review as a conference paper at ICLR 2022
1.0
Gradient
Smooth Gradient
Bank correlation
Uniform Gradient
P smoothing Gradlent
0.6
0.4
0.2
0.0
no rand
fcB (logits)
1c7
feG
conv 5-3
conv 5-1
cone 4-3
cone 4-2
comv 4-1
conv 3-3
conn 3-1
conv 2-2
conv 2-1
cone 1-2
con 1-1
conv
conv
Figure 2: Spearman rank correlation between the original and randomized explanation derived for randomization up to the layer indicated by the x-axis. A higher rank correlation value indicates higher similarity, i.c, the higher the curve of an explanation method, the less sensitive it is to model parameters. The results are averaged over 1000 Images from Imagenet and the shaded area around each curve indicates the standard deviation.
Table 1: Ratio of the explanation top-20 values included in the segmentation mask of each object. Note that the columns "obj 1" and "obj 2" are for the explanations computed for the top predicted class and the class corresponding object in the image respectively. The results are averaged over 60 samples.
Dog
the second
Tiger
Explanation Method
obj
obj
Gradient Smooth Gradient
0.6
0.49
0.54
0.37
Figure 3: An example of 3-smoothing explana- tion generated for the target category "dog" and "cat" (left), and the segmentation mask of each object from the COCO dataset (right).
Uniform Gradient
0.57
0.38
B-smoothing Gradient
0.62
0.45
the samples for which one of the objects in the image is the top predicted class by the network and the other object is among the top 20 predicted classes by the network. We compute the explanation maps for each of the class labels corresponding to the objects. Using the segmentation mask of the objects provided in the dataset as ground truth, we compute what percentage of the top-20 values in the explanation maps generated for each target category tion masks. The results of this test are shown in table [ and a visual depiction of this test is given in Figure 3 These results indicate that the smoothed explanation methods are less discriminatory when generated for the target class label that has a lower probability. This suggests that in terms of class discriminativeness of explanations, the post-hoc smoothing approaches investigated in this paper are inferior to the Gradient method.
inside the corresponding segmenta-
Sparseness of explanations. To create explanations that are numan-accessibee is advantageous to have sparse explanation map (Molnar) 2019), i.c, only the features that are truly predictive of the model output should have significant contributions, and irrelevant features should have negli- gible contributions. Sparse explanations are more concise because they only include features with significant contribution making simpler for end-users to understand the reasons for a specific pre- diction of the model (Chalasani et al. 2020). To measure the sparseness of an explanation map, we applied the Gini Index on the absolute value of the flattened explanation maps. The Gini Index is a metric that measures the sparseness of vector with non-negative values (Hurley & Rickard 2009). By definition, the Gini Index take values in [0, 1] with higher values indicating more sparseness. Table 2 shows the average Gini Index of the Gradient, Smooth Gradient, Uniform Gradient, and
5

---

Under review as a conference paper at ICLR 2022
Table 2: Average Gini Index for the explanations of VGG-16 network (averaged over 1000 sam-
ples). A larger Gini Index indicates more sparseness and hence a more concise explanation.
Gradient
Smooth Gradient
Uniform Gradient
8-smoothing
Gini Index 0.56 + 0.038
0.34 + 0.053
0.35 + 0.058
0.65 + 0.067
Table Average Infidelity for the explanations of a VGG-16 network (averaged over 100 samples). A lower Infidelity value indicates better fidelity of the explanation to the predictor function.
Gradient
Smooth Gradient
Uniform Gradient
ß-smoothing
Infidelity 1.43 1.52
1.42 + 1.52
1.42 + 1.52
1.00 + 0.88
3-smoothing computed for 1000 randomly sampled images from ImageNet. The results show that compared to the Gradient method, Smooth Gradient and Uniform Gradient provide less concise ex- planations, whereas 3- s-mooothing actually improves the sparseness of the explanations as compared to the Grudient method.
Explanation Infidelity. Introduced in Yeh et al. (019)), this metric captures how the predictor function changes in response to significant perturbations to the input and is defined as the expected difference between the two terms: 1) the dot product of the input perturbation and the explanation
and 2) the difference between function values after significant perturbations to the input. The metric
generalizes the completeness axiom (Shrikumar et al. 2017 Sundararaaan et al., (017) because allows for different types of perturbations which could be of interest depending on the problem and
the dataset. We use the infidelity metric to compare the effect of post-hoc smoothing approaches on the fidelity of explanations to the predictor function. As suggested in (Yeh al., 2019), we used the square removal perturbation to compute the infidelity of explanations for randomly selected images from ImageNet. Table 3 shows the results for the post-hoc approaches. A lower infidelity value indicates better fidelity of the explanation to the predictor function. The results suggest that the degree of smoothing used to robustify explanations, also improves their infidelity Therefore with respect to the Infidelity metric, all of the smoothed explanations investigated in this section are superior to the Gradient method. This finding also line with the results of Yeh et al. 2019), i.e., that modest smoothing improves the infidelity of explanations.
3.2 ROBUSTNESS OF EXPLANATIONS OF POST-HOC APPROACHES
Now, will evaluate the robustness of Smooth Gradient, Uniform Gradient, and 8-smoothing explanations. We present attacks composed of additive and non-additive perturbations, and show that they are more effective than additive attacks manipulate explanations. The non-additive attacks we employed are spatial transformation attacks (Xiao et al. 2018), and recoloring attacks (Laidlaw) & Feizi| 2019). See the Appendix B for brief description of each of these attacks. In the rest of this paper, we refer to the additive attack as Delta, spatial transformation attack as StAdy, and recoloring attack as Recolor.
We used the projected gradient descent (PGD) algorithm to optimize the objective function I In our experiments, we evaluate three combinations of attacks, namely Delta, Delta+StAdv, and Delta=StAAd==Recolorr against the explanation of VGG-16 network trained on ImageNet (Rus-) sakovsky al., 2015). See Appendix C.I for the details about the foo norm for each type of the perturbations and the hyper-parameters used in each attack setting.
We use two metrics to evaluate the attacks: (1) The Cosine Distance metric (cosd) to evaluate the similarity between the target and manipulated explanations (Wang et al.) (020). A lower cosine dis- tance corresponds to a lower le distance between the target and manipulated explanations indicating higher similarity. The range of the values for cosd is between 0 and 1. (2) The LPIPS metric for
2 As discussed in Ghorbani et al, 2019
Dombrowski et al. 2019 to avoid zero-valued gradients when activation with its smooth approximation. In this work, we used
optimizing ( we have to replace the ReI.U soft-plus function with 100.

---

Under review as a conference paper at ICLR 2022
Delta
0.30
StAda+StAdV
cosd( (5))
StAdv Recolor
0.25
LPIPS x)
0.15
0.10
0.05
0.00
Gradient lundefended)
Smooth Gradient
Uniform Gradient
A-smoothing +Gradient
Gradien: (undefended)
Smooth Gradient
Uniform Gredient
B-smoothing GGradient
Figure 4: Evaluation with cosd (left) and L.PISS metries for attacks against post-hoe approaches. For both cosd and LPIPS, smaller values indicate higher similarity. All results are averaged over 200 images from ImageNet. The black bars indicate standard deviation.
quantifying the perceptual similarity between images (Zhang et al. 2018). A lower LPIPS value indicates higher similarity.
Figure 4 shows the cosine distance between the target and manipulated explanations, and the per- ceptual similarity (LPIPS) between the perturbed and original images for each attack setting. We can observe that Delta+StAdv, and Delta=SSAAv+RRcoll attacks more effective than Delta at- tacks to manipulate B-smoothing explanations, i.e, with a less perceptible perturbation (lower LPIPS value),
reach cosd value between manipulated and target explanations very close to the cosd value when attacking the Gradient method. The effect of the non non-additive attacks is less significant on the Smooth and Uniform Gradient methods, however we can still observe improvements in the cosd values under these attacks. Taken together, these results show that Smooth Gradient, Uniform Gradient, and B-smoothing explanations are more vulnurale to son-additive attacks and hence such attacks should be considered as threat to the robustness of these methods. As example, we can visually see the effectiveness of Delta+StAdy+ Recoolr attack against differnt explanation methods in Figure
Original explanations
Delta
Deltai StAdv: attack
Original image
Gosdient
Smooth Grndlent
Target image
Uniform Goddeet
p-moothing
Figure 5: An example visual comparisons of explanation attacks against different methods. Under each attack setting, the left column shows the perturbed image and the right column shows the corresponding perturbed explanation.
4 EVALUATING AD-HOC APPROACHES
Here, we recreate the experiments Section 3 for the ad-hoc approaches. We study the explana- tions of networks trained with curvature regularization (CURE) (Moosavi-Deztoolii al. 2019

---

Under review as a conference paper at ICLR 2022
and adversarial training (Madry al. 2018). Training with CURE, regularizes the eigenvalue of the input hessian with maximum absolute value and is similar to SSR, which was shown to improve the robustness of explanations against additive attacks (Wang et al. 2020). Adversarial training also smooths the decision surface and can provide more robust explanations.
For the experiments in this section, used ResNet-18 network trained with CURE and ad- versarially trained ResNet-16 18 network trained on adversarial examples with &oe norm of the pertur- bations upper bounded by 8/255 (Engstrom al. 2019). Both networks are trained CIFAR-I0 dataset (Krizhevsky 2012).
4.1 QUALITY OF EXPLANATIONS OF AD-HOC APPROACHES
Cascade randomization of model parameters. We evaluate the sensitivity of explanations of the networks trained with CURE and adversarial training using the cascade randomization of model parameters test.
The Spearman rank correlation between the original and randomized explanations is shown in Fig- ure These Results show the explanation of an adversarially trained network is less sensitive to model parameters. This suggests that the explanation of an adversarially trained network cannot be helpful debug a model when making wrong prediction.
Standard training
CURE
Rank correlation
0.8
Adverserial training
0.6
0.2
no rand
layer4-1
layeria
Layyes 1
12/4600
bnz.
Figure 6: Spearman rank correlation between the original and the randomized explanations (aver- aged over 1000 Images from CIFAR-10), A higher rank correlation value indicates a higher similar- ity, i.e the higher the curve of an explanation method, the less sensitive it is to model parameters.
Sparseness of explanations. We compare the sparseness of the explanations derived by ad-hoc approaches, using the Gini Index metric. Table 4 compares the Gini Index for the explanations of networks trained with different training objectives. These results show that adversarial training helps to improve the sparseness of explanations as compared to standard training. Hence the explanations of an adversarially trained network are more concise. This in line with the results of Chalasani al. (2020) as well. However, the rsults of Table 4 indicates that training a network with CURE does not help to improve the sparseness of explanations as compared to standard training.
Table Gini Index for the explanations of ResNet-18 network (averaged over 1000 images from CIFAR-10). A larger Gini Index suggests a more concise explanation.
Standard training
CURE
Adversarial training
Gini Index
0.54 + 0.035
0.54 + 0.045
0.71 + 0.054
Explanation Infidelity. To compare the fidelity of explanations derived by ad-hoc approaches to the predictor function, we used the Infidelity metric with square perturbation (Yeh et al.) 2019). Showss the results for randomly selected images from CIFAR-10. A lower infidelity value in- dicates better fidelity of the explanation to the predictor function. From these results, we can observe that training a network with CURE and adversarial training helps to improve the explanation Infi- delity. Therefore with respect to the Infidelity metric, the ad-hoc smoothing approaches investigated in this section improve the explanation Infidelity as compared to standard training.
8

---

Under review as a conference paper at ICLR 2022
Table 5: Infidelity of the explanations of a ResNet-11 network (averaged over 1000 images from CIFAR-10). A lower Infidelity value is better.
Standard training
CURE
Adversarial training
Infidelity
5.69 + 4.44
0.59 + 0.34
1.56 + 0.76
4.2 ROBUSTNESS OF EXPLANATIONS OF AD-HOC APPROACHES
Now, evaluate the improvement of robustness via ad-hoc approaches. We present results for Delta, Delta+StAdv, and Delta=StAdv+Reeoolo attacks against explanations of the networks trained with CURE and adversarial training. Figure 7 shows the results of these attacks. For the adver- sarially trained network, we can observe that non-additive attacks can more effectively manipulate explanations compared the additive attacks. However, even with the strongest attack setting we still cannot get close the cosd value reached by attacking the explanation of the network trained in standard way. For the attacks against the explanation of the network trained with CURE, the effect of non-additive attacks are less significant in terms of the cosd value, however can still observe that such attacks can reach similar cosd values with perceptually less visible perturbations.
0.200
Delta
DSAtd+
(h( h9)
0.30
Stlla+SUAdv+ Recolor
0.150
(Xays x)
0.25
0.125
0.100
0.15
0.10
0.05
0.025
0.00
0.000
Standard training
CURE
Adversarial training
Standard training
CURE
Adversarial
training
Figure 7: Evaluation with cosd (left) and LPIPS (right) metrics for the attacks against ad-hoc ap- proaches. For both cosd and LPIPS, smaller values indicate higher similarity. All results are aver- aged over 1024 images from CIFAR-10. The back bars indicate standard deviation.
5 CONCLUSION
We have evaluated two aspects smoothed explanations: a) explanation quality, and b) robustness of cxplanation. In terms of explanation quality, we performed thorough evaluation of four quality aspects: sensitivity to model parameters, class discrininativeness, sparseness, and infidelity. Our results show that the smoothed explanations investigated in this paper perform worse than those of the Gradient method in terms of sensitivity to model parameters and class discriminativenes.. On the other hand, show that using such smoothing methods helps improve explanation Infidelity and sparseness.
We further looked the robustness of explanations, when inputs are perturbed by combination of additive and non-additive attacks. To the best of our knowledge, this is the first time such attacks are used to manipulate explanations. Our experimental results highlighted the fact that non-additive attacks are still threat for explanation methods, including the smoothed ones. These results also point us the fact that many problems explanation robustness can be addressed by making analogies with the area of prediction robustness. As these two areas are closely related, the solutions already explored in prediction robustness can be potentially helpful to study explanation robustness. This will be the focus of our future work.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò

---

Under review as a conference paper at ICLR 2022
Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9525 9536, 2018. URL http://ps//ppoccceennns/narareea
/
Christopher J. Anders, Plamen Pasliev, Ann-Kathri Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fair- washing explanations with off-manifold detergent. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine URL http::/proceedingsmmprpppp
Learning Research, pp.
PMLR, 2020.
anders2Ua.htm.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojeiech Samek. On pixcl-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 1067):1-6,, 07 2015. doi: 10. (0.137//journal.pog) pone.013014...
URL https:
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. J. Mach. Learn. Res., 11:1803-1831 2010. URL
http: http://ppprtaa.....ooooo//aaaaaa
Prasad Chalasani, Jiefeng Chen, Amrita Ray Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of neural networks using adversarial training. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1383 1391. PMLR, 2020. URL http:::/proceedhiiitte
r.press/vll1
MwllasaniiOa.at..o
Jiefeng Chen, Xi Wu, Vaibhav Rastogi. Yingyu Liang and Somesh Jha. Robust attribution regulariza- tion. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florenee d' Alché-Buc, Emily B. Fox, and Roman Gamett (eds.), Advances Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver BC, Canada, pp. 14300-14310, 2019. URL https: http:///ppoceeee............. www.peeeingg....riiiii.....e..se hash/177ef5539tb44aaa htt.:/a4aa..aaa22222622222223330 120e1110c222 0tt.
Abstract.html
Ann-Kan Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Hanna M. Wallach, Hugo Larochelle. Alina Beygelzimer, Florence d' d'Alché Buc, Emily B. Fox, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural In- formation Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 13567- 2019. URL httthtt/////pperaaaa www.papppsssspsc.a..oo..pp pttcc/pppcrr.p//ppppppppppppp.pp http//ppper////// /951 -explexpaneee anaplaaatiinn ionationn--o wwwoonnnnn--n cann-caeeeeeee man pul ppl.ppulttttt atdllaned-d-dd-ae
Ann-Kathrin Dombrowski, Christopher J. Anders, Klaus-Robert Müller, and Pan Kessel. Towards Robust Explanations for Deep Neural Networks, arXiv e-prints, art, arXiv:2012.10425, December 2020.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL httttt:://gtthhheee httpt/uuu...ccccaaaattttaeee rww.usouuuuutto tness
Amirata Ghorbani, Abubakar Abid, and James Y. Zou. Interpretation neural networks fragile. In The Thirty-Thine AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applica- tions of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pp. 3681- 3688. AAAI Press, 2019. doi: 10.1609/aaai, 333101.30013681. URL
1aai.v33.00.3301368
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamesssing adversarial exam- ples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.ozg/ abs/ 1412.6372
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence A'Ahe Buc, Emily B. Fox, and Roman Garnett (eds.), J. Advances in Neural Information Processing Systems 32; Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pp. 2921-2932 2019. URL https: pttp://eeeeeeeeeeee.e.ppopppppee www.neerripp.ccppppprp..p Tfea637fd6d02e (tw.hea00aaaffffm www.eddd3--bstttcttt.t
htw.s/////ppprt//2//hhhhaa 9/hash/
10

---

Under review as a conference paper at ICLR 2022
Niall P. Hurley and Scott T. Rickard. Comparing measures of sparsity. IEEE Trans. Inf. Theory,
55(10)4473 4741, 2009.
doi: (0.1109/T1122002222753 URL /ttppss//dd...rrg/0011////
FU0.2009.2027522
Adam Ivankay, Ivan Girardi, Chiara Marchiori, and Pascal Frossard. FAR: A general framework for attribu- tional robustness. CoRR, abs/2010.07393,, 2020. Utt.hctps://aaxxiv.oo httpr////aa/22/200033303
Matt Jordan, Naren Manoj. Surbhi Goel, and Alexandros G. Dimakis. Quantitying perceptual distortion of adversarial examples. CoRR, abs/1902.08265. 2019, URL BtLLtttpp/a//.v..o./ http://r///a///12..000922 190...//299.2233333
aps
Pieter-Jan Kindermans. Sara Hooker, Julius Adebayo. Maximilian Alber, Kristof T. Schütt, Sven Dähne, Du- mitru Erham, and Been Kim. The (un)rcliability of saliency methods. Wojeicch Samek, Grégoire Mon- tavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (cds.), Explainable Al: Interprering, Explaining and Visualizing Deep Learning, volume 11700 of Lecture Notes in Computer Science, pp. 267- 280. Springer, 2019. doi: 10.100079778-3-00-288544 _,1,, URL httttps://do.ergg//// 3-030-28554-611
9978-
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexan- der Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion and generie model interpretability library for pytorch, 2020.
Captum: A unified
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Cassidy Laidlaw and Soheil Feizi. Functional adversarial attacks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Bue, Emily B. Fox. and Roman Garnetl (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, http://ppppcecccinnnnne..r....pe wttpcccccpppp///111111 2019/hahh////hhh
pp. 10408-10418, 2019. URL
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. In Pro- ceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings Machine Learning Research, pp. 5628-5638. PMLR, 2020. URL
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays. Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. arXiv e-prints, art. arXiv:1405.0312, May 2014.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver BC, Canada, April 30- May 3. 2018, Conference Track Proceedings. wttp///rrmmdddra/22...
OpenReview.net, 2018. U
Christoph Molnar. Interpretable Machine Learning. 2019.
Scyed-Mohsed Dezzooi- Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 2574-2582. IEEE Computer Society, 2016. doi: (T..///VR..000.22... U https: : www.oo...g///... wt..///..0000. 1011///001110///2200.0600
Seyed-Mohsen Moosavi-Dezfooll: Alhussein Fawzi. Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp- 9078-9086. Computer Vision Foundation / IEEE, 2019. doi: 10. N..1109/VVR.2019900 URL
http:://opeaaccss.theeee Dezfooli htw..ao.sssssssss.a..a.uauaueee.
wwt.c/cccccoonntt.coo wwwpeeetttcvcrcereee htt.9//htmm//oosav.ii. wwwp://www.nneeeeeeeeeeaaaaaaaea Str.eeeeeeerree......aapp.....ar paper.html
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local lineariza- tion. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-BBuc Emily B.
Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Conada, pp. 13833, 2019. / htt http/httss//p/rceediiie http::///oooceeegg.eeeeeee wwtp:ppppppppppppppppppppm hash ff 1 ed0 D.cccddbffffo 993e -

---

Under review as a conference paper at ICLR 2022
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. Balaji Krishnapuram. Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, Pp. 1135- 1144, ACM, 2016. /ttt:pp:/ddo.orggg./ http::/ddoooogggooo 0Tt...11110/99977622227777
doi: 10. 1155//99972..99999799 URL UtLpstts::/
Leonid L. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithuss. In Proceedings of the Eleventh Annual International Conference of the Center for Nonlinear Studies on Experimental Mathematix:: Computatoonal Isxues in Nonlinear Science: Computational Issues in Nonlinear Science, pp. 259-268, USA, 1992. Elsevier North-Holland, Inc.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjecv Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):111252 2015. doi: 10. 1007/s1 1263-015-0816-y.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEFE In- ternational Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 618- 0.11199/
626. IEEE Computer Society, 2017. doi: 10.1109/ 2017.74. URL /octps://doi..o ICCV.2011.74
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 1145-31552 PMLR, 2017. URL kttp: \/proceedi ngs.ml www.ppsss//////// http://hrikuuuarrrom kumar 7a.ht.]
Karen Simonyan, Andrea Vedaldi. and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yamn LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014, URL httl/http//xxxxae/
abs abs/aaas/.1///// 6312.6034
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattemberg, Smoothgrad: removing noise by adding noise. CoRR, abs// 706.738255 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy. Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Reprexentations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Workshop Truck Proceedings, 2015. URL htllbttpp//axxxiee
abs/ 1412.6806
Mukund Sundararajan. Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedingy of the 34th International Conference on Machine Learn- ing, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume Proceedings of Machine Learning Research, pp. 3319-3328. PMLR, 2017. URL http:://proceddnng........ wttpsngggs.....p.///..... sundarara janl Va.htm.l
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. htRp:ttp::// httttp:://axxi..oogg.. www.ariwwwoorrooooooooooooooooaa
wwt.g//..//////ooooooooooooo.. abs//////////// httpg//bb///11////. 1312.6119
Tom J. Viering, Ziqi Wang, Marco Loog, and Elmar Eisemann. How to manipulate cnns to make them lie: the gradeam case. CoRR, abs//990.11090) 2019. htl/attpp//axxxv.crt/ wttp:///o//..b.////11001.0 11977.1091111
Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Sys- tems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, Decem- 2020/ /ta0h/hah////////
ber 6-12, 2020, virtual, 2020. URL httpppp:/pprceeccggn.eeeeeeeeeee 2adeee222LAAd222efeeeeeeeeeeeeee
Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed ad- versarial examples. In 6th International Conference on Learning Representations, ICLR 2018, Vancou- ver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018 URL. htttps://ppent wwweeerwwveeeeee...eeee forrum/ffff d-
12

---

Under review as a conference paper at ICLR 2022
Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Raviku- mar. On the (in)fidelity and sensitivity explanations. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alcé-BBue, Emily В. Fox, and Roman Gareett (cds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019. Vancouver, BC. Canada,
pp. 10965-10976, 2019. URL https: http:s/s///oocceiinngg...o
6777 Abstracchhi
Jiamming Zhang. Zhe L. Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, 543 559. Springer. 2016. doi: 10.1007/ 978-3 1-311-46493-00 .33. utthttps: /ttp:///oo......10////////////// 9978 31 -14-33199-999-99 46493-449900 0.33
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In 2018 IEEE Conference Computer Vision and Puttern Recogni- rion, CVPR 2018, Salt Lake Ciry, UT, USA, June 18-22, 2018, pp. http://cc...coooetcc....0000 doi: 10. 10..110/CVPR.... 2018 00068. URL /
IEEE Computer Society, 2018.
html/ let.aaagw........cccooooaoll.eaa ott.://co......................r papprr.hmm
13


Reviews:
[{"review_id": "clx0T6GO-np", "reviewer": "Reviewer_xHez", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty and Significance:\nThe novelty and significance of this paper are in the experimental evaluation of \"smoothing explanation approaches\" using diverse evaluation measures. The major novelty and significance of this paper can come from section 3.2 that aims to show non-robustness of the smoothing methods. However, the experimental results do not show a significant difference between the three attacks, which degrades the novelty and significance. Section 3.1. experiment and result are similar to those in prior works, but still can provide somewhat meaningful observations  (i.e. insensitivity) about the smoothing approaches if error bar is provided.\n\nTechnical soundness: The experimental evidence is not fully supporting the authors' claims. Additional experiments and in-depth discussion are needed.\n- Table1-3, 5: Please provide error bar (i.e. empirical confidence interval). If error bar is not small enough to distinguish the difference, please use more samples for evaluation.\n- Figure 4, 7. The two metrics are not consistent with each other. For example, in figure 4, the non-additive attack is shown as effective for Smooth gradient and $\\beta$-smoothing in the left figure but not in the right figure. Why do those two metrics show different results? \n- Figure 4, 7. Each smoothing approach shows no significant differences between the three attacks. It is quite hard to say that those methods are less robust to other types of attacks based on this experiment. Please provide more experimental evidence or use more samples to reduce the error bar. \n\nWriting and Clarity: Minor corrections are needed.\n- Page 1. The first sentence 'Explanation methods attribute a numerical value ... ' is not always true. There are other explanation methods not using attribution scores. Please modify this sentence. \n- Page 2. Infidelity -> infidelity\n", "summary_of_the_paper": "This paper evaluates the quality and robustness of explanations of three post-hoc smoothing approaches (Smooth Gradient, Uniform Gradient, B-smoothing), and two ad-hoc smoothing approaches (CURE, Adv). It evaluates the quality of explanations based on the model parameter sensitivity, class sensitivity, sparseness, infidelity. It also evaluates the robustness of explanations to combinations of additive, spatial transformation, and recoloring attacks by comparing similarities between target and explanation maps. All evaluation is performed on publicly available benchmark datasets such as ImageNet and MS COCO. Based on their experimental results, the authors made several claims about the quality and robustness compared to the vanilla gradient method. For example, the authors claim that most of the smoothing methods are less sensitive to perturbation of model parameters so they may not be helpful to debug a model, and they may not be \"robust\" to non-additive types of attacks.  ", "summary_of_the_review": "The novelty and significance of this paper are in the experimental evaluation of robustness of the smoothing explanations that were known to be robust and any quality degradation trade-off. However, the experimental evidence is not fully supporting the authors' claims. The paper should provide experimental evidence showing significant robustness reduction. Also in-depth discussion about the experimental results is needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CuxckkZV6u8", "reviewer": "Reviewer_Aokc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- a broad set of evaluation metrics is used to measure the quality\n- hyper-parameters are well-documented, reproducibility is high\n- methods from prior work are studied thoroughly for their application\n- the paper is structured and written well\n\n\n### Weaknesses\n- the sample set of almost all experiments is very small and should be increased\n    - currently for post-hoc approaches on ImageNet:\n        - 128 samples for the similarity to the target map\n        - 100 samples for cascade randomization\n        - 20 (!) samples for the segmentation\n        - 100 samples for sparseness with the GINI index\n        - 100 samples for infidelity\n    - currently for the ad-hoc approaches on CIFAR-10:\n        - 320 samples for the similarity to the target map\n        - 100 samples for cascade randomization\n        - 100 samples for sparseness with the GINI index\n        - 100 samples for infidelity\n\n- errors should be reported for the segmentation experiment in table 1, the\n  cascade randomization (Figure 2 & 6), the GINI index (Table 2 & 4) and the\n  infidelity (Table 3 & 5)\n\n- only a single model on a single dataset is used for each experiment, which should be increased\n    - VGG16 on ImageNet for post-hoc approaches\n    - ResNet-18 on CIFAR-10 for ad-hoc approaches\n\n- only a single run of the experiments is conducted with only a single seed\n  (model parameters)\n    - to test for statistical significance, the experiments should, in addition\n      to more models and a higher sample size, be conducted multiple times with\n      multiple seeds\n\n- are the error bars in Figures 4 and 7 percentiles or the standard deviation?\n- the errors in Figures 4 and 7 are very large (\"compatible results\" most of\n  the time), but this is not discussed\n\n- in the segmentation experiment in Table 1 only the 20 pixels with highest\n  attribution values are investigated\n    - why do the authors not use all pixels' attribution values?\n    - if there is any reason behind this, it should be discussed, otherwise all\n      pixels should be used\n    - this approach measures spikey attribution maps in a different way than\n      smooth ones, since the spikey attribution maps will have more of their\n      total attribution scores measured\n\n- in Table 3 the infidelity for Smooth and Uniform are only marginally better\n  (1.43 vs. 1.42), yet in the text this is simply described as an improvement\n  (this marginal improvement will probably even become less significant with\n  errors reported)\n\n- it seems a little suspicious that Smooth and Uniform Gradient have the exact\n  same values for GINI Index and infidelity in Tables 2 & 3, though this may\n  just be by chance\n\n- for the similarity in Figures 4 & 7, it may give more insights to also\n  compute the distance to the original, ie. dist(h(x_{adv}, h(x)))\n\n- why is the segmentation experiment is not conducted for the ad-hoc methods?\n\n- minor: a label \"higher is better\" for figures 4 & 7 could improve readability\n", "summary_of_the_paper": "The authors empirically evaluate the quality and robustness of 3 post-hoc and\n2 ad-hoc approaches for robustification of gradient-based attributions under a\ncombination of 3 adversarial attack approaches which they use to target the\nattributions.\nThey evaluate the robustness via cosine-distance and LPIPS to a target-heatmap,\nand the quality via spearman correlation to the original under cascade\nrandomization of model parameters, the sparseness of the attribution maps using\nthe GINI index, and the explanation infidelity.", "summary_of_the_review": "While the results of this work are potentially interesting, this paper has many\nshort-comings in its empirical evaluation, which is especially problematic\nsince it is the paper's main contribution. With small sample sizes, most errors\nunreported, only a single model and dataset, and only a single run, there is a\nlot of room to improve this work. As these changes involve running many more\nexperiments, for the sake of improving this work, I recommend a score of 3:\nreject, not good enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9zZoUqu4-rv", "reviewer": "Reviewer_Sfxr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\nThis paper presents sanity check, non-Lp adversarial attacks, and sparsity measurement on SmoothGrad, Uniform Gradient and $\\beta$-smoothing. The results are generally interesting to the community. \nUsing the Gini Index to measure the sparsity is an interesting idea that might be useful for the follow-up works. However, I have several questions regarding this metric. I will elaborate in the next subsection.\n\n### Weakness\n- It is not surprising that Lp-based defenses (smoothing) are not robust to non-Lp threat models in the community. The authors aim to present evidence that smoothing techniques are not robust enough because the proposed non-Lp attacks can break the explanations, which by itself seems to be an unfair comparison for the prior work. In fact, the contributions showing the proposed non-Lp attacks can break Lp defense is not a new observation.\n\n- The proposed attacks do not seem to produce very different results. In Fig 5, resulting adversarial attacks from new technologies remain very similar to Delta attack. In Fig 4, it seems that the new attacks only make more than 0.05 difference with the Delta attack on $\\beta$-smoothing in cosine distance. On other attributions, the improvement seems to be minimal.\n\n- Gini Index for evaluating sparseness. The motivation to use Gini Index seems to be fair, however, a lot of prior works have proposed several different metrics [1, 2, 3, 4] in measuring the sparseness (or the concentrations and the localizations on the relevant features) of attributions. At least some discussions and justifications of the proposed metrics should be included. The robustness-related and fidelity-related evaluations are motivated from the paper’s main idea, understanding if smoothing techniques provide faithful explanations, however, the transition to study sparsity is somewhat sudden to me and it seems to be unconnected from the previous content, only because the following reason authors provide: “To create explanations that are human-accessible, it is advantageous to have a sparse explanation map”\n\n- Some related prior work, i.e. ROAR [5] , that finds smoothing techniques do not create significant degeneration to explanations should be included and discussed, especially when the paper is trying to provide the shortcoming of the smoothing techniques.\n\n\n[1] A. Chattopadhay, A. Sarkar, P. Howlader and V. N. Balasubramanian, \"Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks,\" 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), 2018, pp. 839-847, doi: 10.1109/WACV.2018.00097.\n\n[2] Poppi, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2021). Revisiting The Evaluation of Class Activation Mapping for Explainability: A Novel Metric and Experimental Analysis. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2299-2304.\n\n[3] Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., & Hu, X. (2020). Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 111-119.\n\n[4] Fong, R., Patrick, M., & Vedaldi, A. (2019). Understanding Deep Networks via Extremal Perturbations and Smooth Masks. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2950-2958.\n\n[5] Hooker, Sara et al. “A Benchmark for Interpretability Methods in Deep Neural Networks.” NeurIPS(2019).\n", "summary_of_the_paper": "The main contributions of this paper are a series of empirical results on smoothed attribution methods that are designed to show several smoothing techniques in the previous literature may produce worse explanations and these smoothing techniques are also not robust to non-Lp attacks. \n", "summary_of_the_review": "In summary this paper presents a series of interesting empirical results for smoothing techniques. However, I find the empirical results for the goodness of the smoothed attributions are not convincing to me, and the conclusions from the non-Lp attacks are not very new to the community, I remain negative for this paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0wvWru6kLCA", "reviewer": "Reviewer_dEQe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper touches on a very important problem - the quality and understanding of (smoothed) explanations. It is very well written and an easy to follow. \n\nIt has many positives. I particularly liked the approaches to assess robustness as they seems very reasonable and widely applicable, for example using LPIPS and perception and a way of giving an interesting new perspective. The main to areas for improvement / clarification are as follows.\n\n- Generalization. How do the results extend beyond the specific networks chosen? It is difficult to understand how relevant the conclusions are in general and whether the example configuration in influencing the conclusions too much. \n\n- Formalization. For me, the paper lack in the formalization of the problem. After reading the paper I don't fully have a clear notion of what makes an explanation of this form high quality and the exact properties that one should be looking for to assess it. Of course, the metrics reported are proxies to it, but the actual objetive is not clear to me. \n\n", "summary_of_the_paper": "The paper presents experiments for both post-hoc and ad-hoc explainers to better understand their quality and robustness.", "summary_of_the_review": "Good paper but I expected a bit more in terms of formalizing quality. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper conducts a series of empirical studies to evaluate the robustness of smoothed attribution methods. Although the reviewers think this is an important direction, there are several concerns about the experimental settings, such as the sample size and the models to be tested. Also, one of the main finding that Lp based smoothing methods are non-robust to non-Lp norm perturbations is well known and is not that surprising.
Decision: Reject