Robust Models Are More Interpretable Because Attributions Look Normal
Authors: Zifan Wang
Matt Fredrikson
Anupam Datta
Venue: ICLR 2022 Submitted (2022)
Keywords: Explainability
Decision Boundary
Attribution
Adversarial Robustness

Abstract: Recent work has found that adversarially-robust deep networks used for image classification are more interpretable: their feature attributions tend to be sharper, and are more concentrated on the objects associated with the image's ground-truth class. We show that smooth decision boundaries play an important role in this enhanced interpretability, as the model's input gradients around data points will more closely align with boundaries' normal vectors when they are smooth. Thus, because robust models have smoother boundaries, the results of gradient-based attribution methods will capture more accurate information about nearby decision boundaries. This understanding of robust interpretability leads to our second contribution: \emph{boundary attributions}, which aggregate information about the normal vectors of local decision boundaries to explain a classification outcome. We show that by leveraging the key factors underpinning robust interpretability, boundary attributions produce sharper, more concentrated visual explanations---even on non-robust models.

Under review as a conference paper at ICLR 2022
ROBUST MODELS ARE MORE INTERPRETABLE BECAUSE ATTRIBUTIONS LOOK NORMAL
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has found that adversarially-robus deep networks used for image classification are more interpretable: their feature attributions tend to be sharper, and are more concentrated on the objects associated with the image's ground- truth class. We show that smooth decision boundaries play an important role in this enhanced interpretability, as the model's input gradients around data points will more closely align with boundaries' normal vectors when they are smooth. Thus, because robust models have smoother boundaries, the results of gradient- based attribution methods, like Integrated Gradients and Deepl .ift, will capture more accurate information about nearby decision boundaries. This understanding of robust interpretability leads to second contribution: boundary attributions, which aggregate information about the normal vectors of local decision bound- aries to explain classification outcome. We show that by leveraging the key fac- tors underpinning robust interpretability, boundary attributions produce sharper, more concentrated visual explanations-evee on non-robust models.
INTRODUCTION
Feature attribution methods are widely used to explain the predictions of neural networks (Binder)
al. 2016; Dhamdhere al. 2019 Fong & Vedaldi 2017 Leino et al., 2018 Montavon et al. Selvaraju et al. 2017 Shrikumar et [ Simonyan et al. 2013 Smilkov et al. 2017
2015
Springenberg et al. [2014] Sundararaaan et al. 2017). By assigning an importance score to each input feature of the model, these techniques help to focus attention parts of the data most responsible for the model's observed behavior, Recent work (Croce et al. 2019: Etmann et al. 2019) has observed that feature attributions in adversarially-robur image models, when visualized, tend to be more interpretable-ttt the attributions correspond more clearly to the discriminative portions of the input.
One way to explain the observation relies the fact that robust models do not make use of non- robust features (lyas et al., 2019) whose statistical meaning can change with small, imperceptible changes in the source data. Thus, by using only robust features to predict, these models naturally tend to line up with visibly-relevant portions of the image. Etmann et al. take a different approach, showing that the gradients of robust models' outputs more closely align with their inputs, which explains why attributions on image models are more visually interpretable.
In this paper, we build on this geometric understanding of robust interpretability. With both analyti- cal (Sec. 3 and empirical (Sec. 5) results, we show that the gradient of the model with respect to its input, which is the basic building block of all gradient-based attribution methods, tends to be more closely aligned with the normal vector of nearby decision boundary in robust models than in "nor- mal' models. Leveraging this understanding, we propose Boundary-based Saliency Map (BSM) and Boundary-based Integrated Gradient (BIG), two variants of boundary atributions (Sec. 4). which base attributions on information about nearby decision boundaries (see an illustration in Fig. [Ta. While BSM provides theoretical guarantees in the closed-form, BIG generates both quantitatively and qualitatively better explanations. We show that these methods satisfy several desireable formal properties, and that even non-robust models, the resulting attributions more focused (Fig. 1b) and less sensitive to the "baseline" parameters required by some attribution methods.
To summarize, our main contributions as follows. (1) We present analysis that sheds light phenomeon of robust interpretability showing that alignment between
on the previously-obeers the normal vectors of decision boundaries and models' gradients is key ingredient (Proposition) T.

---

Under review as a conference paper at ICLR 2022
Integrated Gradient
X hd
H2
Boundary- ba sed Integrated Gradient
xg
(a)
Figure 1: (a) Visualizations of geometrical interpretations of Saliency Map (SM), Boundary-based Saliency Map (BSM), Integrated Gradient (IG) and Boundary-based Integrated Gradient (BIG). Gra- dient computations can be viewed as projecting the input onto particular decision boundary. While SM projects to a nearby boundary (H1), BSM projects the nearest one (H2). IG (the red dashed path) from global baseline Xg. i.c. zeros, aggregates boundaries in colorful shaded areas; BIG (the green dashed path) integrates from the point Ina on the nearest boundary H₂ to x and therefore aggregates nearby boundaries, H1 and H2 in gray shaded areas. (b) Visualizations of Integrated Gra- dient and the proposed improvement of it, Boundary- -basd Integrated Gradient, which is sharper, more concentrated and less noisy.
In particular, we derive a closed-form result for one-layer networks (Theorem D and empirically validate the take-away of our theorem generalizes deeper networks. (2) Motivated by our anal- ysis, we introduce boundary attributions, which leverage the connection between boundary normal vectors and gradients yield explanations for non-robust models that earry over many of the fa- vorable properties that have been observed of explanations on robust models. (3) We empirically demonstrate that one such type of boundary attribution, called Boundary- based Integrated Gradi- ents (BIG), produces explanations that are more accurate than prior attribution methods (relative to ground-truth bounding box information), while mitigating the problem baseline sensitivity that known to impact applications of Integrated Gradients Sundararajan et al. (0177) (Section 6)
2 BACKGROUND
We begin by introducing our notations. Throughout the paper we use italicized symbols x to denote scalar quantities and bold-face x to denote vectors. We consider neural networks with ReLU as activations prior to the top layer, and a softmax activation at the top. The predicted label for a given input x is given by F(x) arg max, fx(x), x IRdd where F(x) is the predicted label and (x) is the output on the class i. As the softmax layer does not change the ranking of neurons in the top layer, we will assume that fx) denotes the pre-softmax score. Unless otherwise noted, we use ||x to denote the εg norm of x, and the & neighborhood centered with radius B(x,
Explainability. Feature attribution methods are widely-used to explain the predictions made by DNNs, by assigning importance scores for the network's 's output each input feature. Convention- ally. scores with greater magnitude indicate that the corresponding feature was more relevant to the predicted outcome. We denote feature attributions by g(x, f), z, Rd When is clear from the context, we simply write g(x). While there is an extensive and growing literature on attribution methods, our analysis will focus closely on the popular gradient based methods, Saliency Map (Si- monyan et al. 2013), Integrated Gradient (Sundararajan et al. 2017) and Smooth Gradient (Smilkov et al. 2017 shown in Defs 33
Definition 1 (Saliency Map (SM)) The Saliency Map 9s(x) is given by gs(x) af(x) "xx
Definition 2 (Integrated Gradient (IG)) Given a baseline input Xt. the Integrated Gradient giG(x; Xb) is given by (IG(x; xb) : (x Xx) fo af((x-xx))+t) dl.
2

---

Under review as a conference paper at ICLR 2022
2/2
R1
(a)
(b)
(c)
Figure 2: Different classifiers that partition the space into regions associated with apple or banana. (a) A linear classifier where ń the only faithful explanations and v is not. (b) A deep network with ReLU activations. Solid lines correspond to decision boundaries while dashed lines correspond to facets of activation regions. (c) Saliency map of the target instance may be normal to the closest decision boundary (right) normal to the prolongation of other local boundaries (left).
Definition 3 (Smooth Gradient (SG)) Given zero-centered Gaussian distribution N with stan- dard deviation a, the Smooth Gradient gsa(x; 0) is given by 9sG(x; 0) = (e 8x
af(ate)
Besides, we will also include results from DeepLIFT (Shrikumar al. 2017) and grad i input (element-wise multiplication between Saliency Map and the input) (Sioonyan et al. 2013 in our empirical evaluation. As we show in Section 3.2 Defs J linearity of ReLU networks, and in the case of randomized smoothing (Cohen et al. 2019), their robustness to input perturbations. We further discuss these methods relative others Sec.
satisfy axioms that relate the local
Robustness. Two relevant concepts about adversarial robustness will be used in this paper: pre- diction robustness that the model's output label remains unchanged within a particular l norm ball and attribution robustness that the feature attributions are similar within the same ball. Recent work has identified the model's Lipschitz continuity as a bridge between these two concepts (Wang et al.) 2020c) and some loss functions in achieving prediction robustness also bring attribution robustness
CChllaaani et al. 2020). We refer to robustmess as prediction robustness not otherwise noted.
EXPLAINABILITY, DECISION BOUNDARIES, AND ROBUSTNESS
In this section, we begin by discussing the role of decision boundaries in constructing explanations of model behavior via feature attributions. We first illustrate the key relationships in the simpler case of linear models, which contain exactly one boundary, and then generalize to piecewise-lineal classifiers as they are embodied by deep ReLU networks. We then show how local robustness causes attribution methods to align more closely with nearby decision boundaries, leading to explanations that better reflect these relationships.
3.1 ATTRIBUTIONS FOR LINEAR MODELS
Consider a binary classifier C(x) = sign w T x + b) that predicts a label (-1, 1} (ignoring "tie"
cases where C(x) O, which can be broken arbitrarily). In its feature space, C(x) is hyperplane H that separates the input space into two open half-spaces S₁ and S, (see Fig. 2a Accordingly, the normal vector ń of the decision boundary is the only vector that faithfully explains the model's classification while other vectors, while they may describe directions that lead to positive changes in the model's output score, are not faithful in this sense (see v in Fig. 2a for an example). In practice, to assign attributions for predictions made by C, SM. SG, and the integral part of IG (see Sec. 2 return a vector characterized by = k + k2 (Ancona et al. 2018). where k * and by E R, regardless of the input x that is being explained. In other words, these methods all measure the importance of features by characterizing the model's 's decision boundary, and are equivalent up to the scale and position of ñi.
3

---

Under review as a conference paper at ICLR 2022
3.2 GENERALIZING TO HECEWISE-LINEAN BOUNDARIES
In the case of a piecewise-linear model, such as a ReLU network, the decision boundaries comprise a collection of hyperplane segments that partition the feature space, as H1, H2 and H3 in the example shown in Figure 2b Because the boundary no longer has a single well-defined normal, one intuitive way to extend the relationship between boundaries and attributions developed in the previous section is to capture the normal vector of the closest decision boundary to the input being explained. However, as we show in this section, the methods that succeeded in the case of linear models (SM, SG, and the integral part of IG) may in fact fail to return such attributions in the more general case of piecewise- linear models, but local robustness often remedies this problem. We begin by reviewing key elements of the geometry of ReLU networks (Jordan et al. 2019).
ReLU activation polytopes. For a neuron u in a ReLU network fxx,, we say that its status is ON if its pre-activation u(x) 2 0, otherwise it is OFF. We can associate an activation pattern denoting the status of each neuron for any point x in the feature space, and a half-space A, to the activation constraint u(x)= 2 0. Thus, for any point x the intersection of the half-spaces corresponding to its activation pattern defines polytope P (see Fig. 2b). and within /' the network is a linear function such that Vx E P. (x)
w _ + bp, where the parameters Wp and bp can be computed by differentiation (Fromherz et al., 2021 Each facet of P (dashed lines in Fig. 2b) corresponds to a boundary that "flips" the status of its corresponding neuron. Similar decision boundaries are precewise-linear because each decision boundary corresponds to constraint [x)x 2 [x) for two classes j (Fromherz et al. 2021 Jordan et al. 2019).
activation constraints,
Gradients might fail. Saliency maps, which we take to be simply the gradient of the model with respect to its input, can thus be seen as a way to project an input onto a decision boundary. That is, a saliency map is a vector that is normal to a nearby decision boundary segment. However, as others have noted, a saliency map is not always normal to any real boundary segment the model's geometry (see the left plot of Fig. [c). because when the closest boundary segment is not within the activation polytope containing x, the saliency map will instead be normal to the linear extension of some other hyperplane segment (Fromherz et al. (02T). In fact, iterative gradient descent typically outperforms the Fast Gradient Sign Method (Goodfellow et al. 2015) as an attack demonstrates that this is often the case.
When gradients succeed. While saliency maps may not be the best approach in general for captur- ing information about nearby segments of the model's decision boundary, there are cases in which serves as a good approximation. Recent work has proposed using the Lipschitz continuity of an attri- bution method to characterize the difference between the attributions of an input x and its neighbors within a Ep ball neighborhood (Def. 4) (Wang et al., 2020c). This naturally leads to Proposition T which states that the difference between the saliency map at an input and the correct normal to the closest boundary segment is bounded by the distance to that segment.
Definition 4 (Attribution Robustness) An attribution method g(x) is (A, -looclly robust at the evaluated point x if'v' E B(x, 119(')) 9(x)ll < X||x' - x||.
Suppose that I has (X, 5-robbustt saliency map gs at x, x' is the closest point on the closest decision boundary segment to x and ||x' x|| S 6. and that n is the normal vector of that boundary segment. Then (x)(x)) I A x||x x'I.
Proposition 1
Proposition therefore provides the following insight: for networks that admit robust attribu- tions (Chen et al. 2019; Wang et al. 2020c), the saliency map is a good approximation to the boundary vector. As prior work has demonstrated the close correspondence between robust predic- tion and robust attributions (Wang et al., 2020c Chalasani et al. (020)), this in turn suggests that explanations on robust models will more closely resemble boundary normals.
As training robust models can be expensive, and may not come with guarantees of robustness, post-processing techniques like randomized smoothing (Cohen et al., 2019). have been proposed as an alternative. Dombrowski et al.) (011)) noted that models with softplus activations (y 1/3 log(1 + exp (5x))) approximate smoothing, and in fact give an exact correspondence for single- layer networks. Combining these insights, arrive Theorem I which suggests that the saliency map on a smoothed model approximates the closest boundary normal vector well; the similarity is inversely proportional to the standard deviation of the noise used to smooth the model.
4

---

Under review as a conference paper at ICLR 2022
Theorem Let m(x) = (W be one-layer network and when using randomized smooth- ing, we write (x). Let g(x) be the SM for m,(x) (x) and suppose Vx" E [xxx|xx-''|]) where x' is the closest adversarial example, we have the following statement holds: l19(x)) g(x')l| & À where X is monotonically decreasing w.r. a.
2
Theorem [ suggests that when randomized smoothing is used, the normal vector of the closest decision boundary segment and the saliency map are similar, and this similarity increases with the smoothness of the model's ' boundaries. We think the analytical form for deeper networks exists but its expression might be unnecessarily complex due that we need to recursively apply ReLU before computing the integral (i.e., the expectation). The analytical result above for one-layer network and empirical validations for deeper nets in Figure [1] if taken together, shows that attributions and boundary-band attributions are more similar in a smoothed model.
4 BOUNDARY-BASED ATTRIBUTION
Without the properties introduced by robust learning or randomized smoothing, the local gradient, Le. saliency map, may not be good approximation of decision boundaries. In this section, we build on the insights of our analysis to present a set of novel attribution methods that explicitly incorporate the normal vectors of nearby boundary segments. Importantly, these attribution methods can be applied to models that are not necessarily robust, to derive explanations that capture many of the beneficial properties of explanations for robust models.
Using the normal vector of the closest decision boundary to explain classifier naturally leads to Definition 5 which defines attributions directly from the normal of the closest decision boundary.
Definition 5 (Boundary-based Saliency Map (BSM) Given I and an input x, we define Boundary-based Saliency Map Bs(x) as follows: Bx(x) def 0f((x))(xx, where x' is the closest ad- versarial example to x, i.e. = F(x) # F(x') and Vxm ||mm x|| < - + F(x) = F(xm).
Incorporating More Boundaries. The main limitation of using Definition local explanation is obvious: the closest decision boundary only captures one segment of the entire decision surface. Even in small network, there will be numerous boundary segments in the vicinity of relevant point. Taking inspiration from Integrated Gradients, Definition proposse 6 the Boundary-based Inte- grated Gradient (BIG) by aggregating the attributions along a line between the input and its closest boundary segment.
Definition 6 (Boundary-basee -bsse Integrated Gradient(BIG)) Given f. Integrated Gradient gIG and an input x, we define Boundary-basee -aased Integrated Gradient Bs(x) as follows: BIG(x) == 9IG x') where x is the nearest adversarial example to i.e. F Fxx) + F(x') and V / x||
|x' x F(x) = F(Xm)-
Geometric View of BIG. BIG explores a linear path from the boundary point to the target point. Because points on this path are likely to traverse different activation polytopes, the gradient of intermediate points used compute gig normals of linear extensions of their local boundaries. As the input gradient is identical within a polytope P., the aggregate computed by BIG sums each gradient w; along the path and weights by the length of the path segment intersecting with P.. In other words, one may view IG as an exploration of the model's global geometry that aggregates all boundaries from fixed reference point, whereas BIG explores the local geometry around x. In the former case, the global exploration may reflect boundaries that are not particularly relevant to model's 's observed behavior at point, whereas the locality of BIG may aggregate boundaries that are more closely related (a visualization is shown in Fig.
Finding nearby boundaries. Finding the exact closest boundary segment identical to the problem of certifying local robustness (Fromherz et al.] [021 Jordan et al., 2019 Kolter & Wong, 2018 Lee et al. 2020 Leino et al. [021b] Tjeng et al.) 2019: Weng et al., 2018), which is NP-hard for piecewise-linea models (Sinha et al. 2020). To efficiently find an approximation of the closest boundary segment, we leverage and ensemble techniques for generating adversarial examples, i.e. PGD (Madry et al. 2018), 1 AutoPGD (Croce& & Hein 2020) and CW (Carlini & Wagner (017), and use the closest one found given a time budget. The details of our implementation are discussed in Section 5| where we show that this yields good results in practice.
5

---

Under review as a conference paper at ICLR 2022
CIFARIO
standard
Eg 0..5
SM-BSM.
59.96
1.23
IG-AGI
28.20
1.43
IG-BIG
31.22
2.73
Corr.
Loc.
EG
PP
Con.
SM-BSM
0.40
0.46
-0.19
0.07
ImageNet
standard
E 3.0
Loo 255
255
IG-AGI
0.24
0.25
0.05
-0.03
SM-BSM
8,48
0.41
2.25
1.61
IG-BIG
0.35
0.30
0.20
-0.03
IG-AGI
13.52
0.36
1.19
0.86
IG-BIG
17.07
0.69
1.74
1.45
(a)
(b)
Figure 3: (a): Eg differences between SM, IG and their boundary variants for robust models. The heading of each column reports the respective training epsilon and the corresponding L, norm con- straint; Appendix B.4 reports the corresponding boxplot. (b): Linear correlation coefficients be- tween the alignment of SM and IG with nearby boundary vectors, and the localization metrics. For each row starting with X-Y, the alignment is defined as -| Y||. For each column, the localiza- tion results measured with approach in bold font, a.k.a X.
Model
Metrics
BIG
BSM
AGI
SM
GTI
IG
DeepLIFT
Loc.
0.38
0.33
0.33
0.33
0.35
0.34
0.34
0.34
standard
BG
0,54
0.47
0.48
0.47
0.46
0.55
0.5
0,49
PP
0.87
0.50
0.58
0.50
0.50
0.50
0.51
0.53
Con.
4.35
3.88
4.01
3.92
3.94
4.06
3.97
3.93
Loc.
0.39
0.33
0.39
0.33
0.33
0.34
0.33
0.33
f2/3.0
EG
0.74
0.6
0.64
0.6
0.63
0.62
0.65
0.64
PP
0.92
0.50
0.88
0.50
0.55
0.51
0.65
0.77
Con. 5.03
4.12
4.32
4.10
4.25
4.23
4.37
4.34
Table 1: Results of several attribution methods over 1500 images ImageNet using standard and robust ResNet50 (training is reported in the first column). BIG: Boundary-based Integrated Gra- dient. BSM: Boundary-basee Saliency Map. AGI: Adversarial Gradient Integration. SM: Saliency Map. GTT: gradx nput. SG: Smoothed Gradient. IG: Integrated Gradient. See Appendix E for the corresponding boxplot.
5 EVALUATION
In this section, we first validate that the attribution vectors are more aligned to normal vectors of nearby boundaries robust models(Fig. Bap. We secondly show that boundary- -oundary-based attributions provide more "accurate" explanations attributions highlight features that are actually relevant to the label - both visually (Fig. and 5 and quantitatively (Table TT. Finally, we show that in a standard model. whenever attributions more align with the boundary attributions, they are more "accurate".
General Setup. We conduct experiments over two data distributions, ImageNet (Russakovsky et al,, 2015) and CIFAR-10 (Krizhevsky et al.). For ImageNet, choose 1500 correctly images from ImtgeNette (Howard), a subset of ImageNet, with bounding box area less than 80% of the original source image. For CIFAR-10. 10, We use 5000 correctly classsiie images. All standard and robust deep classifiers are ResNet50. All weights are pretrained and publicly available (Engstrom al. 2019). Implementation details of the boundary search (by ensembling the results of PGD, CW and AutoPGD) and the hyperparameters used in our experiments, are included in Appendix B.2
5.1 ROBUSTNESS + BOUNDARY ALIGNMENT
In this subsection, we show that SM and IG better align with the normal vectors of the decision boundaries in robust models. For SM, use BSM as the normal vectors of the nearest decision boundaries and measure the alignment by the & distance between SM and BSM following Propo- sition] For IG, BIG as the aggregated normal vectors of all nearby boundaries because

---

Under review as a conference paper at ICLR 2022
IG also incorporates more boundary vectors. Recently, Pan et al. (2021) also provides Adversarial Gradient Integral (AGI) as an alternative way of incorporating the boundary normal vectors into IG. We first use both BIG and AGI to measure how well IG aligns with boundary normals and later compare them in Sec. 5.2 followed by a formal discussion in Sec. 7
Aggregated results for standard models and robust models are shown in Fig. It shows that adversarial training with bigger c encourages a smaller difference between attributions and their boundary variants. Particularly, using E2 norm and setting e - 3.0 are most effective for ImageNet compared to &o norm bound. One possible explanation is that the f2 space is special because training with foe bound may encourage the gradient to be more Lipschitz in f because of the duality between the Lipschitzness and the gradient norm, whereas Eg is its own dual.
5.2 BOUNDARY ATTRIBUTION + BETTER LOCALIZATION
In this subsection, we show boundary attributions (BSM, BIG and AGI) better localize relevant features. Besides SM, IG and SG, we also focus on other baseline methods including Grad x Input (GTI) (Simonyan et al., 2013) and DeepLIFT (rescale rule only) (Shrikumar et al. 2017)
that are reported to be more faithful than other related methods (Adebayo et al. 2018 2020).
In an image classification task where ground-truth bounding boxes are given, we consider features within bounding box more relevant to the label assigned to the image. Our evaluation is per- formed over ImageNet only because no bounding box is provided for CIFAR- 10 data. The metrics used for our evaluation are: 1) Localization (Loc.) (Chattopadhyay
al. 2017) evaluates the in- tersection of areas with the bounding box and pixels with positive attributions; 2) Energy Game (EG) (Wang al.] (020a) instead computes the portion of attribute scores within the bounding box. While these two metrics are common in the literature, we propose the following additional metrics: Posiitve Percentage (PP) evaluates the portion of positive attributions in the bounding box be- cause naive assumption is all features within bounding boxes are relevant to the label (we will revisit this assumption in Sec. b): and 4) Concentration (Con.) sums the absolute value of attri- bution scores over the distance between the "mass" center of attributions and each pixel within the bounding box. Higher Loc., FG, PP and Con. are better results. We provide formal details for the above metrics in Appendix BB1
We show the average scores for ResNet50 models in Table T where the corresponding boxplots can be found in Appendix [B.4] BIG is noticeably better than other methods on Loc. EG, PP and Con. scores for both robust and standard models and matches the performance of SG on EG for a standard model. Notice that BSM is not significantly better than others in standard model, which confirms our motivation of BIG that we need to incorporate more nearby boundaries because a single boundary may not be sufficient to capture the relevant features.
We also measure the correlation between the alignment of SM and BSM with boundary normals and the localization abilities, respectively. For SM, we use BSM to represent the normal vectors of the boundary. For IG, we use AGI and BIG. For each pair X-Y in (SM-BSM, IG-AGI, IG-BIG), we measure the empirical correlation coefficient between in standard ResNet50 and the result is shown in Fig. 3b Our results suggest that when the attribu- tion methods better align with their boundary variants, they can better localize the relevant features in terms of the Loc. and EG. However, PP and Con. have weak and even negative correlations. One possible explanation is that the high PP and Con. of BIG and AGI compared to IG (as shown in Table may also come from the choice of the reference points. Namely, compared to a zero vector, a reference point on the decision boundary may better filter out noisy features.
X Y|l2 and the localization scores of X
We end our evaluations by visually comparing the proposed method, BIG, against all other attri- bution methods for the standard ResNet50 in Fig. 4 and for the robust ResNet50 in Fig. 5 which demonstrates that BIG can easily and efficiently localize features that are relevant to the prediction. More visualizaitons can be found in the Appendix E
Summary. Taken together, we close the loop and empirical show that standard attributions in robust models are visually more interpretable because they better capture the nearby decision boundaries. Therefore, the final take-away from our analytical and empirical results is if more resources devoted to training robust models, effectively identical explanations can be obtained using much less costly standard gradient- based methods, i.e. IG.
7

---

Under review as a conference paper at ICLR 2022
DeepLIFT
French horn
Figure 4: Visualizations of attributions for two examples classified by standard ResNet50.
Parachute
SM
GTI
IG
SG
DeepLIFT
AGI
BIG
Figure 5: Visualizations of attributions for two examples classified by robust ResNer50 (62/3.0). The second example from Fig. 4 is not correctly classified so we replace it with another image.
DISCUSSION
Baseline Sensitivity. is natural to treat that BIG frees users from the baseline selection in ex- plaining non-linear classifiers. Empirical evidence has shown that IG is sensitive the baseline inputs (Sturmfels et al., (020). We compare BIG with IG when using different baseline inputs, white or black images. We show example in Fig 6b For the first two images, when using the baseline input as the opposite color of the dog, more pixels on dogs receive non-zero attribution scores. Whereas backgrounds always receive more attribution scores when the baseline input has the same color as the dog. This is because GIG (x); с (x Xb); (see Def. 2 that greater differences in the input feature and the baseline feature can lead to high attribution scores. The third example further questions the readers using different baselines in IG whether the network is using the white dog to predict Labrador retriever. We demonstrate that conflicts in IG caused by the sensi- tivity to the baseline selection can be resolved by BIG. BIG shows that black dog in the last row is more important for predicting Labrador retriever and this conclusion further validated by our counterfactual experiment in Appendix D Overall, the above discussion highlights that BIG is significantly better than IG in reducing the non-necessary sensitivity the bascline selection.
Limitations. We identify two limitations of the work. 1) Bounding boxes not perfect ground- truth knowledge for attributions. fact, we find a lot of examples where the bounding boxes either fail to capture all relevant objects bounding boxes still remain an open question and should benefit more expandability research in gen- eral. 2) Our analysis only targets attributions that based on end-to-end gradient computations. That is, we are not able to directly characterize the behavior of perturbation-based approaches, i.e. Mask (Fong Vedaldi 2017). and activation- approaches, i.e. GradCAM (Selvaraju al. 2017) and Feature Visualization (Olah et al. 2017).
big capture relevant features only. Fixing mislabeled
RELATED WORK
Ilyas et al. (019) shows an alternative way of explaining why robust models are more interpretable by showing robust models usually leam robust and relevant features, whereas work serves geometrical explanation to the same empirical findings in using attributions to explain deep models. Our analysis suggests need capture decision boundaries in order to better explain classifiers,
8

---

Under review as a conference paper at ICLR 2022
black 1G
white iG
DIG
Properties
black IG
AGI
BIG
Malbese
Boundary-based
X
Boundary Search
N/A
PGD
Any Local
Geometry
Global
Local
vetrie ver
Symmetry
X
V V
Completeness
Labrador netrie ver
(a)
(b)
Figure 6: (a): Qualitative comparisons between IG with black baseline, BIG and AGI. BIG can use any boundary search approaches or an ensemble of them while AGI uses PGD only. AGI fails to meet the symmetry axiom (Sundararajan al., 2017) where BIG satisfies all axioms that IG satisfies, i.c. completeness. (b): Comparisons of IG with black and white baselines with BIG. Predictions are shown in the first column.
whereas similar line of work, AGI (Pan et al. (021) that also involves computations of adversarial examples is motivated find non-linear path that is linear in the representation space instead of the input space compared to IG. Therefore, AGI uses PGD to find the adversarial example and aggre- gates gradients on the non-linear path generated by the PGD search. We notice that the trajectory of PGD search is usually extremely non-linear, complex and does not guarantee to return closer adver- sarial examples without CW or AutoPGD (see comparisons between boundary search approaches in Table B.2 We understand that finding the exact closest decision boundary is not feasible, but our empirical results suggest that the linear path (BIG) returns visually sharp and quantitative better results in localizing relevant features. Besides, non-linear path should cause AGI fail to meet the symmetry axiom (Sundararajan et al., 2017) (see Appendix C for an example of the importance of symmetry for attributions). We further summarize the commons and differences in Table
In the evaluation of the proposed methods, we choose metrics related to bounding box over other metrics because for classification we interested in whether the network associate relevant features with the label while other metrics (Adebayo et al. 2018 Ancona et al.) 2017 Samek et al.) 2016 Wang et al. 2020b; Yeh al., 2019). e.g. infidelity (Yeh al.) 2019). mainly evaluates whether output scores are faithfully attributed to each feature. Our idea of incorporating boundaries into explanations may generalize to other score attribution methods, e.g. Distributional Influence (Leino et al. 2018) and DeepLIFT (Shrikumar et al., 2017). The idea of using boundaries in the explanation has also been explored by T-CAV (Kim al. 2018). where linear decision boundary learned for the internal activations and associated with their proposed notion of concept.
When viewing our work as using nearby boundaries as a way of exploring the local geometry of the model's output surface, a related line of work is NeighborhoodSHAAP (Ghalebikesabi et al.) [20T)) local version of SHAP (Lundberg & Lee 2017). When viewing our as different use of adversarial examples, some other work focuses counterfactual examples (semantically meaningful adver-
sarial examples) on the data manifold (Chang et al.) 2019 Dhurandhar et al. 2018 Goyal et al. 2019).
8 CONCLUSION
In summary, we rethink the target question an explanation should answer for a classification task, the important features that the classifier uses to place the input into specific side of the decision boundary. We find the answer to our question relates to the normal vectors of decision boundaries in the neighborhood and propose BSM and BIG boundary attribution approaches. Empirical evaluations on STOA classifiers validate that our approaches provide more concentrated, sharper and more accurate explanations than existing approaches. Our idea of leveraging boundaries to explain classifiers connects explanations with the adversarial robustness and help to encourage the community to improve model quality for explanation quality.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.
Julius Adebayo. Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model expla- nations, 2020.
Gunjan Aggarwal, Abhishek Sinha, Nupur Kumari, and Mayank Kumar Singh. On the benefits of models with perceptually-alignn gradients. ArXiv, abs/2005.01 01900990..99,.... 2020.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks, 2017.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-basee attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin. Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.
Müller, and Wojeiech
Nicholas Carlini and D. Wagner. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise expla- nations of neural networks using adversarial training. In International Conference on Machine Learning. pp. 1393.1391. PMLR, 2020.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and D. Duvenaud. Explaining image classifiers by counterfactual generation. In ICLR, 2019.
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian, Grad- cam++: Generalized gradient-hased visual explanations for deep convolutional networks. arXiv preprint wwxiv:1710.11000 2017.
Jiefeng Chen, Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization. In Advances in Neural Information Processing Systems, 2019.
Junsuk Choe and Hyunjung Shim. Attention-based dropout layer for weakly supervised object lo- calization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 2219-2228. 2019.
Jeremy M. Cohen, Elan Rosenfeld, a.d J.Z. I. Kolter. Certified adversarial robustness via randomized smoothing. In ICML, 2019.
Francesco Croce and Matthiss Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu net- works via maximization of linear regions. AISTATS 2019, 2019.
Kedar Dhamdhere, Mukund Sundararajan, and Qigi Yan. How important is a neuron. In Interna- tional Conference on Learning Representations, 2019. URL https:://peereeiee
forum?id=SylKoooeee
A. Dhurandhar, P. Chen, Ronny Luss, Chun-Chen Tu, Pai-Shun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In NeurIPS, 2018.
Ann-Kathrin Dombrowski, M. Alber, Christopher J. Anders, Marcel Ackermann, K. Müller, and P. Kessel. Explanations can be manipulated and geometry is to blame. NeurlPS, 2019.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python
2019. URL http:///w.////gg.ooo...////////a
10

---

Under review as a conference paper at ICLR 2022
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola Schoenlieb. On the connection between adversarial robustness and saliency map interpretability. In Proceedings of the 36th International Conference Machine Learning, 2019.
R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3449-3457, 2017. doi: 10.1109//CCV2201..3371.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In Proceedings of the IEEE international conference on computer vision, pp. 3429-3437, 2017.
Aymeric Fromherz, Klas Leino, Matt Fredrikson, Bryan Parno, and Corina Pāsäreanu. Fast ge- ometric projections for local robustness certification. In International Conference on Learning Representations (ICLR), 2021.
Sahra Ghalebikesabi, Lucile Ter-Minassian, Karla Diaz-Ordaz, and Chris C. Holmes. On locality of local explanation models. arxiv, a0s/2106.11440i 14648, 2021.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In Proceedings of the 36th International Conference on Machine Learning, pp- 2376-2384, 2019.
Jeremy Howard. imagenette. URL /tt.://pp///gttt...o////////////
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, 2019.
Matt Jordan, J. Lewis, and A. Dimakis. Provable certificates for adversarial examples: Fitting ball in the union of polytopes. In NeurIPS, 2019.
Been Kim, M. Wattenberg, Gilmer, C. J. Cai, James Wexler, Viégas, and Rory Sayres. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: unified and generic model interpretability library for pytorch, 2020.
J. Z. Kolter and E. Wong. Provable defenses against adversarial examples via the convex outer adversarial polytope. In /CML, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). http::t:://wws: httktt::/www..o. www.://wwcccsoo/coco http://wwww.t...tttto// toronto......o...o.... httpp//eww.....iii...aar..tt...
Sungyoon Lee, Jaewook Lee, and Saerom Park. Lipschitz-certtffablee training with a tight outer bound. Advances in Neural Information Processing Systems, 33, 2020.
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. dirreted expla- nations for deep convolutional networks. In 2018 IEEE International Test Conference (ITC), pp. 1-8. IEEE, 2018.
Klas Leino, Ricardo Shih, Matt Fredrikson, Jennifer She, Zifan Wang, Caleb Lu, Shayak Sen, Di- vya Gopinath, and Anupam. truera/trulens: Trulens, 2021a. URL htttt:://///// potttp://zeeodo... zenodo.
Record/449588
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks, 2021b.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference on neural information processing systems, pp. 4768-
4777, 2017.
11

---

Under review as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Grégoire Montavon, Sebastian Bach, Alexander Binder, Wojeiech Samek, and Robur- Müller. Explaining nonlinear classification decisions with deep taylor decomposition, 2015.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.2391 10distii//ll..o00 00000..
http://ddittil.pub//00 http:///ill.pp2///////eaaaaaaiaa 7/ffeauureviisulliztiin
Deng Pan, Xin Li, and Dongxiao Zhu. Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017. URL http:://aaaaate wwwivv.ivvvvv..////va/aaaaaaaaaa erg/ abs/
1707. 04131
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensor- flow, and jax. Journal of Open Source Software, 5(53):2607, 2020. doi: 10.21 10211005/osss026077 URL httphttt://doo http:::///oooog/////oooooo 10. 21105/ 02607
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2211225 252, 2015. doi: 10.100 1263-015-0816.y.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28( (1 (1)):2666-2667, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3 3145- 3153. PMLR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2013.
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad:
removing noise by adding noise, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net, 2014.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution basclines. Distill, 2020. doi: 0.223915/distill0000222 http:://istill.pub22220aatrrrorn baselines................
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference Learning Representations, 2019.
12

---

Under review as a conference paper at ICLR 2022
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Work- shops, pp. 24-25, 2020a.
Zifan Wang, Piotr Mardziel, Anupam Datta, and Matt Fredrikson. Interpreting interpretations: Orga- nizing attribution methods by criteria. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.
Zifan Wang, Haofan Wang. Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13623- 13634. Curran Associates, Inc., 2020c.
Tsui- Weng, Huan Zhang, H. Chen, Zhao Song, C. Hsieh, D. Boning, I. Dhillon, and L. Daniel. Towards fast computation of certified robustness for relu networks. In ICML, 2018.
Chih-Kuan Yeh, Cheng-Yu Yu Hsieh, Arun Sai Suggala, David Inouye, and Pradeep Ravikumar. On the (in) fidelity and sensitivity for explanations. In Advances in Neural Information Processing Systems, 2019.
13

---

Under review as a conference paper at ICLR 2022
A THEOREMS AND PROOFS
A. PROOF OF PROPOSITION
Proposition | Suppose that has (X, -r-ruustt saliency map gs at x, x' is the closest point on
the closest decision boundary segment to X and ||x -xx] x|| S ö, and that n is the normal vector of that boundary segment. Then ||n- (inn-gs(x)), gs(x)ll I X||x- X1I-
To compute n can be efficiently computed by taking the derivatice of the model's output w.r. to
the point that on the decision boundary such that as(x)) and Vxm IR F(xm) F(xx 8x
| x|| S ||x' -x x||.
Because we assume ||X x(I $ 6, and the model has (A, -obbusttt Saliency Map, then by Def. we have
gs(x)ll S A|Ix x'Il
A.2 PROOF OF THEOREM
Theorem Let m(x) (Wx) be one-layer network and when using randomized smoothing, we write (x).). Let g(x) be the SM for mz(x) and suppose Vx" E B(x, |x,||x (19)|1l((x)11 2 where x' is the closest adversarial example, have the following statement holds: l19(x) (xx')l= S where is monotonically decreasing w.r.t. a.
Proof:
We begin our proof by firstly introducing Randomized Smoothing.
Definition (Randomized Smoothing (Cohen et al., 2019) Suppose F(x) = arg max. fc (x),
the smoothed classifier G(x) is defined as
G(x) == arg max Pr [F +) =d
(1)
where O0(0022
Now the rest of the proof of is three-fold: 1) firstly will show that there exist non-linear activa- tion function Er(x) such that the output of the smoothed ReLU network m (x) is equivalent when replacing the ReLU activation with Er activation; 2) secondly derive the difference between the
saliency map of the network with Er activation; and 3) lastly, we show that the difference between SM and BSM of the network with Er activation bounded, which is inversely proportional to the standard deviation used to create the smoothed ReLU network m (x()..
(1) Step I: Error activation (Er) function and randomized smoothing.
Randomized Smoothing creates a smoothed model that returns whichever the label that the base classifier most likely to return under the perturbation generated by the Gaussian noise. Now take a look at the output of each class under the Gaussian noise. Consider ys is the output of the i-th class of the network ReLU(Wxx) x), that is
y1 = T2 =
To simplify the notation, we denote Be-.N(,,02) as E. We expand Equation 2
yi E |ReLU(w, + w e) ww E ReLLU(u+e))
where we denote W x and é' w, u is scalar and é' follows zero-centered univariate Gaussian with the standard deviation s x a because the dot production between the constant weight vector w; and the random vector can be seen as linear combination of each dimension of and the covariance between each dimension of is 0 for the Gaussian noise used for randomized smoothing
We appreciate the discussion with the author Pan Kessel of Dombrowski et al. 2019 for the derivations from Equation (6) (7)
14

---

Under review as a conference paper at ICLR 2022
in the literature (Cohen al. 2019). By expanding the expectation symbol to its integral form, obtain:
14 SV2
exp(
Ru e')dé'
(4)
2x²
Let = u and notice that (eLU(r) = if < O, the equation above can be rewritten as:
(T u)²
y1
exp(
TdT
(5)
8/2T
J0
2s²
u
expl
)8 +
1 Erf(
(6)
2t
2x
2s
where Erf is the error function defined as Erf(x) = / Jee exp(-13)dl. We therefore define an Er activation for an input with the standard deviation
u²
u
u 22s
Er(u; s) = 22T
exp(
)s
+
1
+
Erf(
(8)
25
2
and we show that
Y: = U [eeLU(w; (x c)) = Er(w, x; s(r))
That is, to analyze the gradient of the output for a smoothed model w.r. the input, we can alterna- tively analyze the gradient of an equivalent Er network. We plot three examples of the Er activations in Fig. 7 for the readers to see what does the function look like.
2) Step II: the Saliency Map for Er network.
By the definition of Saliency map (Def.
D and the chain rule, we have:
Öy:
dy: Ōu
SM(x)
(Let u == =., x)
(10)
dx
Ou dx
(Er(u; s)) w;
(11)
Iu
1
Erf(
w;
(12)
2
228
The transition between Equation (11) to (12) is based on the fact that the derivative of Erf(x: 2 exp(-x3)
3) Step III: the difference between SM and BSM for an Er network.
Let x' be the closest point on the decision boundary for the smoothed classifier m, and ||x-|'XX x'|| =r (for the closed-form expression of see Cohen et al. (209)). Based on the definition of BSM, have
Oy: (x')
v' Erf(
4')
BSM(x)
Wis n' w.T x'
(13)
Ox!
/28
The difference between SM and BSM therefore is computed
15

---

Under review as a conference paper at ICLR 2022
12
12
1.0
0.8
0.6
0.6
0.2
0.2
0.2
-10
-0.5
0.5
1.0
-1.0
-05
0.5
1.0
-1.0
-0.
0.5
1.0
(a)
0.035
0.3
(c)
1.0
Figure 7: The graph of Er(u; s) w.r.t different standard deviations s.
[BM((x) SM(x)]I -11
1
Erf(
W;
1 [rf
will
(14)
V28
v2s
u¹
Erf(
Erf(
>1 //we|l
(15)
2
V28"
V28"
1
u1
Erf(
)1
|rr((
(Triangle Inequality)
(16)
2
/28"
/28
We notice that the u' is bounded because w/ = W x
w |x||| < ||wa|l (s/ + +r and
similarly for u such that u = w. x < ||w/]| [||x| r). Because Erf function increasing w.r.t the
input and > O, we arrive at the following inequality:
[BSM(x) SM(x)]I s
(17)
where
[will (1|x| + r)
X Erf(
||will
(18)
v 22
We take the absolute symbol out because the output of an Erf is positive when its input is posi- tive. Now, given that ||w,ll:r r and ||x|| are constants when a given input x, the upper-bound Erf( [|wall-([>[|++r) |wall is monotonically increasing as s decreases. From the Step I, we know v2s
that x a, therefore prove there exist upper-bound À of the difference between the SM and BSM for smoothed classifier and À is monotonically decreasing w.r.t the standard deviation of the Gaussian noise.
B EXPERIMENT DETAILS AND ADDITIONAL RESULTS
B. METRICS WITH BOUNDING BOXES
We will use the following extra notations in this section. Let X, Z and U be of indices of all pixels, set of indices of pixels with positive attributions, and set of indices of pixels inside the bounding box for target attribution map gxx). We denote the cardinality
set as (.S].
Localization (Loc.) (Chattopadhyay et al.] (017) evaluates the intersection of areas with the bounding box and pixels with positive attributions.
Definition 8 (Localization) For given attribution map g(x), the localization (Loc.) is de- fined as
Z N
U
Loc :
(19)
U
2 (X
U\
16

---

Under review as a conference paper at ICLR 2022
Localization Hladd
Energy Game (GB)
1.0
0.6
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
81G
85M
AdI
SM
G1
DeepLiFT
85M
AGI
SM
56
18
DeepLiFT
PP (PP.1
Concentration (Con.)
6.0
5.5
0.8
0.7
0.6
4.0
0.5
3.5
0.4
0.3
BIG
SM
DeesLIFT
Figure 8: Localizaiton performance for attributions on a standard ResNet
Energy Game (EG) Wang et al. 2020a) instead evaluates computes the portion of attribute scores within the bounding box.
Definition 9 (Energy Game) For given attribution map g(x), the energy game EG is defined
EG Eieznu g(x);
(20)
Σiex max(g(x))) 0)
Positive Percentage (PP) evaluates the sum of positive attribute scores over the total (absolute value of) attribute scores within the bounding box.
Definition 10 (Positive Percentage) Let V be of indices pf all pixels with negative attribution scores, for given attribution map g(x), the positive percentage PP is defined as
Eieznu g(x);
PP = ΣiEZnu 9(x); Eievnu g(x);
(21)
Concentration (Con.) evaluates the sum weighted distances by "mass" between the "mass" center of attributions and each pixel within the bounding box. Notice that the computation of er and ey can be computed with scipy. ndimage. cent er_offrass This definition encourages that pixels with high absolute value of attribution scores to be closer the mass center.
Definition 11 (Concentration) For given allribution map g(x), the concentration Con. is defined as follws
Con. = E(x)) / (1e cx)² (iy -cy)²
(22)
where 9 the normalized attribution map so that g. gi/ Eiev Igil. in by are the coordinates of the pixel and
Eieu (x)( Eiev g(x); ,. = Eisu (x)(
(23)
Eieu g(x);
Besides metrics related to bounding boxes, there are other metrics in the literature used to evaluate attribution methods (Adebayo et al.] 2018 Ancona et al.] 2017 Samek et al., 2016 Wang et al. 20206b Yeh et al., 2019). We focus on metrics that use provided bounding boxes, as we believe that they offer clear distinction between likely relevant features and irrlevvnt ones.
17

---

Under review as a conference paper at ICLR 2022
Localization HLacd
Energy Game (GB)
1.0
0.7-
0.8
0.5
0.6
0.3
0.4
0.2
0.1
0.0
0.0
81G
85M
AdI
SM
G1
DeepLiFT
85M
AGI
SM
56
18
DewpLiFT
Postive Percentage
Concentration (Con.)
0.9
0.5
BIG
SM
DeesLIFT
Figure 9: Localizaiton performance for attributions on a robust ResNet (42)3.0)
Pipeline
Avg Distance
Success Rate
(ImageNet) Standard ResNet50
PGDs
0.549
72.1 %
CW
0.548
72.1% 1%
AutoPGD
0.548
72.1%
(ImageNet) Robust ResNet50 (6a 3.0)
PGDs
2.870
74. 1%
CW
2.617
74.
AutoPGD
2.617
74, 1%
(ImageNet) Robust ResNet50 (Lso 4/255)
PGDs
2.385
98.9%
CW
2.058
98.
CIFARIO
standard
robust
AutoPGD
2,058
98.9%
0.5
1.0
(ImageNet) Robust ResNet50 (2.0
3/255)
topk
10
10
PGDs
2.378
99, 19.1%
max iters
15
15
+ CW
1,949
99.1%
AutoPGD
1,949
99.15
ImageNet
standard
robust
(CIFAR-I0) Standard ResNet50
2.0
6.0
PGDs
0.412
98.7%
+ CW
0.120
98.7%
topk
15
15
AutoPGD
0.120
98.7%
max iters
15
15
(CIFAR-10) Robust ResNet50 (&z 0.5)
PGDs
1,288
99.9%
+ CW
1,096
99.9%
AutoPGD
1.096
99.9%
(a)
(b)
Figure 10: (a): Pipeline: the methods used for boundary search. Avg Distance: the average & distance between the input to the boundary. Success Rate: the percentage when the pipeline returns an adversarial example. Time: per-instance time with batch size of 64. We are using much bigger es for robust models, so the success rates are higher than a standard model. (b): Hyper-paramette used for AGI. We the default parameteres from the authors' implementation for ImageNet and make minimal changes for CIFAR-10.
B.2 IMPLEMENTING BOUNDARY SEARCH
Our boundary search uses a pipeline of PGDs, CW and AutoPGD. Adversarial examples returned by each method are compared with others and closer ones are returned. If an adversarial example is not found, the pipeline will return the point from the last iteration of the first method (PGDs in our case). Hyper-pane parametrssssrss for each attack can found Table 2 The implementation of PGDs and CW based Foolbox (Rauber et al. 2020 2017) and the implementation of AutoPGD is based
18

---

Under review as a conference paper at ICLR 2022
CIFARIO
standard
robust
68
[0.2, 0.4, 0.6, 0.8, 1.0) 100
0.225, 0.5, 1.0, 1.5, 2.0] 100
steps
Step size
5e-3 standard
5e-3
PGDs
ImageNet
robust
e$ max steps
[36/255., 64/255., 0.3, 1.1]
2.00, 3.0, 5.0, 6.0) 100
100
step size
adaptive
adaptive
CIFARIO
standard
robust
1.0
2.0
max steps
100
100
step size
le-3
le-3
CW
ImageNet
standard
robust
1.0
6.0
max steps
100
100
step size
le-2
5e-2
CIFAR10
standard
robust
1.0
2.0
max steps
100
100
step size
6e-3
1.6e-2
AutoPGD
ImageNet
standard
robust
1.1
6.0
steps
100
100
step size
2.3e-2
1.2e-1
Table 2: Hyper- paremmearessssss used for adversarial attacks. adaptive means the actual step size is determined by 2 * e max steps.
on the authors' public repository (we only use apgd-ce and apgd-dlr losses for efficiency reasons). All computations are done using a GPU accelerator Titan RTX with memory size of 24
GB. Comparisons on the results of the ensemble of these three approaches are shown in Fig. 10a
B.3 HYPER-PARAMETERS AITRIBUTION METHODS
All attributions are implemented with Captum (Kokhlikyan al. 2020) and visualized with Tru- lens (Leino et al. 2021a). For BIG and IG, we use 20 intermediate points between the baseline and the input and the interpolation method set to riemann trapezoid. For AGI, we base on the authors' public repository The choice of hyper-paramters follow the default choice from the authors for ImageNet and we make minimal changes adapt them to CIFAR-10 (see Fig. 1Ob
To visualize the attribution map, we use the HeatmapViss ievvissalli qualizerr izer with blur=10, no rmal wtteiiooontpesssssisssss max " and default values for other keyword arguments from Trulens.
B.4 DETAILED RESULTS ON LOCALIZATION METRICS
We show the average scores for each localizaiton metrics in Sec. 5 We also show the boxplots of the scores for each localization metrics Fig. for the standard ResNet50 model and Fig. 9 for the robust NesNe (&2 3.0). All higher scores are better results.
htteas://tthbb.ccm//rraa/atatttt https://ggtthbb.comm///95//////
19

---

Under review as a conference paper at ICLR 2022
-0.5
-1.0
1 distance in log
-1.5
-2.0
-2.5
-3.0
0.15
0= 0.2
0= 0.25
0= 0.3
0= 0.35
Figure 11: & distances in logarithm between SG and BSG against different standard deviations of the Gaussian noise. Results are computed on ResNet50.
black IG
white IG
BIG
Maltese
Labrador retrie retriever
Labrador retriever
English
foxhound
Labrador retriever
Figure 12: Full results of Fig. 6b in Sec. 6 For the third, fourth and fifth example, we compute the attribution scores towards the prediction of the third example, Labrador retriever. IG with black white attributions show that masked area contribute a lot to the prediction while BIG "accurately" locate the relevant features in the image with the network's prediction.
B.5 ADDITIONAL COMPARISON WITH AGI
We additionally compare the localization ability of relevant features between BIG and AGI if we only PGDs return closest boundary points, that is recursively increase the norm bound and perform PGD attack until the first time we succeed to find an adversarial point. We denote this approach BIGp. Note that BIGp is still different from AGI by the type of the path, i.e. lines
20

---

Under review as a conference paper at ICLR 2022
Model
Metries
BIG
BIGp
AGI
IG
Loc.
0.38
0.38
0.33
0.34
standard
EG
0.54
0.54
0.48
0,5
PP
0.87
0.87
0.58
0.51
Con.
4.35
4.35
4.01
3.97
Loc.
0.39
0.39
0.39
0.33
f/3.00
EG
0.74
0.76
0.64
0.65
PP
0.92
0.96
0.88
0.65
Con.
5.03
5.10
4.32
4.37
Table 3: Comparisons among BIG, BIGp (BIG using PGD only to the boundary search), AGI and IG.
and curves, over which the integral is performed. That AGI also aggregates the path integral starting from set of adversarial points found by the targeted PGD attack, where BIGp starts from the adversarial pointed returned by untargeted PGD attack. We use the same parameters for both PGD and AGI from Fig. 2 and we the experiments reference, we also include the results of IG. The results are shown in Table. We notice that after removing CW and AutoPGD, BIGp actually performs better than AGI, and even slightly better than BIG for the robust model. One reason to explain the tiny improvement from BIGp might be that for a robust network. the gradient at each iteration of the PGD attack more informative and less noisy compared to standard model so that the attack can better approximate the closest decision boundary. The results in Table. 3 therefore demonstrates that BIG and BIGp are able to localize relevant features better than AGI.
the same dataset used in Sec. 5.1 For
B.6 ADDITIONAL. LOCALIZATION METRIC
Besides the localization metrics used in Sec. 5.1 we discuss an additional localization metric fre- quently used for evaluating attention and CAM-based explanations: Topl-LocChhoe Shim (001)) Aggarwal et al.) (220). Topl-Loe is calculated as follows: an instance is considered as Top1-Loc correct given an attribution if 1) the prediction is Top1 ccorectr:ect; and 2) GT-Loc correct namely, the IoU of the ground-truth bounding box and area highlighted by the attribution is more than 50 %. When only using the images that are Topl -correct,, Top1-Loc reduces to GT-Loc. Topl-Loc different from other localization metrics used for evaluating attribution methods because it takes the prediction behavior of the target model into the account, which in general is not an axiom when motivating pradient-based attribution method. In the previous evaluations, in images that the model makes correct Top1 predictions, in this section we will use the same images mea- that are true-positives. In this case, Top1-Loo accuracy reduces to GT-Loc accuracy, and sure the GT-Loc directly. To determine the which part of the image is highlighted by the attribution, compute threshold for each attribution map and pixel is considered within the highlight region if and only if the attribution score is higher than the threshold. For given attribution map, we con- sider threshold value t as the q-th percentile for the absolute values of attribution scores. We plot the GT-Locc accuracy against in Fig. [3] We notice that attention-basse and CAM-based attribu- tions usually produce cloud-like visualization because of the blurring technique or upsample layers used to compute the results. To ensure GT-Loc will work from gradient-based attributions interested in this paper, we also include results (Fig. 14) where we apply a Gaussian Blur (a = 3.0) to the attribution map first before calculating the GT-Loc accuracy. The results are aggreated over 1500 images from ImageNette on a standard ResNet50 and a robust ResNet50, respectively. Higher GT-Loc scores are better.
only interested
Behavior of BIG. The results in Fig. 13 and 14 show that BIG is better than all other attributions on standard models excluding SG and uniformly better including SG on a robust model. Before we provide some explanations about the behaviors of SG (green curves) standard models in the next paragraph, we also observe that: 1) blurring only changes the GT-Loc scores but not the overral rankings across attributions; 2) threshold corresponding to percentile near 40% provides the best GT-Loc scores for all methods; 3) gradient- based attributions generally provide worse GT-Loc (or Topl-Loc) scores compared CAM-based and attention-hased approaches in the literature Choe &
21

---

Under review as a conference paper at ICLR 2022
Shim 2019); Aggarwal et al. 2020), which is not surprising because gradient- baasee approaches to be faithful to the model. Therefore, is expected that the model will more or less learn spurious features from the imput, which makes the gradient- based attributions noisy than attention and CAM-based ones. Therefore, when localizing relevant features, users may want consult activation- birsaation--beed approaches, i.e. CAMs, but when debugging and ensuring the net- work learns less spurious and irrelevant features, users should instead use gradient-basee approaches because of the axioms behind these approaches.
usually axiomatically-justifi
Behavior of SG in Standard Models. SG is uniformly better than all other approaches in terms standard model, which is surprising but not totally unexpected. We
of the Gt-Loc accuracies beleive the reason behind this result is that, SG actually the gradient from a smoothed counterpart Theorem D which is
of the standard model (see discussions not seem to be an apple-to-apple comparison between SG and other approaches because it can be less faithful to the standard model namely SG is more faithful the smoothed classifier. That is very likely why SG is worse than BIG in Fig, 13p and [4p when the smoothing technique becomes marginal for improving the robustness for model that has already been robustly trained.
robust. Therefore, it does
B.7 SANITY CHECK FOR BIG
We perform Sanity Checks for BIG using Rank Order Correlations between the absolute values of BIGs when randomizing the weights from the top layer to bottom (Adebayo et al. 2018)) To ensure the output of the model does not become NaN, when randomizing the weights of each trainable layer, we ensure that we replace the weight matrix with a random matrix with the same norm follows.
def __randdmized_modeee ():
all_parameters []
for param model paw.ell.aaammmeerssearae
all all_parameters. append (param)
for step, param in enumerate [ [al1_parameteers[[:=
random_w
like (param)
# make sure the randomized weights have the same prevent the network to output nan results param. data - torch.nn. m.r.e..m.paaamteer.eeeeee parameter randem_w * torch. norm apparamddataa / torch.nomm(randort if step num_blocks == step len (all para
yield model
12
For each iteration, we continuously replace randomized layers in the reversed sequence returned by model. parameters () and the results are plotted in Fig. 15 We consider BIG passes the sanity check the results are similar compared with the top row of Fig 4 in Adebayo al. (2018).
B.8 ADDITIONAL EXPERIMENT WITH SMOOTHED GRADIENT
Theorem demonstrates that for one-layer network, Gaussian distribution used for creating the smoothed model m, (Cohen et al. 2019), the difference between the saliency map and the boundary- saliency map computed in m. bounded by constant X, which monotonically decreasing w.r.t a.. That is, greater o will produce a smoothed model, where the saliency map (SM) explanation of m. is good approximation for the boundary- based saliency map (BSM). However, as the depth of the deep network increases, a closed-form analysis may be difficult derive. Therefore, in this section, we aim empirically validate that the take-away from Theorem should generalize to deeper networks.
increase the standard deviation of the
Computing SM for ma. One practical issue to compute any gradient-related explanations for the smoothed model m, is that m, defined in an integral form, which can not be directly built with terker.. However, Theorem 2 shows that the smoothed gradient of the original model is equivalent to the saliency map of the smoothed model m., Namely, the order of smoothing and integral is exchangeable when computing the gradient.
22

---

Under review as a conference paper at ICLR 2022
SM
0.35
35M
BSM
0.30
5G
0.25
GT
0.20
AGI
BBG
0.2
0.15
0.10
0.1
0.05
0.00
0%
10%
20%
30%
40%%
50%
60%
70%
m2%
90%
0%
10%
20%
30%
40%
50%
60%
70% 80%
90%
Threshold
Threshold
(a) standard
(h) robust (62 (.33)
Figure 13: GT-Loc scores for different attributions. GT-Loc measures the portion of instances where the loUs of between the groudtruth bounding box and the bounding box generated by thresholded the attributions greater than 0.5. The x-axis is the percentile used to threshold attribution map to determine the highlighted area and y-axis is the GT-Loc score aggregated over all the instances.
SM
9.4
0.3
Bt
0.2
0.0
10%
20%
30%
40%
50%
60%
70%
80%
90%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Threshel
(a) standard
(b) robust (..3)
Figure 14: GT-Loc scores for different attributions when applying a Gaussian blur kernel (a 3.0) to the attribution maps before thresholding the attribution maps.
Theorem (Proposition from Wang et al. (2020c)
Suppose
model f(x)
satisfies
max [f()) < co. For Smoothed Gradient gsG (sG( we have
9sc(x) Ø(f @ (x)
(24)
where ((x) = N(,,0 and @ denotes the convolution operation.
Computing BSM for my. Another practical issue is computing the decision boundary for a smoothed model m, defined deterministic way as randomized smoothing provides a prob- abilistic guarantee. In this paper, we do the following steps to approximate the decision boundary of smoothed model. To generate the adversarial examples for the smoothed classifier of ResNet50 with randomized smoothing, we need to compute back-propagation through the noises. The noise sampler usually not accessible to the attacker who wants to fool model with randomized smooth- ing. However, goal in this section is not to reproduce the attack with similar setup in practice, instead, what we are after the point on the boundary. We therefore do the noise sampling prior to PGD attack, and
the same noise across all the instances. The steps are listed as follows:
1. We use numpy. random. randn as the sampler for Gaussian noise with its random seed set 2020. We use 50 random noises per instance.
2. In PGD attack, we aggregate the gradients of all 50 random inputs before we take regular step to update the input.
3. We set 3.0 and
most 40 iterations with step size of 2 e/40.
23

---

Under review as a conference paper at ICLR 2022
Rank Correlation with ABS
34-
0--
341
Origha15
15
95 150 120 of Layers with Random Weights
140
250 155
Figure 15: The rank order correlations of the absolute values of BIGs against the number of layers (counting from top to bottom) where trainable weights are replaced with random matrices.
4. The early stop criteria for the loop of PGD is that when less than 10% of all randomized points have the original prediction.
5. When computing Smooth Gradient for the original points for the adversarial points, use the same random noise that we generated to approximate the smoothed classifier.
Results. We run the experiment with 500 images from ImageNet on ResNet50 as this computation
significantly more expensive than previous experiments. We compute the &g distances between SM and BSM obtained from the steps above for several values as shown in Fig. [I] Notably, the trend of the log difference against the standard deviation used for the Gaussian noise validates that the qualitative meaning of Theorem holds even for large networks. That is, when the model becomes more smoothed, saliency map explanation is good approximation for the boundary-based saliency map.
C SYMMETRY OF ATTRIBUTION METHODS
Sundararajan et al. (017)) prove that linear path is the only path integral that satisifes symmetry; that is, when two features' orders are changed for a network that not using any order information from the input, their attribution scores should not change. One simple way show the importance of symmetry by the following example and we refer Sundararaaan et al. (2017) to readers for more analysis.
Example Consider a function fxx y) = min(x, y) and to attribute the output of the inputs
at x. 1, = 1 we consider baseline
0, 0. An example non-linear path from the
baseline the input can be (x = 0,y 0) (xr 1,y = -)- (a 1, 1). On this path, f(z,y) min(æ, y) y after the point (x L, 0); therefore, gradient integral will return 0
for the attribution score of x and 1 for (we ignore the infinitesimal part of (æ = 0, = 0) + (x
1, 0)). Similarly, when choosing path (z: 0. 0)
0, y 1) (z 1, 1), we find x is more important. Only the linear path will return 1 for both variables in this case.
D COUNTERFACTUAL ANALYSIS IN THE BASELINE SELECTION
The discussion in Sec. 6 shows an example where there are two dogs in the image. IG with black baseline shows that the body of the white dog also useful to the model to predict label and the black dog is mix: part of the black dog has positive attributions and the rest is negatively contribute to the prediction. However, our proposed method BIG clearly shows that the most important part the black dog and then comes to the white dog. To validate where the model is actually using the white dog, we manually remove the black dog or the white dog from the image and see the model retain its prediction. The result is shown in Fig. 12 Clearly, when removing the black dog, the model changes its prediction from Labrador retriever to Englist foxhound while removing the white dog does not change the prediction. This result helps convince the reader that
24

---

Under review as a conference paper at ICLR 2022
BIG is more reliable than IG with black baseline in this case more faithful explanation to the classification result for this instance.
E ADDITIONAL VISUALIZATIONS FOR BIG
More visualizations comparing BIG with other attributions can be found Fig. 16 and 17 show several examples in Fig. 18 when there are more than one objects in the input and we explain the model's Top1 prediction, where we show that BIG able to localize the objects that are actually relevant to the predicted label.
We
25

---

Under review as a conference paper at ICLR 2022
DeepLIFT
Figure 16: Visualizations of different attributions for standard ResNet50
26

---

Under review as a conference paper at ICLR 2022
DeepLIFT
Figure 17: Visualizations of different attributions for robust (2/3.00 ResNet50
27

---

Under review as a conference paper at ICLR 2022
vidi
SM
G11
16
5G
DecpUFT
AGI
BIG
!!!!!!
||1111
teapst
Labradoe
jeaamr
Figure 18: Visualizations of different attributions for a standard ResNet50 where there are usually more than objects in the input. We also label each input with the Top I prediction made by the classifier.
28


Reviews:
[{"review_id": "rHW42cPDhuI", "reviewer": "Reviewer_oVRP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n+ the general idea of alignment with the nearest decision hyperplane normal and the specific modification of IG seem quite novel and plausible.\n+ In the experiments, BIG achieves significantly better results using various explanation metrics.\n\n**Questions to authors**\n- Smoothening the learnt function, at some point, should start losing the discrmination ability of the learnt function. Has the authors pushed enough to find some indication of this trade-off?\n- From theorem 1, I can understand why a smoother learned functions can give rise to a more faithful saliency-based explanation but I cannot see how it advocates smoothgrad as explanation. Wouldn't smoothgrad be faithful to a very-likely different function than the actual learned function and thus not necessarily faithful to the true learned function? \n- The text before definition 6, argues for BIG based on the existence of multiple boundary segments near a point and proposed definition 6 that integrates over the segment connecting a point x to its nearest adversarial $x'$. However, shouldn't the nearest decision boundary segment for all points along the line segment $x\\rightarrow x'$ remain the same? The integral is taken over standard saliency g which of course can change linear regions but the rationale (of wanting to find different decision boundary hyperplanes) does not seem to hold for the proposal.\n- The previous question could be simply rectified if the meaning of \"boundary segments\" is the linear regions' boundary segments as opposed to the decision boundary segments but then I think \"boundary\" has been used as \"decision boundary\" at occasions before this definition, *e.g.*, in def 5. Am I mistaken? If not, the text needs a rewrite to distinguish between \"regions boundary segments\" and \"decision boundary segments\".\n- Due to the approximation using an ensemble of adversarial example methods, we should expect that the found segment is very likely not the closest decision boundary segment (since we know from many works that the density of linear regions are extremely high in the input space). In light of that, how reliable are the observations in the experiments section? Especially, with regards to the deviation from the normal vector (Figure 3.a). \n- Following up on the previous question, could the fact that BSM does not show improvement on standard models be due to this approximation?\n\n**Minor points**\n- on many occasions, when referring to boundary facets of a polytope, better to use hyperplane as opposed to segment to avoid confusion with line segments that are used as linear path.\n- In definition 3, $f(\\alpha + \\epsilon) \\rightarrow f(x + \\epsilon)$\n- In Theorem 1, $\\forall x'' \\in B(...).$ better to replace $. \\rightarrow ,$ (although a minor point, it makes reading the statement challenging in the first glance. )\n- In Theoretm 1, it might be better to use $O(\\frac{1}{\\sigma c})$? \n- two different notations are used for definition (:= or else)\n- better to refer to Figure 3.a and 3.b as tables\n- page 7: \"a smaller difference between the difference between\nattributions\"\n- page 7: \"instead evaluates computes\"\n- page 8: \"It is naturally to treat BIG frees users from the baseline selection\"\n\n", "summary_of_the_paper": "The idea is that as one smoothens the decision boundary of a piecewise linear function $f$ (e.g., from ReLU-Net) its saliency map ($g$) obtained by $g=\\frac{df}{dx}$ gets closer ($||g - n||_2$) to the normal of the closest boundary hyperplane ($n$). The authors then propose two variants of explanation techniques based on the nearest decision boundary hyperplane and try it on explaining trained image deep classifiers. The results seem to corroborate as for a better alignment with the nearest boundary hyperplane's normal. Also, the proposed methods achieve better explanations as measured by locality and overlap with ground-truth bounding boxes.", "summary_of_the_review": "The paper has an interesting and original idea which brings consistent improvement to the established explanations techniques such as gradient-based saliency maps and integrated gradients. However, there are some questions that makes me keep my rating only at borderline accept.\n\n---\n**Post-Rebuttal Comments**\n\n*General rationale for the updated score*: after some more thoughts and the discussions during the rebuttal phase, the reviewer remains unconvinced about the claim of the normality of SM to (the extension of) a segment in the decision boundary. In this regard there are two significant concerns: (i) there are explicit statements about this both in the revised paper (see discussion with the authors for some instances) as well as the authors arguments during the discussion, and (ii) the motivation for the proposed method BIG is based on this claim. Furthermore, the paper cites other papers that as far as the reviewer understands do not (explicitly) discuss this claim. Therefore, I do not think I can vouch for accepting the paper claiming (and building on) some formal statements that I cannot personally verify. Consequently, I reduce my rating from 6 to 3. Since there might be a simple point that I am missing here which would prove the claim, I reduce my confidence as well from 4 to 3.\n\n*Summary of the technical discussion*:  The authors, at various points, implicitly suggest or explicitly claim that SM which is the gradient of the network's function w.r.t. the input ($\\frac{df}{dx}$) is perpendicular to (the extension of) a segment in the decision boundary. This is then used to motivated a variant of IG, called BIG which integrates SM over a line path from the sample to the nearest adversarial example. For the reviewer it is possible to see: (i) how $\\frac{df}{dx}$ for a linear binary classifier will always be orthogonal to the decision boundary since the decision boundary is by definition a hyperplane with the SM as its normal (as described in section 3.1), (ii) how $\\frac{d(f_i-f_j)}{dx}$ is orthogonal to the surface $f_i-f_j=0$. However, it is unclear to the reviewer how $\\frac{df}{dx}$ can be guaranteed to be prependicular to the decision boundary of a general function $f: \\mathbb{R}^d\\rightarrow\\mathbb{R}^K$ with K being the number of classes. In fact, I believe for the simplest case of linear binary classification, as soon as we (redundantly) model each class with a separate linear model (to become analogue to the multiclass setup), the gradient of each of the linear functions i.e., $\\frac{df_1}{dx}$ and $\\frac{df_2}{dx}$ will no more be orthogonal to the decision boundary.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "lg5vuaqzb9N", "reviewer": "Reviewer_NMCL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has an interesting topic: adding theoretical insights to explainability methods. The paper does especially well on providing a good intuition about the relationships of normals, polytopes and decision boundary (content of 3.1 and first part of 3.2). I also found the paper overall well written (some minor typos and duplicates are listed below). The paper's story of first analyzing the limitations of gradients, fixing the errors, and then evaluating the methods is also good. I address my concerns about the generality and rigor of theorem 1, the evaluation, and the limitations below.\n\n### Proof of theorem 1\n\nTheorem 1 contains an $\\lessapprox$ sign. After checking the appendix, it turns out that the proof is only correct for the case that\n\n> (Dombrowski et al., 2019) points out that the random distribution $p_β (\\epsilon_{i}) =\\frac{β}{(exp (β \\epsilon_{i} / 2)+exp ( β \\epsilon_{i} / 2))^2}\n> $closely resembles a normal distribution with a standard deviation $\\sigma = \\sqrt{\\log 2 \\frac{\\sqrt{2 \\pi}}{\\beta}}$.\n\nHowever, under which conditions does it resemble a normal distribution? (Dombrowski et al., 2019) only made this comment to explain a possible connection to SmoothGrad (see page 8 in Dombrowski et al., 2019). No concrete conditions are given on when or how close the distributions matches. I did not even found how  $\\sigma$  was derived in (Dombrowski et al., 2019) (if you know where, please point me to it). I did a small experiment myself and plotted the distributions. For each plot, the corresponding $\\beta$ is given on top and the normal distribution has $\\sigma = \\sqrt{\\log 2 \\frac{\\sqrt{2 \\pi}}{\\beta}}$ (for the notebook with the code see this [link](https://f002.backblazeb2.com/file/nnnnnnnn/iclr2022/Robust+Models+Are+More+Interpretable+Because+Attributions+Look+Normal.ipynb)).\n\n[[Plots for different $\\beta$s]](https://f002.backblazeb2.com/file/nnnnnnnn/iclr2022/beta_plots.png)\n\n\nAs you can see, it is only close for $\\beta \\approx 1$. Two solutions exist: either provide a theorem with $\\leq$ or give a rigorous discussion on the cases where only $\\approx$  or even > holds. \n\nThe other limitation of Theorem 1 is that it only holds for one-layer ReLU networks. I would find a short discussion helpful why it does not hold for n-layer ReLU networks. In addition, it should be emphasized throughout the paper that Theorem 1 is only for one-layer networks. For example, in the last paragraph of the introduction:\n\n> We present an analysis that sheds light on the previously-observed phenomeon of robust interpretability, showing that alignment between the normal vectors of decision boundaries and models’ gradients is a key ingredient (Proposition 1, Theorem 1)\n\nPlease make clear in that sentence and others that Theorem 1 only addresses one-layer networks. \n\nAt the end of section 3.2, Figure 10 is referenced as empirical validation of Theorem 1, but I do not understand the figure and caption:\n\n> distances in logarithm between SG and BSG against different standard deviations σ of the Gaussian noise. Results are computed on ResNet50. Notice the first column corresponds to σ = 0.\n\nPlease, clarify what you want to evaluate with this figure, e.g. the first column says $\\sigma=0.15$.\n\n### Evaluation\n\nI think the evaluation of the normality to the decision boundary can be improved. In Figure 3, pairs of gradient attribution method and the corresponding boundary attributions (e.g. IG vs. BIG) are compared to evaluate how normal the attributions are. However, why not measure the normality in the feature space $z(x)$ directly.  $z(x)$ is defined such that $f_i(x) = w_i^T z(x)$. We know that $w_i$ must be normal to the decision boundary, as shown in Figure 2a. The corresponding change in z-space of an attribution $g(x)$ would be $\\Delta z = z(x) - z(x + \\alpha g(x))$. Now, we can measure the similarity of the normal $w_i$ and the different attributions: just compute $\\cos(\\Delta z,  w_i)$ for all the different attributions. This evaluation would relate the estimated directions in $x$-space to the ground-truth normals in $z$-space. The current evaluation of attributions methods against their boundary equivalent cannot provide such a ground-truth reference. \n\nThe evaluation using the ground-truth bounding boxes is a good proxy task and seems to be executed correctly. It might make sense to only use images where the bounding box covers less than 50% of the image, as done in (Schulz et al., 2020). The attribution method in (Schulz et al., 2020) might also be an interesting candidate for the evaluation as it was also able to outperform int.grad. and smooth grad. I would also suggest focusing on one or two metrics for the bounding box task instead of four.\n\nI would also encourage the authors to include the sanity check for weight reinitialization (Adebayo et al., 2018). It is easy to implement and should be passed by any new attribution method. \n\n### Limitations\n\nWhile I do not think that the paper requires a human-subject evaluation, its lack should be mentioned in the limitation section. Also the saliency maps look more concentrated, would humans actually profit from it? Even if there is a significant difference, would you expect a large effect size? Please also list that theorem 1 is only for one-layered networks in the limitations. Limitation 2 (not applicable to perturbation attributions) arises from focus of the paper and I think there is not need to mention it. \n\n### Minor Comemnts:\n\n* \"In fact, the fact\" (page 4)\n* smaller difference between the difference between (page 7)\n* Thefore (page 7)\n* Lost clause: \"It is naturally to treat \" (page 8)\n* It should be Table 3 and not Figure 3\n* IamgeNet (page 6)\n* I think it should be \"The RHS of the above equation is Smoothed Gradient\" (page 15)\n\n### References:\n\n(Schulz et al., 2020) https://openreview.net/forum?id=S1xWh1rYwB\n\n(Adebayo et al., 2018) https://arxiv.org/abs/1810.03292\n\n## After Rebuttal Update\n\nThe authors were able to rectify their proof and also provided details to my other questions. While the initial submission was a clear reject, the rebuttal was well done. I agree with the concerns of the others reviewers about novelty. Overall, I increased my rating to marginal above acceptance.", "summary_of_the_paper": "The paper focuses on the intersection of gradient attribution and adversarial robustness. First, it analyzes the weaknesses of vanilla gradients: the gradient does not have to point towards the decision boundary of an n-layer ReLU network. Then the paper provides some insights into the smoothing of one-layer ReLU networks (Theorem 1). Finally, a boundary-based saliency map and an extension of integrated gradients are proposed and evaluated in terms of boundary alignment and object localization. \n", "summary_of_the_review": "While I like that the paper aims to provide a more theoretical justification on attributions, I am not satisfied with the rigor of the theory and the empirical evaluation. I am not convinced that the proof is correct and the alignment with the normals should be checked using ground-truth knowledge. Overall, The paper is well-written, but please fix the grammar. I cannot recommend the paper in its current form for acceptance. If the proof were corrected and the evaluation extend to a ground-truth assessment of the normal, I would reconsider my rating.\n\n**Technical Novelty:** The papers technical contribution is novel. I am less convinced about the significance in its current form. \n\n**Empirical Novelty:** The paper does not present new empirical evaluations or datasets.\n\n**Confidence:** I am confident about my assessment. I read the proof and investigated the issue of resembles-a-normal-distribution in depth. I still might have missed other issues of the proof. While I did looked at the referenced literature about adversarial examples, I am more familiar with the interpretability side of the related work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "gqK1Pb5a1Wp", "reviewer": "Reviewer_qvFb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths -   \n1) The motivation of the idea is well explained in the paper.\n2) The mathematical foundation required for understanding is also well explained.\n3) I like the effort put in the paper in understanding the reasoning behind interpretable attributions for robust models and then using the info to devise new attribution methods.\n4) For both claims, the paper does extensive qualitative and quantitative experiments.\n\nWeakness - \n1) The new attributions devised in the paper seem very similar to the AGI attribution(mentioned in the paper) approach. In BIG, the attributions are computed along interpolations of x and its closest adversarial image, whereas in AGI the attributions are computed along each step of the adversarial image generation.\n2) In Table1 mentioned in the paper, the improvements along the two metrics used in other papers are not really significant. The improvement only comes along with the two new metrics proposed in this paper. I would like to see a comparison against some other metrics used in the related works such as top-1 localization accuracy as used in [1] and [2].\n3) For a fairer comparison with the AGI method, can the authors use only the PGD attack for the adversarial image generation? Or, the authors can also incorporate other adversarial images (and not just PGD) in AGI. For instance, the AGI method can be used to compute the attributions along each step of PGD, CW, and AutoPGD attacks and the final attribution is just the mean attribution of all three approaches.\n4) [3] showed that their attribution technique works well with even multiple objects in the image. Can the authors show some qualitative results of comparison for multiple objects across different attribution methods.\n\nReferences -   \n[1] Attention-based Dropout Layer for Weakly Supervised Object Localization. Choe et al. 2019. \n[2] On The Benefits Of Models With Perceptually  Aligned Gradients. Aggarwal et al. 2020. \n[3] Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks. Wang et al. 2020. ", "summary_of_the_paper": "The paper has two main contributions.   \na) First it shows that one reason behind the attributions being more interpretable for adversarial robust models is that for these models, the gradient with respect to the input is more closely aligned with the normal direction to a close decision boundary. They empirically verify this claim, by showing that the l2-distance between attributions and their boundary variants(attributions computed at a close point on the decision boundary) are lower for robust models than for standard models.   \nb) Using the previous fact, they devise two new attribution methods, BSM and BIG which can be used to get more interpretability/explanation from even a normal (non-robust) model. They again verify this claim empirically through various quantitative metrics aimed at finding the relation between positive attributions inside a localized bounding box of an object in the image.", "summary_of_the_review": "I have a few concerns regarding the quantitative experiments in the paper, which are mentioned in the Weakness section. I will be willing to update my ratings if the authors address all my points. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O950apcHslM", "reviewer": "Reviewer_t25u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Table 1 shows the empirical results are good.\n\nWeakness:\n\n1. My major concern for this paper is that the conclusion has already known. For example, Ilyas et al shows that robust models can produce better perceptual aligned features when gradient descent, and adversarial robust models are known to have smooth decision boundary [1].\n\n[1] Theoretically Principled Trade-off between Robustness and Accuracy. ICML 2019. ", "summary_of_the_paper": "This paper introduces boundary attributions, which leverage the connection between boundary normal vectors and gradients to yield explanations for non-robust models that carry over many of the favorable properties that have been observed of explanations on\nrobust models. It also proposes a BIG to explain models. ", "summary_of_the_review": "The conclusion is not novel to me, which is already known to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes the following contributions -- 1) it shows that one reason behind the attributions being more interpretable for adversarial robust models is that for these models, the gradient with respect to the input is more closely aligned with the normal direction to a close decision boundary. 2) Using the previous fact, the authors devise two new attribution methods -- BSM and BIG -- which can be used to get more reliable explanations from even a normal (non-robust) model. While the reviewers agree that the premise of this paper is interesting, some concerns remain post the rebuttal. More specifically, some reviewers opine that the AGI and BIG methods are somewhat similar, and other reviewers are not very convinced about some of the details e.g., the generalization of the orthogonality of SM to the decision boundary from the binary classification case (section 3.1) to the more general case of ReLU-Net's multi-class classifiers (section 3.2). Given this, we are unable to accept this paper at this time. We hope the authors find the reviewer feedback useful.
Decision: Reject