Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The idea is that as one smoothens the decision boundary of a piecewise linear function $f$ (e.g., from ReLU-Net) its saliency map ($g$) obtained by $g=\frac{df}{dx}$ gets closer ($||g - n||_2$) to the normal of the closest boundary hyperplane ($n$). The authors then propose two variants of explanation techniques based on the nearest decision boundary hyperplane and try it on explaining trained image deep classifiers. The results seem to corroborate as for a better alignment with the nearest boundary hyperplane's normal. Also, the proposed methods achieve better explanations as measured by locality and overlap with ground-truth bounding boxes.
main_review: **Strengths**
+ the general idea of alignment with the nearest decision hyperplane normal and the specific modification of IG seem quite novel and plausible.
+ In the experiments, BIG achieves significantly better results using various explanation metrics.

**Questions to authors**
- Smoothening the learnt function, at some point, should start losing the discrmination ability of the learnt function. Has the authors pushed enough to find some indication of this trade-off?
- From theorem 1, I can understand why a smoother learned functions can give rise to a more faithful saliency-based explanation but I cannot see how it advocates smoothgrad as explanation. Wouldn't smoothgrad be faithful to a very-likely different function than the actual learned function and thus not necessarily faithful to the true learned function? 
- The text before definition 6, argues for BIG based on the existence of multiple boundary segments near a point and proposed definition 6 that integrates over the segment connecting a point x to its nearest adversarial $x'$. However, shouldn't the nearest decision boundary segment for all points along the line segment $x\rightarrow x'$ remain the same? The integral is taken over standard saliency g which of course can change linear regions but the rationale (of wanting to find different decision boundary hyperplanes) does not seem to hold for the proposal.
- The previous question could be simply rectified if the meaning of "boundary segments" is the linear regions' boundary segments as opposed to the decision boundary segments but then I think "boundary" has been used as "decision boundary" at occasions before this definition, *e.g.*, in def 5. Am I mistaken? If not, the text needs a rewrite to distinguish between "regions boundary segments" and "decision boundary segments".
- Due to the approximation using an ensemble of adversarial example methods, we should expect that the found segment is very likely not the closest decision boundary segment (since we know from many works that the density of linear regions are extremely high in the input space). In light of that, how reliable are the observations in the experiments section? Especially, with regards to the deviation from the normal vector (Figure 3.a). 
- Following up on the previous question, could the fact that BSM does not show improvement on standard models be due to this approximation?

**Minor points**
- on many occasions, when referring to boundary facets of a polytope, better to use hyperplane as opposed to segment to avoid confusion with line segments that are used as linear path.
- In definition 3, $f(\alpha + \epsilon) \rightarrow f(x + \epsilon)$
- In Theorem 1, $\forall x'' \in B(...).$ better to replace $. \rightarrow ,$ (although a minor point, it makes reading the statement challenging in the first glance. )
- In Theoretm 1, it might be better to use $O(\frac{1}{\sigma c})$? 
- two different notations are used for definition (:= or else)
- better to refer to Figure 3.a and 3.b as tables
- page 7: "a smaller difference between the difference between
attributions"
- page 7: "instead evaluates computes"
- page 8: "It is naturally to treat BIG frees users from the baseline selection"


summary_of_the_review: The paper has an interesting and original idea which brings consistent improvement to the established explanations techniques such as gradient-based saliency maps and integrated gradients. However, there are some questions that makes me keep my rating only at borderline accept.

---
**Post-Rebuttal Comments**

*General rationale for the updated score*: after some more thoughts and the discussions during the rebuttal phase, the reviewer remains unconvinced about the claim of the normality of SM to (the extension of) a segment in the decision boundary. In this regard there are two significant concerns: (i) there are explicit statements about this both in the revised paper (see discussion with the authors for some instances) as well as the authors arguments during the discussion, and (ii) the motivation for the proposed method BIG is based on this claim. Furthermore, the paper cites other papers that as far as the reviewer understands do not (explicitly) discuss this claim. Therefore, I do not think I can vouch for accepting the paper claiming (and building on) some formal statements that I cannot personally verify. Consequently, I reduce my rating from 6 to 3. Since there might be a simple point that I am missing here which would prove the claim, I reduce my confidence as well from 4 to 3.

*Summary of the technical discussion*:  The authors, at various points, implicitly suggest or explicitly claim that SM which is the gradient of the network's function w.r.t. the input ($\frac{df}{dx}$) is perpendicular to (the extension of) a segment in the decision boundary. This is then used to motivated a variant of IG, called BIG which integrates SM over a line path from the sample to the nearest adversarial example. For the reviewer it is possible to see: (i) how $\frac{df}{dx}$ for a linear binary classifier will always be orthogonal to the decision boundary since the decision boundary is by definition a hyperplane with the SM as its normal (as described in section 3.1), (ii) how $\frac{d(f_i-f_j)}{dx}$ is orthogonal to the surface $f_i-f_j=0$. However, it is unclear to the reviewer how $\frac{df}{dx}$ can be guaranteed to be prependicular to the decision boundary of a general function $f: \mathbb{R}^d\rightarrow\mathbb{R}^K$ with K being the number of classes. In fact, I believe for the simplest case of linear binary classification, as soon as we (redundantly) model each class with a separate linear model (to become analogue to the multiclass setup), the gradient of each of the linear functions i.e., $\frac{df_1}{dx}$ and $\frac{df_2}{dx}$ will no more be orthogonal to the decision boundary.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on the intersection of gradient attribution and adversarial robustness. First, it analyzes the weaknesses of vanilla gradients: the gradient does not have to point towards the decision boundary of an n-layer ReLU network. Then the paper provides some insights into the smoothing of one-layer ReLU networks (Theorem 1). Finally, a boundary-based saliency map and an extension of integrated gradients are proposed and evaluated in terms of boundary alignment and object localization. 

main_review: The paper has an interesting topic: adding theoretical insights to explainability methods. The paper does especially well on providing a good intuition about the relationships of normals, polytopes and decision boundary (content of 3.1 and first part of 3.2). I also found the paper overall well written (some minor typos and duplicates are listed below). The paper's story of first analyzing the limitations of gradients, fixing the errors, and then evaluating the methods is also good. I address my concerns about the generality and rigor of theorem 1, the evaluation, and the limitations below.

### Proof of theorem 1

Theorem 1 contains an $\lessapprox$ sign. After checking the appendix, it turns out that the proof is only correct for the case that

> (Dombrowski et al., 2019) points out that the random distribution $p_β (\epsilon_{i}) =\frac{β}{(exp (β \epsilon_{i} / 2)+exp ( β \epsilon_{i} / 2))^2}
> $closely resembles a normal distribution with a standard deviation $\sigma = \sqrt{\log 2 \frac{\sqrt{2 \pi}}{\beta}}$.

However, under which conditions does it resemble a normal distribution? (Dombrowski et al., 2019) only made this comment to explain a possible connection to SmoothGrad (see page 8 in Dombrowski et al., 2019). No concrete conditions are given on when or how close the distributions matches. I did not even found how  $\sigma$  was derived in (Dombrowski et al., 2019) (if you know where, please point me to it). I did a small experiment myself and plotted the distributions. For each plot, the corresponding $\beta$ is given on top and the normal distribution has $\sigma = \sqrt{\log 2 \frac{\sqrt{2 \pi}}{\beta}}$ (for the notebook with the code see this [link](https://f002.backblazeb2.com/file/nnnnnnnn/iclr2022/Robust+Models+Are+More+Interpretable+Because+Attributions+Look+Normal.ipynb)).

[[Plots for different $\beta$s]](https://f002.backblazeb2.com/file/nnnnnnnn/iclr2022/beta_plots.png)


As you can see, it is only close for $\beta \approx 1$. Two solutions exist: either provide a theorem with $\leq$ or give a rigorous discussion on the cases where only $\approx$  or even > holds. 

The other limitation of Theorem 1 is that it only holds for one-layer ReLU networks. I would find a short discussion helpful why it does not hold for n-layer ReLU networks. In addition, it should be emphasized throughout the paper that Theorem 1 is only for one-layer networks. For example, in the last paragraph of the introduction:

> We present an analysis that sheds light on the previously-observed phenomeon of robust interpretability, showing that alignment between the normal vectors of decision boundaries and models’ gradients is a key ingredient (Proposition 1, Theorem 1)

Please make clear in that sentence and others that Theorem 1 only addresses one-layer networks. 

At the end of section 3.2, Figure 10 is referenced as empirical validation of Theorem 1, but I do not understand the figure and caption:

> distances in logarithm between SG and BSG against different standard deviations σ of the Gaussian noise. Results are computed on ResNet50. Notice the first column corresponds to σ = 0.

Please, clarify what you want to evaluate with this figure, e.g. the first column says $\sigma=0.15$.

### Evaluation

I think the evaluation of the normality to the decision boundary can be improved. In Figure 3, pairs of gradient attribution method and the corresponding boundary attributions (e.g. IG vs. BIG) are compared to evaluate how normal the attributions are. However, why not measure the normality in the feature space $z(x)$ directly.  $z(x)$ is defined such that $f_i(x) = w_i^T z(x)$. We know that $w_i$ must be normal to the decision boundary, as shown in Figure 2a. The corresponding change in z-space of an attribution $g(x)$ would be $\Delta z = z(x) - z(x + \alpha g(x))$. Now, we can measure the similarity of the normal $w_i$ and the different attributions: just compute $\cos(\Delta z,  w_i)$ for all the different attributions. This evaluation would relate the estimated directions in $x$-space to the ground-truth normals in $z$-space. The current evaluation of attributions methods against their boundary equivalent cannot provide such a ground-truth reference. 

The evaluation using the ground-truth bounding boxes is a good proxy task and seems to be executed correctly. It might make sense to only use images where the bounding box covers less than 50% of the image, as done in (Schulz et al., 2020). The attribution method in (Schulz et al., 2020) might also be an interesting candidate for the evaluation as it was also able to outperform int.grad. and smooth grad. I would also suggest focusing on one or two metrics for the bounding box task instead of four.

I would also encourage the authors to include the sanity check for weight reinitialization (Adebayo et al., 2018). It is easy to implement and should be passed by any new attribution method. 

### Limitations

While I do not think that the paper requires a human-subject evaluation, its lack should be mentioned in the limitation section. Also the saliency maps look more concentrated, would humans actually profit from it? Even if there is a significant difference, would you expect a large effect size? Please also list that theorem 1 is only for one-layered networks in the limitations. Limitation 2 (not applicable to perturbation attributions) arises from focus of the paper and I think there is not need to mention it. 

### Minor Comemnts:

* "In fact, the fact" (page 4)
* smaller difference between the difference between (page 7)
* Thefore (page 7)
* Lost clause: "It is naturally to treat " (page 8)
* It should be Table 3 and not Figure 3
* IamgeNet (page 6)
* I think it should be "The RHS of the above equation is Smoothed Gradient" (page 15)

### References:

(Schulz et al., 2020) https://openreview.net/forum?id=S1xWh1rYwB

(Adebayo et al., 2018) https://arxiv.org/abs/1810.03292

## After Rebuttal Update

The authors were able to rectify their proof and also provided details to my other questions. While the initial submission was a clear reject, the rebuttal was well done. I agree with the concerns of the others reviewers about novelty. Overall, I increased my rating to marginal above acceptance.
summary_of_the_review: While I like that the paper aims to provide a more theoretical justification on attributions, I am not satisfied with the rigor of the theory and the empirical evaluation. I am not convinced that the proof is correct and the alignment with the normals should be checked using ground-truth knowledge. Overall, The paper is well-written, but please fix the grammar. I cannot recommend the paper in its current form for acceptance. If the proof were corrected and the evaluation extend to a ground-truth assessment of the normal, I would reconsider my rating.

**Technical Novelty:** The papers technical contribution is novel. I am less convinced about the significance in its current form. 

**Empirical Novelty:** The paper does not present new empirical evaluations or datasets.

**Confidence:** I am confident about my assessment. I read the proof and investigated the issue of resembles-a-normal-distribution in depth. I still might have missed other issues of the proof. While I did looked at the referenced literature about adversarial examples, I am more familiar with the interpretability side of the related work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper has two main contributions.   
a) First it shows that one reason behind the attributions being more interpretable for adversarial robust models is that for these models, the gradient with respect to the input is more closely aligned with the normal direction to a close decision boundary. They empirically verify this claim, by showing that the l2-distance between attributions and their boundary variants(attributions computed at a close point on the decision boundary) are lower for robust models than for standard models.   
b) Using the previous fact, they devise two new attribution methods, BSM and BIG which can be used to get more interpretability/explanation from even a normal (non-robust) model. They again verify this claim empirically through various quantitative metrics aimed at finding the relation between positive attributions inside a localized bounding box of an object in the image.
main_review: Strengths -   
1) The motivation of the idea is well explained in the paper.
2) The mathematical foundation required for understanding is also well explained.
3) I like the effort put in the paper in understanding the reasoning behind interpretable attributions for robust models and then using the info to devise new attribution methods.
4) For both claims, the paper does extensive qualitative and quantitative experiments.

Weakness - 
1) The new attributions devised in the paper seem very similar to the AGI attribution(mentioned in the paper) approach. In BIG, the attributions are computed along interpolations of x and its closest adversarial image, whereas in AGI the attributions are computed along each step of the adversarial image generation.
2) In Table1 mentioned in the paper, the improvements along the two metrics used in other papers are not really significant. The improvement only comes along with the two new metrics proposed in this paper. I would like to see a comparison against some other metrics used in the related works such as top-1 localization accuracy as used in [1] and [2].
3) For a fairer comparison with the AGI method, can the authors use only the PGD attack for the adversarial image generation? Or, the authors can also incorporate other adversarial images (and not just PGD) in AGI. For instance, the AGI method can be used to compute the attributions along each step of PGD, CW, and AutoPGD attacks and the final attribution is just the mean attribution of all three approaches.
4) [3] showed that their attribution technique works well with even multiple objects in the image. Can the authors show some qualitative results of comparison for multiple objects across different attribution methods.

References -   
[1] Attention-based Dropout Layer for Weakly Supervised Object Localization. Choe et al. 2019. 
[2] On The Benefits Of Models With Perceptually  Aligned Gradients. Aggarwal et al. 2020. 
[3] Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks. Wang et al. 2020. 
summary_of_the_review: I have a few concerns regarding the quantitative experiments in the paper, which are mentioned in the Weakness section. I will be willing to update my ratings if the authors address all my points.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces boundary attributions, which leverage the connection between boundary normal vectors and gradients to yield explanations for non-robust models that carry over many of the favorable properties that have been observed of explanations on
robust models. It also proposes a BIG to explain models. 
main_review: Strengths:

1. Table 1 shows the empirical results are good.

Weakness:

1. My major concern for this paper is that the conclusion has already known. For example, Ilyas et al shows that robust models can produce better perceptual aligned features when gradient descent, and adversarial robust models are known to have smooth decision boundary [1].

[1] Theoretically Principled Trade-off between Robustness and Accuracy. ICML 2019. 
summary_of_the_review: The conclusion is not novel to me, which is already known to the community.
review_title: null

Meta-review: This paper makes the following contributions -- 1) it shows that one reason behind the attributions being more interpretable for adversarial robust models is that for these models, the gradient with respect to the input is more closely aligned with the normal direction to a close decision boundary. 2) Using the previous fact, the authors devise two new attribution methods -- BSM and BIG -- which can be used to get more reliable explanations from even a normal (non-robust) model. While the reviewers agree that the premise of this paper is interesting, some concerns remain post the rebuttal. More specifically, some reviewers opine that the AGI and BIG methods are somewhat similar, and other reviewers are not very convinced about some of the details e.g., the generalization of the orthogonality of SM to the decision boundary from the binary classification case (section 3.1) to the more general case of ReLU-Net's multi-class classifiers (section 3.2). Given this, we are unable to accept this paper at this time. We hope the authors find the reviewer feedback useful.
Decision: Reject