Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of work: a new saliency map method is presented, along with empirical work showing that it beats SoTA by some metrics.

Recommendation: I lean toward accepting this paper.

Reasoning: 

Strengths: the paper represents an advance in state of the art, based on a variety of quantitative metrics. The authors have come up with an interesting new idea, and done a fair amount of work to back up their claims. It would be useful to have this paper in the literature.

Weak points: Interpretability methods ultimately stand or fall based on how useful they are to humans, not quantitative metrics. To my eye, the results for images and sentences aren't noticeably better than the alternatives. Furthermore, I found the writing fairly confusing--I needed to read the "decoy" definition several times.

Areas for improvement:
* I'd like to see a better description of the basic decoy method, maybe with some figures walking us through the construction. I would also like to see some motivation for what feels like a fairly complex technique, beyond Theorem 1.
* Theorem 1 is suggestive, but I'd like to see a critical analysis of how it applies in the real world. Do we have any idea how big the constant $C_1$ is, for instance? Furthermore, $F^c$ is only piecewise differentiable, and on very small pieces in practice. (Also, if it weren't for the softmax layer, the function would be piecewise linear--it's not clear to me what the Hessian tells us about anything that happens before that final layer.) Given this, it's not obvious that the Hessian at a given point captures significant information about robust inter-feature interactions.
* I find Figure 3 confusing (e.g., there are undefined abbreviations) and visually it's not clear that the decoy method is better. Maybe having one large diagram, with callout arrows and annotations, would help make it clear what the new method is adding.
* I did not check the math in the appendix (if I understand correctly, this is not necessary in a review, but I wanted to make that fact explicit.)
review_title: A saliency map method based on "decoys"

final_score: rating: 6: Marginally above acceptance threshold
review_content: (I have updated my review to raise my reviewer score by two points after discussion with the authors; this is still a preliminary evaluation as I have not discussed the paper with the other reviewers)

--

SUMMARY OF METHOD

The paper discusses an approach to create "decoys", which are slightly altered versions of the input that preserve the activations of an intermediate layer. The proposed "decoy enhancement" consists of applying a saliency method to calculate importance scores for the decoys, and then subtracting the maximum saliency across decoys from the minimum saliency. The authors benchmark their method on several datasets and argue that their saliency maps are superior according to the fidelity metric (Dabkowski & Gal, 2017). They also demonstrate that the decoy-enhanced maps are more robust to adversarial attacks.

STRENGTHS

- This approach of enhancing saliency maps is very unique compared to other methods in the literature. The generation of decoys could be useful in other contexts (e.g. to understand which sets of inputs can compensate for or "buffer" each other).
- The authors have benchmarked their method on several types of data - images, the Stanford sentiment treebank, and network intrusion detection.
- The authors have included several valuable experimental analyses, such as making sure the method is robust to the choice of hyperparameters, performing sanity checks to make sure the saliency maps respond to cascading randomization (a sanity check from Adebayo et al.), investigating different baselines for integrated gradients, comparing the decoys to a constant perturbation, etc.
- To me, the authors have made the case that the maps are more robust to adversarial attacks (caveat: I am not an expert on adversarial attacks)

WEAKNESSES

(1) The biggest weakness of this method would be the runtime to compute the saliency maps on the decoys. The authors generate $2n$ decoys, where $n$ is the number of unique masks. The number of masks is determined by the patch size $K$, and the authors write (in section A16 in the supplement) "given a constant stride 1, the number of decoys $n$ is unique to the patch size $K$. Specifically, $n$ is inversely proportional to $K$". The claim that $n$ is inversely proportional to $K$ is a little confusing given that the stride is 1 - I would expect the number of unique masks to be $(\text{image_len} - $K$ + 1)^2$, but the phrase "inversely proportional" implies that $n$ is proportional to $1/K$. Assuming that the authors meant  $(\text{image_len} - $K$ + 1)^2$ (which is consistent with a stride of 1) - this could potentially be a very large number of decoys per image, given that the patch sizes $K$ are fairly small ({3, 5, 7, 9, 11} according to section A16 in the supplement) - e.g. the ImageNet images are resized to 227 x 227 (as per section A10 in the supplement). The authors say in section A15 of the supplement that the "saliency map computation can be run parallelly in batch mode", but regular saliency maps can also be computed in batches on a GPU, so I am confused how the authors claim that their method  "introduces about 2X ~ 5X overhead over the existing saliency methods" when it seems like the number of decoys per image is much larger than 5. Can the authors explicitly state how many decoys are used for particular tasks, and (if the number is larger than 5) how they reconcile this with the claim of "2X - 5X overhead"? It would additionally be good to clarify whether the interpretation improvements from having the decoys are greater than what one would achieve by instead investing the equivalent computation time in other techniques (e.g. more samples for SmoothGrad or ExpectedGradients - have SmoothGrad and ExpectedGradients saturated in the number of samples?).

(2) I have a set of concerns pertaining to the evaluation using the fidelity metric. The authors define a normalized saliency score as follows: "First, we follow the existing methods (Simonyan et al., 2013) and compute the absolute saliency scores...to avoid outlier features with extremely high saliency values leading to almost zero saliency scores for the other features, we then winsorized outlier saliency values to a relatively high value (the 95th percentile)...before linearly scaling to the range [0,1]". They then define the fidelity metric as $- \log \frac{F^c (E(x; F^c) \odot x) }{F^c (\bar{E(x; F^c}) \odot x) }$, where $c$ is the predicted class of the input. The authors write "by viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will weight important features more highly than less important ones and thus give rise to higher predicted class scores and lower metric values. Note that we subtract the mean saliency $\bar{E(x; F^c)}$ to eliminate the influence of bias in $E(x; F)$ and exclude trivial cases such as $E(x; F^c) = 1$". Here are my concerns:

2a) In Figure 1, the fidelity scores are predominantly all positive - this implies that $F^c (E(x; F^c)$ was *lower* than $F^c (\bar{E(x; F^c}) \odot x)$, which is concerning - the average saliency map across all images was better than a saliency map that was generated specifically for the example at hand? To me, this suggests that the fidelity metric is not working as intended on image data. My guess is that there is a lot of *high-frequency noise* in the saliency maps, which is causing $E(x; F^c) \odot x$ to be out-of-distribution, and that is the reason $F^c(E(x; F^c) \odot x)$ is so poor. According to this hypothesis, I suspect the authors may find that simply *smoothing* the saliency map scores would achieve better results via the fidelity metric. The authors should thus include simple smoothing as a baseline. I note that the fidelity metric values are predominantly negative for the SST dataset (as shown in Figure 2c), which a sign that the fidelity metric makes sense there; however, in Table A2 (pertaining to the network intrusion dataset), the fidelity scores are once again predominantly positive (also Grad performs better than IntGrad and SGrad, which is unexpected). 

2b) The authors acknowledge that extreme values can result in near-zero importance for many features in the normalized saliency maps and propose clipping as a way to address this; however, even with clipping, the problem could still persist; if the 95th percentile of a saliency method is more on the extreme side, then the resulting masks would have a **lower L1 norm after the linear scaling normalization** - thus, saliency maps that had less extreme variation in the scores would get an unfair advantage in the evaluation (because they would be able to retain a larger portion of the original image after applying the mask $E(x; F^c)$). It is not clear to me that subtracting the mask created by the mean saliency would be enough to correct for this potential advantage; the rigorous way to correct for it would be to retain something like the top $k$ positions as ranked by a particular saliency method (with sign information included for the methods that provide them), or to find some other way of ensuring that the L1 norm of the masks were comparable.

2c) As alluded to in 2b, the approach of taking the absolute value of the saliency scores would handicap methods where the sign information is valuable. Given that the authors are specifically trying to identify regions that are important for the target class, they should retain the sign information for the methods that provide them (in conjunction with switching to a rank-based masking strategy described above). I recognize that Simonyan et al., 2013 discarded the sign information, but they were not trying to optimize the fidelity metric. I also recognize that the proposed decoy-enhanced saliency scores are always positive, but this is a shortcoming of the decoy-enhanced saliency scores (in that they don't distinguish regions that are contributing positively to a class from regions that are contributing negatively to the class); I don't think that's a good enough reason to handicap methods that are providing valuable information in the sign of the saliency.

2d) (this point is lower priority) Again regarding the benchmarking on images: it has frequently been observed that pixel-level attribution methods do not perform very well on image data (e.g. they tend to be outperformed by Grad-CAM, which computes the attributions at a higher conv layer). Accordingly, to make a compelling case that the proposed method can be helpful in practice, the authors should show that the decoy enhancement helps even when the saliency maps are computed at some higher convolutional layer (rather than computed at the pixel level). Alternatively, they could add Grad-CAM (or a related method - see [1]) as a baseline. A suggestion regarding benchmarking on image data: the authors could consider using the BAM dataset (https://github.com/google-research-datasets/bam) to sidestep the issues with the fidelity metric altogether.

(3) In section A7 of the supplement, the authors write "from the above equation, we can see that, for a linear model, the linearity zeroes out the gradients of the decoys, causing the ill-defined behavior of our method. This does not dilute the significance of our method because linear models are self-explainable and do not need extra post-hoc explanation". I think this is a bit misleading (either that, or it could use more clarification); the equation in question is $\nabla_{\tilde{\boldsymbol{x} }} F^c(\tilde{\boldsymbol{x}})^T \Delta \approx - \frac{1}{2} \Delta^T \boldsymbol{H}_{ \tilde{\boldsymbol{x} } } \Delta$, where $\Delta = \boldsymbol{x} - \tilde{ \boldsymbol{x} }$ and $ \tilde{ \boldsymbol{x} }$ denotes the decoy of $x$. To me, this equation just seems to convey that decoys are constructed such that the change in the output due to first-order effects (the left-hand side) is approximately balanced by the change in the output caused by second-order effects (the right-hand side). If the right-hand side were zero (as is the case in a linear model), then that would just mean the first-order effects would have to be such that they canceled out - why does this make the method's behavior "ill-defined"? However, more to the point: it seems clear that the decoy-construction process will only perturb a feature if they effect of this perturbation can be cancelled out by perturbing other features. But *a feature can be important even if the effect of perturbing the feature is not easily cancelled out by perturbing other features* - particularly given that the decoys are constructed in order to preserve the representation at some intermediate hidden layer. Also, regarding the assertion that "linear models are self-explainable and do not need extra post hoc explanation" - a more complex model is still capable of learning linear effects for some of its input features, and it would not be obvious that the complex model has learned these linear effects without some post-hoc explanation.

(4) Since these decoy-enhanced saliency maps look at $\max(\tilde{E}) - \min(\tilde{E})$, where $\tilde{E}$ is a population of saliency scores, it seems appropriate to benchmark against a method that also looks at the variation across a population of saliency scores - e.g. the VarGrad method, which was one of the top-performing methods in https://arxiv.org/abs/1806.10758. Also, while I appreciate that the authors added a baseline of the "constant perturbation" decoy, I would be interested to see a similar baseline that consists of random perturbations that are of comparable magnitude to the perturbations made by decoys.

MINOR

- Why do the authors write (in the introduction, when discussing their proposed method): "By design, this score naturally offsets the impact of gradient saturation" - if the gradients on a particular set of inputs is zero, would the decoy-generation process perturb those inputs? My understanding is no, since the decoy generation proceeds via gradient descent. In what sense is the saturation problem addressed?
- In table A3, how many samples were used for ExpectedGradients?
- It would be good if the authors clarified what $\delta_i$ is in Proposition 1 in the main text itself, without the reader having to refer to the supplement
- There are a few typos/grammatical errors - "it exhibits ill-defined on linear models" on page 4, "can obtain where" in Proposition 1, "obtain, we can further" in the subsequent paragraph.
- I did not read the supplementary proof of the adversarial robustness in detail (I am also not an expert on these adversarial attacks), so one high-level question I have for the authors is whether we expect *any* method that is based on an ensemble of saliency maps to be more robust to adversarial perturbations compared to the original (non-ensembled) saliency maps, and if so, is the proposed decoy-enhanced approach *more* robust than what would be provided by devoting the equivalent computational resources to drawing more samples for an ensemble (say with a method like VarGrad)?

SUMMARY OF RATING

I think the concept of using decoys to enhance saliency scores is unique, and the authors have clearly invested considerable effort in the work. However, the concerns regarding the benchmarking (point (2) under weaknesses) mean that I hesitate to fully believe the empirical improvements (the results of course look aesthetically pleasing to the human eye, but we as a field have learned our lesson that aesthetically pleasing results can be deceiving). If the authors can make the benchmarking more compelling, I would be willing to revise my score towards acceptance. Overall, my current feeling is that the decoy generation process proposed by the authors may turn out to have good uses (e.g. in identifying sets of inputs that can compensate for each other), but I'm not yet convinced that the application presented here of enhancing saliency maps is the right fit for the method (particularly when coupled with the runtime concerns in point (1) and the potential failure cases in point (3) where important features might not be highlighted).

[1] https://openaccess.thecvf.com/content_CVPR_2020/papers/Rebuffi_There_and_Back_Again_Revisiting_Backpropagation_Saliency_Methods_CVPR_2020_paper.pdf

review_title: Interesting method, but some concerns regarding the benchmarking/runtime

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper introduces a technique to improve current gradient-based attribution methods by 1. generating perturbed data, i.e. decoys, that maintain the same feature maps as the original data and 2. an aggregation strategy of the saliency maps across the decoys. They provide theoretical as well as empirical support for their method. This paper show a great improvement over saliency maps and integrated gradients with a smaller gain for smoothgrad. This is well written and an interesting technique that could add to the repertoire of interpretability methods that exist. 

Comments:
* One questions that arose while reading: why is there a significant boost with range aggregation compared to mean aggregation? What is the intuition here? 
* Visually, smoothgrad (without decoys) seems to make a big improvement over saliency maps and integrated gradients — almost as much as decoy-enchanced saliency maps. Smoothgrad perturbations are drawn from Gaussian noise, not optimized, with a mean aggregation. The performance gain of smoothgrad seems to be better than the decoys with mean aggregation (Fig. 1). Certainly the decoys with range aggregation improve Grad (saliency maps). Does it make sense to try smoothgrad with the range aggregation strategy to see if it improves its performance?  This experiment can help isolate which attributes are more noteworthy, the decoy generation method itself or the aggregation strategy.  Also, since smoothgrad can be applied to integrated gradients, a similar comparison should be made for integrated gradients as well. 
* The fidelity metric is a way to quantitatively compare the attribution methods, but it is unclear (at least visually) how sensitive the score is. For instance, in Fig. 1, the picture of the alps has a SF score of 7.57 for SGrad_Decoys, while the the volcano image has an SF score of 0.72. Moreover, the intgrad_decoys for the volcano has an SF score of 0.36, but it seems to be capturing the base of the mountain in addition to the sky. Are there certain aspects of the saliency maps that boosts/biases this fidelity score? A brief mention of the limitations of this SF score would be helpful to put this in context for a reader. 
* The comparison between grad and intgrad with deocys is visually very clear. However, the difference between sgrad and sgrad+decoys is much less pronouced. Perhaps a difference plot would be better at revealing what the decoy methods are able to better capture. 
* The robustness to adversarial attacks on images is not as convincing as the other aspects of the story. In Ghorbani et al. 2017, they demonstrate that adversarial attacks can maintain the same predictions but lead to very different attribution maps.  Each attack that is explored here (Top-k, mass center, and target) largely leads to the same feature importance maps as the original image without any adversarial attacks (see Fig. 3a). This may be why the sensitivity (while consistently lower) is very similar to the feature importance maps without decoys.
* On a smaller note, this paper mentions that the three hyperparameters are swappable feature size, network layer, and initial lagrange multiplier. They never mention lagrange multiplier in the main text — it is first introduced in A6.  It should probably be mentioned in the main text for clarity.  

review_title: improving saliency maps with in-distribution noisy decoys

Meta-review: This paper presents a novel approach to producing saliency maps for interpreting deep neural networks.  In general this paper seems quite close to borderline, although on the positive side, with some low confidence reviews.  The reviewers felt that the proposed approach could be useful to the community and they seemed to feel that the qualitative results in the experiments demonstrated convincing saliency maps.  There were some concerns, however, about the quantitative experiments as the reviewers (e.g. AnonReviewer2) found that while the mean results seemed better, it wasn't clear if they were statistically significant.  Naturally, the qualitative experiments are highly subjective and there was disagreement between reviewers whether the proposed approach did indeed produce better saliency maps than existing approaches such as smoothgrad.  One reviewer indicated that they found it difficult to follow the paper and to understand the decoy concept given the writing.  During discussion AnonReviewer2 updated their score (and very thorough review) by 2 points to indicate a weak preference toward accept.  None of the reviewers argued particularly strongly for acceptance and "championed" the paper.

The low confidence, slightly above borderline reviews seem to suggest that the reviewers thought the paper was above the bar but were reluctant to argue strongly for acceptance.  The method seemed like it could be useful to them but they weren't clearly convinced that it set a new state-of-the-art given the quantitative and qualitative empirical results.  
Decision: Reject