Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new gradient-based feature attribute method Distilled Gradient Aggregation (DGA) that extends FullGrad (FG). On quantitative and qualitative evaluations the authors demonstrate that their method produces better explanations than previously proposed methods.
strengths_and_weaknesses: Overall, I think the ideas in this work are interesting. However, as-is I have significant concerns with the evaluation of the method that prevent me from recommending acceptance. If the authors are able to address my concerns with additional experimental results I would be happy to raise my score.

Strengths:

1. The authors tackle a high-impact problem space (i.e., interpreting neural network predictions).
2. The authors' proposed method is, to my knowledge, novel.
2. Both qualitative and quantitative experimental results indicate that the method produces higher-quality attributions than previously proposed methods.

Weaknesses (in order of importance to my score):

1. Lack of ablations: The authors' proposed method is comprised of multiple moving parts (e.g. the redefined post-processing function $\Phi$, the masking functions, and the aggregation of multiple attribution maps), and it's not clear to me exactly which component(s) led to the improvements over previous methods. For example, based on the data provided in Figure 8 in the supplement, it appears that DGA without the EPC mask (i.e., the purple line with $q=1$), already outperforms many of the baseline models. In my view, the manuscript would be much stronger if additional ablation results were included. On a similar note, it would further strengthen the paper if the authors demonstrated that the individual pieces of their method can't trivially be applied to previous work to obtain similarly strong results. For example, how does the performance of DGA compare to applying integrated gradients with an EPC mask?
2. Influence of the ReLU in aggregation: The authors propose to apply a ReLU function to the individual attribution maps before aggregating them, with the justification of only wanting to retain features that push the network _towards_ a given prediction (rather than away). However, the application of the ReLU function makes the comparison with baseline methods unfair. For example, qualitatively (e.g. in Figure 5), we can see that many of the features with negative attribution values from baseline methods lie in the object of interest of the image (e.g. on the dog's face for the dog image). As such, it's not surprising that DGA performs much better than baselines on the LeRF metric, since the ReLU in DGA would set such attributions to 0. Do these results change if e.g. ReLU is applied to the Integrated Gradients attributions before computing LeRF values?
3. Axiomatic justification (or lack thereof): Many of the previously proposed methods cited in this work (Integrated Gradients, FullGrad, GIG) are guaranteed to satisfy various desirable axioms for attributions. Could the authors comment on which, if any, of the axioms satisfied by previous methods are also satisfied by DGA? If none, is DGA guaranteed to satisfy another set of axioms?
4. Not self-contained: The authors' proposed method is an extension of FullGrad, yet the authors do not provide any background on FullGrad in their manuscript. As such, I frequently needed to refer to the original FullGrad manuscript to be able to understand this work. 
questions: See "Weaknesses" section.
limitations: The authors briefly discuss the limitations of their work in their conclusion section. They do not address potential negative social impact, which is fine (I don't see any obvious negative impacts with this work).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This manuscript proposes a new attribution based DNN explanation method that distill input features using weak and extremely positive contributor masks. This Distilled Gradient Aggregation (DGA) approach combines the advantages of both global IG and local FG approaches via a new sequential feature distillation algorithm, distilling irrelevant features from the input. Authors performed quantitative and qualitative evaluation by comparing to various attribution methods.  
strengths_and_weaknesses: Strengths: The DGA approach effectively addresses the limitations of local approach (FullGrad) that uses a single anchor point to compute attribution. It also addresses counter-intuitive behavior of global approach (IG) with arbitrary integral path. Authors also evaluated performance using ROAR and LeRF/MoRF and demonstrated superior performance. The overall idea of aggregating local attributions via knowledge distillation is novel and experimental evaluations are sound and adequate.      

Weaknesses: (1) Authors could discuss the robustness of your explanation approaches against input perturbation since it is a known problem of IG type of methods. (2) Author could perform an evaluation on the percentage of input ablation on model explaining. (3) The sequential distillation process can be time consuming considering the explanation is on per sample basis.      

questions: Have you discussed the pros and cons of your iterative tuning approach to simple baseline such as saliency map in terms of cost-effectiveness? 
limitations: Just like other methods discussed in this manuscript, it is a post hoc explanation method that attempting to explain a trained model’s output. It does not address the issue of model selection/training for improving explanation quality. Since pre-trained models are used, so the comparison is fair.

Other limitations are time and tuning complexity. It can be cost-prohibitive to explain each single image.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper identifies pitfalls in two attributions methods, then proposes a novel attribution method that purports to solve the problem.
1) The paper identifies issues with popular attribution methods Integrated Gradients and FullGrad. IG suffers from attribution noisiness and unintuitive results, while FullGrad has weak dependency.
2) The paper proposes using FullGrad with a different post-processing function, and engaging in sequential feature distillation. Essentially they calculate a FullGrad attribution; then they mask a constant, small proportion of the top-ranked and an increasing proportion of the bottom-ranked inputs according to the attributions. They repeat this process until the image is fully masked.
3) They run comparative tests on other attribution methods. They run quantitate ROAR and Perturbation tests, and they run qualitative tests. These tests indicate the advantages of their method.
strengths_and_weaknesses: Strengths:
- Clear presentation, easy to follow with good examples.
- Strong qualitative and ROAR results support the conclusion.
- Masking and reattributing successively seems novel, to my understanding.

Weaknesses:
- Motivation seems weak. Why did the authors identify those two issues in two seemingly arbitrary methods as motivation for their method? Why did they decide to use FullGrad with the successive masking, as opposed to any other method?
- Grammar/typos need polish.
- Some experimental choices seem arbitrary/unexplained (see questions).
questions: Q: Have you addressed the issue of not knowing the ground truth? How do you know this method is reliable if the ground truth is unknown? Pixel Perturbation and ROAR tests address this question, but perhaps another test that could help. Are there any instances where the ground truth is known, i.e., we know what part of the image contributes to the model classification?

Q: It seems your method consists of 1) using successive masking and reattributing, and 2) using Fullgrad to get attributions between iterations. Why use FullGrad and not any other attribution method for part 2? Have you tried your algorithm with other attribution methods? I wonder if the real novelty and success here is in part 1, your masking approach, and using FullGrad between iterations is replaceable.

Q: What made you choose the methods you used for comparison? Why not SmoothGrad, LRP, Deconvolutional Networks, DeepLift, GradCam, or CAMERAS (see references)? I am not saying you should use them all, and understand it would require more work to compare them all. I am wondering how you justify the comparisons you choose to make and which you leave out. Perhaps a blurb on why you choose to compare some and not others?

Q: Which baseline do you use for IG? It seems you are not using black or white baselines, since IG attributes to the dumbbells and white cloth in the first and second images of figure 2, and IG gives zero attribution to pixels with the same value as the baseline. Have you considered trying other baselines, as in [Pascal Sturmfels, 2020]? It is possible a single baseline IG is not the strongest formulation. If this is the case, It would be interesting to see what the comparative metrics are to a stronger version of IG that uses a better baseline. It would strengthen the claim that this method outperforms IG.

Q: It seems that the counter-intuitive behavior of IG is due to the choice of baseline. An unanswered question is: why choose that baseline? It makes sense to put it in the middle, between the two curves. However, another side-effect of IG is that if you leave the baseline at (0,0), then any point of the form (0,x_2) will have 0 attribution in the first component, which does not seem correct either. This is not a problem unique to this paper: the issue of how to choose a baseline is a generally unanswered for baseline-attribution methods. It has to do with the idea of missingness in game-theory, which is not present in object recognition. See references for treatment of baseline options.

Q Are there any references needed for section 3.1, "Vulnerability of FullGrad" and "Counter-intuitive behavior of IG"? These observations are completely original?

References for the Author:
- DeepLift
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.
Learning important features through
propagating activation differences.
- Deconvolution Networks
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks.
- CAMERAS
Mohammad AAK Jalwana, Naveed Akhtar, Mohammed Bennamoun, and Ajmal Mian. Cameras:
Enhanced resolution and sanity preserving class activation mapping for image saliency.
- An investigation of various baselines used for IG.
Pascal Sturmfels, Scott Lundberg, S.-I. L. Vi-
sualizing the impact of feature attribution base-
lines, 2020. 
- Using a distribution of training-set images as a baseline for IG.
Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M.,
and Lee, S.-I. Improving performance of deep learning
models with axiomatic attribution priors and expected
gradients.
- Uses a Gaussian blur process to alter the path of IG.
Xu, S., Venugopalan, S., and Sundararajan, M. Attribution


Small issues/typos:
- Autonomous Driving (22-23)
- "so that humans" (24)
- no "the" before "critical (26)
- "evidence for the "decisions" (27)
- a heatmap (30)
- What is Clever Hans (32)
- "Obtaining a trustworthy input" (32)
- Missing an arrow from "Input x" down to the first dot in Figure 1 (A)
- derives the attribution to be vulnerable (108)
- let we have two inputs (108-109)
- Figure 2 sentence 2 is incomplete.
- Please describe MoRF and LeRF in caption of Figure 2.
- What is x-bar in 121-12? I believe it is the baseline but I don't see in defined before.
-  each ablated features (133)
- that masking out irrelevant features using IG(138)
- as uniformly distributing function (142-143)
- to remain the relevant features (175)
- In Figure 2 b, I assume you pruned k%. What is k?
- Similarly, in Table 1, what is k? Can you give multiple k's in a graph? This is a snapshot using one k, and could not be representative.
- For pixel perturbations, did you calculate attribution rankings for each of 50k images for each method and model, then clip inputs according to the rankings and re-evaluate the confidence? If so, does table 1 report average new confidence, over all 50k images, after k% removed?
- while almost methods (234)
- concentrates to the person (235)
- the relevant patch has less sharp than patch of DGA (235-236)
- in the Appendix (236)
- we provide the vulnerability (239)
- rework sentence on line 241
 - there would exist... (256)
limitations: Some limitations mentioned. Other limitations could be addressed refereeing to Questions section.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper introduces a new explanation method called DGA for attributing a function’s output w.r.t its input features. DGA is motivated by problems of some related work, i.e. Integrated Gradient and FullGrad. 
strengths_and_weaknesses: ### Strength
Visualizations are helpful to convey the authors’ message and explain the pipeline of the proposed method.

### Weakness

**Summary of Weakness.** My main concerns for this paper are problems in its motivation and the way the new method is evaluated. Although I find the current way the method is presented to the audience is quite confusing, this can be fixed in an easy way. I will vote for rejection for now. 

**Motivations are not convincing.** A short summary for this paper’s motivation is the problems in existing methods, IG and FullGrad. Namely, FullGrad is vulnerable to local changes to the input (line 105)  and IG is counter-intuitive (line 114). However, none of the comments (or blames) made on the prior work makes sense to me. I will elaborate in detail. 

- The fact that FullGrad (or gradient-based methods) may change for some $x’$ that is near to x does not mean this is a vulnerability of the explanation method. In fact, this is a problem of the underlying model and the explanation method is just being faithful to the model’s behavior. **In another word, being faithful is not a problem.** With $f(x) = f(x’)$ and $||x-x’|| \leq \epsilon$, it is unclear to me why $FullGrad(x)$ and $FullGrad(x’)$ must be the same if the model indeed treats $x$ and $x’$ differently in a significant way. In fact, a lot of work in adversarial vulnerability has shown that this is often the case: 1) linear regions are often tiny and very different from their neighboring regions [1]; and 2) models are learning noisy or spurious features, e.g. using background instead of the object, to make the prediction [2, 3, 4]. All these observations point out that making a more robust model is the way that one will observe that explanations for neighboring points look similar in the end [4, 5, 6]. If the underlying model does not behave similarly for similar input, why a **good** explanation method, e.g. FullGrad, should behave similarly? All in all, the so-called “vulnerability” is not a problem for FullGrad but a problem for the model.

- The comment about IG being counterintuitive is counterintuitive to me as the authors do not provide any justifications for what behavior is intuitive/expected. In the example shown in Figure 3c, the reason why IG is different in region A and B is that the baseline point is in A so that any integral path in A returns the same IG scores because the function is linear in A. In this case, the integral part of IG returns nothing but just the weight of the function, w^{(A)}, for every point in the path integral. Namely, $\forall x \in R^{(A)}, IG(x) = w^{(A)} \odot x$. And because you only move x on one axis, only the attribution score of IG for another axis will change. But for points in B, the baseline is outside the region B so IG(x) no longer just returns the weight $ w^{(B)}$: it is an aggregation of all $w^{(i)}$ the path integral traverse weighted by the length of the intersection between the path integral and the linear region. Thus, you observe IG also changes even when you only move the point on one axis. I have spent a long paragraph just showing that this is just a matter of baseline choice in IG and without any theoretical justifications or geometric-based analysis, I don’t see there is any problem in IG. Alternatively, if the authors argue that IG is using “bad” baseline points, then this observation should motivate the paper to find better baselines instead of developing DGA. 

**Empirical evidence is not able to support the authors’ claims.** In the paper, the authors adopt metrics based on the ablation study using feature attributions to show that the proposed method, DGA, has a better performance on these metrics. Notice that the problems brought out earlier in this paper are **vulnerability** and **counterintuitive behavior** instead of something like “the current methods cannot identify features that are actually important to the model.” Without justifications about why the metrics used in the current paper, e.g. MoRF or ROAR, are able to measure **vulnerability** and *counterintuitive behavior**, the current empirical evidence is not related to the topic from my perspective. 

**The way DGA is introduced is a bit confusing.** I find myself lost when transitioning from Section 3.1 to 3.2. I don't see clearly how RISE solves **vulnerability** and **counterintuitive behavior** in IG and FullGrad and I hope to see more principled justifications about that. Then the authors quickly start to explain the technical details of feature distillation, without a high-leveled discussion of what the method is going to do and why this approach solves the problem in a more efficient way compared to RISE. Also, since RISE also solves the problem as the authors mentioned in line 131, why is it missing from the evaluation section? 


[1] Croce, Francesco et al. “Provable Robustness of ReLU networks via Maximization of Linear Regions.” AISTATS (2019).

[2] Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "" Why should i trust you?" Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.

[3] Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." Advances in neural information processing systems 32 (2019).

[4] Wang, Zifan et al. “Robust Models Are More Interpretable Because Attributions Look Normal.” (2021).

[5] Chen, Jiefeng, et al. "Robust attribution regularization." Advances in Neural Information Processing Systems 32 (2019).

[6] Singh, Mayank Kumar et al. “On the Benefits of Attributional Robustness.” ArXiv abs/1911.13073 (2019): n. pag.

questions: My questions are discussed together with my comments in the weakness part. I will summarize them here and please refer to my comments above to see the relevant context.

- Why is the so-called “vulnerability” of FullGrad not a problem of the underlying model?

- What does “counter-intuitive behavior of IG” mean when an intuitive behavior is not discussed at first.

- How do the current metrics measure “vulnerability” and the so-called “counter-intuitive behavior”?

- Why RISE is missing from the evaluation section?

limitations: The authors have a separated section to discuss the limitations.
review_title: null

Meta-review: metareview: This paper proposes a new gradient-based attribution method Distilled Gradient Aggregation (DGA), which combines the strengths of both local and global attribution methods. The reviewers and meta-reviewer found the method novel, and supported by promising results both qualitatively and quantitatively. 

During the rebuttal phase, the authors made a _thorough_ effort in response to each reviewer's comments. As recognized by two reviewers (hnze, cvDy), the newly added results and discussions have significantly strengthened the contribution. 

The AC recommends acceptance given the paper tackles a critical problem, and presented an effective and convincing method that advances the field of explainable AI. 

Authors are strongly recommended to include these new results and writing changes suggested by the reviewers for the final version of the manuscript. 








confidence: Certain
Decision: Accept