Distilled Gradient Aggregation: Purify Features for Input Attribution in the Deep Neural Network
Authors: Giyoung Jeon
Haedong Jeong
Jaesik Choi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Explainable AI
Input Attribution

Abstract: Measuring the attribution of input features toward the model output is one of the popular post-hoc explanations on the Deep Neural Networks (DNNs). Among various approaches to compute the attribution, the gradient-based methods are widely used to generate attributions, because of its ease of implementation and the model-agnostic characteristic. However, existing gradient integration methods such as Integrated Gradients (IG) suffer from (1) the noisy attributions which cause the unreliability of the explanation, and (2) the selection for the integration path which determines the quality of explanations. FullGrad (FG) is an another approach to construct the reliable attributions by focusing the locality of piece-wise linear network with the bias gradient. Although FG has shown reasonable performance for the given input, as the shortage of the global property, FG is vulnerable to the small perturbation, while IG which includes the exploration over the input space is robust. In this work, we design a new input attribution method which adopt the strengths of both local and global attributions.
In particular, we propose a novel approach to distill input features using weak and extremely positive contributor masks. We aggregate the intermediate local attributions obtained from the distillation sequence to provide reliable attribution. We perform the quantitative evaluation compared to various attribution methods and show that our method outperforms others. We also provide the qualitative result that our method obtains object-aligned and sharp attribution heatmap.

Distilled Gradient Aggregation: Purify Features for Input Attribution in the Deep Neural Network
Giyoung Jeon 1,3+
Haedong Jeongl.3. 1.2+ Jaesik Chof2.3
UNIST KAIST *INEEII
gi giyounguuistt.cm ac.kr
haedong www.gjkaist. jaesik. cha.ikkkthoikiis..aac
Abstract
Measuring the attribution input features toward the model output is one of the popular post-hoc explanations the Deep Neural Networks (DNNs). Among various approaches to compute the attribution, the gradient-based methods are widely used to generate attributions, because of its ease of implementation and the model-agnostic characteristic. However, existing gradient integration methods such Integrated Gradients (IG) suffer from (1) the noisy attributions which cause the unreliability the explamation, and (2) the selection for the integration path which determines the quality of explanations. FullGrad (FG) is another approach to construct the reliable attributions by focusing the locality of piece-wisg wise linear network with the bias gradient. Although FG has shown reasonable performance for the given input, as the shortage of the global property, FG is vulnerable to the small perturbation, while IG which includes the exploration over the input space is robust. In this work, we design a new input attribution method which adopt the strengths of both local and global attributions. In particular, we propose novel approach to distill input features using weak and extremely positive contributor masks. We aggregate the intermediate local attributions obtained from the distillation sequence to provide reliable attribution. We perform the quantitative evaluation compared various attribution methods and show that our method outperforms others. We also provide the qualitative result that our method obtains object-alignec and sharp attribution heatmap.
Introduction
Deep Neural Networks (DNNs) are increasingly applied to many fields in human-life such as self- driving, medical predictions and time-series forecasts. Along with these improvements, the recent models get bigger and more complicated that humans cannot investigate and understand the internal decision mechanism important because the malfunction or the groundless decision of the model can cause the critical problems, As an effort provide the evidences on the decisions, the input attribution has been well-studied, especially in visual tasks. Input attribution method aims to measure how much each input feature contributes to the model prediction. Because the output of this method takes the form of heatmap to provide relative importance among input features, it helps to locate the appearance of human-level semantics the input. Previous work also applies the attribution to debug the model by wiping Clever Hans of DNNs [Lapuschkin al., (001)). However, obtaining the trustful input attribution is still challenging because (1) the highly nonlinear structure modern DNNs makes difficult to correctly track the relationship between the input and the output, and (2) quantifying the reliability of attribution methods is non-trivial because the ground-trutt is not available.
them. Identifying and analyzing the reasons for the model predictions are
"Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

NO Aosk Doerlenr
WC
Eacd
Maat
Lack
Leca
Madk
Lacd
Plok
Plot
Port
BPC
Post Procesiay
LendAtabike
(a) Distilled Grddient A ggregation (DGA)
(b) Modules (detailled)
An illustration of Distilled Gradient Aggregation (DGA) and details of modules included in the computation of DGA. DGA generates the discrete sequence of anchor points by distilling the input features. We identify that aggregating the local attributions along the distillation sequence provides the class (Dingo) oriented attribution heatmap. The details are described in the Section 3.2
Figure 1:
tradient-based input attribution is one of the main techniques to derive the relationship between the model decision and the input features. The partial derivative of the output with respect to the input provides the measure of sensitivity, which is easily computed in DNNs. Integrated Gradients (IG) (Sundararajan al.) 2017]
commonly used gradient-based method, which provides the axiomatic properties to support the reliability of attributions. However, IG inheres the problem of noisy attribution, which originates from the gradient integrating path, and several variants of IG have been proposed to alleviate this issue (Smilkov al. 2017 Kapishnikov ef al.) 2019 2021 Pan et al.) 2021]. FullGrad (FG) [Srinivas and Fleuret) 2019) also raises the counter-intuitive behaviors IG. FG avoids this problem by considering only the local gradients instead of the path integration and proposes to use the bias gradient. But FG vulnerable to the small perturbation in the inputs due to its locality.
In this work, we provide the analysis on the weakness of (1) FG method which is unavoidable if considers only single anchor point, and (2) IG method which the continuous path based gradient integration may fail quantify the intuitive attribution. To complement the shortcomings of the two methods, we propose to aggregate the attribution from the multiple anchors. For the selection of anchor points, we devise an algorithm to sequentially distill the irrelevant features to generate reliable attribution. The main contributions of our work are,
Propose a novel feature distillation algorithm based on the intermediate local attribution to generate the sequence of meaningful anchor points.
Devise Distilled Gradient Aggregation (DGA), an attribution method by aggregating the intermediate local attributions from the distilled input sequence for the reliable attributions.
Qualitative and quantitative evaluations to validate the proposed method outperforms existing gradient-basee attribution methods.
Related Work
Input attribution is one of the post-hoc explanation methods, which aims identify the influence of each input feature to the model output on the trained model. There exists considerable variety of techniques to derive the input attribution. With the property that the feature map obtained by the convolutional layers includes the spatial information, Class Activation Mapping (CAM) methods compute the attribution by the weighted sum of the feature maps [Zhou el al. 2016) Selvaraju el al.) 2017]. Layer-wise Relevance Propagation (LRP) method propagates the model output backward to the input [Bach es al. 2015 Nam el al, 2020 LRP extends the Taylor decomposition the DNNs and distributes the relevance in layer- wise sense.

---

0.2
0= 0.4
d=
6 0.8
MoRF is better)
Engan (nney
0.25
FG
0.15-
Ours
LeRF (1 is better)
13
0.8
0.4
Ouz
0
0
02
0= 0.4
0.6
== 0.8
Figure 2: Attribution heatmaps for the Gaussian noise perturbed images obtained by FullGrad (FG), Integrated Gradient (IG) and Ours. Heatmaps from FG shows inconsistent results while other two methods are consistent, which utilize the global perspective for the attribution. Corresponding pixel perturbation scores also show that FG loses reliability against the simple noise perturbation. The procedure pixel perturbation described Section 4.1
There are another approach by measuring the behavior of the model by perturbing the input features. Optimizing the input by gradient ascent gives an example which would maximally activates the target neuron [Erhan er al. (2100] Nguyen er al. [016 Olah et al., 20188. Instead of maximizing the target neuron activation, Extremal Perturbation optimizes the mask which removes or reveals the part the input to localize the attributed part of input [Fong er al. 2019]. It is extended by using Integrated Gradients (Sundararajan et al.] 2017) for the optimization make the optimization more stable [Qi err al. 2020]. By collecting the pair of partially removed inputs and corresponding model ouptuts, training linear model to resembles the mapping would give the feature importance in terms of linear weights [Ribeiro et al. [2016). Rather than training new model, Randomized Input Sampling Explanation (RISE) computes the attribution by aggregating multiple randomly masked inputs, weighted by the model outputs [Petsiuk el al.) 2018
Based on Jumann-Shaples value [Aumann and Shapley 2015). which is one solution of the fair distribution in the cooperative game theory, Integrated Gradients (IG) has been proposed [Sundarara jan al. 2017]. IG is equipped with axiomatic properties which are desirable for the attribution methods. IG is computed by integrating gradients over the straight path from the predefined baseline to the input. As the attribution corrupted by the noisy information along the path, alternatives for the different paths have been proposed [Smilkov et al.) [017] Kapishnikov ef al.) [219 2021]. FullGrad (FG) [Srinivas and Fleuret, 2019], which utilizes the bias-gradient, is proposed to suppress the counter-intuitive behavior of IG which is caused by the weak-dependency between the local linear regions.
3 Distilled Gradient Aggregation Method
In this section, we propose our gradient aggregation method, Distilled Gradient Aggregation (DGA). We first provide an example and analysis about the inconsistency observed by FullGrad, which uses single anchor point to compute attributions. We also provide the countr--ittuiiive behavior of IG caused by the continuous gradient integration path. To complement both shortcomings, we propose an aggregation method, which ensembles the local attribution from the sequence of inputs. To reduce computational cost and reinforce the features that are in charge of the model decision, we suggest sequential feature distillation algorithm, which distills irrelevant features from the input.
3.1 Analyzing FG and IG on Simple Models
Assume we have the input vector xER² and simple neural network f equipped with partial linear activation (e.g., ReLU). This network f can be regarded as the combination of piece-wise linear functions Montufar et al. 2014). Each piece-wise linear function is only defined and feasible in

---

UNTE -X
Baxzinee
A
Athiltation
(a)
(b)
(c)
(d)
Figure 3: (a) contour of logit values for trained f. (b) The linear regions which the trained network comprises. Each colored linear region corresponds to each piece-wise linear function. (c) Selected two linear regions (A and B) and the zero baseline. The dotted lines indicate the perturbations for Ij axis in the same linear region. (d) Attribution of IG and FG for each lincar region. We identify that for linear region (include the baseline), the global attribution (IG) is same as the local ones (FG). However, for linear region B, the global and local attribution has different attributions for input samples.
corresponding linear region R(a,, where U₁ R(( R² and R((k)) n (R(a)))) 2 for any ki and k2- Such piece-wise linear function is formulated as follow,
((1) x + ((1)
R
f(x)
w(K) x (K)
x E
where =R) and b ( denote weight and bias of k-th linear region respectively. Figure depicts an illustrative example of the function f For the network f. FullGrad (FGG)Srrnivar and Fleuret 2019 and Integrated Gradient (IG) (Sundraaaaaan et al. 2017] are given follow,
FG(x) ((Vxf(x) @x)
(V)o.
TEL
IG(x) -f V (x(a)((((a) Var(a)da
Vulnerability of FullGrad FullGrad suggests that the attribution should be same inside the same linear region (8),, and this reduces the dependency between the attribution and the input x. This property is introduced as weak dependency. However, such weak dependency derives the attribution be vulnerable the model perturbation. For example, let have two inputs, x and x' x where e be any small enough random perturbation. If we find any x and x', such that the model output same, fx) fx'), but the region is different, then the attribution on each input should be different. This can be visualized by the simple experiment by generating noise perturbed image + and measuring the attribution, where e NO, of). Figure 2 shows an example where the FullGrad generates inconsistent attribution along with the simple Gaussian noise is added.
Counter-intuitive behavior of IG To visualize the counter-intuitive behavior of IG, select two linear regions (A, B) in Figure 3c and calculate the attribution in each region. In particular, we select sequence of data from a (white dot) b (green dot), which is only shifted in 3 dimension. Figure illusttrrte corresponding attribution for two selected linear regions. We observe that only attribution of 20, changes for the sequence of region A in both IG and FG methods. However, for the sequence of region B, the IG attribution of both I1 and changes at the same time, while FG attribution shows attribution change only in ø1. We conjecture this -ounter-intuitive behavior of IG is caused by the baseline selection. With the zero-baseline (x = 0), the integration paths of samples in region traverse only single region to compute IG. On the other hand, paths of samples in region B traverse through multiple regions. When traversing multiple regions, the counter-intuitio behavior can be induced due to passing the undesirable linear regions. From this observation, we can identify that the selection of baseline determines (1) which linear regions are traversed by the
"The implementation details described in the Appendix J.

---

Altriiation
(a) The sequence & with WC mask.
(b) The sequence & with WC and EPC mask.
Figure 4: Distillation sequence X' with WC mask and EPC mask for the target class French horn in the pre-trained VGG-16. The bottom row of (a) and (b) indicates the local attribution & SUPG (X(n)) for each colored box of first row.
path, and (2) how much portion of the path is included in each selected linear region. Although the proper selection of baseline can be an one option adjust (1) and (2) for the reliable attributions, is still non-trivial to control the sequence of the meaningful linear regions and each weight by only changing the baseline.
Based on these insights, we raise the weakness of the local and global attribution methods: (1) vulnerability and (2) counter-intuitive behavior. To alleviate these issues, we desire an attribution method to combine the strengths of each type of attribution methods. To resolve such desire step by step, we start from the local attribution which is free from the baseline selection. Then, the remained problem is how to select the meaningful linear regions to generate the reliable attributions.
3.2 Sequential Feature Distillation
For the strategy to select linear regions, RISE PPetsiuk er al. 2018 suggests the random perturbation- based approach. RISE explores the multiple linear regions with randomly ablated masks to measure the importance of each ablated features. However, the randomized ablation includes the stochastic process which requires expensive computational cost to achieve reliable attributions. We are also inspired from the adaptive selection for the perturbed inputs in Guided IG (GIG) (Kapishnikov et al. 2021]. which can improve the final attribution. Thus, we hypothesize that the adaptive exploration of linear regions based the intermediate local attribution can reduce the cost of randomized exploration and follow the spirit of the adaptive exploration. Finally, we propose the sequential feature distillation
algorithm to obtain a sequence of ablated inputs, the sequence of inputs X' [x()), x(1) x(2),
where the irrelevant features are distilled. Motivated by previous work that masking out the irrelevant
features using IG [ong et al. 20199 Qi et al.) 2020), we propose to distill the impurities by using the intermediate local attribution obtained along the sequence.
For reliable local attribution, FG proposes the bias gradient with post- pest-processing (((-) which includes the normalization and upsampling. As (-(.) suggested by FG is usually over-estimated by the bias gradient in deeper layers function for the bias gradient to alleviate the estimenni problem.
Barwinska et al. 2021]. we redefine (-()) as uniformly distributing
vT dim(v)
V (v)
1 dim(x)
(4)
dim(x)
where 1dd denotes a d-dimensional all-ones vector. We call FG with redefined post-processing V (-) as Uniform FullGrad (UFG), (G))..) Then we use UFG as the intermediate local attribution method throughout the remained paper. We note that different local attribution, such as Grad (Shrikumar el af., 2016), can be used in our method. We provide the comparison of selecting different local attribution in Appendix G.
To distill off the uninformative features, we build mask zero out the features with low magnitude of local attribution. In the sequence X,, the relation between and n+11th ablated input is formalized as,
x(n + 1) M(x(n)) x(0)
where x(0) = and M(.) a mask extractor. We define this mask extractor as the Weak Contributor (WC) mask, WCC The level of the WC mask increases along the distillation sequence X' with the
5

---

Algorithm Distilled Gradient Aggregation
Input: Model Input
# of steps N, EPC threshold 9, Negative scale 3 Output: Attribution &6 D
Parameter:
1: Let x() = X,
2: for in .0...N do
3: Φ & U (
4: M= N N () (MPM))))))))))
5:
x(n + + x(0) OM
6: end for
7: DGA (x) Eseo ReLU
8: return (x)
pre-defined number of steps N and finally the entire pixels become zero (i.e., X(N) = 0). We define M (.) for each feature j as,
(x) {
1ok
FFFF (xx
-
10
(6)
So if 1s (x)] d)m(w)
M M Nx,
S N
otherwise
where SW (x) is set of feature indices that the magnitude of corresponding local attribution is smaller than ( Practically, M (a, n; .N) can be equivalently derived by thresholding with n/N quantile of absolute local attributions. To implement the smooth change features, we gradually apply the mask with the scale factor proportional to the current step n. The sequential relation with WC mask is defined as,
x(n+1) MWC (x(1),;;; x(D). N
(8)
Figure 4 (a) depicts the sequence of distilled inputs X with WC mask. We can identify that the distillation by WC mask can remove the uninformative information (e.g., human body) to predict the object class, French horn.
However, we observe that considering only WC mask can be difficult to distill irrelevant features, if the strong local attribution is temporarily assigned to such features. For example, in Figure (a), the human face (red box) remains until the end of the distillation with strong attribution. This phenomenon can be caused when the noise exists in the gradient or the value of feature itself is large, which results in extremely high contributions for pixels without relevant information to predict the target class. Once such pixels are assigned with high attribution values, WC mask repeatably reassigns the masks to the same pixels and makes the overall distillation sequence be saturated. The saturated distillation sequence X' disturbs the strength of multiple ablated inputs to build reliable attribution.
We define additional mask to reduce the saturation by filtering out features with extremely strong attribution. We call this mask as Extreme Positive Contributor (EPC) mask M F.P.C.) and it is formulated as,
0z { LUFG S of r
SxPO
(9)
{1 if ISEP (x) dimixx) <q
MEPO (x:q)
(10)
otherwise
where q is EPC threshold to control the ratio of ablation. Finally, we combine two masks for our distillation algorithm with relative weights w.r.t. the current step

---

Table 1: Comparison of various attribution methods with LeRF and MoRF on three models.
GaI
GBP
IG
FG
GIG
DGA
VGG-16
0.078
0.113
0.096
0.415
0.110
0.434
LeRF (t is better)
ResNet- 18
0.114
0.145
0.158
0.448
0.185
0.533
Inception- V3
0.171
0.162
0.243
0.558
0.255
0.691
VGG-16
0.045
0.094
0.036
0.110
0.029
0.023
MoRF (} is better)
ResNet- 18
0.050
0.124
0.038
0.131
0.029
0.019
Inception V3
0.105
0.145
0.066
0.175
0.061
0.041
x(n + 1)
M M (x(n),;; (1- JM BMMPP (MPP(((())))) @ x()) N
Th
(II)
We note that in early distillation step, EPC mask takes high weight reduce saturation too highly attributed feature. the late stage, WC mask gains high weight to remain the relevant features. The distillation sequence & with WC and EPC masks is shown Figure (b). identify that using both WC and EPC masks removes irrelevant features (red box) and iteratively assign the attribution to relevant features (blue box).
3.3 Attribution Aggregation
With N distillation steps, we obtain N local attributions. The remaining question is how to aggregate these local attributions acquire the final attribution. Likewise previous studies (Selvaraju al. 2017 Kindermans et al. 2018; Bach er al. 2013), we desire to take the positive contribution from each local attribution. Thus, we take the ReLU before the aggregation.
(GA( = Σ R (x(n)))
(12)
Finally, call the unification of preceding modules Distilled Gradient Aggregation (DGA) method. DGA method consists of the distillation algorithm with WC and EPC masks to generate the ablated inputs achieve the local attribution, and the aggregation process considering features with the positive attributions. The illustration of the overall structure is depicted in Figure | and pseudo code provided in Algorithm To analyze the influence of each module (WC/EPC mask and ReLU), we perform the ablation study for each module and the results are available in Appendix and
4 Experimental Results
In this section, we validate the effectiveness of DGA by both quantitative and qualitative comparison. We note that evaluating the attribution method is still challenging due to the absence of the ground truth. With this difficulty, we verify the proposed method appropriately reflects the behavior the model prediction by providing the quantitative comparison using three metries: (1) pixel perturbation
(Samek et al.) 2016). (2) sensitivity-n [Ancona et al. 2018) and (3) GemOve-And-Rent
(ROAR) [Hooker al. 2019 Then also provide the qualitative comparison among different attribution
methods. In the following experiments, we set the hyperparameters for DGA as N-30 and q=0.9 with simple grid search. The details for the hyperparameter exploration available in Appendix K. We select various gradient-based attribution methods as the baselines: Gradient Input (G*I), Guided BackPropagation (GBP), Integrated Gradients (IG), FullGrad (FG), and GuidedIG (GIG).
4.1 Pixel Perturbation
Pixel perturbation widely used method benchmark the attribution methods if they correctly capture the relevance between the input features and the model output. To quantify the relevance
7

---

Table 2: Comparison of various attribution methods with sensitivity-n on ResNet-18
9%
10
20
30
40
50
60
70
80
90
G*1
-0,006
-0.009
-0.009
-0.013
-0.007
-0.014
-0.016
-0.022
-0.037
GBP
0.022
0.032
0.025
0,024
0.022
0.027
0.024
0.024
0.020
IG
0.013
0.022
0.020
0.027
0.032
0.037
0.039
0.047
0.067
GIG
0.006
0.007
0.004
0.002
0.003
0.002
0.002
0.002
-0.001
FG -0.045
-0.017
-0.005
0.001
0.003
0.003
0.008
0.008
0.004
DGA 0.095
0.096
0.101
0.098
0.095
0.089
0.083
0.080
0.079
between the input features and the model output, pixel perturbation method removes the pixel values in order of relevance obtained by attribution methods. Then it measures the AUC of the softmax output change for the target class with the perturbation. There are two orders of removal, Most- Relevant-First (MoRF) to remove the pixels with top k% relevance and Least-Relevant to remove the pixels with bottom 4% relevance. If input feature is actually highly related to the model prediction, the softmax output should decrease steeply when removed. Thus, MoRF is better if lower. In the same manner, higher LeRF is better.
(LeRF)
We use 50k images of the validation set provided by ImageNet [Russakovsky et al. 2015 We use
three publicly available pre-trained models: VGG-16 [Simonyan and Zisserman 2015 Inception-v3 SZegedy er al.) 2016), ResNet-18 [He et al., 2016]. Table indicates MoRF and LeRF results for the various attribution methods and model architectures. We identify that DGA shows the best performance in both MoRF and LeRF measure on entire architectures. We also provide additional comparison with different attribution methods (e.g., LRP) in Appendix I.
4.2 Sensitivity-n
Sensitivity-n Ancona et al. 2018 has been suggested to quantify the generalization of previously suggested properties, Completeness (Sundararajan al. (017) and Summation to Delta [Shrikumar et al. 2017]. In general, since not all deep learning models can satisfy Sensitivity-n, we use empirical approximations determine whether there is algorithmic bias in the process of calculating attribution. To measure how much the attribution method satisfics this properties, the metric quantifies the correlation between the sum of the attributions (Eres de (x)x) for any subset of features (S), and the change of the model output when the input subset is ablated (Xs-). The metric can be represented
as,
COITS Σ f() #ES
PO COSTM, (
(13)
where the subset S is uniform randomly sampled with its cardinality |S| and SC denotes its complement. The higher correlation value indicates that the attribution method empirically satisfies the sensitivity-n.
For computational efficiency, instead of ablating the entire individual input features (n), perform
patch-wise ablation to compute sensitivity-n with randomly sampled binary masks M2 E {0. 1) 1 x (100 masks for each input) where the portion of I in each mask is constrained be q. After the mask selection, we multiply the upsampled mask (V,(M)) to the attribution and the input to compute the correlation. The Table shows the correlation values for various percentage of selection (q) each attribution method for ResNet-18. We identify that DGA has the highest correlation values in entire cases.
4.3 RemOve-And-Retrae (ROAR)
ROAR is another metric to evaluate how well the attribution method captures the relevance of feature the perspective the model training. ROAR is performed by measuring the performance of the
re-trained model with inputs modified according to relative ordering of the attribution. Each input in the dataset is modified by removing pixels with top 1% attribution and replacing them with the average pixel value of the input. We perform ROAR experiment with simple CNN (6 Conv +

---

61
GBP IG
GIG
R6
DGA
Acuurayy (%)
50
30
10
70
Removed %
Figure 5: Comparison of ROAR experiment results on CIFAR-10 10 dataset among various attribution methods. The test accuracy for corresponding the percentage of removal.
Linear) trained on 50k images of training set provided by CIFAR-10 dataset (Krizhevsky 2009 using Adam optimizer with learning rate 3e-4 and 100 epochs. After training, the performance of the model is quantified using the standard test dataset with 10k images. We note that the attribution method captures more relevant features the test accuracy is lower. We provide the average performance over 10 trials for each attribution method, where the parameters are random initialized each trial and fixed between attribution methods. Figure [5] shows the test accuracy measure in the ROAR experiment for each attribution method. The result indicates that the model trained on the modified dataset with DGA steeply decreases the test accuracy even with 10% removed. We conclude that DGA can extract the features which are relevant training procedure in DNNs.
4.4 Qualitative comparison
We qualitatively compare the various attribution methods by visualizing the attribution heatmap and top 10% most relevant features at the same time. In Figure 6 provide the result of randomly selected images from the validation set of ImageNet with the pre-trained VGG-16. We can identify that the attributions the right-top row, DGA focuses the person who grabs a paddle while almost methods distribute the relevant pixels to sky and ocean. Although FG concentrates the person, the relevant patch has less sharp than patch of DGA. We provide more examples and results for different models in Appendix B-D.
aligned with the object comparing other methods. For example, in
5 Discussion
In this paper, we propose a novel gradient-based attribution method, Distilled Gradient Aggregation (DGA). We provide the vulnerability FG against the input perturbation and the counter-intuitive behavior of IG. To complement the weakness of both methods, we propose the gradient aggregation method along the distillation sequence that generates the inputs which the impurities are distilled. Our method obtains high quality attributions with its sharpness and object-alignment, and we verify the method through pixel perturbation, sensitivity-ni, and ROAR evaluation metrics. We believe that our DGA method can be broadly applied to explain a decision of various DNNs.
Broader Impact Transparency of deep models is matter of the highest priority for the application of such models in the real world, e.g., medical diagnosis [Caruana et af.) [2015] and autonomous driving (Yurtsever et a [020]. We believe that providing the evidence which well-aligned with the model decision would help the users of such applications to place great trust and the developers to improve or fix the model for better performance. Discovering unintended biases in the model is another issue [Stock and Cisse 2018). Such biases may occur from the dataset [Kim et al. 2018 or the model itself. Identifying the root cause and removing such biases would be another expected future work, beyond the explanation the input features.
Limitation Although our method has empirically outperformed qualitative comparison and various quantitative experiments compared to previous work, the notion of better input attribution method still vague. In this work, we adaptively find the sequence of inputs by using local attribution, but there would exist better justification of the sequence or the set of inputs that are essential clues for identifying the core features in the input.

---

Image
1
GBP
FG
DGA (0urs)
Most Relevant Features (10%)
Most Relevant Features (10%)
0
Most Relevant F Features (10%)
Most Relevant Features (10%)
Figure 6: Qualitative comparison among various attribution methods for VGG-16 in the validation dataset of ImageNet. Upper rows describe the heatmaps obtained by each methods and lower rows shows top 10% most relevant input features. DGA generates sharp and object-oriented attribution heatmap in the almost examples. See more examples Appendix B-D.
Acknowledgement
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAl) grant funded by Defense Acquisition Program Administration (DAPA) and Agency for Defense Develop- ment (ADD) (UD19003 and partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022- 0-00984, Artificial Intelligence, Explainability, Personalization, Plug and Play, Universal Explanation Platform), (No.2019-0 0-000-0-- Artificial Intelligence Graduate School Program (KAIST)), (No. 2022- 0-00184, 14, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethies).
10

---

References
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference Learning Representations, 2018.
Robert Aumann and Lloyd S Shapley. Values of non-atomic games. 2015.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise wwise relevance propagation. PLoS ONE, 10, 2015.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2/th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pages 1721-1730 2015.
D. Erhan, Aaron C. Courville, and Yoshua Bengio. Understanding representations learned in deep architectures. 2010.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks, 2019.
Agnieszka Crabska-Barwingt Amal R Omar Rivasplata, and András György, Towards better visual explanations for deep image classifiers. In eXplainable Al Approaches for Debugging and Diagnosis., 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attributions through regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4948-4957. 2019.
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5050-5058, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 2668-2677, 2018.
Picter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Er- han, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternatribution. In International Conference on Learning Representations, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müllerr. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1-8, 2019.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in Neural Information Processing Systems, 27, 2014.
Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong- Whan Lee. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2501-2508. 2020.

---

Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons neural networks via deep generator networks, In Advances in Neural Information Processing Systems, volume 29, 2016.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018.
Deng Pan, Xin Li, and Dongxiao Zhu. Explaining deep neural network models with adversarial gradient integration. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 2876-2883, 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black-box models. In British Machine Vision Conference, 2018.
Zhongang Qi, Saeed Khorram, and Fuxin Li. Visualizing deep networks by optimizing with integrated gradients. Proceedings of the AAAI Conference Artificial Intelligence, 34(07):11890-) 1900118999--1888 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255)) 2015.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned, IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-25) 26000022673,-1 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE International Conference Computer Vision, pages 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713. 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145-3153, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv::706.000 03825, 2017.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. Advances in Neural Information Processing Systems, 32, 2019.
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Proceedings of the European Conference Computer Vision, pages 498-512, 2018
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference Machine Learning, volume 70, pages 3319- 3328, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-58469, 2020.
12

---

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2921-2929, 2016.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] See Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] Figure 5 includes the error range obtained by multiple runs
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "YEWXzi6AgWA", "reviewer": "Reviewer_cvDy", "summary": "This manuscript proposes a new attribution based DNN explanation method that distill input features using weak and extremely positive contributor masks. This Distilled Gradient Aggregation (DGA) approach combines the advantages of both global IG and local FG approaches via a new sequential feature distillation algorithm, distilling irrelevant features from the input. Authors performed quantitative and qualitative evaluation by comparing to various attribution methods.  ", "questions": "Have you discussed the pros and cons of your iterative tuning approach to simple baseline such as saliency map in terms of cost-effectiveness? ", "limitations": "Just like other methods discussed in this manuscript, it is a post hoc explanation method that attempting to explain a trained model’s output. It does not address the issue of model selection/training for improving explanation quality. Since pre-trained models are used, so the comparison is fair.\n\nOther limitations are time and tuning complexity. It can be cost-prohibitive to explain each single image.     ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The DGA approach effectively addresses the limitations of local approach (FullGrad) that uses a single anchor point to compute attribution. It also addresses counter-intuitive behavior of global approach (IG) with arbitrary integral path. Authors also evaluated performance using ROAR and LeRF/MoRF and demonstrated superior performance. The overall idea of aggregating local attributions via knowledge distillation is novel and experimental evaluations are sound and adequate.      \n\nWeaknesses: (1) Authors could discuss the robustness of your explanation approaches against input perturbation since it is a known problem of IG type of methods. (2) Author could perform an evaluation on the percentage of input ablation on model explaining. (3) The sequential distillation process can be time consuming considering the explanation is on per sample basis.      \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XK0PlAjnTsz", "reviewer": "Reviewer_hnze", "summary": "The authors propose a new gradient-based feature attribute method Distilled Gradient Aggregation (DGA) that extends FullGrad (FG). On quantitative and qualitative evaluations the authors demonstrate that their method produces better explanations than previously proposed methods.", "questions": "See \"Weaknesses\" section.", "limitations": "The authors briefly discuss the limitations of their work in their conclusion section. They do not address potential negative social impact, which is fine (I don't see any obvious negative impacts with this work).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think the ideas in this work are interesting. However, as-is I have significant concerns with the evaluation of the method that prevent me from recommending acceptance. If the authors are able to address my concerns with additional experimental results I would be happy to raise my score.\n\nStrengths:\n\n1. The authors tackle a high-impact problem space (i.e., interpreting neural network predictions).\n2. The authors' proposed method is, to my knowledge, novel.\n2. Both qualitative and quantitative experimental results indicate that the method produces higher-quality attributions than previously proposed methods.\n\nWeaknesses (in order of importance to my score):\n\n1. Lack of ablations: The authors' proposed method is comprised of multiple moving parts (e.g. the redefined post-processing function $\\Phi$, the masking functions, and the aggregation of multiple attribution maps), and it's not clear to me exactly which component(s) led to the improvements over previous methods. For example, based on the data provided in Figure 8 in the supplement, it appears that DGA without the EPC mask (i.e., the purple line with $q=1$), already outperforms many of the baseline models. In my view, the manuscript would be much stronger if additional ablation results were included. On a similar note, it would further strengthen the paper if the authors demonstrated that the individual pieces of their method can't trivially be applied to previous work to obtain similarly strong results. For example, how does the performance of DGA compare to applying integrated gradients with an EPC mask?\n2. Influence of the ReLU in aggregation: The authors propose to apply a ReLU function to the individual attribution maps before aggregating them, with the justification of only wanting to retain features that push the network _towards_ a given prediction (rather than away). However, the application of the ReLU function makes the comparison with baseline methods unfair. For example, qualitatively (e.g. in Figure 5), we can see that many of the features with negative attribution values from baseline methods lie in the object of interest of the image (e.g. on the dog's face for the dog image). As such, it's not surprising that DGA performs much better than baselines on the LeRF metric, since the ReLU in DGA would set such attributions to 0. Do these results change if e.g. ReLU is applied to the Integrated Gradients attributions before computing LeRF values?\n3. Axiomatic justification (or lack thereof): Many of the previously proposed methods cited in this work (Integrated Gradients, FullGrad, GIG) are guaranteed to satisfy various desirable axioms for attributions. Could the authors comment on which, if any, of the axioms satisfied by previous methods are also satisfied by DGA? If none, is DGA guaranteed to satisfy another set of axioms?\n4. Not self-contained: The authors' proposed method is an extension of FullGrad, yet the authors do not provide any background on FullGrad in their manuscript. As such, I frequently needed to refer to the original FullGrad manuscript to be able to understand this work. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QjpOEUy82yC", "reviewer": "Reviewer_H2WE", "summary": "The paper identifies pitfalls in two attributions methods, then proposes a novel attribution method that purports to solve the problem.\n1) The paper identifies issues with popular attribution methods Integrated Gradients and FullGrad. IG suffers from attribution noisiness and unintuitive results, while FullGrad has weak dependency.\n2) The paper proposes using FullGrad with a different post-processing function, and engaging in sequential feature distillation. Essentially they calculate a FullGrad attribution; then they mask a constant, small proportion of the top-ranked and an increasing proportion of the bottom-ranked inputs according to the attributions. They repeat this process until the image is fully masked.\n3) They run comparative tests on other attribution methods. They run quantitate ROAR and Perturbation tests, and they run qualitative tests. These tests indicate the advantages of their method.", "questions": "Q: Have you addressed the issue of not knowing the ground truth? How do you know this method is reliable if the ground truth is unknown? Pixel Perturbation and ROAR tests address this question, but perhaps another test that could help. Are there any instances where the ground truth is known, i.e., we know what part of the image contributes to the model classification?\n\nQ: It seems your method consists of 1) using successive masking and reattributing, and 2) using Fullgrad to get attributions between iterations. Why use FullGrad and not any other attribution method for part 2? Have you tried your algorithm with other attribution methods? I wonder if the real novelty and success here is in part 1, your masking approach, and using FullGrad between iterations is replaceable.\n\nQ: What made you choose the methods you used for comparison? Why not SmoothGrad, LRP, Deconvolutional Networks, DeepLift, GradCam, or CAMERAS (see references)? I am not saying you should use them all, and understand it would require more work to compare them all. I am wondering how you justify the comparisons you choose to make and which you leave out. Perhaps a blurb on why you choose to compare some and not others?\n\nQ: Which baseline do you use for IG? It seems you are not using black or white baselines, since IG attributes to the dumbbells and white cloth in the first and second images of figure 2, and IG gives zero attribution to pixels with the same value as the baseline. Have you considered trying other baselines, as in [Pascal Sturmfels, 2020]? It is possible a single baseline IG is not the strongest formulation. If this is the case, It would be interesting to see what the comparative metrics are to a stronger version of IG that uses a better baseline. It would strengthen the claim that this method outperforms IG.\n\nQ: It seems that the counter-intuitive behavior of IG is due to the choice of baseline. An unanswered question is: why choose that baseline? It makes sense to put it in the middle, between the two curves. However, another side-effect of IG is that if you leave the baseline at (0,0), then any point of the form (0,x_2) will have 0 attribution in the first component, which does not seem correct either. This is not a problem unique to this paper: the issue of how to choose a baseline is a generally unanswered for baseline-attribution methods. It has to do with the idea of missingness in game-theory, which is not present in object recognition. See references for treatment of baseline options.\n\nQ Are there any references needed for section 3.1, \"Vulnerability of FullGrad\" and \"Counter-intuitive behavior of IG\"? These observations are completely original?\n\nReferences for the Author:\n- DeepLift\nAvanti Shrikumar, Peyton Greenside, and Anshul Kundaje.\nLearning important features through\npropagating activation differences.\n- Deconvolution Networks\nMatthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks.\n- CAMERAS\nMohammad AAK Jalwana, Naveed Akhtar, Mohammed Bennamoun, and Ajmal Mian. Cameras:\nEnhanced resolution and sanity preserving class activation mapping for image saliency.\n- An investigation of various baselines used for IG.\nPascal Sturmfels, Scott Lundberg, S.-I. L. Vi-\nsualizing the impact of feature attribution base-\nlines, 2020. \n- Using a distribution of training-set images as a baseline for IG.\nErion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M.,\nand Lee, S.-I. Improving performance of deep learning\nmodels with axiomatic attribution priors and expected\ngradients.\n- Uses a Gaussian blur process to alter the path of IG.\nXu, S., Venugopalan, S., and Sundararajan, M. Attribution\n\n\nSmall issues/typos:\n- Autonomous Driving (22-23)\n- \"so that humans\" (24)\n- no \"the\" before \"critical (26)\n- \"evidence for the \"decisions\" (27)\n- a heatmap (30)\n- What is Clever Hans (32)\n- \"Obtaining a trustworthy input\" (32)\n- Missing an arrow from \"Input x\" down to the first dot in Figure 1 (A)\n- derives the attribution to be vulnerable (108)\n- let we have two inputs (108-109)\n- Figure 2 sentence 2 is incomplete.\n- Please describe MoRF and LeRF in caption of Figure 2.\n- What is x-bar in 121-12? I believe it is the baseline but I don't see in defined before.\n-  each ablated features (133)\n- that masking out irrelevant features using IG(138)\n- as uniformly distributing function (142-143)\n- to remain the relevant features (175)\n- In Figure 2 b, I assume you pruned k%. What is k?\n- Similarly, in Table 1, what is k? Can you give multiple k's in a graph? This is a snapshot using one k, and could not be representative.\n- For pixel perturbations, did you calculate attribution rankings for each of 50k images for each method and model, then clip inputs according to the rankings and re-evaluate the confidence? If so, does table 1 report average new confidence, over all 50k images, after k% removed?\n- while almost methods (234)\n- concentrates to the person (235)\n- the relevant patch has less sharp than patch of DGA (235-236)\n- in the Appendix (236)\n- we provide the vulnerability (239)\n- rework sentence on line 241\n - there would exist... (256)", "limitations": "Some limitations mentioned. Other limitations could be addressed refereeing to Questions section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear presentation, easy to follow with good examples.\n- Strong qualitative and ROAR results support the conclusion.\n- Masking and reattributing successively seems novel, to my understanding.\n\nWeaknesses:\n- Motivation seems weak. Why did the authors identify those two issues in two seemingly arbitrary methods as motivation for their method? Why did they decide to use FullGrad with the successive masking, as opposed to any other method?\n- Grammar/typos need polish.\n- Some experimental choices seem arbitrary/unexplained (see questions).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LI6EdOavsnW", "reviewer": "Reviewer_5oaM", "summary": "This paper introduces a new explanation method called DGA for attributing a function’s output w.r.t its input features. DGA is motivated by problems of some related work, i.e. Integrated Gradient and FullGrad. ", "questions": "My questions are discussed together with my comments in the weakness part. I will summarize them here and please refer to my comments above to see the relevant context.\n\n- Why is the so-called “vulnerability” of FullGrad not a problem of the underlying model?\n\n- What does “counter-intuitive behavior of IG” mean when an intuitive behavior is not discussed at first.\n\n- How do the current metrics measure “vulnerability” and the so-called “counter-intuitive behavior”?\n\n- Why RISE is missing from the evaluation section?\n", "limitations": "The authors have a separated section to discuss the limitations. ", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\nVisualizations are helpful to convey the authors’ message and explain the pipeline of the proposed method.\n\n### Weakness\n\n**Summary of Weakness.** My main concerns for this paper are problems in its motivation and the way the new method is evaluated. Although I find the current way the method is presented to the audience is quite confusing, this can be fixed in an easy way. I will vote for rejection for now. \n\n**Motivations are not convincing.** A short summary for this paper’s motivation is the problems in existing methods, IG and FullGrad. Namely, FullGrad is vulnerable to local changes to the input (line 105)  and IG is counter-intuitive (line 114). However, none of the comments (or blames) made on the prior work makes sense to me. I will elaborate in detail. \n\n- The fact that FullGrad (or gradient-based methods) may change for some $x’$ that is near to x does not mean this is a vulnerability of the explanation method. In fact, this is a problem of the underlying model and the explanation method is just being faithful to the model’s behavior. **In another word, being faithful is not a problem.** With $f(x) = f(x’)$ and $||x-x’|| \\leq \\epsilon$, it is unclear to me why $FullGrad(x)$ and $FullGrad(x’)$ must be the same if the model indeed treats $x$ and $x’$ differently in a significant way. In fact, a lot of work in adversarial vulnerability has shown that this is often the case: 1) linear regions are often tiny and very different from their neighboring regions [1]; and 2) models are learning noisy or spurious features, e.g. using background instead of the object, to make the prediction [2, 3, 4]. All these observations point out that making a more robust model is the way that one will observe that explanations for neighboring points look similar in the end [4, 5, 6]. If the underlying model does not behave similarly for similar input, why a **good** explanation method, e.g. FullGrad, should behave similarly? All in all, the so-called “vulnerability” is not a problem for FullGrad but a problem for the model.\n\n- The comment about IG being counterintuitive is counterintuitive to me as the authors do not provide any justifications for what behavior is intuitive/expected. In the example shown in Figure 3c, the reason why IG is different in region A and B is that the baseline point is in A so that any integral path in A returns the same IG scores because the function is linear in A. In this case, the integral part of IG returns nothing but just the weight of the function, w^{(A)}, for every point in the path integral. Namely, $\\forall x \\in R^{(A)}, IG(x) = w^{(A)} \\odot x$. And because you only move x on one axis, only the attribution score of IG for another axis will change. But for points in B, the baseline is outside the region B so IG(x) no longer just returns the weight $ w^{(B)}$: it is an aggregation of all $w^{(i)}$ the path integral traverse weighted by the length of the intersection between the path integral and the linear region. Thus, you observe IG also changes even when you only move the point on one axis. I have spent a long paragraph just showing that this is just a matter of baseline choice in IG and without any theoretical justifications or geometric-based analysis, I don’t see there is any problem in IG. Alternatively, if the authors argue that IG is using “bad” baseline points, then this observation should motivate the paper to find better baselines instead of developing DGA. \n\n**Empirical evidence is not able to support the authors’ claims.** In the paper, the authors adopt metrics based on the ablation study using feature attributions to show that the proposed method, DGA, has a better performance on these metrics. Notice that the problems brought out earlier in this paper are **vulnerability** and **counterintuitive behavior** instead of something like “the current methods cannot identify features that are actually important to the model.” Without justifications about why the metrics used in the current paper, e.g. MoRF or ROAR, are able to measure **vulnerability** and *counterintuitive behavior**, the current empirical evidence is not related to the topic from my perspective. \n\n**The way DGA is introduced is a bit confusing.** I find myself lost when transitioning from Section 3.1 to 3.2. I don't see clearly how RISE solves **vulnerability** and **counterintuitive behavior** in IG and FullGrad and I hope to see more principled justifications about that. Then the authors quickly start to explain the technical details of feature distillation, without a high-leveled discussion of what the method is going to do and why this approach solves the problem in a more efficient way compared to RISE. Also, since RISE also solves the problem as the authors mentioned in line 131, why is it missing from the evaluation section? \n\n\n[1] Croce, Francesco et al. “Provable Robustness of ReLU networks via Maximization of Linear Regions.” AISTATS (2019).\n\n[2] Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. \"\" Why should i trust you?\" Explaining the predictions of any classifier.\" Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.\n\n[3] Ilyas, Andrew, et al. \"Adversarial examples are not bugs, they are features.\" Advances in neural information processing systems 32 (2019).\n\n[4] Wang, Zifan et al. “Robust Models Are More Interpretable Because Attributions Look Normal.” (2021).\n\n[5] Chen, Jiefeng, et al. \"Robust attribution regularization.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[6] Singh, Mayank Kumar et al. “On the Benefits of Attributional Robustness.” ArXiv abs/1911.13073 (2019): n. pag.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept