Beyond Single Path Integrated Gradients for Reliable Input Attribution via Randomized Path Sampling
Authors: Giyoung Jeon
Haedong Jeong
Jaesik Choi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainable AI
Input Attribution

Abstract: Input attribution is a widely used explanation method for deep neural networks, especially in visual tasks. Among various attribution methods, Integrated Gradients (IG) is frequently used because of its model-agnostic applicability and desirable axioms. However, previous work has shown that such method often produces noisy and unreliable attributions during the integration of the gradients over the path defined in the input space. In this paper, we tackle this issue by estimating the distribution of the possible attributions according to the integrating path selection. We show that such noisy attribution can be reduced by aggregating attributions from the multiple paths instead of using a single path. Inspired by Stick-Breaking Process, we suggest a random process to generate rich and various sampling of the gradient integrating path. Using multiple input attributions obtained from randomized path, we propose a novel attribution measure using the distribution of attributions at each input features. We identify proposed method qualitatively show less-noisy and object-aligned attribution and its feasibility through the quantitative evaluations.

Under review as a conference paper at ICLR 2023
BEYOND SINGLE PATH INTEGRATED GRADIENTS FOR RELIABLE INPUT ATTRIBUTION VIA RANDOMIZED PATH SAMPLING
Anonymous authors Paper under double-bling brind review
ABSTRACT
Input attribution a widely used explanation method for deep neural networks, especially in visual tasks. Among various attribution methods, Integrated Gradients (IG) (Sundararajan et al. (0077) is frequently used because its model-agnostic applicability and desirable axioms. However, previous work (Smilkov et al.) 2017] Kapishnikov et al., 2019: 202I) has shown that such method often produces noisy and unreliable attributions during the integration of the gradients over the path defined in the input space. In this paper, we tackle this issue by estimating the distribution of the possible attributions according the integrating path selection. We show that such noisy attribution can be reduced by aggregating attributions from the multiple paths instead of using single path. Inspired by Stick-Breaking Process (Sethuraman) 1991). we suggest a random process to generate rich and various sampling of the gradient integrating path. Using multiple input attribu- tions obtained from randomized path, we propose a novel attribution measure using the distribution of attributions each input features. We identify proposed method qualitatively show less-noisy and object-aligned attribution and its feasibil- ity through the quantitative evaluations.
INTRODUCTION
Along with the steep improvement and the real world application of the deep learning models (Caruana
et al. 2015; Yurtsever et al. 2020), discovering the evidence of the black-box model decision is considered to be important for debugging the malfunction (Lapuschkin et al.) 2019)) and promise
the safety and the fairness DDoshi-Velez & Kim (017) of the models. Within the vast literature of explaining the decision of the deep models, input attribution (Simonyan et al.) 2013 Bach et al.)
2015 Shrrkkmmrr al. 2016)] (Sundararajan al. 2017) is one of widely used methods quantify the relative contribution of each features to the model output. Input attribution provides the explanation in the form of heatmaps, which is useful indicate the spatial existence of evidences, especially in
visual tasks.
Among various approaches to compute the input attributions, Integrated Gradient (IG), one of widely used methods, and its variants (Sundararajan et al. [2077] |aan et al.) [02 Kapishnikov et al.) 2021 are of particular interest in our work. These methods explore the input space along the predefined
path and integrate the gradients to provide the reliable attributions. The integration path of such methods consists of a baseline which represents the missingness of features and a connecting line between the input and the baseline. With different desired properties, various paths can be used to compute the attribution. For example, Guided IG (Kapishnikov et al. (202I) proposes the adaptive path to alleviate the high and noisy gradients unrelated to the prediction. The selection of baseline can also affect to the attribution results (Strumbelj & Kononenko 2014).
While the above methods address the importance of selecting appropriate integration path, in this paper, we claim that the single path is not reliable enough to interpret the decision of neural networks. We provide a simple example that the attribution computed by a single path provides high variance according different path selection. For better reliability, we propose novel attribution method to take the expectation of the path-ing antt-intterrtedd attribution over the distribution of possible paths. To sample from the distribution over the vast variety of possible paths, adopt the notion of Stick-

---

Under review as a conference paper at ICLR 2023
Basc Disiitatio Coreetntrr
Input
CDE: Curulattre Distiluton Fuactin Breaking
Vor, (ory)
UQ0,11
Random A itribution Samples
Steli-Prnsking Prexesse
O(y
6(9
P(
E, [o(o))))
5ty
Beadice
400
3/rr62 31
Path madiods with SP
Stick Brosking Path
Entegration with Bendom Pulls
Suck-Breakis Brea--Beggggg Path Integration
SPI Probabiiity
Figure 1: An illustration of Stick- Path Integration (SPI) for the given imput Using the realized distribution G from SBP, we randomly generate the integration path in the input domain by taking CDF of each distribution (colored lines in the left-bottom). From the sampled paths, we apply the gradient integration along each path to gather the multiple attribution samples. By taking the average, the attribution of SPI can be obtained.
Breaking Process, which is one sort stochastic processes that samples the probability distribution. The main contributions of our work are summarized as.
Address the inconsistency of attribution according to the selection the integration path, and propose novel attribution method that takes the expectation over the distribution of random paths to retain the reliability of attribution.
Propose sampling method to generate a random integration path inspired by the Stick- Breaking Process. From the proposed method, we can generate the vast integration paths efficiently.
Evaluate the attribution in qualitative and quantitative measure to validate the reliability of the proposed method on various architecture of the networks.
RELATED WORK
Attribution methods The input attribution methods aims to measure the relative sensitivity of the model output with respect to the input features. Saliency method (Simonyan et al. (2013) is simple approach to use the gradient as attribution. Then Grad method (Shrikumar et al.] 2016) is proposed to multiply the input with the gradient for better input alignment. FullGrad (Srinivas & Fleuret 2019) proposes to use the bias gradient in addition to Grad#Input. Guided Backpropagation (Springenberg al. 2014) suggests
consider only features that positively contributes to the prediction by ignoring the negative backpropagated gradients. Layerwise Relevance Propagation (LRP) (Bach et al., 2015 Nam et al. 2020)) is another method modify the backward propagation. LRP proposes the relevance propagation rules based on the Taylor decomposition. There exists family of attribution methods which do require the access to the internal properties (e.g., gradients, parameters). LIME (Ribeiro et al. 2016) trains a surrogate linear model, which resembles the original model for the data that features are masked out from the input to be explained. RISE (Petsiuk et al.) 2018) computes the attribution by aggregating the model outputs from the multiple random masked inputs.
Path-based attribution methods Integrated Gradients (IG) (Sundararajan et al.) 2017) is one of widely used input attribution method. is built upon the same-theoretic notion of pay-off distribution method, Aumann-Shaple value (Aumann & Shapley 2015). IG has several desirable properties, called axioms, which supports the reliability of the attribution. IG is calculated by integrating the gradients along the path from the baseline to the input. Based this work, several extended research has been performed. To reduce the noise in the attribution, SmoothGrad (Smilkov et al. 2017) takes
the average of the multiple random noise added inputs and NoiseGrad (Bykov al.
2021 inject
2

---

Under review as a conference paper at ICLR 2023
Unit length Stick
Fà, (t)
Input Space
Input Space
Input
Inputxx
Duration
Uniis Prit--lenghh Stick
Fo,(t) (t)
Sampled Partition
Baseline
Baseline
SBP(H,
Stick-breaking Paths
a
(b)
Figure 2: (a) An illustrative example of various path integrated attributions with difference path choices. Each colored region indicates the passed different decision region along the integrated path. We identify that the complexity of integration (e.g., the number of decision regions) can be determined by the shape the integrated path. (b) An illustrative example of converting the realized partitions of stick to the integration paths.
noise to the weight parameters. From the observation that the sum of gradient of multiple features
more stable than the individual gradients, XRAI (Kapishnikov al. 2019) proposes the method
to merge the attribution of neighboring features using multi-level superpixels. Some researches try
modify the integration path with their own desired characteristics (Kapishnikov al. 2021 Pan
et al. 2021).
Evalutation of attribution methods Even though a plethora of attribution methods are proposed, evaluating the reliability or the correctness are still challenging. Pixel flip is an causal metric between the input and the prediction confidence (Samek al. 2017). Pixel flip calculated by setting the value of each feature to zero in order of increasing or decreasing order of attributions. As an extension, insertion and the deletion game is proposed by starting the insertion game from the blurred image avoid the spurious effects when small number of pixels are inserted (Petsiuk et al. (018). RemOve-Ane -en---e-Ann-BBRR And-Rerrnnn (ROAR) measures the performance drop when the dataset is reorganized by perturbing the imput images with the average pixel value in decreasing order of attribution (Hooker ęt al. 2019). Steep performance drop in ROAR indicates that the attribution method correctly points important features which actually used to train and inference.
STICK-BREAKING PATH INTEGRATION
In this section, we propose our new path- based attribution method, Bicck-rrakinng Path Integration (SPI). We first provide some backgrounds about Integrated Gradients (IG) (Sundararajan al. 2017) and its family of attribution methods, path methods. From the insight that the attribution highly depends the path selection, we propose to take the expectation over the distribution of random paths for reliability of attribution. Due to the difficulty in defining the distribution of paths, we propose sampling process which is motivated by the Stick-Breaking Process (SBP) (Sethuraman 1991). Using the multiple attributions computed by randomly sampled paths, we finally propose SPI. In addition, we propose visualization method for SPI. SPI-P, which utilizes the stochastic property of the sampling process.
3.1 PATH METHODS AND INTEGRATED GRADIENTS
Integrated Gradient (IG) (Sundararajan et al. 2017) is proposed as an explanation method for neural networks by adopting the Aumann- Aumann-Shapley Shapley value (Aumann & Shapley 2015), which is credit allocation method developed for the cooperative game theory. Based on such allocation strategy, IG aims compute the contribution of input features the model prediction. For differentiable model
3

---

Under review as a conference paper at ICLR 2023
function f. the input and the baseline x, IG on i-th feature is given
(x(7) - =So (s(((()))))))))))))))))))))))))) ax((l)
0r:(1) δL dt, -(t)xx+((--))
(I)
where y() represents a continuous path from the baseline x to the input x and 7/ (t) refers to the value of i-th feature at step along the path. Previous work has shown that integrating over different path y yields different type of attribution method (Pan al., 202T; Kapishnikov al. 202T). ). We call this group of attribution methods with arbitrary selection of path as path methods. However, the intermediate gradient in the single path likely to be noisy and it reduces the reliability and interpretability of the obtained attribution. Several work has shown such noise can be reduced by averaging IG over randomly perturbed inputs (Smilkov et al. 2017 randomly perturbed weights (Bykov al. 2021 average pooling over spatial location (i.e., local pixels for image) (Kapishnikov iet al. 2019).
In this work, we propose to aggregate the attribution from multiple paths with fixed baseline sampled from the proposed distribution of paths to reduce the noise and improve the confidence of the attribution. We first define the integration path and its desired properties.
Definition 3.1 (Integration Path). The integration path is a mapping function from t E [0, 1] to the input domain X, 7 [0, 1] X. The path should satisfy two properties; (1) the path starts from the baseline, y00)=x and ends at the input to be attributed, y(1)= and (2) the path monotonically proceeds from
i.e., dfr(t) dt
C(a; Te) for C C20 0.
In the rest of the paper, we denote the bold symbols for the vector (e.g.. x). We use the subscript to represent the indexed component (c.g.. 2;). We use the superscript represent the different indexed instances (e.g., w(1) w()) ......
3.2 RELATIONSHIP BETWERN PATH AND ATTRIBUTION
A neural network equipped with the partial linear activation, such as ReLU, is known to have the form of the piece-wise linear function (Montufar et al. 2014). The piece-wise linear function is defined by multiple linear functions, where each linear function is only feasible in corresponding linear region, R(0,, Such piece-wise linear function f can be formulated as follow with each linear region R(),
w (1)xx x + (1)
x E (1)
f(x)
WOT x b6)
X R()
Then the path method provides the attribution in terms of weighted sum of corresponding weight vectors in each region,
*,(7) Σ ( (1) (7) of (
dru(t) ()()))))))))) dt
(3)
Σ,
where a. (7)
(7) represents the projected length of the path to the i-th feature that passes through the region R() The delta function 8(5) (r()) returns y(t RG) and otherwise O. Even though Φ;
in Equation sums over every region, we note that it is equivalent to the summation over the regions that path passes through, because (7) (r) = 0 for the region R() that y does not pass through.
has been shown that to increase the expressivity of the DNNs, the number of linear regions should also increase (Xiong et al. 2020). With the high dimensional input space and the vast number of linear regions, using a single path for the path methods would induce high uncertainty and variance to the resulting attributions. Figure 24 illustrates that even in 2-dimensional space with several linear regions, different selection of path aggregates mostly different weight vectors of corresponding regions. To alleviate such uncertainty with selecting integration path, propose compute the expectation of path method over the distribution possible paths as follow,
Byy (dx(7)) f [' (s({(()) 0x((1)
dx.(t) Øt dy)(y
(4)
4

---

Under review as a conference paper at ICLR 2023
10
69
0.01
1
C 10
α 100
Figure 3: Visualization of CDF of SBP realized distributions G' with different concentration parameter a (colored lines). The base distribution is given as uniform distribution, (0, 1), and its CDF is given the straight line (black line). The realized CDF converges the CDF of the base distribution when the concentration hyperparameter a increase.
3.3 RANDOMIZED PATH SAMPLING
As the distribution of the path, P(y). difficult be defined, we propose an alternative approach to randomly generate the path with the variety. When generating path, we have two choices to select; (I) whether proceed the destination or not and (2) how far proceed. We reduce this problem to the Stick- Bieck-Bng Process (SBP) (Sethuraman 1991). SBP is generative process to obtain random fractions of a stick with its initial length of The fraction obtained by SBP can be regarded as how much portion from the baseline to the input should move at each step. In this configuration, the number of fraction can regarded as the number of steps the path and each length of fractions can be regarded as each length of steps. Figure depicts how the sampled partitions of the unit length stick can be used to build the integration path. Each sampled fraction is represented a Probability Mass Function (PMF), and such PMF is defined follow,
A-1 T T II B.), Bx Beta(L, tz ~ H (5)
G(t)= Σ Tr SBP(H,
where 6cg (t) is delta function that returns zero everywhere, except for 5, (t= tx) = and H is a base distribution. The expectation of G sampled by SBP is desired estimate the base distribution H. The hyperparameter o, also known as concentration parameter, controls the realized distribution G to be more similar the base distribution if a: takes larger value.
From the sampled PMF (t)), we define the CDF Ft(t) which can be used to build the integration path function. Let the integration path be
(t( 5; Fo(() -z))-
We note that the Equation 7 satisfies the properties of Definition 3.1 (1) (0)==ē, and (1())=r; because (0) O and FG = 1, and (2) dFF((t)ddt = G(t) 2 0. With taking the base distribution H to be the uniform distribution, H (0, 1), whose CDF is equivalent the IG straight path. Figure deepicts the CDF of SBP realized distributions using different a. When the value of increases, the realized CDF converges the CDF of U (O, 1) (black line), and we call this CDF as the base path.
Definition 3.2 (Stick-breaking Path (SP)). Given hyperparameter o; > O, the Stick-breaking Path (SP) of i-th feature from the baseline Σ; to the input is defined as the multiplication of the CDF of the distribution G, obtained by SBP and the difference between E; and F,,
71 ox) + Fa, (1(x; Tz), G SBP(U(O, 1),
With the sampling process of SP and hyperparameter a, we finally propose new attribution method, Stick-breaking Path Integration (SPI). Definition 3.3 (Stick-breake breaking Path Integration). For model and baseline x, SPI is defined as the expectation of the path method over the path distribution,
S.'
0ʃ(x + Fa(() @ (x R)) dFa. (1)
(x;,(x;)) (+:)) EG (xr (o:-):)_
dt
(8)
0(. Fo, (t) k,)) dt
5

---

Under review as a conference paper at ICLR 2023
Integraaed Gradients
16)
G
Input Image
0.01
==1
10
100
Figure 4: Input attribution obtained by proposed SPI method with varying the hyperparameter a.
With increasing a, SPI-E becomes noisy and resembles the attribution of IG. With decreasing a, the noise is reduced on the objects, but the checkered pattern appear in the background of the image. SPI-P with low a also returns high probability for the background.
where G is vector with each index follows SBP, G; SBP(U (0, a) and Fe [0, 1] + R is stack of SP that returns a vector the input space according to the current step t.
3.4 VISUALIZATION WITH SCORE BASED ON STATISTICS OF ATTRIBUTIONS
As we sample SP from the random process SBP, the attribution computed using SP, (v(7), also can be regarded as random variables. Assuming that d,(r) follows the Gaussian distribution, the estimated Gaussian distribution of Φ:(y) is given as follow
N Σ N
(:(7) (), μ
(k)) 1 Σ
(1)() (o
(9)
We observe that the input attributions obtained by integrating gradient over each randomly sampled path deviates in their own distribution. We categorize such attributions three; (1) the one with low variance and low mean, (2) the with low variance with high mean and (3) the with high variance. For the first and the second cases, we can identify that those features are consistently show low/high contribution with less dependent to the path. However, for the third case, need to measure how probable that such feature would contribute much.
With this insights, we propose an additional method by assigning the probability of attribution each feature is sufficiently positive along SBP. Assuming that the attributions for each feature are larger
distributed normally, we propose SPI-P, which measures the probability of attribution than some threshold O. We use the top 5% quantile value of SPI(x: a:) as the threshold.
Definition 3.4 (SPI Probability (SPI-P)). Assume that the attribution at i-th feature follows the Gaussian distribution with estimated μ and Equation Then SPI-P defined by the CDF of the Gaussian distribution,
exp (_(0-
(0
11)² o²
SPI-P.(x:a) = P(D. 0) = 1
d0
(10)
OV 2n
3.5 ANALYSIS ON a
We note that a controls the variance of sampled paths in Section 3.3 In this section, we provide the analysis how the attribution obtained by proposed method difters according to the value of a. showss the qualitative comparison of the proposed two methods, SPI-E and SPI-P. The first row in Figure 4 indicates that increases, the attribution becomes similar to the attribution of

---

Under review as a conference paper at ICLR 2023
Input Image
IG
GuidedIG
FullGrad
SPI
SPI-P
Figure 5: Qualitative comparison among various attribution methods for VGG-16 in the validation dataset ImageNet. Each column describes the heatmaps obtained by each attribution methods. SPI generates more object-oriented and less noisy attribution heatmaps. With SPI-P, the heatmaps are more distinguishable between the important features and the irrelevant background.
IG. As previously described, if a increase, the realized paths converges to the base path, which is the straight line of IG. The attribution with with high shows noisy results that the positive and the negative values are alternatively placed nearby. Such noisy attribution has been raised as a problem in gradient-based attribution methods (Kapishnikov al. 202T). In contrast, the result with low shows loss noisy attribution. Comparing the attribution from the high and the low a, With low a, the attribution shows less noisy and more consistent on the object.
4 EXPERIMENTS AND RESULTS
In this section, we verify the effectiveness of SPI by the qualitative analysis and the quantitative comparison. We first provide the qualitative comparison among different attribution methods, by providing the attribution heatmaps the randomly chosen examples. We note that the quantita- tive evaluation on the attribution methods is challenging because there no ground truth of the attribution and the ground truth changes if we train new model. To alleviate the absence of
7

---

Under review as a conference paper at ICLR 2023
Table
Insertion/Deletion game of various attribution methods on three models.
Insertion (1 is better)
Deletion (L is better)
VGG-16
Inception-v3
ResNet-18
VGG-16
Inception-v3
ResNet-18
GrautI
0,078
0.171
0.114
0,045
0.105
0.050
GuipedBProp
0,094
0.145
0.124
0,113
0.162
0.145
IG
0.096
0.243
0.158
0,036
0.066
0.038
FulIGrad
0,415
0.558
0,448
0,110
0.175
0.131
GuidedIG
0.110
0.255
0.185
0.029
0.061
0.02
SPI (Ours)
0.443
0.704
0.515
0.019
0.051
0.018
the ground truths, we provide the quantitative comparison using two widely used metries: (1) pixel insertion/deletion game (Samek al., 2017; Petsiuk et al. (018)), and (2) GemOve-And-Retran (ROAR) (Hooker et al.) 2019) to identily that the proposed method can assign attributions which are more relevant to model behavior. We select various gradient- based attribution methods for the comparison: Gradient Srradient*Inpae IInput (Shrikumar et al.) 2016), Guided BackPropagation (GuidedBProp) (Springenherg et al. 2014), Integrated Gradients (IG) (Sundararajan et al. 2017). FullGrad (Srinivas) & Fleuret 2019), and GuidedIG (Kapishnikov et al. 2021).
4.1 QUALITATIVE COMPARISON
For the visual inspection, which is the main application of the input attribution, we qualitatively compare the attribution heatmaps obtained by various methods. For the heatmap visualization, we take the color range to be bounded by top 5% absolute value of each attribution. Figure provides attribution heatmaps of randomly selected images from the validation set of ImageNet with pre-trained 16.-16. We identify that the attributions obtained by SPI show more aligned to the target class object and less attributed to the unrelated background. For example, in the third row, SPI correctly focuses on the necklace, while other methods distracted by the irrelevant pixels in the background. With SPI-P visualization, we also identify that the attribution better isolates the important features from the input image.
4.2 PIXEL INSERTION AND DELETION GAME
Pixel flip is first proposed to benchmark if attribution methods correctly capture the relevance between the input features and the model output (Samek et al. 2017). It later introduced as evaluating the effect on the model prediction caused by the pixel perturbation (Petsiuk et al. 2018). To quantify the relevance between the input features and the model output, pixel flip modifies the pixel values in order of relevance obtained by attribution methods, from high to low. Then it measures the change of softmax output for the target class with the perturbation. Pixel flip consists of two evaluations, the insertion game and the deletion game. The insertion game inserts the original pixel value the predefined baseline, and the deletion game deletes the original pixel value the input. If the input attribution is correctly related to the model prediction, then the insertion game would give high score by steep increase of output with the insertion of highly attributed pixels. In the same manner, the deletion game would give low score with correctly related attribution. When modifying the pixel, several choices arise which value to replace. We follow the configuration of previous work (Petsiuk) al. 2018), which use the blurred input for the insertion and the zero value for the deletion.
We use 50k images of the validation set provided by ImageNet |Russakovsky et al. 2015). We use three publicly available pre-trained models: VGG-16 (Simonyan & Zisserman (2015), 5), Inception-v3 (Szegedy et al. 2006)), ResNet-1e 18 (He et al. (016)). Table [ indicates the insertion and the deletion results for the various attribution methods and model architectures. We identify that SPI shows the best performance in both games on entire three architectures.
4.3 REMOVE-AND-RETRAN (ROAR)
ROAR (Hooker et al. 2019 is another metric to evaluate the attribution method correctly indicates the features that important in the perspective of the model training. ROAR is calculated by
8

---

Under review as a conference paper at ICLR 2023
Table ROAR evaluation of various attribution methods on ResNet-18 trained with CIFAR-10.
Removed %
10
30
70
90
Grad*Input
52.76:1:0.55%
1.00%%
33.22-+1.339
29.25-:0.43%
24.92++1.395
GropedBProp
67.51+0.409
63.70=0..005
61.91=0.85%
59.99=0.93%
52.73=0.52%%
IG
39.70=t0.796
26.23=1.11%
21.46:+0.53%
17.73=0.58%
15.73=0.82%%
FullGrad GuidedIG
67.49.10.79%
56.84_10.77%
42.31110.75%
26. 10.11.19%
14.82.11.09%9
38.90.11.63%
24.23111011%
2..64
18.11110.75%
15.66.10.81%
SPI (Ours)
31.00=:0.48% %
19.70=0.40%
16.05
13.09=0.72%
10.88++1.06%
measuring the performance drop when the model is re-trained with modified training data. Each sample in the training dataset is modified by removing pixels with top k% attribution and replacing them with the average pixel value of the input. For the ROAR experiment, we use ResNet-18
architecture and train the model on 50k images of training set provided by CIFAR-10 10 dataset (Krizhevsky et al. 2009). For the training, we use the Adam optimizer (Kingma & Ba 2014) with learning rate 3e-4 and 100 epochs. After training with each modified dataset, the performance of the trained model is measured with the standard test dataset with 10k images in CIFAR-IO. We note that the attribution method captures more relevant features the test accuracy is lower. Table shows the test accuracy measure in the ROAR experiment for each attribution method. We repeatably conduct trials the experiments, where the parameters are random initialized at each trial and fixed between attribution methods. Table 2 shows the test accuracy measure in the ROAR experiment for each attribution method. The result indicates that the model trained on the modified dataset with SPI steeply decreases the test accuracy with 10% removed. We conclude that SPI is effective in identifying the input features which are important to train the DNNs models.
5 DISCUSSION
In this paper, we proposed the novel attribution method, Stick-breaking Path Integration (SPI), to provide more reliable explanation the relationship between the input features and the model decision. Based on the path method, which is computed by integrating the gradients along the path in the input space, first provide the affect of path selection the computation of the input attribution. By raising the necessity of considering multiple paths for the reliable attribution, we propose to average the wath-based attribution over the distribution of paths. We qualitatively show that our method provides attributions with more object-aligned and less noisy. We also provide the quantitative evaluations, pixel flip and ROAR, to identify that our method well-aligned with the model behavior.
Our work also sheds light for several future works that would provide more reliable or meaningful attributions. The selection of the base distribution would be different to the uniform distribution. For example, one may use the Gaussian distribution to control the path to proceed at early steps or late steps by managing the mean of the distribution. Using different path sampling method would be another approach. Our method assumes the paths are distributed uniformly, but one may define distribution with several modes. Randomizing the baseline would also diversify the sampled integration path. We expect our work would provide new aspect of attribution methods to inspect the box models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Robert Aumann and Lloyd S Shapley. Values of non-atomic games. 2015.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-lineat classifier decisions by layer-wise relevance propagation. PloS one, 2015.
Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M-C Höhne. Noisegrad: enhancing explanations by introducing stochasticity to model weights. arXiv preprint 11X1:22006.0015, 2021.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1721-17361 2015.
Finale Dosei-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv: 070.0000020000 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Sara Hooker, D. Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In NeurlPS, 2019.
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attributions through regions. In ICCV, 2019.
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In CVPR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 2019.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong-Whan Lee. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks. In AAAI, 2020.
Deng Pan, Xin Li, and Dongxiao Zhu. Explaining deep neural network models with adversarial gradient integration. In IJCAI, 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint 1rXiv:1806.077 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In ACM SIGKDD. ACM, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and -obert Müller. Evaluating the visualization of what a deep neural network has learned, IEEE Transactions Newral Networks and Learning Systems, 2017.
10

---

Under review as a conference paper at ICLR 2023
Jay Sethuraman. A constructive definition of dirichlet priors. 1991.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01711 017600 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprini arXiv: urXiv:/706.000 7706.03825, 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. In NeurIPS, 2019.
Erik Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICML. PMLR, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Huan Xiong. Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. In International Conference on Machine Learning, pp. 10514-10523. PMLR, 2020.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 558443--58669 2020.


Reviews:
[{"review_id": "g19vQ0PICg3", "reviewer": "Reviewer_jggk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the attribution method ``Stick-breaking Path Integration'' which averages the attribution maps obtained by integrating over several different paths using the Integrated Gradient attribution method. They show superior performance qualitatively and quantitatively (pixel flipping, ROAR) compared to other explanation methods (Gradient*Input, GuidedBackProp, IntegratedGradients, FullGrad, GuidedIG).", "summary_of_the_review": "While the results show superior performance on quantitative metrics, the overall contribution does not seem very significant. I cannot recommend publication at ICLR.", "strength_and_weaknesses": "*** Strengths\nAttribution maps are less noisy and show good performance on the evaluated metrics.\n\n\n*** Weaknesses\nMy main concerns are with respect to the significance and novelty of the contribution. There are already many attribution methods out there, so I think the minor extension of an already existing method might not be significant enough to be published at ICLR. There is little theoretical motivation/analysis/guarantees for the proposed methods. Comparison to other popular methods (LIME, SmoothGrad) is missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wwx_P9Byox_", "reviewer": "Reviewer_tTr4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Attribution methods address the issue of explainability by quantifying the importance of an input feature for a model prediction. In this paper, the authors propose a variant of the integrated gradients (IGs) method through a strategy of \"selection of the paths to be integrated / aggregated\". The authors propose to generate several possible attribution estimates through a stochastic process, then, they show that attribution is improved by aggregating attributions from multiple paths instead of using a single path.\n\nThe main contribution of the paper is a new strategy that allows to improve attribution maps.\n\n\n", "summary_of_the_review": "I'm satisfied with the response of the authors. The different points I had raised were taken into account and my doubts were clarified. \n\n--\n\nThe paper is well written and clear, there is an interesting contribution which consists in extending the IGs method by keeping the input fixed and generating a set of random estimates in order to integrate the different paths and obtain a single attribution map.\nThe weakness of the paper is in the experimental section which does not contain sufficient details to analyze the results.", "strength_and_weaknesses": "**Strength**\n\n- Integration of different paths keeping the input fixed\n- sampling method to generate a random integration path\n- Good results on qualitative examples\n\n\n**Weaknesses**\n\n- More details in the experiments section would help with comprehension and reading.\n\nFor example, the pixel ins/del game. Is the input tiled or is each pixel used as a test unit?\n\nIf I understand correctly, given an attribution heatmap, each \"unit (=pixel/tile)\" is ranked according to the attribution values, in deletion, the highest-ranked 'unit' is replaced with a constant value (=0), the modified input is fed through the network, and the resulting drop in target class score is measured. \n\n- Does the \"insertion\" test start from a black image?\n- Which score is reported in Table 1? The first output or the AUC of the resulting curve?\n- If the score is the AUC of the curve, it might be interesting to have a plot of some curves to see the different trends.\n\nThe two tasks, del/ins seem exactly complementary. I'm not sure what information is adding the insertion task. On the other hand, it is useful to find the pixels/tiles that  disrupt the output of the network as quickly as possible. Neural networks are sensitive to subtle changes in input, and pixels or tiles don't necessarily have to contain meaningful information to break the network (this is also one of axioms in the IG paper). Therefore, Ancona [1] and others [2,3,4] proposed to reverse the deleting direction, by first removing pixels or tiles ranked as less relevant by the attribution method.\n\nRemoving 1) the most important pixels first and 2) the least important pixels first, leads to two different curves, the integral of the two curves provides an estimate of the performance of an attribution method in a quantitative way.\n\n\n\n\n\n[1] Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. A unified view of gradient-based ¨ attribution methods for deep neural networks. In NIPS 2017-Workshop on Interpreting, Explaining and Visualizing Deep Learning. ETH Zurich, 2017.\n\n[2] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating Feature Importance Estimates. arXiv e-prints, 2018.\n\n[3] Pieter-Jan Kindermans, Kristof T. Schutt, Maximilian Alber, Klaus-Robert Muller, Dumitru Erhan, Been Kim, and Sven Dahne. Learning how to explain neural networks: Patternnet and ¨ patternattribution. In International Conference on Learning Representations, 2018.\n\n[4] Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert ´ Muller. Evaluating the visualization of what a deep neural network has learned. ¨ IEEE transactions on neural networks and learning systems, 28(11):2660–2673, 2016.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QUtr1dygmYz", "reviewer": "Reviewer_KHRW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a refinement of Integrated Gradient (IG) by considering several paths from the baseline to the test example. The paths are sampled from a specific discrete distribution, and the results show improvement in some of the established metrics like insertion/deletion tests and ROAR.", "summary_of_the_review": "After author response:\nI am changing my score from 6 to 7. Since that option doesn't exist in the UI (it goes from 6 to 8), I am adding this note.\n\n-------\nOverall, the paper makes a sensible change to IG by including path averaging and the results look convincing. On the other hand, the specific path sampling method seems like an over-specification, and the result section glosses over a competitive baseline. In its current draft, I am only moderately inclined for the paper to be accepted at the venue.", "strength_and_weaknesses": "IG is a well-established attribution technique and there have been a fair amount of follow-up work to improve it. The authors clearly argue how using a single path can lead to sampling error, and propose a sensible way to sample paths. The results section shows the proposed technique improves on several metrics and is convincing enough. Some critical thoughts follow:\n\n1. The development of section 3.3 is unclear to me. I am also not sure how much the choice of the distribution matters, and this uncommon choice seems unjustified. I would think similar results might be observed by using another discrete distribution (say hypergeometric normalized to a max value of 1), since the CDF is always monotonic and meets the required constraint. I would appreciate authors thought on this and any justification for this specific choice as opposed to a general discussion around any parametrized probability distribution. \n\n2. The result section would benefit from inclusion of smoothgrad as the only other technique that performs some sort of averaging and thus could offer a more competitive baseline.\n\n3. In definition 3.1, the definition of monotonicity is incorrect. It defines the much more constrained linear path. Similar error appears in the description of equation 1 which is constrained to be linear instead of a general path.\n\n4. Many minor editing errors: “we address the single path is not reliable”, “consider the features only increase the value”, “the attribution is an”, “averaging pooling over”, “path is monotonically proceed”, “if \\alpha assigned with bigger value”, “with alleviating the absence”, “with DGA” (which is undefined).\n\n5. I also don’t see the clear value of including the visualization method with the proposal of the technique diluting the focus, but this is more subjective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "K-c2hcqzNSu", "reviewer": "Reviewer_yFiz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to reduce the attribution nioses in integrated gradients method by utilizing multiple intergrating paths. Specifically, the Stick-Breaking Process is utilized for aggregating the attribution map from large number of random choosing paths.", "summary_of_the_review": "The score of this paper can be changed according to the rebuttal response.", "strength_and_weaknesses": "Weaknesses:\n\n1. Given the successful work of IG method, this submission extends the attribution map integration from single path to distributed multiple paths, which is a little bit incremental.\n\n2. There are some incomplete sentences and typos in the submission. For example, 'Sanity checks on the attribution is an'.\n\n3. For equation(2), ReLU, which is the most commonly used activation in vision, only with two linear regions, thus the proposed path choosing method may not important. Moreover, the discussion about the non-linear activations, such as sigmoid, should be discussed.\n\n4. When the value of the hyperparameter /alpha is small, how will the path choosing process be affected? With a less constrained random choosing? Or pushed far away from the base distribution H?\n\n5. Eq(9) and Eq(10) based on the assumption that the \\phi follows the Gaussian Distribution. Therefore, the proof of the distribution of \\phi is a Guassian should be involved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject