Path Choice Matters for Clear Attributions in Path Methods
Authors: Borui Zhang
Wenzhao Zheng
Jie Zhou
Jiwen Lu
Venue: ICLR 2024 poster (2024)
Keywords: explainable AI
integrated gradient
attribution
path method
Primary area: visualization or interpretation of learned representations

Abstract: Rigorousness and clarity are both essential for interpretations of DNNs to engender human trust. Path methods are commonly employed to generate rigorous attributions that satisfy three axioms. However, the meaning of attributions remains ambiguous due to distinct path choices. To address the ambiguity, we introduce Concentration Principle, which centrally allocates high attributions to indispensable features, thereby endowing aesthetic and sparsity. We then present SAMP, a model-agnostic interpreter, which efficiently searches the near-optimal path from a pre-defined set of manipulation paths. Moreover, we propose the infinitesimal constraint (IC) and momentum strategy (MS) to improve the rigorousness and optimality. Visualizations show that SAMP can precisely reveal DNNs by pinpointing salient image pixels.
We also perform quantitative experiments and observe that our method significantly outperforms the counterparts.

Published as a conference paper at ICLR 2024
PATH CHOICE MATTERS FOR CLEAR ATTRIBUTION IN PATH METHODS
Borui Zhang, Wenzhao Zheng + Jie Zhou Jiwen Lu" Department of Automation, Tsinghua University, China
(zhang-br21) zhengwz1 @ma 1a. testtinnggi ainnnghaa... edu fizhou, jujiwen) Stsinnuua
inghaa edu. cn
ABSTRACT
Rigorousness and clarity are both essential for interpretations DNNs to engen- der human trust. Path methods are commonly employed to generate rigorous at- tributions that satisfy three axioms. However, the meaning of attributions remains ambiguous due to distinct path choices. To address the ambiguity, we introduce Concentration Principle, which centrally allocates high attributions to indispens- able features, thereby endowing aesthetic and sparsity. We then present SAMP, a model-agnostic interpreter, which efficiently searches the near-optimal path from pre-defined set of manipulation paths. Moreover, we propose the infinitesimal constraint (IC) and momentum strategy (MS) to improve the rigorousness and op- timality. Visualizations show that SAMP can precisely reveal DNNs by pinpoint- ing salient image pixels. We also perform quantitative experiments and observe that our method significantly outperforms the counterparts.
INTRODUCTION
The lack of transparency in deep neural networks (DNNs) hinders understanding of how these complex models make decisions (Bodria et al., 2021; Zhang Zhu, 2018: Gilpin et al., 2018), which poses significant risks in safety-critical applications like autonomous driving and healthcare. Numerous interpretation methods (Zeiler Fergus, 2014; Bach et al., 2015; Zhou et al., 2016; Sel- varaju al., 2017) have been proposed to shed light on the underlying behavior of DNNs. These methods attribute model outputs to specific input features to reveal the contributions. attribution methods serve valuable debugging tools for identifying model or data mistakes. How- ever, despite these efforts, users often lack confidence in attributions, which can be blamed on lack of rigorousness and clarity in current methods. Attributions are influenced by three types of arti- facts (Sundararajan et al., 2017), namely data artifacts, model mistakes, and interpretation faults. To enhance user trust, it is crucial sever the impact of the last factor.
this way,
One way to enhance the reliability of interpretations is ensuring their theoretical rigorousness. Given a complex mapping function f: X' H> TK, we define the target point xx X and the baseline point æ". Interpretations aim at explaining how the baseline output y gradually becomes y" when base- line x changes to x²". Early interpretation methods (Selvaraju et al., 2017; Montavon et al., 2018) employ Taylor expansion the baseline y² y + V (x"")"((>""" the local linear approximation can hardly interpret nonlinear models due to non-negligible errors of the Lagrangian remainder (x""), which makes attributions less convincing. An intuitive solution is to split the path from x to x into small segments, each of which tends to be infinitesimal. In this formulation, the variation Ay can be formulated in integral form Ay = y yy = V fx=)" da. The attributions a; of each feature x; is gradually accumulated through the line integral, which is commonly referred to path methods (Friedman, 2004; Sundararajan et al., 2017; Xu al., 2020; Kapishnikov et al., 2021). Game theory research (Friedman, 2004) has proved that path methods are the only method satisfying three axioms, namely dummy, additivity, and efficiency.
x²) + R2 (æ="'. However,
Ensuring rigorousness alone is insufficient for convincing interpretations. Distinct path choices in existing path methods highly impact attributions and lead to ambiguity in interpretations. Integrated Gradients (IG) (Sundararajan et al., 2017) adopts simple straight line from x to x" for symmetry. BlurIG (Xu al., 2020) defines a path by progressively blurring the data xF adhering additional
"Corresponding author.
CCode: https: http:////ttttttttt www.:bb...cccm.brcoo/ee 17/MAMP

---

Published as a conference paper at ICLR 2024
seale-space axioms. GuidedIG (Kapishnikov et al., 2021) slightly modifies the straight line to by- pass points with sharp gradients. However, the question of which path choice is better remains unanswered. The lack of research on the optimal path selection hampers the clarity of attributions.
To the best of our knowledge, we are the first to consider the optimal path for clarity. To start with, we define the Concentration Principle. This prin- ciple guides the interpreter to identify the most es- sential features and allocate significant attributions to them, resulting in aesthetic and sparse interpre- tations. Subsequently, we propose SAMP (Salient Manipulation Path), which greedily searches the near-optimal path from pre-defined set of manip- ulation paths. Moreover, we constrain the 11-norm of each manipulation step below an upper bound to ensure the infinitesimal condition for the line inte- gral and employ the momentum strategy to avoid converging to local solutions. Visualizations on MNIST (Deng, 2012), CIFAR-IO (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009) demon- strate the superiority of SAMP discovering salient pixels. We also conduct quantitative experiments and observe clear improvement compared with other interpretation methods as shown in Figure 1. We highlight our contributions as follows:
lnpt
SAMP (Ches)
Figure 1: we propose SAMP to eliminate ambiguity of attributions by path methods, which can precisely pinpoint important pix- els and produce clear saliency maps. Quanti- tative results show a consistent improvement in Deletion/Insertion metrics.
Concentration Principle for Clear Attributions. We introduce Concentration Principle, which enhances the clarity attributions by prioritizing sparse salient features.
A Model-agnostic Interpreter, SAMP. The proposed interpreter SAMP is able to effi- ciently discovers the near-optimal path from pre-defined set of manipulation paths.
Two Play-and-plug Auxiliary Modules. We design infinitesimal constraint (IC) and the momentum strategy (MS) to ensure rigorousness and optimality.
Consistent Improvement in Explainability. Qualitative and quantitative experiments show SAMP pinpoints salient areas accurately and consistently outperforms counterparts.
2 RELATED WORK
Considerable attempts expect to reveal the mysterious veil of DNNs by different techniques. Ad-hoc methods (Zhang et al., 2018b; Liang et al., 2020; Agarwal et al., 2021; Wan et al., 2020; Wang & Wang, 2021; Shen al., 2021: Barbiero et al., 2022) try to observe or intervene latent variables of DNNs, which rely on specific model types. On the contrary, post-hoe methods (Simonyan et al., 2014; Bach et al., 2015; Zhou et al., 2016; Selvaraju et al., 2017: Lundberg Lee, 2017) ignore con- crete implementations and focus on imitating the outside behavior. According to how attributions generated, we mainly divide post-hoc methods into two categories: perturbation methods (Ribeiro et al., 2016; Fong & Vedaldi, 2017; Petsiuk et al., 2018) and back-propagation methods (Zeiler & Fergus, 2014; Bach et al., 2015; Selvaraju et al., 2017).
Perturbation Methods. An intuitive idea for attributions is to perturb the inputs and observe the output variations. Occlusion method (Zeiler & Fergus, 2014) simply covers up partial areas of input and examines the score change. LIME (Ribeiro et al., 2016) interprets the local space around the prediction by linear regression. Prediction difference analysis (Zintgraf et al., 2017) describes the output variation from probabilistic perspective. Mcaningful perturbation (Fong & Vedaldi, 2017) aims at discovering the deletion regions with compact information, which further extended by RISE (Petsiuk et al., 2018) by the weighted average of multiple random masks. DANCE (Lu al., 2021) introduces a subtle perturbation to input without influence on internal variables. Most pertur- bation methods require multiple iterations, which leads to a heavy computation burden. Moreover, most of these methods lack rigorous axiomatic guarantees.
Back- paek-propaggtionn Methods. Another kind of interpretations recovers signals generates attri- butions by prop-aarpggg information layer by layer. Early research (e.g. Deconvolution (Zeiler & Fergus, 2014) and Guided-BP (Springenberg et al., 2015)) reverses the forward procedure and
2

---

Published as a conference paper at ICLR 2024
recover active signals in input space. Recent attempts generate attributions by propagating gradi- ents (Shrikumar et al., 2016), relevance (Bach et al., 2015), and difference-fromm from-reference (Shriku- mar et al., 2017). Most methods choose gradients the propagation intermediary for ease. Grad- CAM (Selvaraju et al., 2017) and its variants (Chattopadhay et al., 2018) directly interpolate gradi- ents from the top layer input size as the saliency map. SmoothGrad (Smilkov al., 2017) aims at removing noise by averaging multiple gradients at neighbor points. The first-order Taylor decompo- sition (Montavon al., 2018) assigns attributions by linearization with the gradient around the given root point. Since the difference between the data and the root is often not infinitesimal, expansion based on single-point gradient results in large error (Lagrangian remainder), which damages the rigorousness of interpretations. Path methods (Sundararajan et al., 2017; Xu et al., 2020; Kapish- nikov et al., 2021) fix this issue by dividing the integral path into small segments. Game theory guarantees that path methods are the only method satisfying three fundamental axioms (see Propo- sition/ in Friedman et al. (Friedman, 2004)). However, different path choices (c.g., straight line in space (Sundararajan et al., 2017) frequency (Xu et al., 2020) and guided path along flat land- scape (Kapishnikov et al., 2021)) indicate distinct attribution allocations, which makes the meaning of attribution ambiguous. Therefore, we introduce the Concentration Principle and discuss how to obtain a near-optimal path through the proposed SAMP method.
3 METHOD
In this section, we first summarize the canonical path methods (Friedman, 2004). Then we define Concentration Principle in Section 3.2. Subsequently, we propose the Salient Manipulation Path and derive an efficient algorithm under Brownian motion assumption in Section 3.3. Finally, we introduce infinitesimal constraint (IC) for rigorous line integrals and momentum strategy (MS) to
escape from local sub- sub-optimal solutions in Section 3.4.
Concentration Phinciple
Spoce
Scame lanisaaoo
ag
Canecy Step Dinction
10-0000 B Hertal
SAMP
Gradien Direction
16
A-
ag
a1
Sundlne
(a)
(b)
Figure 2: (a) Concentration Principle prioritizes attributions (green point A) with large distance from mean point P. (b) SAMP chooses the directions with max gradient projection (colored in red), and attributions allocated along this path mainly concentrate on salient pixels.
3.1 PRELIMINARY: PATH METHOD
Path methods (Friedman, 2004) for additive cost-sharing methods are derived from Aumann- Shapley (Aumann & Shapley, 1974), which is first introduced machine learning by IG (Sundarara- jan et al., 2017). We define the many-to-one mapping as ʃ X' - IR, where input x" E X has d fea- tures and y" denotes its output. An intuitive idea of interpreting models is to analyze how the output y turns to y when gradually changing baseline x to x². Considering the difference between x and x! not infinitesimal, the naive Taylor decomposition y = y +V (= (=:")((>"" =x²)+++)+++ Rí (x=") suffers from large Lagrangian remainder Rí (x:"). Therefore, it is natural improvement to divide path from 0 TT into multiple segments, which should be small enough. Assuming the model is differentiable, the output variation Ay can be expanded as
f Of({(o)) 0r(p) dp,
Ay = x"
P= 0Y(p) Op
where 7(p) is path function æ 7(p) and 7(0) x", 7(1) क We define each feature's attribution as a; and Ay equals sum of ai (namely completeness (Sundararajan et al., 2017)):
af(r(p)) @m(()) dp, Ay 0m((p) Op
0:
a
(2)
P=
1=1
3

---

Published as a conference paper at ICLR 2024
Game theory research (Friedman, 2004) has proved that the path method is the only interpretation method satisfying three fundamental axioms (i.e., completeness, additivity, and dunmy). However, choices of path function y(p) highly impact the attribution allocation, which hampers the clarity of the interpretations. In this paper, we explore an explicit selection criterion among candidate paths.
3.2 CRITERION AND CANDIDATE SET
Existing path methods lack clarity due to various path choices. In Eq. (2). the attribution is a function of the selected path y a = g(n) given x² z"" However, conventional interpreta- tions often scatter the attributions over all pixels (Kapishnikov et al., 2021; Smilkov et al., 2017) due to unpredictable distractors. To address this, we propose the Concentration Principle, which introduces selection preference for the allocation of attributions. Instead of scattering attributions across all features, we aim to concentrate them centrally on the indispensable features.
Definition (Concentration Principle). A path function Y is said to satisfy Concentration Prin- ciple if the attribution a achieves the max Var(a) = Ei=((a= a)².
Remark, Considering Σi=1 ai is a constant C = Ay. the variance of a could depict the concen- tration degree. For 3-feature case, this principle prefers a (0.7, 0.2, 0.1) to (0.4, 0.3, 0.3). For image input in Figure 1, this principle achieves aesthetic and sparsity. Our method clearly pinpoints important pixels while IG (Sundararajan et al., 2017) spreads attributions over most pixels. We also conduct a counting model example in Appendix A.2 to illustrate the potential challenge.
Under this principle, we explore to formulate a tractable optimization problem. To maintain consis- tency in our formulation, we introduce the start point 2:S and the end point 5. To approximate the line integral in Eq. (I), we use Riemann sum by dividing the path into n segments as:
Ay
V ) dx
(3)
where dæ is the ht segment along the path and x* S + Et da'. Analogous to Eq. (2), calculate each attribution a₂ as a, > Ik=I( (Vf(x" fxx))) (dxx However, it is intractable to directly find the optimal path from the infinite set I' of all path functions. Thus we construct a finite Manipulation Path set r S r, along which we manipulate images by inserting deleting s = d/n pixels per step. The formal definition is as follows:
Definition (Manipulation Path). The th segment dx* of manipulation path 7 I' satisfies
= -
x -
E S2
(dx"), = =10,
(4)
Otherwise
where [Sx| = and all Os consist of non-overlapping partition of all pixel indices which satisfy that Vk A L, S2g 0 = Ø and Uh- S2 (1,...,d}
Remark. F, a finite set and [Fa equals dl/(s!)".
Following Definitions and 2. formulate the optimal path selection problem follows:
d
"" = arg max V (a) =
ā
1
Σ
d
(5)
*ET.
Solving Equation 5 directly is computationally challenging. To overcome this, propose SAMP algorithm, which leverages Brownian motion assumption to efficiently search for the optimal path.
3.3 SALIENT MANIPULATION PATH
The intuition of Eq. (5) is enlarge the distance Dap between (limited in the hyperplane
ES a. C) and the center point (C/d,...,C/O in Figure 2a. We need to introduce prior knowledge to accelerate the search process. Without loss of generality, we set = for ease of derivation. Since the attributions are assigned sequentially, we regard the allocation process as a stochastic process? 2, We define the partial sum Uy as Σi= a; and make the following assumption:
We use lowercase letters to denote random variahles for consistency.
4

---

Published as a conference paper at ICLR 2024
Assumption (Allocation as Brownian motion). We assume the additive process {u,, 2 the Brownian motion and u ~ (0, at) if without any constraint condition.
0} as
We now explain the rationality of the assumption. In the model-agnostic case, we consider the input space to be isotropic. If without any constraint condition, we assume E(a.) = E(dy') = 0 with randomly sampled step dæ' and ag aj for any i # are independent. It is important to note that
do NOT directly assume the a, aj as conditional independent given the condition > ai = C. (0, o)). If we
Then we assume that every ag complies with Gaussian distribution (i.e., subdivide time infinitely, the additive process {u,t t 2 0} tends to a Brownian motion.
Proposition 1. By Brownian motion assumption, the conditional joint distribution P(ā|C)
P(a1,.. 0g Udd = C) is a multivariate Gaussian distribution as:
{
C d
P(ā(C)
(2T) d-1
cxp
a-
à
(6)
E
5-1
where E -= R(d-))xdd and J is all-one matrix.
Remark. See proof in Appendix A.I. Eq. (6) reveals that the conditional distribution is centered at point P Figure 2a. For any + J. Covla; az (ug = C)= -aafd indicates that allocating more to results in less to aj. Moreover, E(uglua- = C) kC/d reveals that randomly selected path tends to produce a linear variation in output. Surprisingly, we observe the curve shapes of IG (Sundararajan et al., 2017), XRAI (Kapishnikov el al., 2019), and Grad-CAM (Selvaraju et al., 2017) in Figure are nearly straight lines, which is consistent with theoretical analysis.
As the dimension of images is always high, we investigate the asymptotic prop- erty of P(ä C) as:
Algorithm 1: The SAMP++ algorithm.
Input: Start point S End point ".; Upper bound ny: Momentum coefficient X. Output: Attribution a; Path segments D.
Proposition 2. Since lim E =I,
conditional covariance Cov(ai, az aylua C) is nearly zero with high dimension Thus we can approximate Eq. (6) as:
Reset k = and set of path segments TD =
2 Initialize x² = xx. a = 0, g (x(x));
3 while x² * x do Increase index 1s by 1;
D exp
P(äIC)
2x
4-11
Update g >g x-1
(2TT)
Σ
Compute a; = (1 x7)) if of +
6
Remark. P(ā(C) = P(ā Pe As
and -0 otherwise;
the last attribution dd tends to if is high enough, the approximation error of P(ä|C) is tolerable.
7
Construct M, = {i i top, {aj{
Compute (dx*), )5 = x. x if E Mc and
O otherwise;
If ] dar > my: dæ. = Td= dx
n
Since image dimension is always high, we regard any two attributions a, 0j as nearly independent by Proposition we can maximize each attribution sepa- rately with negligible error while redueing the computational complexity from facto- rial (dd) to linear (d). Specifically, we choose & pixels with the largest projection of gradient V fax*) onto & We name this greedy selection strategy as Salient Manipulation Path (SAMP)
Move current point x²* x + da
10
11 Update attribution a = a k-1 + gb dx
Therefore,
12 Expand D = DD U(dx*);
13 Return a², DD.
and take insertion direction as an example to formulate SAMP as:
E Mlg
(dax*)
(8)
0,
Otherwise
where M, = {il top, (aj}) (+) means the largest elements) and aj = (VI(x*)) 2/) if of # 24 and -oo otherwise. It is obvious that the path defined above belongs to I,,.
3.4 TOWARDS RIGOROUSNESS AND OPTIMALITY
Two potential issues still remain in our proposed SAMP interpreter. First, if the step size | |dx is too large, the infinitesimal condition may be violated, thereby breaking the completeness axiom in
5

---

Published as a conference paper at ICLR 2024
Eq. (2). Besides, most greedy algorithms tend to get stuck in the local sub-optimal solution. To address these, we propose the infinitesimal constraint and the momentum strategy respectively.
Infinitesimal Constraint (IC). To ensure the completeness axiom, we need to restrict each step size below given bound 7) > 0. Therefore we rectify dæ in Eq. (8) as:
|| clal1111 dx* if dx< 1l1 >1
"1
dæ
clæ".
Otherwise
Note that the above constraint does not affect the convergence of SAMP. According to the definition of manipulation paths, it is easy to know the sum of L.1 norm of all steps is a constant value as Σx=1 || |dx*I 11 = ||I55 111111 1 = C. As long as v) > 0, the constrained SAMP can certainly converge after finite iterations.
Momentum Strategy (MS). Due to the nature of greedy algorithms, SAMP runs the risk of falling into a local optimum. Inspired by the gradient descent with momentum, we incorporate the momen- tum strategy to coast across the flat landscape through the inertia mechanism as follows:
ge = >gg k- + (1 - fa*)
(10)
By substituting dx with dæ and V fa(*)) with g", we formulate SAMP+ in Algorithm 1.
4 EXPERIMENT
In this section, we conduct qualitative and quantitative experiments to demonstrate the superiority of our proposed SAMP method. Due to the wide variety of interpretability methods, they often need to be evaluated from multiple dimensions (Nauta et al., 2022). Our proposed SAMP method belongs to attribution methods. We first perform qualitative experiments to verify the Concentration Prin- ciple claimed above and compare the visualization results with other counterparts in Section 4.2. Subsequently, we employ Deletion/Inserticn metrics (Petsiuk et al., 2018) to examine SAMP quan- titatively and conduct completeness check with the Sensitivity-N metric (Ancona al., 2018) in Section 4.3. Extensive ablation studies demonstrate the effectiveness of each feature in Section 4.4.
4.1 EXPERIMENTAL SETTING
Datasets and Models. We evaluate SAMP on the widely used MNIST (Deng, 2012), CIFAR- 10 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009). For MNIST and CIFAR-10 datasets, we simply build two five-layer CNNs (c.f. Appendix A.3) and train them to convergence using AdamW optimizer (Loshchilov & Hutter, 2017). For ImageNet dataset, we use the pre-trained ResNet-50 model (He et al., 2016) from PyTorch torchvision package (Paszke et al., 2019).
Metrics. Interpretations should faithfully reveal the attention of model decisions. One evaluation for judging attributions is to check whether features with large attribution have a significant effect outputs. Therefore, choose the Deletion//nsertion metries (Petsiuk al., 2018) for quantitative comparison. We delete/insert pixels sequentially in the descending order of attributions, plot the output curve, and calculate the area under the curve (AUC). For Deletion, smaller AUC indicates better interpretability; for insertion, larger AUC is expected. Moreover, we wish to examine the effect of the infinitesimal constraint (IC) on the rigorousness of SAMP. Therefore, we adopt the Sensitivity-N metric (Ancona et al., 2018) by calculating the Pearson correlation between the sum of attributions and the model output for completeness check.
Implementation Details. We compare Deletion/Insertion metrics of SAMP with 12 mainstream interpretation methods. Following the configuration (Petsiuk et al., 2018), we set the baseline point as zero-filled image for Deletion and -aussian-bburred image for Insertion. We randomly select 100 images from each dataset and report the mean and standard deviation of AUCs. Specifically, for MNIST and CIFAR-I0, we set the Gaussian blur kernel size Sy to 11 the variance og to and the step size for calculating metrics sm to 10; for ImageNet, Sy = 31, Jg 5, and 8m = 224 x 8. If without special specifications, we fix the step size s in SAMP as 224 x 16 for ImageNet and 10 for other datasets, the ratio of the infinitesimal upper bound 17 to || as 0.1, and the momentum coefficient À as 0.5. We perform all experiments with PyTorch on one NVIDIA 3090 card.
"The benchmark code will be released together with SAMP.

---

Published as a conference paper at ICLR 2024
096
5%
10%
25%
50%
73%
100%
Atbrixatioe
Soon Carm
Imzage
(a)
(b)
Figure 3: Verification of Concentration Principle. (a) Visualizations of intermediate points and corresponding attributions along the path solved by SAMP. (b) The output score curve from the baseline point to the target image.
Shop
Iropp
Figure 4: Visualizations MNIST, CIFAR- 10, and ImageNet compared with other methods.
4.2 QUALITATIVE VISUALIZATION
4.2.1
VERIFICATION PROPERTIES
We first verify whether SAMP can reach an expected path towards Concentration Principle. For clear visualization, we set the baseline point as zero-filled, and choose the manipulation direction from æ" to x²". Along the path solved by SAMP, the intermediate points and corresponding attributions at different stages are visualized separately, as shown in Figure 3a. We can see that the first 25% of the path has precisely pinpointed the subject animal. Besides, stages along the manipulation path Figure 3b. A rapid rise can be observed at the start, which indicates that SAMP tends to capture the most salient pixels first. At the same time, there is small drop at the end. We ascribe this to background pixels, which interfere with the output score.
plot the output scores at different
4.2.2 VISUALIZATION COMPARISON
We compare the visualization results of SAMP with other mainstream interpretation meth- ods (Ribeiro et al., 2016; Selvaraju et al., 2017; Sundararajan et al., 2017; Smilkov et al., 2017; Shrikumar et al., 2017; Bach et al., 2015; Petsiuk et al., 2018; Kapishnikov et al., 2019; Xu al., 2020; Kapishnikov et al., 2021). After randomly selecting input images on MNIST, CIFAR-10, 10, and ImageNet, we calculate the attribution results each method. We first convert the attributions to grayscale image for visualization and also superimpose the attribution values with the original im- age. Figure 4 shows the comparison of the SAMP method with existing methods. As can seen, the attribution results allocated by our method pinpoint important pixels and localize all pixels on salient objects most completely. Additionally, the attribution results of the SAMP++ approach are broadly similar to SAMP, but the results of SAMP++ are more fine-grained due the infinitesimal constraints (for instance, the subject is more separated from the background).
4.3 QUANTITATIVE ANALYSIS
We conducted quantitative experiments to assess the performance of SAMP, including metrics such as Deletion/Insertion and Sensitivity-N check. In addition, we also carried out evaluations such as uFidelity (Novello al., 2022) and pointing game (Zhang al., 201 8a) in Section A.5.4.

---

Published as a conference paper at ICLR 2024
Table 1: Deletion/Insertion metrics on MNIST, CIFAR-10, and ImageNet.
MNIST
CIFAR-10
ImageNet
Method
Deletion
Insertiont
Deletion
Insertion
Deletion
Insertion
LRP
-0.003 (10013)
0.808 (100100)
-0.257 (10049)
1,452 (::0.37)
0.210 (1:0.13
0,575 (1:0.15)
CAM
0.221 (=0.15)
0.715 (1:0.1)
0.314 (20.31)
0.863 (1.0.33)
0.313 (1:0.129)
0.897 (:10.13)
LIME
0.282 (=0.14)
0.597 (#0.09)
0.479 (=0.29)
0.722 (#0.24)
0.312 (#:0.13)
0.898 (#:0.14)
Grad-CAM
0,221 (=0.15)
0.715 (+0.11)
0.314 (=0.31)
0.863 (#0.23)
0.313 (#:0.13)
0.897 (#0.13)
IG
-0.038 (=0.14)
0.795 (=0.11)
-0,372 (#:0,54)
1.452 (#:0.40)
0.197 (#:0.13)
0.725 (+:0.20)
SmoothGrads
0.003 (=0.13)
0.547 (1:0.11)
0.777 (=555))
0.517 (d:0.28)
0,300 (::0.11)
0.605 (1:0.17)
DeepLIFT
-0.025 (10.14)
0.791 (100.1)
-0.300 (10.51)
1.443 (1.0388
0.216 (:10.12)
0.688 (1:0.18)
RISE
0.059 (=0.11)
0.651 (20.12)
0.149 (=0.8)
0.904 (#.0.77)
0.282 (1:0.13)
0.849 (10.15)
XRAI
0.120 (=0.12)
0.754 (#0.10)
0.248 (=0.33
0.910 (#0.21)
0.346 (#:0.16)
0.865 (#0014))
Blur IG
0.021 (=0.02)
0,804 (#0.17)
-0.107
(:0039)
1.407 (#0047)
0,261 (=:0.14)
0,712 (2:0.22
Guided IG
-0.041 (+:0.144
0.762 (+0010)
-0.276 (#:0.47)
1.209 (#:0.35)
0.167 (+:0.13)
0.699 (#:0221)
SAMP (ours)
-0.093 (::0.14)
1.074 (#:0.18)
-0.733
(.:),)))
1.458 (::0.40)
0.154 (::0.12)
0.984 (2:0.20)
SAMP++ (ours)
-0.137 (1:0151)
1.050 (1:0.18)
-0.899 (::0.72)
1.514 (2:0.43)
0.145 (2:0.12)
1.116 (1:0.24)
Seediviity N on MNIST
Seaakiity CIFAR 10
Dekciin Insp
Traerioos TrngeNe
2
Timc
Gundaiion
Thaz
0.57
6:15
19
6:12
Bound" (JAN) (n)
Borrd ((ANN).1
5.20
Monarram
0,00
(a)
(b)
(a)
(b)
Figure 5: Sensitivity-N check for IC.
Figure 6: Impact of momentum coefficient A.
4.3.1 DELETION/INSERTION COMPARISON
To precisely compare the performance, calculate the Deletion/Insertion metries (Petsiuk et al., 2018). We randomly sampled 100 images and report the mean and standard deviation of the AUCs (see Table 1), where "SAMP" represents the original algorithm described in Eq. (8) and "SAMP+4" denotes Algorithm with the infinitesimal constraint (IC) and momentum strategy (MS). Our method consistently outperforms all other methods on three datasets. We ascribe this to Concen- tration Principle that facilitates our method to perform clear saliency rankings. In addition, the improved version significantly improves the original one in most cases. We believe that the momen- tum strategy plays an essential role in prompting the algorithm to break free from the local point (c.f. Section 4.4 for ablation studies.).
4.3.2 SENSITIVITY-N CHECK
In this part, we show the importance of the infinitesimal constraint (IC) on rigorousness (or com- pleteness (Sundararajan et al., 2017)). Sensitivity-N (Ancona et al., 2018) checks the completeness by calculating the Pearson correlation of Σ, ay and Ay. We gradually increase / = || |1/n (i.c., decrease the upper bound 7) in Eq. (9)) and draw the curve of the correlation w.r.t. 3 (see Figure 5). With the decrease of 1). the correlation increases significantly. This is because IC limits each step to be infinitesimal, which ensures that Lagrangian remainder tends to O, thereby enhancing rigorous- ness of Eq. (3). Interestingly, Figure 5a shows that with the further decrease of 1). the numerical error becomes the main error source, and the correlation no longer rises; because 17 is not small enough at the start of Figure 5b, most steps not cropped, thereby leading to flat correlation curve.
4.4 ABLATION STUDY
4.4.1 INFLUENCE OF IC AND MS
We perform ablation studies on the infinitesimal constraint (IC) and momentum strategy (MS), as shown in Table 2. As we can see, the improvement of SAMP in Deletion/Insertion metrics mainly comes from MS. According to Figure 6, SAMP achieves the largest improvement when À zs 0.3. Table shows that IC has no significant impact on Deletion/Insertion metrics, which can be at- tributed to the fact that IC is primarily designed maintain rigor and lacks direct connection with
8

---

Published as a conference paper at ICLR 2024
Table 2: Ablation study on IC and MS.
Table 3: Influence of upper bound 7)
ImageNet
Bound -1
ImageNet
Setting
Deletion
Insertion
(l (1/n)
Deletion
Insertion
SAMP
0.154 (20.118)
0.984 (::0.155)
1/10
0.159 (2:01121)
1.056 (1:0.155)
+MS
0.144 (=0.115)
1.088 (=20.2
1/50
0.218 (20.133)
1.130 (#:0.168)
+IC
0.159 (#0.121)
1.056 (::0.185)
1/100
0.249 (40.142)
1.031 (#0.155)
+MS/IC
0.145 (20.116)
1.116 (::0.241)
1/200
0.279 (#00147)
0.939 (#0.159)
enhancing metrics. In addition, smaller eta (e.g., 7 1/200) leads to finer ther-grained visualization (see Figure 8), which is due to the shortened step size that focuses more on details.
Image
B+B
B+W
B+U
B+G
4.4.2 CHOICE BASELINE POINTS
We wonder whether different baseline choices affect the performance of SAMP. Therefore, we set four different sets of baseline points with ? == ||=l|1/500 550 "B" means padding with zero, "W" means padding with one, "U" denotes uni- formly random initialization, and "G" denotes Gaussian random initialization. In the symbol "X+Y", represents the deletion direction, and Y represents the insertion direction. Figure 7 shows that the impact of different baselines on the explanations not significant compared to different methods in Figure 4. Specifically, dif- ferent baselines have little impact the contour information, but they do significantly affect the overall intensity (e.g., brightness), which leads to visual differences.
Figure 7: Visualization of different baseline.
SAMP (Ours)
Image
1G
GuidedIG
1/50
3452
12000
Figure 8: Results with different upper bound 77.
4.4.3 CHOICE PATHS
The choice of path also has a certain influence on Deletion/Insertion (as shown in Table 4). We discover that only using the path x" + x achieves the best Deletion; only using x æ reaches the highest Insertion. We actually use both directions the same time, and sum attributions generated by two directions to ob- tain a trade-off between two metrics.
Table 4: Influence of path choices.
Path
ImageNet
to
to
Deletion
Insertion
0.108 (#0.107)
0.659 (1.0.711)
V
0.199 (10.135)
1.330 (:20230)
V
0.159 (10.121)
1.056 (=10.185)
5 CONCLUSION
To obtain user trust, interpretations should possess rigorousness and clarity. Even though path method (Sundararajan et al., 2017) identifies fundamental axioms for rigorousness, attributions re- main ambiguous due indeterminate path choices. In this paper, we first define Concentration Principle. Subsequently, we propose Salient Manipulation Path (SAMP), which is a fast and greedy interpreter for solving the approximate optimal path efficiently, To enhance the rigorousness and optimality of SAMP, we propose the infinitesimal constraint (IC) and momentum strategy (MS) respectively. Visualization experiments show that the attribution generated by our method aceu- rately discovers significant pixels and completely pinpoints all pixels of salient objects. Quantitative experiments demonstrate that our method significantly outperforms the current mainstream interpre- tation methods. Moreover, qualitative experiments also reveal that SAMP can obtain higher-quality semantic segmentations by visualizing the attribution values only employing class-level annotations. Investigating the utility of saliency-based explanations in annotation-limi supervised object recognition) and other promising domains (with focus on NLP and medical image analysis) represents exciting direction for further study.
tasks (such weakly-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported in part by the National Key Research and Development Program of China under Grant 20222D0160102, and in part by the National Natural Science Foundation of China under Grant 62125603, Grant 62321005, and Grant 62336004.
REFERENCES
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang. Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. In NeurlPS, pp. 4699-4711, 2021.
Marco Ancona, Enea Ceolini, Cengiz. Öztireli, and Markus Gross. Towards better understanding of gradient- attribution methods for deep neural networks. ICLR, 2018.
Robert J Aumann and Lloyd Shapley. Values of non-atomic games. Princeton University Press, 1974.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(1)xe0130400, 2015.
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-basee logic explanations of neural networks. In AAAI, pp. 6046-6054, 2022.
Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Sal- vatore Rinzivillo. Benchmarking and survey explanation methods for black box models. arXiv, abs/2102.13076 2021.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, pp. 839-847, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pp. 248-255, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In ICCV, 2017.
Eric Friedman. Paths and consistency in additive cost sharing. International Journal of Game Theory, 32(4):501-518, 2004.
Leilani Н Gilpin, David Bau, Ben Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In DSAA, pp. 80 80-89, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Andrei Kapishnikov, Tolga Bolukbasi. Fernanda Viégas, and Michael Terry, Xrai: Better attributions through regions. In ICCV, pp. 4948-4957, 2019.
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In CVPR, pp. 5050-5058, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2024
Haoyu Liang, Zhihao Ouyang. Yuyuan Zeng, Hang Su, Zihao He, Sau-Tao Xia, Jun Zhu, and Bo Zhang. Training interpretable convolutional neural networks by differentiating class specifcc filters. In ECCV, pp. 622-638, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv, awt//a/1..///////. 2017.
Yang Young Lu, Wenbo Guo, Xinyu Xing, and William Stafford Noble. Dance: Enhancing saliency maps using decoys. In ICML, pp. 7124-7133, 2021.
Scott M Lundberg and Su-In Lee. unified approach interpreting model predictions. In NeurIPS, volume 30, 2017.
Grégoire Montavon, Wojciech Samek, and Klaus-Robers Müller. Methods for interpreting and un- derstanding deep neural networks. Digital signal processing, 73:1-15, 2018.
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review evaluating explainable ai. arXiv, abs/2201 1.00 164, 2022.
Paul Novello, Thomas Fel, and David Vigouroux. Making sense of dependence: Efficient black-box explanations using dependence measure. Advances Neural Information Processing Systems, 35:4344-43575 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurIPS, 32, 2019.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In KDD, pp. 1135-1144, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, pp. 118-626,22 2017.
Wen Shen, Zhihua Wei, Shikun Huang, Binbin Zhang, Jiaqi Fan, Ping Zhao, and Quanshi Zhang. Interpretable compositional convolutional neural networks. arXiv, abb/2107.04474. 2021.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv,, abs/1605.01713 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, pp. 3145-3153, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR (Workshop). 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv, abs/ (tts//706008225. 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (Workshop). 2015.
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In ICML, pp. 3319-3328, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
11

---

Published as a conference paper at ICLR 2024
Alvin Wan, Lisa Dunlap. Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: neural-backen decision trees. arXiv, abs/2004.00221, 2020.
Yipei Wang and Xiaoqian Wang. Self-interpretable model with transformation equivariant interpre- tation. In NeurlPS, pp. 2359-2372, 2021.
Shawn Xu, Subhashini Venugopalan, and Mukund Sundararajan. Attribution in scale and space. In CVPR, pp. 9680-9689, 2020.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pp. 818 833, 2014.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 (10):1844-102,, 2018a.
Quan-shi Zhang and Song-Chun Zhu, Visual interpretability for deep learning: survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27-39, 2018.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In CVPR, pp. 8827-8836, 2018b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR. pp. 2921-2929, 2016.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv, abs/1 702.7422.4422 04595, 2017.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 THEORETICAL SUPPLEMENTARY
A.1.1 PROOF OF PROPOSITION
Proof. By assumption, {u,, 2 0} is a Brownian motion and i2 (O ot):
1
ut
P(u))
exp
V2Tor
2at
To treat attributions equally, consider the subset (wt, NJ with uniform time intervals (un
0). According to the Markov property, we can get A E N, us uk-1 ~ (V(, Thus we can formulate the joint probability distribution as follows:
P(u1, U2 ". .2a) ua)
(uux M4A 1 m1)
k=1
P(ux
Ux 1
d
(Uk
2k 1)
exp
12To
20
k=1
(uk
U1 1)
exp
(11)
(2TT)
20
k=
As vk E (1, .. .d,,az a2 ,d],,
Uk Ug-1, I. we have
(a1,
02
ad) [u1, U2
, Ud)
0
0
07
U1
-1
1
0
0
U2
1
0
1t
:
Thus P(ay, (2, az, ... (a) P(u1, U2,
ua)/(J]
P(u1, U2." ...
,.,,," ug)) dividing
P(a a..... ad) by P(ua), we get the conditional distribution P(ā[C) = P(a., ad-1 C' as follows:
P(an 02
as-1lus C)
=P(a1, 02
aalud = C)
=P(ai, 02
aa)) P(ua C)
Vd
11
exp
(2mt) $
2nd
Vd
0
(C
ΣE= d-1
d-1 Σ
az
exp
(2TT)
2
20
2x
k=1
Vd
C2
(2Tо) 1221
exp
20
(C -1aa)
d
Vd
-1
(d-
1)0
(2Tо) =
exp
20
ā
āT 11
201a
(12)
13

---

Published as a conference paper at ICLR 2024
Let - E R(d-1)x(d-1), We can easily get that
|
and Σ- = (I+11²))
Thus we can simplify Eq. (12) as:
P(a1.00 02
0d-
Ud C)
vd
C
11ーー111 }
C2
a Mー 1--1ä à -
1-10 ā
(2TT)
1
exp
Vd
C
(2TT)
1
exp
2
d
1)
-ー
1
1
C
2
(2T)
1
exp
2
d
1
(13)
Therefore, a/ud ~ ( Σ) where E = gdl 117).
A.1.2 SUPPLEMENTARY OF PROPOSITION 2
(1) How to derive Eq. (7): We can reformulate Eq. (13):
P(ā|C) = P(a1, 02 ..... ddd U1 = CC)
vd
1
C
C d
(2TT)
exp
20
à
d
1
à
1
C
C
à
1
11
à
d
1
20
d
(2 11
Vd
exp
D2
2a
ap
2a
1
C
Σ
d-1
kml
Vd
D2
+ 02
ap
(2mT) d=1 T
exp
2a
=P(ā(C)e (20()
(14)
where P(ā(C) is the estimation of P() By Proposition I we know that Valua ~ N C o ) Although the expectation of aa gradually approaches zero with the increase of d, the variance gradu- ally approaches a. Thus the approximation eтror cannot be completely zero. Fortunately, the experi- mental results show that the attribution allocation is not completely homogeneous. Figure 3b reveals that the last allocated ag is usually approximately zero due to the saturation region. Therefore, the approximation error in Proposition 2 is generally acceptable.
Note that lim Σ = al actually means the element-by-element limit rather than the limit in terms of the matrix norm. Given E(d) that
(2) Element-wise limit:
_ [dl 11²), we can easily get
0 i 0, 1 ن
Vi,j E N, lim Σij =
(15)
d=+00
It is hard to find norm ||. such that limd-+00 -E- oI = 0. For example, ||- oI m. (d 1) max.y EEg - oوnن is always 1, not tending to 0. Therefore, Proposition 2 has certain limitations. The continuous mapping such as the determinant (det(-)) and the limit operation (lima-so0 (.)) are not commutative.
A.2 POTENTIAL CHIALLENGE OF CONCENTRATION PRINCIPLE
As a path method, SAMP generates attributions with three axioms (linearity, dummy, and effi- ciency). These axioms ensure that SAMP does not blindly pursue high variance path and only ignores ambiguous elements rather than essential ones. We visualize a blue-pixel counting model in Fig.9, and SAMP achieves consistent attributions.
14

---

Published as a conference paper at ICLR 2024
Imput 1
Attribution
Imput 2
Attribution
Counting Modkl:
f(x)
D1 (ofaeee)) oforace (esesa)
Figure 9: Counting model examples for Concentration Principle.
A.3 MODEL ARCHITECTURE
The architecture of 5-layer CNNs adopted on the MNIST (Deng, 2012) and CIFAR-10 (Krizhevsky et al., 2009) datasets is shown in Table 5. We employ the most common ReLU activation function and an adaptive max-pooling layer to reduce the spatial dimension to
Table 5: CNNs for MNIST and CIFAR-10.
5-layer CNNs
Layer
for MNIST
for CIFAR-0 10 Input 32 x 32
Input
x 28 5 x 5, ReLU
conv1
5 5 ReLU
conv2
2 x 2, 24, stride 2
2 2, 64, stride 2
conv3
x 4, 288 ReLU
x 4, ReLU
conv4
x 2, 864, stride
x 2, 1536, stride 2
conv5
x 3, 2592 +
ReLU
x 3, 4608
ReLU
pool
adaptive max pool
A.4. EVALUATION DETAIL.
A.4.1 EVALUATION PROTOCOL.
We employ classification models for Deletion/Insertion evaluation. Conventional classification mod- els output the final scores with soft-max layer
U4
Σ 695
However, is coupled with the output score y (j * i) of other classes. Therefore, we directly utilize the output score y4 before the soft-max layer to compute metrics.
Given any input data x, we employ the interpretation method to obtain the attribution a. We first rank the attribution in descending order. Then we select s. pixels to delete insert according to the sorting, so as to obtain a series of output scores as:
(0" k Vo..v" .. ...... yk k- y²²
v.
For Deletion
y
For Insertion
where y and n² are path-independent values. We can normalize the series of scores
/,""
and the normalized area under the curve is computed as Sanc K+1 Σ ÿ. Note that Note that Sano is not necessarily constrained between 0 and 1. For Deletion, the intermediate y* may be greater than y²"; for insertion, the intermediate y² may be less than
A.4.2 HYPERPARAMETER CHOICE
setting: Rs 0.3 Section 4.4.1 is only optimal on ImageNet, so we set = 0.5 for all three datasets in Section 4.1.
15

---

Published as a conference paper at ICLR 2024
Score: 0.9940
score: 0 2096
0.9959
0,4050
score: 0.993]
score: 0.5562
Karkiet
score: 0.9856
Score: 0 5.5833
0.6255
Score: 0.9997
score: 0. 0.7360
0.9996
0.7404
score:
score: 1.0000
Score: 0.7677
(
(b)
Figure 10: (a) Visualization results of images with high classification confidence. (b) Images with low classification confidence.
16

---

Published as a conference paper at ICLR 2024
Image
B+B
B+W
B+U
B+G
Figure 11: Additional visualization of different baseline.
Table 6: Influence of baseline choices.
Baseline
ImageNet
x:
x
x: n:
Deletion
Insertion
B
B
0.254 (100.377)
0.779 (1001188)
B
W
0.272 (1001177)
0.743 (1.0.166)
BB
U
0.257 (#:0.130)
0.681 (#:0119)
B
G
0.218 (2-0.133)
1.130 (#-0.168
setting: Since smaller 7 needs more computation, we set a balanced 17 = 0.1 Section 4.1. Besides. set 0.02 for visual comparison because Figure 8 shows that smaller 7) produces more subtle visualizations.
A.4.3 REIMPLEMENTED BENCHMARK FOR FAIRNESS
Results of different works (Bodria et al., 2021; Petsiuk et al., 2018) are quite distinct. To compare fairly, we reimplement all methods and fix some bugs in the public code. We remove final softmax layer to avoid interaction of classes and N 100 can already produce stable results. We will also release benchmark code after the paper is accepted.
A.5 ADDITIONAL EXPERIMENT
A.5.1 CONSISTENCY ON OUTPUTS AND ATTRIBUTIONS
The ultimate goal of attributions is to monitor the potential mistakes of models and data. We nor- malize the final scores through a soft-max layer and visualize the attributions of images with high confidence and low confidence shown in Figure 10a and Figure 10b respectively. It can be
17

---

Published as a conference paper at ICLR 2024
that for images with high confidence, the corresponding attributions concentrated the salient objects, while for images with low confidence, the corresponding attributions are scattered on both the objects and the background.
A.5.2 ADDITIONAL COMPARISON WITH COUNTERPARTS
To further illustrate the significance of the improvement of the SAMP method compared with other interpretation methods, we show more comparison examples, as shown in Figure and Figure 17. We can observe clear improvement in the granularity of attributions.
A.5.3 ADDITIONAL. ABLATION STUDY ON BASELINE CHOICES
We show more visualization here to further illustrate that the choice of baselines has marginal impact Figure 11. We also test the Deletion/Insertion metrics as Table 6. Different from the visualization results, the choice of baselines will significantly affect metries. Specifically, for the Deletion/Insertion metrics, we refer to the open-source code* from RISE (Petsiuk et al., 2018). In calculating these metrics, RISE uses black image (i.e., "B") as the baseline for the Deletion metric and a Gaussian blurred image (i.e.. "G") as the baseline for the Insertion metric. Therefore, based on the results in Table 6, if we also adopt the "B+G" baseline generate explanations, which aligns with the metric calculation setup in RISE, the Deletion/Insertion metric would be best.
A.5.4 ADDITIONAL. VISUALIZATION ON DIFFERENT BOUND 7
SAMP (Ours)
Image
IG
GuidedIG
1,50
1r303
Figure 12: Additional visualization with different upper bound r7.
Here we show more visualization results with different upper bound 77, and compare them with the mainstream IG (Sundararajan et al., 2017) and GuidedIG (Kapishnikov et al., 2021), as shown in
/https://gtt www.cclcccccccooecc
18

---

Published as a conference paper at ICLR 2024
Input
ApproShapley SHAP
SHAP
SAMPH
S
S
2
Figure 13: Visual comparison with Shapley values.
Figure 12. We that SAMP locate salient pixels more clearly, and as r decreases, the fineness of localization is higher.
A.5.5 OTHER EVALUATIONS
Why Deletion/Insertion: Explainabilityy evaluations are still an open problem (Bodria et al., 2021). It is controversial to compare explanations against ground truth since there is no guaran- tee that models use the same features as humans. Therefore, we choose to employ self-comparison evaluations (e.g., Deletion//Insrtionn) even though they have potential information leakage.
FFidelity and Pointing game results: Despite the controversy, we conduct pointing game exper- iments for the test set of PASCAL VOC2007 shown in Table 7. We accurate method. RISE has an inherent advantage in pointing game because the block-level methods like RISE (with 7x7 resolution) are not sensitive to outliers. Compared with other path methods (IG, BlurIG and GuidedIG), SAMP still has significant improvement. We also evaluate the μu uFFdelity on MNIST, We can see that SAMP++ achieves competitive performance compared with other methods.
that RISE is the most
Table 7: pFidelity on MNIST and pointing game on VOC2007.
MWP
SG
RISE IG
BlurIG GuidedIG SAMP++
y
0.327
0.286
0.453
0.292
0.453
0.461
Pointing Acc.1
76.90
87.33
76.31
76.19
72.82
80.75
Time(s) (om MNIST)
0.204 0.191
|02007
0.205
0.344
0.235
A.5.6 COMPARISON WITH SHAPLEY VALUES
Table 8: Comparison with Shapley-based methods on MNIST.
ApproSSapley Deep SHAP (Lundberg & Lee, 2017)
SAMP+
Deletion,. / Insertion 0.014 / 0.808
0.029 / 0.955
-0.137 1.050
Time(s) (on MINIST
14.390
0.748
0.235
compare SAMP with two Shapley- based methods. ApproShapley computes the Shapley value by the Monte Carlo method and Deep SHAP (Lundberg & Lee, 2017) is a high-speed approximation. Tab.8 shows that SAMP++ achieves better Deletion/Insertion with less computation burden and Fig.13 13 visualizes the attributions.
A.5.7 VISUALIZATION ON MULTI-LABEL. CLASSIFICATION
To comprehensively explore the application for more complex scenarios, we constructed 2x2 image combinations using images from the MNIST dataset, where each image contained multiple digits.
19

---

Published as a conference paper at ICLR 2024
$
2
7
S
8
7 S
5
00
S
S
Figure 14: Visualizations on multi-label classification.
Then, we utilized CNNs to classify these images and applied the SAMP method for visualizing the attributions of each classification. The visualization results in Figure 14 demonstrate that SAMP can accurately pinpoint specific data related certain categories (for example, in synthetic images containing 0, 5, and 7, when attributing to the 0 class, SAMP only attributes to the 0 digit). This suggests that the SAMP method can be effectively applied multi-class classification problems.
A.5.8 VISUALIZATION ON INE-GRAINED DATASET: CUB-200-2011
The CUB-200-2011 (Wah et al., 2011) dataset is a widely used fine -rained bird dataset consisting of 200 types of birds. We employ a ResNet50 network pre-trained on the ImageNet dataset to fine-tune on the CUB-200- 200-2011 dataset until training convergence, achieving an accuracy rate above 98% on the training set. To evaluate the impact of the SAMP method fine-grained data, we randomly select images from various categories in the CUB-200-2011 dataset and compute attributions on these images with their corresponding category labels, generating total of 25 visualized images arranged in a 5x5 grid, as shown in Figure 15. In this figure, the diagonal is outlined in green borders, representing the attribution result for the image's corresponding category, while non- diagonal elements are outlined in red borders, denoting attribution results for other categories. It evident that the attribution results for the image's 's corresponding category are significantly more prominent compared to those for other categories. However, we acknowledge the presence of some intensity in the attribution results of other categories, suggesting that attributions for fine-grained datasets have not yet fully decoupled from the category, which is also futher direction aim to further investigate.
A.6 LIMITATIONS AND FUTURE DIRECTIONS
In this section, we will briefly outline the limitations of the SAMP approach and suggest potential future research directions.
Theoretically, our paper presents the concentration principle to address the issue of am- biguous path selection in traditional path methods. However, directly solving the con- centration principle involves an exponentially complex combinatorial optimization prob- lem. To address this, we proposed the SAMP algorithm based on the Brownian motion assumption, which reduces the exponential complexity linear complexity and can obtain a near-optimal solution to the original problem. Nonetheless, this paper does not pro- vide thorough investigation of the optimality of the SAMP algorithm in solving the
20

---

Published as a conference paper at ICLR 2024
Turmet
Turpen
Thrnett
Throet
Thrnet abet 4G
Tarpe-Labet 26
Thrger Latar-Labat
Tarper-Labtt
Tarpet-Labet
Tarpet-Labet 46
Tarpo-Labst
Tarpor Lates
TarpssLaball
TarpooLubbst
Taspoiiuaost 40
Labol
Toogs Lobol 16
Toneo Lobol 52
Tures Lob:l 62
Tupest Lobel 10
Lobst 46
Figure 15: Visualizations on CUB-200-2011.
problem, and intend to conduct rigorous theoretical analysis of the optimality of the SAMP algorithm in the future and explore more efficient and accurate algorithms exist for solving the problem.
Experimentally, the SAMP algorithm reduces the original exponentially complex problem to linear complexity and maintains comparability with mainstream path methods (e.g., IG, BlurIG, GuidedIG). However, the SAMP algorithm still requires multiple forward and backward passes for each sample, leading to a certain computational overhead in practical applications, and its caleulation speed is slower than CAM and Grad-CAM. In the future, we plan to explore if there are more efficient approximation algorithms that can improve exccution efficiency while maintaining explanation accuracy.
21

---

Published as a conference paper at ICLR 2024
LIME
Gradi CAM
JG
Smoodhrrdds
DeepLiPT
XRAII
BiariG
GundedIG
SAMP (Ours)
Figure 16: Additional visualization comparison with counterparts.
22

---

Published as a conference paper at ICLR 2024
Imput
LIME
Gradi CAM
IG
Smoothlrdds
DeepLiFT
RISE
XRAI
BluniC
GuidediG
SAMP (Ourl
Figure 17: Additional visualization comparison with counterparts.
23


Reviews:
[{"review_id": "ZocJJw3Qld", "reviewer": "Reviewer_4ZSP", "summary": "This paper introduces SAMP (Salient Manipulation Path), a novel model-agnostic method for attribution explanation that selects a near-optimal path guided by the proposed Concentration Principle. The authors further enhance SAMP with two additional modules, IC and MS, creating an extended version SAMP++. Performance evaluations of SAMP/SAMP++ are conducted on MNIST, CIFAR-10, and ImageNet datasets using Deletion/Insertion metrics. It outperforms other baseline methods. Additionally, the qualitative results of SAMP and SAMP++ show improvements in visualization quality, offering more precise object location and detailed pixel highlighting. These results demonstrate the advantages of the proposed method.", "questions": "How do we understand the impacts of baselines on two paths? To be concrete, in Appendix A.5.3, “the setting of Deletion/Insertion is consistent with ‘B+G’”. How are they related?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper introduces a novel technique for determining the optimal path for distinct attribution allocations, using Brownian motion as the guiding mechanism. To satisfy the completeness axiom, the Infinitesimal Constraint is employed, while the Momentum Strategy is utilized to avoid local optima. Overall, the algorithm is well-motivated and clearly explained.\n\n2.\tThe evaluation is thorough, and the results highlight the fidelity advantages of both SAMP and SAMP++. Importantly, the efficiency of SAMP and SAMP++ remains relatively high when compared to other integrated gradient methods.\n\n3.\tThe paper is well-organized and clearly written, contributing to its overall quality.", "weaknesses": "1.\tSome parts need more clarification: \n\n    a.\tFigure 8 states that \"as η decreases, the attribution visualization becomes fine-grained.\" However, at η=10, the visualization appears overly dark, making it challenging to understand. Similarly, Table 3 suggests that “since the main role of IC is to ensure rigorousness, its effect in improving performance is not significant”. Have the authors conducted a statistical test to substantiate this claim?\n\n    b.\tIn Section 4.4.2, more details should be added, for instance, “B+W” refers to Bin the path x^T to x^0 and W refers to x^0 to x^T. In Figure 7, the authors claim that “the visual impact of different baselines is not significant”. But Figure 7 demonstrates quite different qualitative results. For instance, B+W is in general less salient than others. Moreover, quantitative results in Table 6 in the appendix also show significant differences between baselines. Could the authors elaborate on their reasoning?\n\n    c.\tDoes SAMP++ provide similar qualitative results as SAMP? Why not present qualitative results of SAMP++ as it has better performance than SAMP? \n\n\n2.\tThe current results support the efficacy of the proposed method, but the paper would benefit from more impactful and insightful analyses. For example, it would be informative to investigate whether SAMP++ can help humans understand models, particularly on complex tasks involving fine-grained data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffgIpfJ7lM", "reviewer": "Reviewer_tbYK", "summary": "This paper proposes a new attribution method. For the baseline of integrated gradient, the author proposed the Concentration Principle and also proposed the SAMP method, which can target the black box model. Advantages over partial IG methods on the challenging Imagenet dataset.", "questions": "Please see weaknesses, if the author can convincingly address my concerns, I'm open to raise my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A new path attribution principle is proposed for path methods (IG class methods).\n- This method can target agnostic models.\n- On the challenging imagenet dataset, this method has better attribution effects compared with traditional attribution algorithms and IG-type attribution algorithms.\n- This method has theoretical guarantee.", "weaknesses": "- Should \"Attributions\" be changed to \"Attribution\" in the title of this paper?\n- It is not recommended to cite references in the abstract.\n- Since the proposed method is attributed to the pixel level, I hope the author can discuss the advantages of this method compared to the method attributed to features, because it seems that the explanation attributed to the region level is more convenient for human understanding.\n- In addition to natural image datasets such as Imagenet, I suggest that the author can try to apply this method to natural language interpretation or medical images. Because a single word in natural language has strong semantics and is easy for humans to understand, it is most critical for medical images to have small areas. This can better reflect the practical application value of this article.\n- The author mentions the case of model agnosticism. If we consider that the internal gradient of the model is accessible, that is, a white-box model, can the method proposed in this article achieve better attribution results? Or why doesn't it work with white box? I hope the author can discuss relevant content.\n- I hope the author can discuss the limitations of this article and future outlook.\n- In Table 1, why does the value of deletion appear negative? Did the authors fail to normalize the network's classification output?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NROOIu7P3Z", "reviewer": "Reviewer_7nkz", "summary": "This work studies the efficacy of optimal path to achieve clarity of attributions. A Concentration Principle is defined to guides the interpreter to identify essential features and allocate attributions to them.", "questions": "- The proposed approach is evaluated on image classification dataset, where there common has a dominant object in the image. For some of the example shown, the attributions could cover several objects (e.g., socks and heaters are both attributed; the dog, tiles line, and shower head are attributed). Does it infer that all the attributed regions/pixels contribute to the decision? Has the author applied this method on fine-grain image classification dataset?\n- Brownian motion is the erratic motion of particles suspended in a medium due to continuous collisions. Assumption 1 assume the additive process as the Brownian motion. Please explain why is this valid and the intuition behind. \n- The manipulation path is pre-defined. How is the path defined? Could these paths sufficiently cover most scenarios for the search of near-optimal path? \n- What is the full name for IG?\n\nMinor comments:\n- Some of the figure (or table) is too far from the text that discussed its result, hence not friendly to cross reference the figure and the discussion.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ This work study how optimal path can improve the clarity for attributions for interpretations of DNNs. A Concentration Principle is proposed, which aim to concentrate the attributions on the dispensable features.\n+ A Salient Manipulation Path approach is proposed to search the near-optimal path (as an approximation of Equation 5). Each manipulation step is constrained below an upper bound with l1-norm, and momentum strategy is proposed to avoid convergence to local solution.\n+ This paper shows quantitative and qualitative results to demonstrate the effectiveness of the proposed method. \n+ The paper is written clearly and easy to follow.", "weaknesses": "- There are some minor typos and latex error for the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)