AttEXplore: Attribution for Explanation with model parameters eXploration
Authors: Zhiyu Zhu
Huaming Chen
Jiayu Zhang
Xinyi Wang
Zhibo Jin
Jason Xue
Flora D. Salim
Venue: ICLR 2024 poster (2024)
Keywords: attribution
transferability
adversarial attack
Primary area: visualization or interpretation of learned representations

Abstract: Due to the real-world noise and human-added perturbations, attaining the trustworthiness of deep neural networks (DNNs) is a challenging task. Therefore, it becomes essential to offer explanations for the decisions made by these non-linear and complex parameterized models. Attribution methods are promising for this goal, yet its performance can be further improved. In this paper, for the first time, we present that the decision boundary exploration approaches of attribution are consistent with the process for transferable adversarial attacks. Specifically, the transferable adversarial attacks craft general adversarial samples from the source model, which is consistent with the generation of adversarial samples that can cross multiple decision boundaries in attribution. Utilizing this consistency, we introduce a novel attribution method via model parameter exploration. Furthermore, inspired by the capability of frequency exploration to investigate the model parameters, we provide enhanced explainability for DNNs by manipulating the input features based on frequency information to explore the decision boundaries of different models. Large-scale experiments demonstrate that our \textbf{A}ttribution method for \textbf{E}xplanation with model parameter e\textbf{X}ploration (AttEXplore) outperforms other state-of-the-art interpretability methods. Moreover, by employing other transferable attack techniques, AttEXplore can explore potential variations in attribution outcomes. Our code is available at: https://github.com/LMBTough/ATTEXPLORE.

Published as a conference paper at ICLR 2024
ATTEXPLORE: ATTRIBUTION FOR EXPLANATION WITH MODEL PARAMETERS EXPLORATION
Zhiyu Zhu', Huaming Chen' Jiayu Zhang', Xinyi Wang", Zhibo Jin', Jason Xue* & Flora D. Salim
University of Sydney', SuZhou Surrhu Universiti Malaya', CSIRO's Data61*, University of New South Wales*
ABSTRACT
Due to the real-world noise and human-added perturbations, attaining the trust- worthiness of deep neural networks (DNNs) a challenging task. Therefore, it becomes essential to offer explanations for the decisions made by these non- linear and complex parameterized models. Attribution methods are promising be further improved. In this paper, for
for this goal, yet its performance the first time, we present that the decision boundary exploration approaches of attribution are consistent with the process for transferable adversarial attacks. Specifically, the transferable adversarial attacks craft general adversarial sam- ples from the source model, which is consistent with the generation of adver- sarial samples that can cross multiple decision boundaries in attribution. Uti- lizing this consistency, we introduce novel attribution method via model pa- rameter exploration. Furthermore, inspired by the capability of frequency ex- ploration to investigate the model parameters, we provide enhanced explainabil- ity for DNNs by manipulating the input features based on frequency information to explore the decision boundaries of different models. Large-scale experiments demonstrate that our Attribution method for Explanation with model parameter eXploration (AttEXplore) outperforms other state-of-tth-aae interpretability meth- ods. Moreover, by employing other transferable attack techniques, AttEXplore can explore potential variations in attribution outcomes. Our code is available at: https///ittubbcom/mMMM//TTTggo
INTRODUCTION
Nowadays, DNNs have achieved state-of-the-art performance in various application scenarios such as medical diagnostics (Ribeiro et al. 2020). autonomous driving (Chen et al.) (021). and senti- ment analysis (Tan al. (022)). Given the usage in safety critical areas. the trustworthiness of such models plays key role which may be affected by real-world noise and the Human-addee pertur- bations (Toreini et al. 2020 Jin et al., 2024 Zhu af.) 2024). Considering the intrinsic nonlinear and complex parameters nature, trustworthy DNN model necessitates both high performance and interpretable decision making process (Adadi & Berrada 2018: Maze et al. 2018 |mall et al. [0233] Zhu et al. 2023a b). Understanding the data propagation from model input to output is essential for Explainable Artificial Intelligence (XAI) (Sokol et al. 2023).
There are two different interpretation methods (Pan et al. 2021). Local approximation methods provide an explanation by approximating the local neighborhood behaviors of the target model at particular point in the input space (Ribeiro et al. 2016; Shrikumar al.) 2017). Alternatively, gradient-baser methods explain the target model via the gradients associated with the model inputs and provide the importance of the input features (Pan et al. [02T (Sundararajan al.) (017]. In this work, we focus on gradient-based based methods, specifically attribution methods, which is to obtain pixel- level explanations determining the importance of each input feature for model decisions. Assuming that a small change to input features may alter the output, these features are considered an important factor aiding the sample in crossing the model's decision boundary, i.c., important features. Some
"Corresponding author; huaming.chen
@yydenyccddnccccuuuuu huaming.heenexydneyycd..aaa

---

Published as a conference paper at ICLR 2024
: Original Boundary
Shifted Boundary
Original Explanation Approach
MPE Explanation A pproach
Original Feature
Overfitiin Position
Modified Feature
Ideal Position
Minimal and Stable Area
Figure I: Decision boundary editing. Points ! to 2 represent the traditional adversarial attack which may lead to overfitting. The region within the green dashed circle denotes the local instability of the decision boundary (where samples are relatively concentrated and cannot be reliably distin- guished by the decision boundary). In order to obtain samples that cross the decision boundary more stably, points I to S adjust the decision boundary to surmount the Shifted Boundary. Points 3& ④ to ② simulate the effect of crossing the Shifted Boundary by generating modified samples.
recent methods are Integrated Gradients (IG) (Sundararajan et al. 2017). Boundary-based Integrated Gradient (BIG) (Wang et al.) 2021b), Adversarial Gradient Integration (AGI) (Pan et al. 2021).
However, need to consider the impact of inaccurate decision boundaries during model training since the training data is generally far from the decision boundaries. For the samples close to the de- cision boundary, they are likely to be OOD samples and are more sensitive. Avoiding such sensitive phenomena is crucial during the process of obtaining interpretive results. Besides, current gradient- based attribution methods may require either a baseline for integration (IG), or a specific linear integration path to quantify feature contributions (BIG). Even for AGI, the implemented adversarial attack is targeted, which may result in crossing multiple decision boundaries before reaching the target decision boundary, thus thwarting the interpretation particularly when there are similarities or overlaps between the decision boundaries of the target category and other categories. Therefore, we propose the first research question: (i) How to construct more general decision boundary exploration approach to ensure that features can explore the current decision boundary?
As shown in Fig. by modifying portion of the features (yellow dots) so that they may cross the decision boundary, the most important features are found with minimal changes. We find that transferable attacks, which aim to obtain more transferable samples to perform black-box attacks, essentially consist of exploring model parameters to generate generic adversarial examples that can cross multiple decision boundaries. The decision boundaries obtained by transferable attacks are likely to be less overfitting, in other words, more accurate. This fits our idea of feature alterations explore the current decision boundary in Fig. I Therefore, boundary exploration method of transferable attacks with the attribution process, namely novel model parameter exploration (MPE) based method, obtain the needed feature changes.
propose combine the decision
To verify the integration of important features by the attribution algorithm, one way is to check whether the model makes correct decisions when only essential features are retained. However, this poses a challenge wherein a significant number of model parameters that should originally be acti- vated remain inactive. is worth noting that the inactive model parameters are primarily responsible for unimportant features, i.e., the decision boundary is shifted, but not too far. This necessitates that the attribution algorithm should exhibit strong stability and adaptability to different decision bound- aries. Therefore, we propose the second research question: (ii) How to construct more robust approach for minimal feature alterations via MPE to ensure the attribution performance?
Motivated by recent research demonstrating DNNs exhibit different sensitivities to different fre- quency domains for the human-added perturbations (Yin et al., 2019 Wang et al., 2020b Guo et al.) 2018), performing spectral transformations inputs for frequency exploration provides new in- sights into model decision boundary exploration (Long et al. 2022). We find that the frequency information can significantly enhance the exploration of model parameters impacts the decision
2

---

Published as a conference paper at ICLR 2024
boundary. Moreover, exploring more model parameters leads to more precise attribution results. Therefore, we use frequency-based transferable attacks to generate minimally altered adversarial examples, and the results in the experimental section demonstrate the effectiveness of our approach. Notably, we are the first to introduce MPE to explore the decision boundaries of different models in generalizable manner. Since different transferable attack methods explore the decision bound- aries to varying degrees, our approach can be combined with other state-of-the-ar transferable attack methods discover potential variations in attribution performance. (See Appendix A for attribution results combining different state-of fate--o-thheasd ti--hhe-rrt art transferable attacks)
The main contributions of this paper are: (1) We uncover, for the first time, the decision boundary exploration approaches of attribution and transferable attacks are consistent. (2) We propose a novel attribution algorithm by performing Attribution for Explanation with Model Parameter Exploration based on transferable attacks, named AttEXplore. the effectiveness of our AuEXplore. (4) We release the code of AttEXplore publicly.
(3) We conduct extensive experiments to verify
2
RELATED WORK
2.1 METHODS FOR INTERPRETING DNNS
Local approximation methods Local approximation methods typically ascertain an approximately interpretable surrogate model, thereby allowing the computation of gradient information and the derivation of attribution outcomes. For example, LIME (Ribeiro et al. 2016) amalgamates approx- imation techniques with weighted sampling methods construct local model for interpretable predictions. We note that LIME's interpretable behavior requires cluster segmentation of images, so it is not point-to-point. Shapley Additive Explanations (SHAP) (Lundberg & Lee 2017) com- putes the contribution of each feature to the prediction outcome using Shapley values then ranks their importance. However, when applied high-dimensional samples, SHAP typically incurs high computational complexity. DeepLIFT (Shrikumar et al. 2017) quantifies the significance of each input feature by elucidating the predictive influence on the deep learning model. However, its inter- pretation of nonlinear models is not necessarily accurate. In this paper, we prioritize gradient-based methods, as they are better suited for providing promising explanations on complex models.
Gradient-based methods Gradient information can be leveraged to visually represent the contri- bution values of image pixels, such as Grad-CAM (Selvaraju et al., 2017) and Score-CAM (Wang) et al. 2020a). Saliency Map (SM) method (Simonyan al.) 2013) can produce interpretable results in non CNN environments where CAM-based methods are not applicable, however, is susceptible to gradient saturation, potentially yielding attribution results of zero.
To provide fine-gras fine-grained pixel-level explanations, IG method (Sundararajan et al. 2017) rectifies the gradient deficiency observed in SM and introduces two axioms: Sensitivity and Implementation Invariance. By strategically selecting reference points as anchors along a linear integration path, IG integrates the continuous gradients to derive the attribution results. Following, BIG (Wang al.) 2021b) introduces a boundary search mechanism, resulting in more precise attribution outcomes. resolves the concern of baseline selection process in IG. However, the integration path remains linear in BIG. AGI (Pan et al.) 202T) further improves the performance by identifying the steepest non-lineat ascending trajectory from the adversarial example z to x. Therefore, the attribution performance and stability hinge upon the quality of the adversarial samples.
Considering the integration path noise in IG, Guided Integrated Gradients (GIG) (Kapishnikov et al., 2021) obviates extraneous noisy pixel attributions by imposing constraints on the input and back- propagating gradients of the neurons, thus retaining only the pixel attributes pertinent to the pre- dicted category. Nonetheless, it is limited to images, where the quality of input features signifi- cantly impacts the results, and the computational complexity is high. Other methods, such as Fast- IG (Hesse et al. 2021) and Expected Gradient (EG) (Erion et al. 2021), have similar concerns.
2.2 TRANSFERABLE ADVERSARIAL ATTACKS
The objective of transferable ad attacks is to craft general adversarial samples from the source model, capable of crossing decision boundaries across different models. Many algorithms have been proven to generate highly transferable adversarial samples. MI-FGSM (Dong al. 2018
3

---

Published as a conference paper at ICLR 2024
and PGD (Madry al. 2017) utilize advanced gradient calculations to improve the transferability of adversarial samples. Based on imput transformation, SINI-FGSM (Lin et al. 2019). DI-FGSM (Xie) et al., 2019), and TI-FGSM (Dong et al.) 2019) adopt the image transformation methods the input image to generate more transferable adversarial samples. As one of feature-level transferable attacks, NAA (Zhang al. (2022) estimates the importance of intermediate layer neurons through neuron attribution, thereby solving the problem of inaccurate estimation of neuron importance by
FDA (Ganeshan et al.] (2019). FIA (Wang et al. 2021a) and other feature-level methods (Huang
et al. 2019 Naseer et al. 2018).
By exploring potential model parameters with frequency information (aang
2020b Guo al.
2018 Yin et al.) 2019), spectral transformation is implemented in (Long et al. [2022) for the input, effecting model augmentation to improve sample transferability. Thus, we consider harnessing the power of the frequency information to further explore different decision boundaries.
3 PRELIMINARIES
3.1 AXIOMS OF SENSITIVITY AND IMPLEMENTATION INVARIANCE
The beauty of attribution methods is from the axioms (Sundararajan al. 2017). Since our method maintains a one-to-one correspondence between model inputs and outputs during attribution, also satisfies these two axioms. Detailed proofs are provided in the Appendix B
Sensitivity An attribution method adheres to the axiom of Sensitivity when, for any given input and baseline instances differing solely in one feature yet yielding distinct predictions, said divergent feature is allocated non-zero attribution.
Implementation Invariance An attribution method conforming the axiom of Implementation Invariance should guarantee that two neural network attributions, when applied to identical input and output values, exhibit consistency.
3.2 DEFINITION OF DECISION BOUNDARIES
The decision boundary refers to hyperplane, curve, or boundary that separates data points of dif- ferent classes or sets in the input data space SShaart-Swwrrz & Ben-David] 2014). The position, shape, and characteristics of the decision boundary depend on the model's structure and its parame- ters. Constructing a robust method for exploring and visualizing the decision boundaries of different DNN models is pivotal for understanding the decision-makinn process.
3.3 IG AND AGI METHODS
Formally, in order to explicate the DNN model denoted as f()), we define the input feature r IR", where is the dimension of the input feature, and the model output is represented Y = fxx). The primary objective attribution lies the determination of E IR"", which to elucidate the corresponding significance of each feature within x. According to Saliency Map (Simonyan al.) 2013), if a DNN model f exhibits continuous differentiabilityy the input feature importance measure A can be derived from the gradient information dz engenders a one-to-one correspondence. For example, denote the input feature importance of IG by 1G, (z), then the formula of IG is expressed in Eq
Øf It is imperative to underscore that this process
af(a' + x (x x')) da
A₁ (z) = (xj 243) x
Oz,
where j = 1...., denotes the j-th input feature, Of(w' ((x==+xxx===))) is the gradient of model f w.r. input feature zg. Here n²G represents the reference input feature. If we denote the input feature importance of AGI by AGT, (z), then the formula is described in Eq
dag
(x))
4 = ACI AGI,-1(=) (z) sign( [x){
Vz, f(( means the gradient corresponding to false class label i. Step size is represented by
Eq 2 integrates along the path until argmaz) (x)) i. We can that, the decision boundary
4

---

Published as a conference paper at ICLR 2024
Teudonale
Pociic
Ohrinbograton pith
kidd hreteen
Otter int gaah
Soura cade decuat beally
Tapatralis boanlig
Anf
A
Ar
Hiues
Figure 2: Decision boundaries
Figure 3: Our nonlinear integration path
exploration approach of IG is linear. For AGI, despite the non-linear decision boundary exploration approach without selecting specific reference points, it still needs to continuously cross the deci- sion boundaries other categories until the decision boundary category becomes i, which could potentially lead to overfitting issues and raise the concern of computation efficiency.
METHOD
4.1 EXPLORE DECISION BOUNDARIES VIA MODEL PARAMETER EXPLORATION (MPE)
Feasibility of MPE We discuss the relationship between model parameter exploration and attri- bution in this section. Since directly exploring the decision boundaries is difficult, we alternatively consider the model parameters obtain the changes in model decision corresponding the changes of a small number of parameters. This can significantly facilitate the attribution process. Assuming model y L(r; w). where y is the model output for input æ with the parameter w. Here we sim- plify the model to w" x. If we consider a two-dimension scenario when w 2), x = [3, 4), y 11. We have two methods explore cases where the first parameter in w" is not activated. One method is to leave w" unchanged and the = 0, i.e., x = [0, 4], at which point = 8. Another method is to leave x unchanged and w = 0, i.e., n² = [0, 2). at which point y = We can see these two methods are equivalent, which means exploring is to some extent consistent with exploring w",, i.e., Lxx; w) can be viewed as L(w;,), æ). Thus, model parameter exploration can be performed by modifying the input feature or adjusting the activation levels of parameters in w.
MPE via transferable adversarial attacks With the discussion of MPE, we understand that still infeasible to make extensive adjustments to the model's parameters, in particular, attribution al- gorithms aims to provide a rigorous explanation of the model's behaviour under current parameters. Moreover, due to the nature of attribution, in scenario where a complete dataset is unavailable (Liu) et al. 2014 Wang et al. 2020c Retsinas et al. 2020). we cannot systematically adjust the parame- ters to explore the decision boundary in controlled manner. Therefore, resort to adjusting input features to explore the model's decision boundary, aiming to obtain more precise attribution results.
We firstly confirm that modifying samples explore different decision boundaries aligns with the methods of transferable adversarial attacks. As illustrated in Fig. 2 the goal of transferable attacks is to generate samples with strong transferability on local surrogate model to launch an attack on the target black-box model. Since different black-box models have different decision boundaries, de- veloping a robust adversarial sample generation method cross the decision boundaries is the core idea. Currently, input transformation-bant
transferable attacks represent the state-of-the-art (Lin) cet al. 2019 Xie et al. 2019 Dong et al. 2019). in which input samples are modified to generate general adversarial samples. This aligns with our idea of modifying features for model parameter exploration. Therefore, we propose to incorporate the transferable attack method in the attribution algorithm to enhance decision boundary exploration, solution to the first research question.
4.2 ATTRIBUTION FOR EXPLANATION WITH MODEL PARAMETER EXPLORATION (ATTEXPLOREE)
Novel nonlinear integration path In AGI Pan et al. 2021), the nonlinear integration path has been proven to be beneficial for attribution results. Specifically, nonlinear integration paths allow for
5

---

Published as a conference paper at ICLR 2024
more accurate assignment of weights to features well capturing the nonlinear behaviour of the model in a more comprehensive way. In order to utilize model parameter exploration for attribution,, design novel nonlinear integration path as in Fig. We use Eq. to mathematically explain our integration path, with detailed proofs in the Appendix
f Ax g(x")dt
A A=
(3)
where Ar" represents the difference in the sample as it varies along the boundary in the decision direction. g(x") denotes the gradient information that needs to accumulated during the integra- tion process. y represents the original label. @ denotes hadamard product. There are two options for g(x*) in the integration process, One is the actual updated gradient obtained after MPE, cor- responding to the black arrow in Fig. 3 The other is the gradient obtained by recomputing the current sample Ty, which corresponds to the blue arrow in Fig. 3
BIG and AGI, it is expressed
@L(mg, () Taking AGI as an example, since is a targeted attack, the model may cross multiple as du decision boundaries of other categories before reaching the decision boundary of a specific category. This results slight biases in AGI's nonlinear integration path before the integration is completed, leading to unnecessary attacks and attributions (i.e., the angle of bias in Fig. 33. Therefore, in order to integrate the attribution results more smooth and robust in our nonlinear integration path, we use MPE from Eq. to explore the decision boundary and update the gradient information of the model.
Frequency- input feature alterations method Frequency domain information can effectively explore model parameters and generate highly transferable adversarial samples (Wang et al. 2020b Guo et al. 2018: Yin et al., 2019), which can assist the attribution process. Inspired by SSA (Long a 2022), we propose frequency -baee input feature alterations method to generate input Iea- tures that can effectively cross different decision boundaries, as detailed in Eq.
x¹y. IDCT(DCT(e + 1()
* N(1,o))
(4)
255
aL v)
Ax' = sign
(5)
Osy
g(x') 1
(L
(6)
N
Ozy;
From Eq. 4 to explore different frequency domains of the input feature first Discrete Cosine Transform (DCT) (Ahmed et al. 1974) to map the features into the frequency space. Then, generate N approximate features sty, of xf by adding noise to the original features and applying random transformations in the firquueccy space. Here is the perturbation rate, and i represents the number of frequency domain explorations. The inverse discrete cosine transformation (IDCT) serves as the reverse operation of DCT, allowing the image to be transformed back to the spatial domain. It is important to note that both DCT and IDCT operations are lossless, and they facilitate the ease of gradient calculations (Ahmed al. 1974). From Eq 5 we randomly select N approximate features and average the results to represent the difference in samples. L represents the target model, sign(-) determines the direction of integration, and 7) the learning rate. Eq. i the specific mathematical formula for gradient information calculation. We address the second research question by utilizing our novel nonlinear integration path and trequency-based input feature alterations method.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Dataset and Models In this study, we employ ImageNet dataset (Deng et al. 2009). We conduct experiments on a selection of 1000 samples from ImageNet, guided by the principles outlined in NAA (Zhang et al. 2022), SSA (Long et al.) 2022). and AGI (Pan et al. 2021). Furthermore, we employ three commonly utilized CNN models in the field of image classification: Inception- v3 (Szegedy et al. 2016), ResNet-50 (He et al. 2016). and VGG16 (Simonyan & Zisserman 20144).
Notably, we also employ the ViT-B/I6 (Dosovisskiy et al. 2020) model to investigate the inter- pretability of our method on transformer-based visual models.

---

Published as a conference paper at ICLR 2024
Houtmas
Heazree
Mextree-
Prodiction: speonkill
ArtESplore
Productenn
AttiEpllrr
Testnup
Hustrup Tirpe:
Hearrep hros
MALIBACYHLP
AGI
SALINEYYNAP
*hstmpp
Phatrspp Tinest
Hedrees
Heairep hou.
Hodeee:
Denplet
Deeplit
Hestrop
Hestrop tirps
Hearrap Yruu.
Heatras
Harrmaa 'rput
Hearrap
Hestrop 11p.4
Figure 4: Visualization Results of our AttEXplore and Other Competitive Methods
Baselines We primarily compare with the state-of-the-an attribution algorithm, AGI (Pan et al. 2021). We also include eight other classical interpretability algorithms for comparative analysis, namely BIG (Wang al. 2021b), DeepLIFT (Shrikumar et al., (017), GIG (Kapishnikov et al.) 2021 EG (Erion et al. (021). Fast-IG (Hesse et al. 2021). ), IG (Sundararajan et al. 2017). SM (Si- monyan et al. 2013). SG (Smilkov et al. 2017). and Grad-CAN (Selvaraju et al.) 2017).
Evaluated Metries We adhere to the evaluation metrics, specifically the Insertion&Deletionn Scores, commonly employed by interpretability algorithms (Pan et al., 2021). Insertion Score quantifies the degree of change in model output when pixels inserted into the input. A higher score signifies perior algorithmic interpretability. Conversely, Deletion Score measures the extent of model output change when pixels are removed from the input. A lower of the algorithm. It is noted that, for attribution algorithms, the importance of the Insertion Score outweighs that of the Deletion Score. This is due to the adversarial nature of neural networks, where Deletion Score may offer unreliable indications (Petsiuk et al.) 2018). Hence, the Insertion Score serves as a more representative performance metric for attribution algorithms, while the Deletion Score can serve as an auxiliary metric to analyze attribution algorithms from multiple dimensions. Additionally, we employ the INFD score (Yeh al.) 2019) to demonstrate the faithfulness of our method to the underlying model. The lower the INFD score, the higher the faithfulness.
indicates enhanced interpretability
Parameters All experiments are conducted using an AMD Ryzen Threadripper PRO 5955WX 16- Core CPU, NVIDIA RTX6000 Ada GPU, and Ubuntu 22.04. Additionally, we apply the following general parameters setting: momentum set to 1.0, mask control parameter & set to 0.5, number of ap- proximate features N set to 20, standard deviation of Gaussian noise (a) set to 16, perturbation rate (e) set to 48/255, and total attack iterations muum_steps set to 10. We notice a significant boosting is achieved without fine-tuning. Further parameter tuning may lead to much better performance.
5.2 EXPERIMENTAL RESULTS
Fig. displays the visual results of our AuEEXplore and other methods on Inception-v3 (See Ap- pendix G for more visualization results). It is clear that the output heatmaps of AttEXplore are denser and clearer compared to methods like AGI, BIG, etc. This implies that the attribution results with high attribute values are more concentrated on the target object. Based on the results presented in Tab. [] it is evident that our proposed method exhibits significant performance improvement over other classical interpretability algorithms. Particularly for the Insertion Score, has surpassed both similar classical interpretability algorithms and AGI. Meanwhile, the Deletion Score remains con- sistently at a relatively low level, consistently outperforming AGI in comparative assessments. To provide specific instances, on the Inception-và model, relative AGI, our method has increase of 4.89% in Insertion Score and a decrease of 1.42% in Deletion Score. When compared with other algorithms average, our method results in improvement of 20.59% in the Insertion Score and

---

Published as a conference paper at ICLR 2024
reduction of 1.86% in the Deletion Score. On the ResNet-50 model, relative to AGI, our method shows an increase of 4.01% % in the Insertion Score and a decrease of 1.72% in the Deletion Score. When compared with other algorithms on average. our method improves by 24.06% in Insertion Score and has a reduction of 2.42% in Deletion Score. Finally, on the VGG-16 model, relative to AGI, our method led the Deletion Score. When compared with other algorithms on average, our method results in an im- provement of 19.05% in the Insertion Score and reduction of 1.81% % in the Deletion Score. Notably, different from traditional CNN models, Vision Transformers (ViTs) process images as sequences of patches, rendering them challenging to interpret. In the Appendix C we conducted additional ex- periments on ViT-B/16 (Dosovitskiy et al.) 2020), and the results further substantiate the superior performance achieved by our method. Also the Appendix D the INFD score tests demonstrate that our method exhibits the highest faithfulness.
increase of 6.01% in the Insertion Score and decrease of 0.93% in
Table DnsertionDDelttioon score comparison of AtEXplore and other competitive baselines
Inception-r3
RosNet-50
VGG-16
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Method
Grad-CAM
0.4496
0.1084
0.2541
0.0942
0.3169
0.0841
BIG
0.356.3
0.0379
0.2272
0.0415
0.1762
0.0303
MaliencyMap
0.3974
0,2089
0.0323
Deepliitt
0.216
0.0314
0.1246
0,0256
0.0827
0.0157
GIG
0,2584
0.02399
0.1308
0.0184
0,0859
0.0142
BG
0,2364
0.256
0.2178
0.1959
0.1797
Fast-GG
0,146
0,0138
0.0889
0,0315
0.0623
0.0213
IG
0.2268
0.0284
1.36
0.0247
0.0701
0.0173
SG
0.301
0.023
0.2357
0.0202
0.1423
0.015
AGI
0.4243
0.0439
0.3796
0,0465
0.2585
0.0319
AlEXplore (ours)
0.4732
0.4197
0.0293
0.3186
0.0226
5.3 ANALYSIS OF TIME COMPLEXITY
We use FPS, the number of frames processed by the algorithms per second (FPS), to evaluate the algorithm processing speed (See Appendix E for the definition of FPS). All experiments are run in the same environment discussed in Section 5.1 We select five methods that closely match the performance of AuEXplore as our baselines. Other methods such as Saliency Map, DeepLIFT, Fast- IG, EG, and Grad-CAl - demonstrate relatively poorer attribution accuracy compared to AttEXplore. Therefore, they are not considered for efficiency comparison. Table 2 demonstrates the superior computational efficiency of AtEXpplrr while also attaining enhanced attribution performance.
Table 2: FPS results for AttEXplore and state-of-the-art methods
Method
BIG
AGI
IG
SG
GIG
AttEXplore
FPS
3.3798
0.8818
19.7461
19.4942
2.2814
47,2805
5.4 ABLATION STUDY
Here we discuss the impact of three parameters, namely the approximate features number (N), the total attack iterations (num_steps), and the perturbation rate (c), on the performance of AttEXplore.
Number of approximate features (N) The total attack iterations are firstly fixed at 10, where the perturbation rate is 16. We change N to values of 10, 20, 30, 40, 50, and 60, to assess the influence of this parameter on the performance of AttEXplore. In Table with an increase in N, the performance of AttEXplore exhibits a gradual enhancement. Specifically, across three distinct models, namely Inception-v3, ResNet-50, and VGG-16, both insertion and delction scores consistently increase as N increases. It indicates that increasing the number of approximate features effectively enhance the performance of AttEXplore. Appendix F.1 contains results with additional values of N.
Total attack iterations (num_stepp) We first keep 16 and N at 20. We then configure num_steps to be 5, 10, 15, 20, 25, and 30, to evaluate the influence on AttEXplore. Table 4 shows that, across three models of Inception-v3, ResNet-50, and VGG-16, there is slight fluctuation in both insertion
8

---

Published as a conference paper at ICLR 2024
Table Insertion&Dellcton score of AtEXplore with different values of N
Inception-v3
ResNet-50
VGG-16
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion Score
10
0.4603
0.0301
0.4004
0.0291
0.3074
0.0228
0.4644
0.0313
0.4022
0.0309
0.3096
0.0237
0,4649
0.0325
0,4033
0.0319
0.3090
0.0243
40
0.4659
0.0325
0.4045
0.0330
0.3108
0.0244
0.4666
0.4032
0.0329
0.4679
0.0335
0.4037
0.0340
0.3107
0.0249
and deletion scores with an increase in num_steps. However, there is no evident trend indicating significant impact of an augmented num steps the performance of AttEXplore. This suggests that, considering a set of and N, variations num steps exert a comparatively minor influence on the performance of AttEXplore. Appendix F22 contains results for different num_steps.
Table 4: DneetiionDeletiooo score of AuEXplore with different values of num_steps
Inception-v3
ResNet-50
VGG-16
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion Score
num steps
5
0.4615
0.0307
0.3986
0.0287
0.3080
0.0224
10
0.4644
0.0313
0.4022
0,0309
0.3096
0.0237
15
0.4651
0.0324
0.4031
0.0322
0.3077
0.0244
20
0,4672
0.0329
0.4024
0.0331
0.3086
0.0244
25
0.4673
0.0332
0.4032
0.0336
0.3081
0.0248
30
0,4663
0.0339
0.4026
0,0339
0.3089
0.0252
Perturbation rate (c) We firstly fix N 20 and the num steps at 10. We separately configured the
perturbation rate (c) to be 8, 16, 24, 32, 40, and 48, to assess the influence on AuEXpplore. Table 5 demonstrates that, across the three distinct models of Inception-v3, ResNet-50, and VGG-16, an increase in the perturbation rate is accompanied by a noticeable rise in the Insertion Score, while the Deletion Score exhibits a declining trend. This implies that in scenarios where the num_steps and N remain relatively stable, higher may be positively correlated with the performance of AttEXplore. The results with additional values of are included in Appendix F3
Table 5: Insertion& Deletion score of AttEXplore with different values of
Inception-vs v3
ResNet-50
VGG-16
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion Score
8
0.4637
0.0325
0.3962
0.0309
0.3065
0.0234
16
0.4644
0.0313
0.4022
0.0309
0.3096
0.0237
24
0.4659
0.0306
0,4071
0.0305
0.3121
0.0233
32
0..6755
0.0305
0.4109
0.0300
0.3142
0,0232
40
0.4714
0.0291
0.4157
0.0296
0.3161
0.0231
48
0.4732
0.0297
0.4197
0.0293
0,3186
0.0226
CONCLUSION
In conclusion, this paper introduces a novel method for Attribution for Explanation with model parameter eXploration (AntXpplore), which significantly advances the XAI results by providing enhanced interpretability for Deep Neural Networks (DNNs). Through the combination of model parameter exploration and frequency based input feature alterations, AttEXplore outperforms state- of-the-art methods, demonstrating substantial improvements both Insertion and Deletion Scores. By uncovering the relationship between attribution and transferable attack methods, we anticipate this work can contribute to a new standard for trustworthiness and explainability in deep neural networks. To achieve this, we also release the replication package of AttEXplore to facilitate im- provements in future works. We hope this work will provide some insights to enhance the attribution method research community for better XAI field.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
Prof. Flora Salim acknowledges the support of the Australian Research Council (ARC) Centre of Excellence for Automated Decision-Making and Society (ADM+S) (CE200100005).
REFERENCES
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: survey on explainable artificial intelligence (xai). IEEE access, 6:52138-52160, 2018.
Nasir Ahmed, T- Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90-93, 1974.
Long Chen, Shaobo Lin, Xiankai Lu, Dongpu Cao, Hangbin Wu, Chi Guo, Chun Liu, and Fei- Yue Wang. Deep neural network based vehicle and pedestrian detection for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 22(():3234 3246, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversar- ial examples by translation-ivaariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2312-4321 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint WrXi::2010.11929 2020.
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 3(7):620-63) 2021.
Aditya Ganeshan, Vivek BS, and Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8069-8079, 2019.
Chuan Guo, Jared S Frank, and Kilian Q Weinberger. Low frequency adversarial perturbation. arXiv preprint 1rXiv:1809.088
2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Robin Hesse, Simone Schaub-Meyer. and Stefan Roth. Fast axiomatic attribution for neural net- works. Advances in Neural Information Processing Systems, 34:19513-19524, 2021.
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhanc- ing adversarial example transferability with an intermediate level attack. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 4733-4742, 2019.
Zhibo Jin, Jiayu Zhang, Zhiyu Zhu, and Huaming Chen. Benchmarking transferable adversarial attacks, 2024.
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Temry and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5050-5058, 2021.
10

---

Published as a conference paper at ICLR 2024
Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.0628.1 2019.
Chao Liu, Zhiyong Zhang, and Dong Wang. Pruning deep neural networks by optimal brain damage. In Fifieenth Annual Conference of the Interaational Speech Communication Association, 2014.
Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV, pp- 549-566. Springer, 2022.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.000 1606.06083, 2017.
Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. larpa janus benchmark-c: Face dataset and protocol. In 2018 international conference on biometrics (ICB), pp. 158-165. IEEE, 2018.
Muzammal Naseer, Salman H Khan, Shafin Rahman, and Fatih Porikli. Task-generalizable adver- sarial attack based on perceptual metric. arXiv preprint arXiv: 1000000, 2018.
Deng Pan, Xin Li, and Dongxiao Zhu. Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of 2018.
black-box models. arXiv preprint
George Retsinas, Athena Elafrou, Georgios Goumas, and Petros Maragos. Weight pruning via adaptive sparsity loss. arXiv preprint arXiv:2006.02768 2020.
Antônio Ribeiro, Manoel Horta Ribeiro, Gabriela MM Paixão, Derick M Oliveira, Paulo R Gomes, Jéssica A Canazart, Milton PS Ferreira, Carl R Andersson, Peter W Macfarlane, Wagner Meira Jr, et al. Automatic diagnosis of the 12-lead eeg using a deep neural network. Nature communications, 11(1):1760, 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-628866 118-626, 2017.
Shai Shalev-Shwarle and Shai Ben-David. Understanding machine learning: From theory algo- rithms. Cambridge university press, 2014.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. International conference on machine learning, pp. 3 3145- 3153. PMLR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint /rXiv:/312.661 2013.
11

---

Published as a conference paper at ICLR 2024
Edward A Small, Jeffrey N Clark, Christopher J McWilliams, Kacper Sokol, Jeffrey Chan, Flora D Salim, and Raul Santos-Rodriguez. Counterfactual explanations via locally-guided sequential algorithmic recourse. arXiv preprint arXiv:2309.042. 04201,, 2023.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1766.0000
2017.
Kacper Sokol, Meelis Kull, Jeffrey Chan, and Flora Dilys Salim. Cross-model fairness: Empirical study of fairness and ethics under model multiplicity, 2023.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Kian Long Tan, Chin Poo Lee, Kalaiarasi Sonai Muthu Anbananthen, and Kian Ming Lim. Roberta- Istm: a hybrid model for sentiment analysis with transformer and recurrent neural network. IEEE Access, 10:21517-21525, 2022.
Ehsan Toreini, Mhairi Aitken, Kovila Coopamootoo, Karen Elliott, Carlos Gonzalez Zelaya, and Aad Van Moorsel. The relationship between trust ai and trustworthy machine learning tech- nologies. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 272-283, 2020.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 24-25, 2020a.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8684-8694, 2020b.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. arXiv preprint arXiv:2012.092433 2020c.
Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importance-aware transferable adversarial attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7639-7648, 2021a.
Zifan Wang, Matt Fredrikson, and Anupam Datta. Robust models more interpretable because attributions look normal. arXiv preprint arXiv:2 /257, 2021b.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L Yuille. Improving transferabilitty of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2730-2739, 2019.
Chih-Kuan Yeh, Cheng- Yu Hsich, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances neural information processing systems, 32, 2019.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances Neural Information Processing Systems, 32, 2019.
Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-bu based attacks. In Pro- ceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14993-
15002,, 2022.
12

---

Published as a conference paper at ICLR 2024
Zhiyu Zhu, Huaming Chen, Zhibo Jin, Xinyi Wang, Jiayu Zhang, Minhui Xue, Qinghua Lu, Jun Shen, and -wang Raymond Choo. Fow: Finding valuable weight on deep neural network for model pruning. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pp. 3657-3666, 2023a.
Zhiyu Zhu, Huaming Chen, Jiayu Zhang, Xinyi Wang. Zhibo Jin, Minhui Xue, Dongxiao Zhu, and -aanKwang Raymond Choo. Mfaba: A more faithful and accelerated soundary-basee attribution method for deep neural networks. arXiv preprint arXiv:2312. 13630, 2023b.
Zhiyu Zhu, Huaming Chen, Xinyi Wang. Jiayu Zhang, Zhibo Jin, and Kim-Kwang wwng Raymond Choo. Ge-advgan: Improving the transferability of adversarial samples by gradient editing-basee adver- sarial generative model. arXiv preprint 2rX1v:2401.06031, 2024.
13

---

Published as a conference paper at ICLR 2024
A ATTRIBUTION PERFORMANCE WHEN USING OTHER SOTA
TRANSFERABLE ATTACKS FOR MODEL PARAMETER EXPLORATION
Table 6: Attribution performance with difference attack methods
Inception-v3
ResNet-50
VGG-16
Method
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Att..plore NoAttack
0.3955
0.0422
0.2584
0.0457
0.2121
0,0312
AtrEXplore
0,4644
0.0313
0.4021
0.0308
0.3097
0,0237
AnEXplore- PGD
0,402
0.037
0,2901
0.0294
00.2258
0.0199
ArEXplore-DI-FOSS
0,4007
0.0416
0.2908
0.0427
0.2283
0.031
AnEXXploreTTI FGSM
0.3943
0.0344
0.3229
0.0348
0.2511
0.0268
AtEEXploe-MI-FGSS
0,398
0.0414
0,2606
0.0447
0.2137
0.0304
AnEXplore-SINI--FO
0,4418
0.03144
0.3134
0.0292
0.2564
0.0227
AnEXplare-NAA
0.436
0.034
0,3058
0.0381
0.2559
0,0248
B DETAILED PROOFS OF TWO AXIOMS
Firstly, during the iterative process, the changes in the gradient along the integration path are cap- tured by the original input information. Furthermore, is not retroactive since feature values in previous iterations are unchanged in subsequent iterations. Therefore, the attribution result must be non-zero, which meets the definition of sensitivity. Here is the mathematical proof.
Poof of Eq.3:
We the first-order Taylor approximation to expand the loss function and combine the information for the path from Ax to Ar?"
OL (x+-1)
L (x²) = L (x"
(xt
x1-1) +e
Oxt-1
T
T-1
T-1
OL (x')
(x') = Σ (x²)
+ Σ
(z - x¹)
Oxt
1=1
1=
1=0
T-1
OL (x')
(7)
A=
- (L(9)
Σ
(z - -x²)
Ozt
1=0
Σ
T-1
g((xr) @ Ax' / Ao' o ggat')t
Here is omitted due to the principle of higher-order Taylor expansions. And Art = rt1l x², g(x") OL(x")
Secondly, it is clear that the computational processes in AttEXplore follow the chain rule of gradi- ents, which meets the definition of implementation invariance.
C ADDITIONAL EXPERIMENTS ON VIT-B/16
See Tab. 7
D INFD SCORE TESTS
See Tab. g
E
FPS DEFINITION
we use Frames Per Second (FPS) as an evaluation metric for our running efficiency. A higher FPS indicates greater number of images generated per second, signifying higher operational efficiency
14

---

Published as a conference paper at ICLR 2024
Table Attribution performance of AtEXxplore and other competitive baselines on ViT-B/16
Model
Method
Insertion Score
Deletion Score
VIT-B/16
Saliency Map
0.373
0.125
VIT-B/16
BIG
0.422
0.093
VIT-B/16
GIG
0.335
0.046
VIT-B/16
DecpLIFT
0.296
0.063
VIT-B/16
BG
0.361
0,329
VIT-B/16
Fast IG
0.216
0.071
B/16
SG
0.428
0.035
VIT-B/16
AGI
0.425
0.069
VIT-B/16
0,346
VIT-B/16
AnEXplore (ours)
0.470
0.062
Table 8: INFD Score
Model
AGI
BIG
DeepLIFF
EG
Fast IG
GIG
IG
Saliency Map
SG
AtEXpplrr
Inception-v3
3.839
3,928
110.158
111.631
111.44
37.67
66.509
4.078
63.659
3.728
ResNet-50
1.003
0.708
18.828
143.593
135.651
39.659
85.834
0.696
42.504
0.671
VGG16
0.88
9.746
220.376
211.104
47.988
124.474
0.499
72.912
of the method.
Number of samples
FPS
(8)
Rumning time of these samples
F ADDITIONAL ABLATION STUDIES
F.1 ABLATION STUDIES FOR THE NUMBER OF APPROXIMATE FEATURES
Table Result for the number of approximate features (W 10)
Inception-v3
ResNet-50
VGG-16
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion
N
Score
0.4536
0.0282
0.3841
0.0262
0.2915
0.0190
2
0.4568
0.0284
0.3931
0,0275
0.2970
0.0196
0.4624
0.0298
0.3957
0.0274
0.3020
0.0210
0.4606
0.0292
0.3987
0.0286
0.3058
0.0214
5
0.4588
0,0300
0.3995
0.0282
0.3041
0.0223
0.4602
0.0288
0.3989
0.0289
0.3059
0.0221
7
0.4597
0.0301
0.3999
0.0288
0.3071
0.0224
0.4619
0,0301
0.4005
0.0289
0.3073
0.0229
9
0.4646
0.0299
0.3995
0.0291
0.3064
0.0225
We observe that when the perturbation rate is set to larger value such as 48, the trend in model performance across different N does not become more clear. This might be attributed the fact that larger perturbation rate represents a larger search space. Although the number of approximate samples increases, it doesn't mean that all these samples are necessarily effective for the attribution result.
15

---

Published as a conference paper at ICLR 2024
Table 10: Result for the number of approximate features N when the perturbation rate is 48
Model
N
8
50
60
Inception-v3
0.168
Deletion scoe
0.077
0.028
0.028
0.027
0,028
0.029
0.83
RexNet-00
Inseriion scone
0.406
0.414
0.417
0.477
0.419
0.42
0.422
0,422
0.422
0.423
0.455
0.427
0.427
0.428
0.488
Deletion score
0.027
0.028
0.028
0.029
0.03
0.029
0.03
0.03
0.03
0.03
0.051
0.031
0.032
0.032
0.032
VGG16
Inseriion
0.298
0,304
0,306
0.308
0.308
0.508
0.31
0.31
0.311
0.311
0.312
0.313
0.313
0.313
0.315
0.022
F.2 ABLATION STUDIES FOR THE TOTAL ATTACK ITERATIONS
Table 11: Result for the total attack iterations num steps
Inception-v3
ResNet-50
VGG-16
Insertion Score
Deletion Score
Insertion Score
Deletion Score
Insertion Score
Deletion
stessteps
Score
1
0.4236
0.0281
0.3469
0.0249
0.2645
0.0195
2
0.4488
0.0291
0.3835
0.0261
0,2929
0.0208
3
0.4557
0.0297
0.3936
0.0273
0.3029
0,0217
4
0.4607
0.0301
0.3966
0.0283
0.3061
0,0218
F.3 ABLATION STUDIES FOR THE PERTURBATION RATE
Table 12: Result for the perturbation rate (e < 8)
Model
2
4
6
Inception-v3
Insertion score
0.459
0.459
0.459
0.461
0.46
0.461
0.46
Deletion score
0.031
0.032
0.032
0.031
0.032
0.031
0.032
ResNet-50
Insertion score
0.397
0.397
0.398
0.401
0.4
0.403
0.403
Deletion score
0.031
0.031
0.031
0.032
0.032
0.032
0.032
Insertion score
0.298
0.299
0.3
0.3
0.3
0.302
0.303
VGGI6
Deletion score
0.021
0.022
0.022
0.022
0.022
0.022
0.022
16

---

Published as a conference paper at ICLR 2024
G ADDITIONAL VISUALIZATION RESULTS OF OUR ATTEXPLORE
Heatmap
Heatmap Input
Prediction: beetle Heatmap * Input
AttEXplore
Heatmap
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap Input
Heatmap
Heatmap Input
IG
GIG
17

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: buckeye Heatmap Input
AttEXplore
Heatmap
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
18

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: lionfish Heatmap Input
AttEXplore
Heatmap
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
19

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: drake
AttEXplore
Heatmap
Heatmap Input
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
20

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: soccer_bal
AttEXplore
Heatmap
Heatmap Input
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
21

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: vulture
AttEXplore
Heatmap
Heatmap Input
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
22

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: grasshopper Heatmap Input
AttEXplore
Heatmap
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
23

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: bulbul
AttEXplore
Heatmap
Heatmap Input
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
24

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: mountain_teme ttnnt Heatmap Input
AttEXplore
Heatmap
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
25

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Goy Lowe
Prediction: schooner
AttEXplore
Heatmap
Heatmap Input
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
26

---

Published as a conference paper at ICLR 2024
Heatmap
Heatmap Input
Prediction: peacock Heatmap Input
AttEXplore
Heatmap
Heatmap
Heatmap Input
AGI
SALIENCYMAP
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
BIG
Deeplift
Heatmap
Heatmap * Input
Heatmap
Heatmap Input
IG
GIG
27


Reviews:
[{"review_id": "OAo7zzHz9k", "reviewer": "Reviewer_H27T", "summary": "This paper introduces a new attribution technology AttExplore that improves over three existing gradient integral methods, IG, BIG and AGI. The main contributions are that 1) the proposed method finds a baseline in the integral that serves as an adversarial input for the current model and some other variations; 2) their empirical results show improved insertion and deleting scores compared to baselines.", "questions": "N/A", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Authors have made it clear how AttExplore computes the attribution scores. Their experiments include 3 networks and a rich amount of baselines. Table 1 shows a good amount of improvement over the best baseline.", "weaknesses": "My main concerns include the motivation and the evaluation of this paper. \n\n### Motivation\nI am not entirely sure I understand the motivation of the paper. It looks like this paper accuses BIG and AGI of using the exact boundary of the underlying model, which fails to be generic to similar models with variations in the decision boundary? This is pretty counterintuitive to me as feature attributions are often perceived as local explanations at the given model and the given input. Why do we need to consider if the explanation generalizes other models? With that being said, generalizing to other decision boundaries may trade in the faithfulness of the underlying model (this would relate to my second question about faithfulness and I will explain later). \n\n### Generalization of Frequency-based Methods?\nThe motivation to alter features in the frequency domain is because there are works showing high-frequency features help the model to generalize better [1]. I am pretty worried the authors explain this conclusion as altering some features in the frequency domain helps to create examples that transfer better, especially for methods just proposed in this paper (Eq. 4 - 5).  Have you verified your adversarial examples actually transfer better? Can you provide some analytical results convincing me this helps better explore more decision boundaries? So far all descriptions are pretty hand-wavy.  \n\n### Evaluations\nIs the method faithful to the underlying model? It looks like to realize “a more general adversarial input” the proposed method manages to find an adversarial example that is much further to the decision boundary compared to the previous methods. I think authors may want to compare how much farther you go. It is pretty concerning to me that no matter how you adjust the feature generations in Table 3 and steps in Table 4, the proposed method has almost identical results on your metrics. Similar, no matter how you decrease $\\epsilon$ in Table 5, the results do not change at all. What should we make out of these results? Are they showing the proposed method may be actually unfaithful to something? BTW, using $\\epsilon=8$ is pretty huge and I think decision boundaries are usually much much closer than $8/255$ for models without adversarial training. \n\nI recommend testing INFD score [2] and run sanity check [3].\n\n### Ending\n\nUnlike doing detection or being more robust, I really think research works in explaining feature importance are not result-driven. Namely, it is not about being state-of-the-art on some scores. Not every attribution method uses deletion and inserting scores in baseline papers cited. There are a lot of discussions for the unreliability of feature attributions that are not cited here. It is fine to say “we do not care about those critiques but just want to improve the current method” but it is important to point out *important flaws* in the existing methods and fix those. \n\n[1] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8684–8694, 2020b.\n\n[2] Yeh, C. K., Hsieh, C. Y., Suggala, A., Inouye, D. I., & Ravikumar, P. K. (2019). On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32.\n\n[3] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. Advances in neural information processing systems, 31.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sgZQBxRdCL", "reviewer": "Reviewer_M36V", "summary": "This paper introduces a novel attribution method called AttEXplore, designed for explaining DNN models through the exploration of model parameters. AttEXplore leverages the concept of transferable attacks as there's consistency between the decision boundary exploration approaches of attributionand the process for transferable adversarial attacks. By manipulating input features according to their frequency information, AttEXplore enhances the interpretability of DNN models.", "questions": "It's shown in Table 1 in the appendix that compared with N = 10, the drop of insertion score  for N = 1is not huge (~0.007 for inception-v3, still better than existing methods ). The perturbation rate was set as 16, while the model performance is better when perturbation rate is 48. Would the trend of model performance under different N be more clear when the perturbation rate is set as a larger value?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. AttEXplore outperforms existing methods on the ImageNet dataset and models such as Inception-v3, ResNet-50, and VGG-16, achieving higher insertion scores and lower deletion scores, which underscores its effectiveness in model evaluation.\n3. AttEXplore exhibits superior computational efficiency in terms of the number of frames processed per second when compared to existing methods.", "weaknesses": "1. This paper appears to focus exclusively on evaluating AttEXplore using image data, it lacks experiments on other data modalities, such as text data on NLP models. Expanding the evaluation to different data modalities could provide a more comprehensive assessment of AttEXplore's applicability.\n2. For image classification models, the authors did experiments on CNN model groups including Inception-v3, ResNet-50, and VGG-16. It is unclear that if the superior explainability of AttEXplore stands still on other model groups, like vision transformers and MLP model groups. Extending the evaluation to diverse model architectures would further validate its effectiveness across different model types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WR4ihVexes", "reviewer": "Reviewer_r7kS", "summary": "The authors present AttEXplore, a novel feature attribution method for explaining Deep Neural Networks (DNNs), designed to address the growing need for transparent and interpretable AI models. AttEXplore is inspired by recent advances in the domain of transferable adversarial attacks, and it combines two essential components: \n\n(1) Feature attributions based on adversarial gradient integration, and \n\n(2) Frequency-based data augmentation techniques to enhance the robustness of explanations.\n\nIn their paper, the authors conduct an extensive evaluation of AttEXplore on the ImageNet dataset, employing three different DNN architectures (Inception-v3, ResNet-50, VGG-16). They compare AttEXplore's faithfulness and time complexity against other attribution methods, providing valuable insights into its performance. Furthermore, the authors perform an ablation study to better understand the impact of the key components within AttEXplore.\n\nTo facilitate the adoption of their method, the authors also provide the associated code, making it accessible for other researchers and practitioners seeking to utilize AttEXplore for their own applications.", "questions": "Have you compared your method against other local XAI techniques that attempt to generate more robust explanations by reducing noisy attributions?\n\nIn the Method section, Equation (4) includes index $i$ without a clear definition. It is unclear whether $I$ refers to a random draw of sample $x$ or a class index. What does $i$ stand for?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper offers several notable strengths:\n\n**Leveraging of Insights from Transferable Adversarial Attack Works:** The authors introduce a unique approach that draws inspiration from the field of transferable adversarial attacks. Particularly, their integration of frequency-based data augmentation techniques, aimed at reducing noise in heatmaps (usually caused by gradient shattering and/or high non-linearity of DNNs), is an innovative contribution. This demonstrates a thoughtful incorporation of knowledge from a related domain into the field of local XAI.\n\n**Comprehensive Model Evaluation:** The authors carry out an extensive evaluation, involving multiple deep neural network models and the ImageNet dataset. This thorough examination of AttEXplore's performance across different DNN architectures enhances the credibility of their findings and demonstrates its versatility and applicability.\n\n**Related Work Section:** The authors effectively contextualize their work within the existing literature, highlighting its relevance and significance in the broader research landscape.\n\n**Open-Source Code Availability:** A notable strength of this paper is the provision of the code associated with AttEXplore. This openness enables other researchers and practitioners to readily adopt and build upon the proposed method, promoting further research and development in the area of local XAI.\"", "weaknesses": "The paper exhibits, however, several weaknesses:\n\n**Clarity:** \nSome sections of the paper lack clarity, making it challenging for readers to grasp the key concepts. The abstract and introduction suffer from being overly specific without providing sufficient context. Additionally, important elements of the proposed method are not adequately introduced, leaving readers without a clear understanding of the approach.\n\nFigure 1, a critical element in conveying the method, could benefit from improved clarity. Enhancements to this figure would help readers better comprehend their motivation for AttEXplore.\n\nFurther, the paper frequently references the appendix but fails to provide specific locations. It would be highly beneficial if the authors included clear references to specific sections within the appendix, making it easier for readers to locate relevant supplementary information.\n\nTo me, some sentences in the paper do not make sense, such as \"DeepLIFT and LIME generate explanations for generic models which limits the understanding of model’s global behaviors.“ Why would it limit the understanding, and what do you mean with global behvior?\n\nIn the Method section, Equation (4) includes index $i$ without a clear definition. It is unclear whether $I$ refers to a random draw of sample $x$ or a class index. This ambiguity needs to be resolved to enhance the precision of the method description.\n\n**Evaluation Shortcomings:**\nThe evaluation has certain limitations. The absence of faithfulness curves is one deficiency as it hinders a comprehensive understanding of the method's performance. Moreover, the paper does not compare AttEXplore against state-of-the-art approaches designed to address noise in heatmaps, such as LRP (composite rules), SHAP, or LIME. Additionally, the time complexity comparison is limited to only a subset of methods, which restricts the scope of the evaluation and limits the ability to assess AttEXplore's competitive performance in terms of speed.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LIbpxCRivh", "reviewer": "Reviewer_UxR1", "summary": "The paper proposes AttEXplore, an attribution framework based on transferable adversarial attacks. The authors claim that there are shared principles between the decision boundary exploration approaches of attribution and transferable attacks. Based on such observation, the framework performs attribution analysis based on the design of a nonlinear integration path. The frequency-based input feature alterations make the framework transferable across different decision boundaries.", "questions": "Please address my concerns listed in the weakness section. I look forward to the authors' response and I will possibly consider revising the rating based on the response.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written with a clear storyline that motivates the work.\n2. The experiments on multiple attribution baselines are comprehensive.\n3. The idea of bridging attribution methods and adversarial attacks is novel.", "weaknesses": "1. Given the transferable claim of this new attribution method, it will be necessary to show sufficient visualizations of how the attribution generalizes to different types of task models other than image classification.\n\n2. The post-hoc explanation methods often suffer from false-positive responses, which highlight the semantic parts that are actually not relevant to the true grounding objects. The paper should elaborate more on how the framework overcomes such challenges. Similarly, the paper shall have more quantitative evaluations on whether the attribution is correctly aligning with the ground truth.\n\n3. Recent years have witnessed the development of white-box transformers (e.g., [1]), whose self-attentions naturally emerge as attributions for the model decision. It remains a question as how will AttEXplore outperform these interpretable-by-design approaches.\n\n[1] Yu et al., Emergence of Segmentation with Minimalistic White-Box Transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)