An Evolutionary Algorithm for Black-Box Adversarial Attack Against Explainable Methods
Authors: Phoenix Neale Williams
Jessica Schrouff
Lea Goetz
Venue: Accepted by TMLR ()

Abstract: The explainability of deep neural networks (DNNs) remains a major challenge in developing trustworthy AI, particularly in high-stakes domains such as medical imaging. Although explainable AI (XAI) techniques have advanced, they remain vulnerable to adversarial perturbations, underscoring the need for more robust evaluation frameworks. Existing adversarial attacks often focus on specific explanation strategies, while recent research has introduced black-box attacks capable of targeting multiple XAI methods. However, these approaches typically craft pixel-level perturbations that require a large number of queries and struggle to effectively attack less granular XAI methods such as Grad-CAM and LIME. To overcome these limitations, we propose a novel attack that generates perturbations using semi-transparent, RGB-valued circles optimized via an evolutionary strategy. This design reduces the number of tunable parameters, improves attack efficiency, and is adaptable to XAI methods with varying levels of granularity. Extensive experiments on medical and natural image datasets demonstrate that our method outperforms state-of-the-art techniques, exposing critical vulnerabilities in current XAI systems and highlighting the need for more robust interpretability frameworks.

Published in Transactions on Machine Learning Research (10/2025)
An Evolutionary Algorithm for Black-Box Adversarial Attack Against Explainable Methods
Phoenix Williams
phoeniz. williamss@skk com
GSK.oi
Jessica Schouff GSK.ai
jessica. Saciiim...schroufff@gs.. com
Lea Goetz
lea.z.goeto Basttttggoooooo com
GSK.oi
Reviewed on OpenReview:
https: // optp:/veeepe..ee...mmeee net/ forum? id-NUUS5
Abstract
The explainability of deep neural nelworks (DNNs) remains a major challenge developing trustworthy AI, particularly in high-stakes domains such as medical imaging. Although explainable Al (XAI) techniques have advanced, they remain vulnerable to adversarial perturbations, underscoring the need for more robust evaluation frameworks. Existing adversarial attacks often focus on specific explanation strategies, while recent research has introduced black-box attacks capable of targeting multiple XAI methods. However, these approaches typically craft pixel-level perturbations that require a large number of queries and struggle to effectively attack less gramular XAI methods such as Grad-CAM and LIME. To overcome these limitations, we propose a novel attack that generates perturbations using semi-transparent, RGB-valued circles optimized via an evolutionary stralegy. This design reduces the number of tunable parameters, improves attack efficiency, and is adaptable to XAT methods with varying levels of granularity. Extensive experiments on medical and natural image datasets demonstrate that our method outperforms statc-of-the-art techniques, exposing critical vulnerabilities in current XAI systems and highlighting the need for more robust interpretability frameworks.
1 Introduction
Deep neural networks (DNNs) have revolutionized the field of computer vision, driving significant advancements across variety of tasks Lin et al. (2014): Simonyan & Zisserman (2015) Springenberg et al. (2015 In
healthcare, artificial intelligence is becoming transformative force, offering groundbreaking solutions for diagnosis, treatment, and patient care Chaddad et al. (2023). Yet, the black-box nature of many DNNs raises
concerns regarding their explainability, accountability, and trustworthiness Quinn et al. (2021] Rane et al.
(223 : Rosenbacke al. (0244b). To address these issues and bolster trust, explainable artificial intelligence (XAI) has emerged as a pivotal area of rescarch. By undorstanding the docision-making processos of complex
DNNs, XAI fosters confidence among healthcare providers and patients Dosilovic al. (2018). Within the
computer vision domain, explanation methods frequently generate attribution maps that visualize feature importance, illustrating how different elements on an image contribute to DNN's predictions Simonyan
et al. 2014) Shrikumar et al. 2017 Selvaraju et al. 2017 Lundberg & Lee 2017
Böhle et al. 20244)
Despite the advancements of XAI, recent studies have revealed that many existing methods remain vulnerable adversarial inputs Tamam al. (002)): Huang et al. (2023); Baniecki Biecok (2024). Such inputs, generated through imperceptible perturbations (as illustrated Figure D.. have shown the ability
simultaneously alter both the XAI attribution maps and the classification outputs of DNNs Huang al.

---

Published in Transactions on Machine Learning Research (10/2025)
EvoAttack
SAFARI
NES
Perturbation Adversarial Imagge
Figure 1: Adversarial images and perturbations generated by the NES Tamam et al. (2023) SAFARI Huang et al. 2023) and the proposed EvoAttack algorithm when attacking an image from the Br35h dataset. Adversarial perturbations generated by NES and SAFART perturb every pixel of the image whereas the perturbation generated by the proposed EvoAttack method is constructed using set of 300 circle shapes
SHAPLEY
Saliency
Ispat x Grucient
DepLiFF
Grad-CAM++
Goud-CAM
LIME
Figure 2: Autribution maps generated by XAI methods. These methods are applied explain 'true' tumour classification of image from the Br35h dataset made by trained VGG-16 classifier. We observe that DeepLIFT, SHAPLEY, Saliency and Input Gradient methods produce attribution maps with high- granularity, emphasizing important pixels. In contrast, Grad-CAM, Crad-CAM++ and LIME generate attribution maps that capture more global features, highlighting broader regions of the image.
(023). The presence of these adversarial examples in real-world settings is particularly concerning in domains where DNN explainability is essential or legally required, such as autonomous driving Omeiza et al. 2022)
and hcaltheare Chaddad et al.] (2023); Hao et al. (2024): van der Velden et al. (2022)- Consequently, the
development adversarial allack techniques has become à crucial research direction for evaluating and
improving the robustness of XAT methods Tamam et al. (2023 Huang al. 2023
Early rescarch efforts primarily focused on crafting adversarial images by targeting specific XAI methods and leveraging knowledge of the underlying DNN architecture and parameters, setting known white-bor
attacks Wang et al. (2023;; Moosavi-Dezfooll et al. (2016); Zhang et al. (2020); Ghorbani et al. (2019)
However, because these approaches depend on access to internal DNN information, they often fail to generalize across different explanation techniques. Consequently, recent work has shifted toward the black- box scenario.
where only
pairs from the DNN and the XAT method are accessible Tamam el. al. (2023): Huang
ct al. (023). In this setting, most cxisting attacks employ meta-houristic approaches Tamam
2023
Huang 2023) inspired by evolutionary algorithms Li et. al. (2224).
While existing methods have successfully generated adversarial images against XAI techniques, they face key limitations. Firstly, these methods often require extensive querying of both the DNN and the XAI method to achieve meaningful distortions in altribution maps. This dependency poses substantia challenges in environments where query budgets are limited or expensive, whether due to financial constraints [[as ct (2018) Dhabliya al. (2024) or time restrictions Keddous al. (2023). As a result, conducting robustness evaluations that involve adversarial attacks with high query budgets becomes costly, impacting both financial resources and development time. This issue stems from the of population-based approaches
2

---

Published in Transactions on Machine Learning Research (10/2025)
SHAPLEY
Saliesey
Tsput x Gradient
DespUIFT
Grid-CAM CAM
GridCCMM+++
LIME
Original
Adverrsiial
PCC 0.81
PCC 0.76
PCC 0.72
PCC 0.85
PCC 0.8
PCC 0.96
PCC - 0.84
Figure 3: Figure shows adversarial images produced by the EvoAttack method when applied to the Task scenario (misclassification with preserved explanation), along with the respective generated attribution maps. Both original and adversarial explanations on the HAM10000 images are visually similar with PCC values of 0.7 and above. Both sets of explanations highlight seemingly relevant regions of the image, however, all adversarial images cause the underlying VGG-16 DNN to misclassify.
combined with the inherently high-dimennsional nature of the search space for instance, attacking an image from the HAM10000 datasel [sccandl al. (2018) with dimensions (450 x 600 x 3) results in searching through a space of 810, 000 dimensions. Secondly, existing attacks often overlook the varying granularity of XAI methods' explanations when dosigning perturbations, shown in Figure Current approaches tend to modify all pixels independently, which is effective when largeting XAI methods that produce detailed, pixel-level explanation maps, such as SHAPLEY or Saliency methods. However, these approaches struggle against XAI methods that emphasize broader, global regions, leading to increased robustness in methods like Crad-CAM and Grad-CAM++ Huang et al. 2023 This oversight highlights the need for more adaptive allack strategies that consider the explanatory granularity of different XAT techniques.
To address these limitations, we propose a novel attack method inspired by image approximation techniques from the computational art community Lambert et al. (2013): Garbaruk al. 2022 Tian & Ha, 2022
Our approach generates adversarial perturbations using set. of semi-transparent, RGB-valued circles whose parameters are optimized through an evolutionary strategy. Constructing perturbations in this manner substantially reduces the search space that remains consistent across images of varying sizes, thereby improving attack efficiency. Moreover, by constraining the size of the shapes, our attack can effectively target XAI method that produce attribution maps with different levels of granularity.
The remainder of this paper organized follows: Section reviews related works, highlighting key contributions and limitations. Section introduces the proposed attack and provides detailed explanation of its implementation. Section presents the evaluation metrics and experimental results, followed by a comprehensive analysis. Finally, Section concludes the paper and discusses potential directions for future research.
2 Related Works
Adversarial attacks on DNNs have become one of the most active research areas within the machine learning community, predominantly focusing targeting DNN classifiers Williams & Li (2023b); Dong et al. (025
Ilyas et al. (0018;; Williams & Li (202aa) Madry et al. (017): Andriushchenko al. (222)). More recently,
there has boen increasing interest in exploring the effects of adversarial perturbations on attribution maps
produced by XAI methods, addressing both white-box Heo el. al. (2019); Moosavi-Deziooli el al. (2016);
Zhang et al. (2020); Ghorbani et al. (2019); Kindermans et al. (2019): Subramanya et al. al. (2019): Dombrowski
et al. (2019); Kuppa Lo-Khad (2020) and black-box Tamam et al.
(2023) Huang & Zhang (2020) scttings.
Our work situates itself within this domain by focusing on the black-box scenario. Unlike existing black- box
approaches that generate pixel-wise perturbations, our method employs semi-transparent circular shapes,
3

---

Published in Transactions on Machine Learning Research (10/2025)
SHAPLEY
Salioncy
Impal x Grackut
DeepUIFT
Grad-CAM
Grad-CAM+++
LIME
Original
Adverarial
POC
POC 0.32
PCC P00--0.0
PCC 2036
PCC
PCC POD -0.17
Figure 4: Figure shows adversarial images produced by the EvoAttack method when applied to the Task 2 scenario (correct classification with distorted explanation), along with the respective generated attribution maps. The original and adversarial explanations on the HAM10000 images are visually dissimilar with PCC values of -0.04 and below. Despite the distortion, all adversarial images are correctly classified by the VGG-16 DNN.
which substantially reduces the number of optimizable parameters and can be easily adapted to various XAI methods, addressing key limitations of current strategies.
As DNNs become increasingly deployed within real-world applications, cnsuring their docision-making processes are interpretable is essential (Doshi-Veles et. al. (2077 underscoring the importance of developing robust and accurate explanations. Adversarial attacks against XAI methods are designed to assess the sensitivity of explanations to minor changes in input images A Iivare-Meels & Jaakkola (218)). While traditional adversarial attacks on DNNs focus on inducing misclassiffcation, attacks against XAI methods aim to challenge the reliability of the provided explanations by either 1) causing a minimal distortion to an attribution map whilst causing misclassification Huang et al. (2023). or 2) maximizing the distortion of an attribution map whilst
maintaining correct classification Tamam et al. (2023) (shown in Figures and
In their pioncering work, Ghorbani et al. introduced the concept of adversarial attacks XAT methods by targeting baadintt-basdd feature attributions of convolutional DNNs. The authors iteratively perturbed inpuls in the direction that altered the explanation's gradient. Concurrently, research by Kindermans el. al. highlighted the sensitivity of explanations to slight input transformations, although they did not directly proposc methods for constructing such attacks. Subsequent. works exploiting gradient information have constructed adversarial images by altering all pixels in the image Zhang el al. (2020) addition to localised regions (patches) Selvaraju et al. (017
Whilst most existing methods focusing the whitc-box setting, recent efforts have shifted toward developing black-box approaches. In the black-box scenario, only the input-output pairs of the DNN and XAI methods are accessible, allowing these techniques to be applied to any explanation method that produces an attribution map Huang et al. (202)). Existing black-box techniques often employ heuristic optimization methods to craft an adversarial image by solving a constructed loss function. Tamam et al. utilize a Natural Evolutionary Strategy (NFS) optimiser WWierstra et al. (2014) to minimize the loss function previously proposed in Dombrowski et al. 2219 This attack itcratively estimated the gradient of the loss function by computing the finite-difforencos between set of points sampled from a normal distribution. Conversely, the attack method by [uang et al. does not rely any gradient information. Instead, they adapt the POBA-GA genetic algorithm developed by CChn et al.] to evolve a population of solutions through crossover and mutation genetic operators. The authors demonstrate the superior performance of their method by targeting both gradient-based and perturbation-basel explanation methods.
Despite recent advances in black-box adversarial attacks against XAI methods, the large number of queries required for both the DNN and the XAI method raises concerns about their practicality in financially Dhabliya et al. 2024 time-constrained Keddous et al. 2023) scenarios. Moreover, the extensive computational effort.
4

---

Published in Transactions on Machine Learning Research (10/2025)
needed to generate adversarial examples limits the realistic evaluation of explanation robustness Wu et al. (2021). In addition, existing approaches overlook the granularity of XAI explanations, typically perturbing input images at the pixel level. While such fine-grained modifications perform well against highly detailed methods such as saliency maps and DeepLIFT, they struggle to succoed when attacking XAI techniques that locus on less granular features, such as Grad-CAM.
For a comprehensive survey of adversarial attacks on XAI methods, readers are referred Baniecki & Blecek 2224 A detailed discussion of black-box adversarial attacks against image classifiers is also provided in Appendix Section A.11
3 Proposed Method
The goal of method is to generate adversarial images that either (i) induce misclassification while minimizing distortion to the attribution map or (ii) maximise the distortion of the attribution map while preserving the model's original classification Huang et al. (2023), all under constrained query budget. We design the perturbation as a set of semi-transparent, RGBB-vllue circles, thereby reducing the search-space dimensionality to the circles' attributes. Similar to existing approaches, we generate adversarial perturbations by optimizing a distance-beatd based objective function. We begin this section by formulating the problem, followed by a detailed description of each component of our proposed method. The overall structure of our approach is summarized in Figure within the appendíx.
3.1 Problem Formulation
Consider a trained DNN classifier f: X S [0, 1]2xwx3 R which takes single benign RGB image we wish to attack x X of height. h and width w, and outputs label y argmax fz(x), where P is the total pE(L.-..Pp
number of class labels. Further, let g( be an explanation function, which takes a trained DNN f and benign image x as imputs. In this work, we assume access to the output probabilities of the classifier f. and attack an XAI method g that outputs an attribution map g(.,)) R where its height and width match the input image
To preserve the semantic integrity of the image, we adhere to existing attack methods by constraining the perturbation size using the loo norm Huang et al. (2023); Tamam et al. (2023); Dombrowski et al. (009 Consequently, we aim to generate a perturbation 8 that solves the following optimization problems:
Task 1:
minimize 5 subject to
D(g(x), 5])
|10 Se, L(f:x + b.34) O, 05x 8
(1)
Task 2:
maximize subject to
D(g(x), g( ((xx((xxx)0 + 5))
|00 Se.
(2)
C(f:x + 8, 0g) > 0,
05 1
where D represents the distance measure between the attribution maps produced for the benign and adversarial image and controls the extent of the perturbation's 's impact on the benign image. We follow the setup of Huang el al. by defining D as the I/PCC where PCC is the Pearson Correlation Coefficient. We ensure that the value of the loss function C(-) negative when x + d results misclassification by defining the loss in the constraint as the marginal loss:
L(J;x 8,y) - S0.'
(3)
5

---

Published in Transactions on Machine Learning Research (10/2025)
Image
Image + Perturbation
After Projection
Figure 5: The process of repairing an adversarial image x + (described in equation the constraints of and where c = 6/255
to ensure it satisfies
where y corresponds to the true label of and y/ argmax fz(x) is label corresponding to class other than the true label y.
3.2
Perturbation Initialization
To alleviate the issues associated with the large number of tunable values of image perturbation, we
construct perturbation by overlaying N AGB-valued remi-transparent circles, drawing inspiration from
techniques prevalent within the computational art field Lambert et al. (2013): Garbaruk et al. (2022: Tian &
Ha (202))
In this work, we construct the adversarial perturbation through the concatenation of N shapes:
8 8². N
(4)
where @ denotes the concatenation operator and 8 the a-th shape applied to a blank array. the proposed attack, each shape 8", for a I,... ,N, represented by a vector consisting of seven elements: the centre's coordinates (2), the radius r" (Max Diameter) x (w- h), where 'Max Diameter' parameter controlling the size of the circles, and w. h the width and height of the attacked image, respectively. Finally, the vector includes the RGB values R Go Ba and the transparency 7 These elements are normalized to continuous values between 0 and 1 and are initially sampled randomly from a uniform distribution, 8 ~ ~0, 1). Constructing the perturbation in this manner reduces the number of optimization variables significantly frrmm h x w x 3 (which totals 150, 528 for a typical (224, 224, 3) ImageNet image) to N x 7, where is the number of circle shapes used to construct the perturbation defined by the user. Importantly, this perturbation construction method is also invariant the image size.
tunable
3.3 Adversarial Image Construction
Given an adversarial perturbation and a benign image x, where both x and d belong Rxxxxx the corresponding adversarial image x" is generated by equation which overlaps all shapes from ō onto To ensure x complies with the constraints of [ and image follows:
project the pixels of the constructed adversarial
X; + if x; + x;
x
x <x-E,
5
otherwise,
where a pixel index. For greyscale images, we convert the RGB values of each circle to their greyscale equivalents before placing them onto the benign image. We visualize the projection process in Figure which ensures the constraints and are satisfied.
3.4 Perturbation Optimization
To optimize the properties of each shape 8ah we utilize a singlc-solution evolutionary strategy known (1 + 1)-FS. In each iteration, child solution 8t is generated by stochastically modifying its parent 5* with

---

Published in Transactions on Machine Learning Research (10/2025)
values sampled from normal distribution I). The parameter is adjustable, allowing balance between iesooaation-sarcchie unexplored regions of the solution space- and explotation-ffnettunnng the current solution. A larger o facilitates exploration, while a smaller value encourages exploitation. The child solution replaces the parent if it demonstrates superior performance on the task at hand.
Traditionally, the selection process based solely on comparing the objective values of the parent and child solutions. However, due to the constraints outlined in equations 1 and (2), more nuanced selection process is warranted. Huang et al.] addressed this issue for (I) by proposing hcuristic method related to the dynamics of the evolutionary population, and resolved (2) by multiplying the objective value of solutions not satisfying the constraint with -1. Alternatively, Tamam ct al.) proposed methodology that combines 'D and & using a weighted sum of its estimated gradients. Nonetheless, the single-solution architecture of our proposed method limits its ability to incorporate these existing strategies.
Instoad, wc draw inspiration from cvolutionary computation rescarch that addresses complex constrained optimization problems using dominance funetions Coello Coello & Mezura-Montes (2002); Williams & Li (223bb: Williams et all 2023)). similar to techmiques employed in the multi-objective optimization domain Deb
2001 For each task, child solution 8" replaces its parent 8 if any of the following conditions satisfied:
Definition 1 (Task 1 Domination)
[(8") 2 and [()*)) < 0
Both L(0*) < 0 and L(8**) < O and 9(x(x).((+++")) + D(x(x), 8*))
Both [(6*) 0 and (()*) and L(6 [(8*)
Definition 2 (Task Domination)
[("*) S and L(8*) > 0
Both C(8*) > 0 and L()*) > and D(((x), > D(x(x),
Both C(8*) < 0 and (()") < and L(8*) < L(8").
These domination definitions are designed to guide our attack method in generating perturbation values that either minimize (for task 1) or maximize (for task 2) the distortion of the explanation, while simultaneously satisfying the respective classification constraints.
4 Experiments
Most cxisting studies assess their methods by attacking explanation techniques applied to ImageNet trained classifiers. In contrast, our work additionally focuses on classification lasks within the medical imaging domain, where the explainability of DNN's decisions is critically important Chaddad al. (2023); Hao et al. 2024]
van der Velden et al. (022). Therefore, target explanation methods applied to DNN classifiers trained
three distinct medical image datasets and ImageNet. The experimental setup is outlined in Section 4.1 which
is followed by comparative analysis of leading attack methods SAFART HHang et al. (2023)
NES Tamam
lt al. [2023] and Square Attack Andriishcheenoo et al. (2020). detailed in Section 4.2 Subsequently. we
utilise our proposed method to rank the robustness of various XAI techniques, as presented in Section 4.3
Section 14 we evaluate XAI methods applied to adversarially trained models that incorporate adversarial
images generated by our attack method within their training processes. Lastly, Scction 4.5
presents an ablation study that examines the significance of different components and parameters of our proposed approach.
4.1 Experimental Setup
Datasets: We evaluate XAI methods on deep neural network (DNN) classifiers trained across three medical imaging datasets well ImageNet. The H0M00000 dataset Tschandl al. (2018) contains approximately

---

Published in Transactions on Machine Learning Research (10/2025)
10, 000 dermatology images of pigmented skin lesions spanning seven diagnostic categories, including malignant
cases. The Br35h dataset Hamada (2020) consists of 3, 000 brain MRT scans, annotated to indicate the presence or absence of tumours. The COVID-QU-Ex dataset Tahhrr et al. (2021) provides 33, 920 chest X-rays classified into three categorics: COVID-19, pneumonia, and normal. Following prior work Huang et al. 2023
Tamam et al. 20023) we randomly sample 100 correctly classified images from each dalasel's 's lest or val validation split to conduct adversarial attacks. All images are resized to (224 x 224 x 3) prior to input into the DNN
models.
Explanation and Classifier Settings: In this study, we explore seven XAT methods to assess their robust-
ness and efficiency. Specifically, we evaluate Grad-CAM Selvaraju et al. 2017), Grad-CAM CAM++ (hattooaddyyy
ie all. 2017). Saliency Maps Simonyan et al. 2014]; Selvaraju et al. (2017). DeepLIFT Shrikumar et al. 2017).
Gradient x Input Shrikumar et al. (2017), LIME Peng & Menzies 2021p, and Shapley Lundberg & Lee (017)) p.
For the classificrs, we employ the architectures of MobileNet Howard et al. (2019). AlexNet Krizhevsky
et al. (2012), and VGG-16 Simonyan & Zisserman (2015). Given the relatively small size of our medical
datasets, we fine-tune models pre-trained on ImageNet using the PyTorch library Paszke et al. (2119) Each
model undergoes fine-tuning over 10 epochs, with a batch size of 32 and learning rate of x 10 utilizing
the ADAM optimizer Kingma & Ba (015) and cross entropy loss. The datasets are divided into training, validation, and testing subsets with a ratio of 00%/10%/209/ Detailed performance metries are provided in Section A.2 in the appendix. All experiments were executed on an NVIDIA RTX A6000 GPU system.
Parameter Settings: To ensure the generated perturbations cause minimal semantic alterations to the attacked images, we set the value of e using previous adversarial attack research targeting image classification DNNs Rusu et al. (022). For RGB images from the ImageNet and HAM10000 datasets, we set 8/255, whereas for greyscale images from the Br35h and COVID-QU QU-Ex datasets, we set 6/225 Dong et al. 2025) Consistent with prior studies, we set K = 5000 Williams & Li (003a b) for all attacks. As outlined in Section g our approach involves three adjustable parameters: o, N, and "Max Diameter'. Here, a governs the exploration of the method, N signilies the number of shapes used to create the perturbation, and 'Max Diameter' determines the largest possible size of the perturbation circles. The specific values for these parameters are listed in Table 4 with justification provided in Section 4.5
For evaluating the effectiveness of the proposed method, we follow previous works and adopt PCC for measuring of the distortion caused to the attribution maps Huang et al. (2023). PCC values near indicate strong positive correlation, values close to 0 imply no correlation, and values approaching -1 suggest strong negative correlation. Additionally, we report the percentage of generated adversarial images that meet the constraints specified in ① and @ To ensure fair comparison, distortion measurements are only conducted on images that satisfy their respective task constraints. For ranking the robustness of XAI methods, utilize the dominance relations described in Definitions I and Z to evaluate and rank the considered XAI methods.
Performance Metrics:
Given the stochastic nature of our method, each experiment is repeated over 10 different random seeds. For each metric, we aggregate its value across all model architectures. To statistically verify whether the results achieved by our method are significantly different to other algorithms, we employ the Wilcoxon signed-rank test Wilcoxon| [992) a 5% significance level, as is standard practice within the evolutionary optimization
field Williams & Li 2023a Storn & Price (9977
Deb 2001
4.2 Results Analysis
To ensure a fair comparison among EvoAutack, SAFARI, and NES, we fix the population size of SAFART and NES to 50, which corresponds to 100 iterations of their optimization cycles. For Square Attack, we adapt its selection strategy by replacing it with the domination functions defined in and 2
Constraint satisfaction: Table D reports statistical results for the proposed method and comparative attacks; arrows indicate whether larger or smaller metric values are preferable. Task enforees image misclassification, whereas Task requires the original correct classification to be preserved. The Task 2 satisfaction rates below 100% for EvoAttack and SAFARI indicate that even random perturbations sometimes
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Table presents the Pearson Correlation Coefficient (PCC) along with the percentage of images that satisty the respoctive task constraint when attacking images from the HAM10000, Br35h and COVID-QU-Ex datasets. We provide the mean each metric over 10 runs and its variance inside the brackets.
DeepLIFT
Task
Task 2
Method
Coustraint Sutisfied (*)
PCC (t)
Constroint Satisfied (t)
PCC (L)
Square Attack
84.87%(2.08
0.52(0.1)
77.34%
(1.977
EvoAttack
83.91%((.718)))
0.7(0.08)
78 28..33%(2 (2. 164)
0. 0.03/0.00 080)
NES
30.84% (1.53)
0.2(0.096)*
61.02%((2.63]]
0.35 105)
SAFARI
36.81%(( 639)1
0.46(0.117)
65.72%
0.53(0.114))
Saliency
Task
Task
Mothod
Coustraint Satisfied (*)
PCC (t)
Constraint Satisfied (1)
PCC (4)
Square-Attace
83.07%(1.99) (1.999)
0.4(0.097)*
81. (6.46%((6.33))
0.24(0.097)f
EvoAttack
81.94 81.94%((2.o) 463)
0.57(0.077)
83 (1 489)
0.0 081)
NES
30.85%(( 1982
0.2(0.092)+
61.28%
0.29(..112)+
SAFART
32.25% (1.14))
(0.088)
64.91% (22.328)
0.49().0999+
Grad- CCM
Task
Task 2
Method
Constraint Satistied (*)
PCC (f)
Constraint Satisfied (1)
POC (+)
Square-Attare
82.71%((.022)
(..69)
86.3%(1.949)
0.3(0.161)
EvoAttack
81.79%(2.063)
0.88(0.) (3))))
87 62% (2 186)
-0.57(0 (995)
NES
31.08%((.88) 802)
(6.26)
59.95% (6.55)
0.4.(
SAFARI
33.64(2.034)-
(.31(0
62.0%((.346))
0.55(0.2147)
Shapley
Task
Task
Method
Constraint Satisfied (*)
PCC (f)
Constraint Satisfed (1)
PCC (L)
Square- Attack
12.69%(..229)
(..2()
75.06%((1.2)
0.29(0.11)
EvoAttack
81.67%(1.777)
0.73(0.092)
75.83% 767)
0.04(0.107)
NES
(0.39%
(.25(0.1
(1.97)) 63.05%([1677) (1.67)
0.39(0.125)
SAPARI
32.98%((2.56) (2
0.46(0.112)
0.56(0.108)#
Input Gradient
Task
Task 2
Method
Constraint, Satisfied (*)
PCC (t)
Constraint Satisfied (t)
POC (L)
Square- Attack
94.17%(1.906)
0.4(0.105)+
75. (5.11%((.963)
0.24(0.103)+
EvoAttack
82.92%((1.9) (1.95)
0.57().072)
76.24% (1.759)
0.05 (0.08)
NES
30.37%(( 302)+
(01.00.
60.33%(1 (1.569+
0.28(0.107))
SAFARI
36.97%((2336))
(0.099
(1..%%(1.301)
0.41(0..04)
Crad-CAMD
Task
Task
Method
Constraint Satisfied (*)
PCC (f)
Constraint Satished (f)
PCC (L)
Square-Attack Attack
19.71%(116722
0.66().144)-
63.69%(2.299)
0.33 168)+
ExnAttack
81.72%(2 (33)
(.89()
84.73%(( 6664)
-0.46(0 243)
NES
30.37%((2.115)
(0.843.
60.52%(2.19)
0. 0.44(0.21)=
SAFARI
35.81%((.819)
(..11))
(5.58%((2.03)
0.6(0.1555
LIME
Task
Task 2
Method
Coustraint Sutished (*)
PCC (T)
Constroint Satished (t)
PCC (1)
Squaro-Attar
86.0 16.06%(4.143)
0.33(
(8.88%(3.893)
0.27(0.1)+
EvoAttack
85.14% (5.216)
0.88(0.071)
90. (5 177)
0.6(0.091)
NES
33.91%(..96)
(0.1(0
69.38%(3 (
0.41(0.103))
SAFARI
38.23%((1
(.28(0.138)
54.195(
0.55(0.09)
denotes the performance of the method significantly outperforms the compared methods according to the Wilcoxon signed-rank test Wilcoxon (1922 at the 5% significance level; denotes the corresponding
method is significantly outperformed by the best performing method (shaded).
cause the DNN to misclassify, preventing those attacks from producing perturbations that meet the Task 2 constraint. For NES, the ability to satisfy either constraint additionally depends the relative weighting used to combine the classification and attribution objectives.
Overall, EvoAttack outperforms both SAFARI Huang et al. 2023) and NES Tamam el al. (023) across XAI methods and datasets, particularly under tight query budgets. As noted in Section ! many existing approaches implicitly assume access to large query budgets, and their effectiveness degrades substantially when queries are limited. In our experiments, NES and SAFARI frequently fail to induce adversarial misclassification (Task 1) across datasets, consistent. with prior observations about the difficulty of generating adversarial
9

---

Published in Transactions on Machine Learning Research (10/2025)
HA
Br55h
COVII-QUEE
XAI Method
Task 1
Task
Task
Task 2
Task 1
Task 2
Dee
4.19 (0.982)
4.15 (1.269)
4.02 (1.535)
4.9 (1.368)
4.04 (1.048)
2.71 (0.825)
Grad-CAM
5.48 (1.055)
5.88 (1.649)
4.14 (1.529)
5.84 (1 (.39)
6.2 (0.965)
5.71 (1.286)
SHAPLEY
4.87 (0.875)
3.12 (0.968)
4.69 (1.467)
5.0 (1.535)
4.31 (1.012)
3,04 (0.943)
Input x Gradient
2.43|0.86] 86)
2.82 9488
3.93(1. .377
4.15(1. 6377
3.34 (0.947)
2.9 (1.02)
Crad-CAM+
6.17 (1.516)
5.49 (1.669)
4.42 (1337)
5.16 (2.013)
6.52 (0.726)
5.89 (1.489)
Saliency
3.38 (0.963)
2.93 (1.037)
4.51 (1.34)
4.95 (1.367)
2. 2.54()
4.35 (1.262)
Table 2: Table presents the robust ranking of XAT methods across each task and dataset using the proposed ExoAttack as the attack method. We provide the mean and variance of cach metric over 10 runs.
images under constrained-query settings. By contrast, EvoAttack achieves substantially higher success rates in producing adversarial examples while better respecting the query and classification constraints.
Square Attack shares conceptual similarities with EvoAttack in that it also employs shape-based perturbations. Our experimental results indicate that Square Attack achieves a comparable success rate in inducing misclassffication This effectiveness can be attributed to two main factors: (i) its perturbations are bounded within the range [-e, c], a constraint known to be effective for generating adversarial examples, and (ii) its square-shaped perturbations have been shown to be particularly effective in misleading classifiers.
When evaluating performance under the Task 2 condition, the differences among NES, SAFARI, and EvoAttack are less pronounced. Since Task 2 requires perturbations that preserve the original classification, this outcome can be attributed to the inherent robustness of the underlying DNN classifiers to small pixel-level perturbations. Compared with Square Attack, EvoAttack achieves higher success under this constraint: although Square Attack is highly effective at inducing misclassification, the strength of its perturbat ions diminishes its ability to maintain correct classifications.
When analyzing constraint satisfaction rates across datasets (see Tables 13
4 in the Appendix), all attack methods exhibit reduced performance on the Br35h dataset. Despite the presence of clearly defined regions of interest across all datasets, diagnostic cues in Br35h images tend to be more explicit- such as the visible presence of a tumour. Consequently, the classifiers may have learned more robust features (e.g,, tumour presence) rather than relying on non-robust or spurious correlations, a phenomenon commonly referred to as shortcut learning Wang et al. 20244
12 and
Finally, only minor discrepancies in constraint satisfaction rales are observed across different XAI methods. This uniformity is expected for EvoAttack, Square Attack, and SAFARI, as all prioritize satisfying the task constraints before optimizing for explanation distortion.
Explanation distortion: In assessing the distortions to explanations caused by the allack methods, EvoAttack consistently outperforms Square-Attade Attack, SAFARI and NES across all experimental scenarios, demonstrating significant superiority in the majority of cases. Notably, there is a pronouneed performance disparity when targeting XAI methods that generate less granular, more region-focused explanations, such as Grad-CAM, Grad-CAM++ and LIME. Conversely, when attacking more granular XAT methods like DeepLIFT, Saliency maps, SHAPLEY, and Input x Gradient, the performance gap between EvoAttack and the other attack methods narrows. This outcome is anticipated due to the less granular perturbation strategy employed by EvoAttack, emphasizing the importance of accounting for the granularity of XAI explanations when designing perturbations. However, it also underscores the adaptability of the proposod method to effectively larget both granular and non-granular XAI methods.
Despite its success in satisfying the task constraints, Square Attack struggled to distort the attribution maps produced by the XAI methods. This limitation can be attributed to several factors. First, Square Attack samples perturbations exclusively on the constraint boundary [-e, which is effective for classification attacks but can be overly restrictive for more complex objectives such as explanation distortion. Second, the attack proceeds by initially perturbing the entire image with vertical strips and then adding one square at a time; however, once a square is accepted, it cannot be removed, making the optimization process being prone to local optima. In contrast, EvoAttack can flexibly adjust perturbations to cover either broad regions
10

---

Published in Transactions on Machine Learning Research (10/2025)
localized areas of the image. Third, as the attack progresses, Square Attack iteratively decreases the size of its perturbation squares. While this strategy is advanlageous for inducing misclassification, it proves less effective for explanation methods that emphasize coarse-grained features (e.g., Grad-CAM), where fine perturbations exert minimal influence. For more fine-grained XAI methods, the sparse structure of Square Attack's perturbations similarly reduces iveness EvoAttack, however, adapts its perturbations to varying levels of granularity through its attack parameters, enabling stronger distortion of attribution maps.
Comparing task performances, the attack methods generally achieve greater success in maximizing the distortion of explanations for correctly classified images rather than minimizing distortion for misclassified ones. This indicates that XAI-generated explanations are more susceptible to Task 2 attacks, where the classification remains accurate, but the explanation is distorted. Although this scenario poses less risk to patient safety due to correct disease classification, it could undermine trust in Al systems among medical practitioners, potentially reducing their willingness to rely on AT for assistance Rosenbacke et al. (2024a).
Additionally, performances vary across different XAI methods among the attack strategies. EvoAttack notably manipulates less granular explanations, such as Grad-CAM, Crad-CAM+ and LIME, more effectively in all experimental setups compared to the gramular methods, underscoring their specific vulnerabilities to structured perturbations. Among gramular XAI methods, the greatest challenge appears in attacking Saliency maps and Input x Gradient, particularly for Task-1. For example, when iargeting images from the HAM10000 and COVID-QU-EX datasets, no attack achieves an average PCC value above 0.6, which is a benchmark for consistency between explanations. Conversely, for Task 2, EvoAttack able to reduce the average PCC of those XAI values below 0.4, indicating inconsistent explanations Huang et al. 20233 highlighting the greater vulnerability of XAI methods when distorting correctly classified images.
More visual comparisons of adversarial images and explanations is provided in Section A.6 of the Appendix. We also compare the performance of the proposed and compared attack methods over varying query budget in Section A.8 along with their attack speed in Section A.10
4.3 XAI Robustness Comparison
The findings in Section techniques, showcasing ils utility for robust.nss evaluations However, the combination of the task's 's objective and constraint (as detailed in Section 3) makes it difficult to use a single metric value for ranking. Therefore, to rank the robustness of the targeted XAI mothods, we employ the task-specific domination relations definod in and
4.2
highlight the superior performance of the proposed method over existing attack
For each attacked image, we utilize non-dominated sorting Deb et al. (0022) to rank each XAI method based the performance of EvoAttack in targeting the image, where lower ranks correspond to poorer attack results, indicating greater robustness of the XAI method. This procedure is repeated across all 100 images for each model architecture. We repeat this procedure over the 10 different random seeds, with the average rank serving as a measure of the XAI method's overall performance.
The final robustness ranking of XAI methods, as presented in Table 2 indicates that Input x Gradient frequently achieves better average rankings across most attack instances. Despite the Saliency method attaining better average PCC values, it achieves higher overall rankings on out of 3 datasets. This discrepancy arises because the metric values in Section 4.2 focused solely on attack instances that satisfied the constraints, whereas this ranking accounts for all evaluated images. For example, during Task-1 attacks on images from the HAM10000 datasct (sce Table 12 within the Appendix), the Saliency method shows greater resistance to allacks, resulting in a higher average PCC value. By employing the domination relation, the ranking considers the attack's ability to satisfy the constraint, demonstrating that EvoAttack achieves lower constraint satisfaction when targeting Input x Gradient compared to Saliency, leading to Input x Gradient's superior average rank. Finally, in Task-1 attacks on COVID-QU-Es images, the Saliency method demonstrates superior robustness, while DeepLIFTT proves most resilient against Task-2 attacks.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Table presents the Pearson Correlation Coefficient (PCC) along with the percentage of images that satisfy the respective task constraint when attacking XAI method applied to adversarial trained classificrs.
We provide the mean and variance of each metric over 10 runs.
DeepLIFT
Task
Task 2
Dataset
Constraint Satisfied (1)
PCC (1)
Constraint Satistied (1)
PCC (4)
HAM10000
98.06 (2.04)
0.71 (0.118)
97.01 (2.544)
0.08 (0.162)
Br35h
56.00 (2.376)
0.66 (0. (48)
56.99 (2. 182)
0.56 (0. 158)
COVID-QU-Ex
98.01 (2.577)
0.57 (0.154)
61.08 (2.508)
-0.0 (0.235)
Saliency
Task
Task 2
Datasel
Constraint, Satisfied (T)
PCC (f)
Constraint Satisfied (1)
PCC (4)
HAM10000
98.07 (2.214)
0.54 (0.112)
98.91 (2.782)
0.17 (0.16)
Br35h
59.29 (2.816)
0.65 (0.124)
59.0 (2.218)
0.51 (0.146)
COVID-QU-EX
98.95 (2.46)
0.46 (0.11)
67.01 (2.86)
-0.1 (0.083)
Grad-CAM
Task 1
Task 2
Dataset
Constraint Satisfied (7)
PCC (1)
Constraint Satislied (1)
PCC (4)
HAM10000
98. 07.07 (2.034)
0.77 (0.238)
99.94 (2.531)
-0.3G (0.391)
Br35h
40.0 (2.46)
0.60 (0.209)
48.0 (2.334)
0.68 (0.365)
COVID-QU-Ex
98.92 (2.31)
0.78 (0.268)
75.06 (2.367)
-0.73 (0.481)
Shapley
Task
Task 2
Dataset
Constraint Satisfied (?)
PCC (1)
Constraiut Satistied (1)
PCC (L)
HAM10000
97.9 (2.185)
0.76 (0.11)
96.91 (2.139)
0.13 (0.176)
Br35h
41.0 (2.687)
0.63 (0.154)
42.0 (2.385)
0.56 (0. 139)
COVID-QU-Ex
98.96 (2.866)
0.59 (0.151)
60.93 (2.174)
-0.0 (0.223)
Imput x Gradient
Task
Task 2
Datasel
Constraint Satisfied (T)
PCC (f)
Constraint Satisfied (f)
PCC (4)
HAM10000
97.94 (2.476)
0.48 (0.123)
96.99 (2.005)
0.12 (0.147)
Br35h
52.03 (2.82)
0.67 (0.131)
41.0 (2.913)
0.56 (0.118)
COVID-QU-Ex
97.95 (2.708)
0.52 (0.1)
60.9 (2.6)
0.01 (0.087)
Grad-CAM++
Task
Task 2
Dataset
Constraint Satisfied (*)
PCC (f)
Coustraint Satisliod (f)
PCC (1)
HAM10000
98 02.02 (2.541)
0.86 (0.154)
99.9 (2.416)
-0.33 (0.401)
Br35h
46.0 (2.105)
0.49 (0.259)
46.0 (2.344)
0.69 (0.34)
COVID-QU-Ex
97.92 (2.327)
0.87 (0.154)
75.06 (2.933)
-0.73 (0.482)
LIME
Task
Task 2
Dataset
Constraint Satisfied (?)
PCC (t)
Constraint Satistied (1)
PCC (L)
HAM10000
99.31 (1.409)
0.77 (0.101)
99.4 (0.18)
-0.41 (0.392)
Br35h
38. 11 (1.201)
0.49 (0:259)
46.0 (2.344)
0.69 (0.34)
COVID-QU-Ex
97.92 (2.327)
0.87 (0.154)
75.06 (2.933)
-0.73 (0.482)
4.4 Evaluation of Adversarial Training
The results in Section 4.2 demonstrate the effectiveness of the proposed attack in compromising XAT methods and outperforming existing attacks. Since adversarial training has emerged as promising defence against adversarial attacks, we next evaluate its potential to improve the robustness of XAI methods against Attactack. A comparison with the black-box defence mechanisms proposed by Qin et al. and Cohen et al. is further provided in Section A.9 of the appendix.
We implement adversarial training on the HAM10000, Br35h, and COVID- QU-Ex datasets following a similar procedure established in prior work Madry et al. (017): Chernyak et al, (2021). In each training iteration, all images are augmented with random EvoAttack perturbations producing batches that contain both benign and adversarial examples and thereby doubling the effective batch size. To expose the DNN to a diverse range of perturbation structures, the parameters N and Maximum Diameter are randomly sampled for each perturbation from the grid used in our ablation study. A detailed description of the adversarial training procedure, along with the performance metrics of the adversarially trained classifiers, is provided in Section A.4
Adversarial Training Setup:
Task 1:
The results presented in Table 3 illustrates the impact of adversarial training on different XAI methods. First, we see that adversarially trained HAM10000 DNN classifiers become more vulnerable to
12

---

Published in Transactions on Machine Learning Research (10/2025)
adversarial attacks. This phenomenon is likely attributed to the large class imbalance in the HAM10000 dataset, which has shown to be an issue for adversarial training Wang et al. (2022). Comparing with Br35h and COVID-QU-Ex classifiers, we see adversarial training has improved their robustness against the proposed EvoAttack.
Comparing the performance of the different XAT methods, we observe that adversarial training consistently improves robustness across all methods. The effect is particularly pronounced for the less gramular methods, such Grad-CAM, Grad-CAM++, and LIME, which exhibit larger gains in robustness. Nevertheloss, these methods remain among the least vulnerable overall. For the more granular XAI methods, adversarial training substantially enhances the robustness of Saliency Maps and Input x Gradient, reducing their PCC values to around or below 0.6 across all three datasets. This suggests that, while adversarial explanations are not entirely inconsistent with their benign counterparts, their similarity falls close to or below the threshold of consistency, implying that distortions in the attribution maps may become perceptible.
Task 2: Similar to Task 1, we witness an enhancement. in robust.ness across all XAT methods which can be described by the increased average PCC values. In the case of Br35h images, EvoAttack's ability to meet the constraint is diminished, reflecting improved robustness across XAI methods, with all average PCC values exceeding 0.5. This suggests that EvoAtlack faces challenges in altering the original explanation to be inconsistent while ensuring the classifier predicts accurately.
Similar obscrvations arise with the COVID-QU-Ex dataset, where EvoAttack's success in mocting the constraint. is reduced, while the average PCC values for most XAI methods increase. This indicates that adversarial training has degraded EvoAttack's ability to distort explanations while retaining accurate classifier predictions.
Despite the impact of adversarial training in improving the robustness across XAI methods, EvoAttack is still able to distort attribution maps to PCC values of below 0.4, which indicates that inconsistency was achieved.
4.5 Ablation Study
The proposed method incorporates three tunable parameters: N, a and the maximum circle diameter 'Max Diameter' expressed as a percentage of the image. We employ a grid search over the parameter space to determine their optimal values. Specifically, we explore N E 100, 300, a E 0.1, 0.2, 0.3, and maximum circle diameters with ranges 20%, 30%, 40%, 50% of the original image size. These parameter ranges are based commonly set values used in the evolutionary [Skiscim & Golden (1983) and computational art Tian & Ha (2022) fields. To evaluate the performance of each parameler configuration, we conduct atlacks on each explanation for each task using VGG-16 ImageNet classifier with 100 correctly classified images from the validation set. To compare the performance of different paramcter configurations, we employ the same methodology as described in Section 4.3
Configuration Performance Analysis: As illustrated in Table i the optimal configuration varies significantly across XAI methods and tasks. Similar to previous studies that highlight the lack of consensus among XAI methods, this suggests that the vulnerabilities bet them may differ. Nonetheless, some patterns emerge across different XAI methods. Firstly, for Task 1, which aims to induce misclassification while preserving the explanation, the best-performing configurations consistently feature a maximum circle diameter of 80% of the image size. Conversely, configurations with smaller circle diameters perform worse. This indicates that utilizing larger local perturbations (i.e., larger circles) is more effective in influencing the DNN classifier while minimizing distortion in the attribution map.
Another observed pattern is that all optimal configurations use = 100 circles. A likely explanation for this is that adding more shapes increases the number of variables involved, which might demand a larger computational budget for effective optimization Williams et al. 2021 Eltaeib & Mahmood (2018). Keeping a constant. budget might lead to prematurely halting the optimization process, thus affecting performance negatively.
=
For Task 2, which aims distort the explanation while maintaining correct classification, we observe greater variation in the best performing configurations. However, similar configuration performances seen across
13

---

Published in Transactions on Machine Learning Research (10/2025)
Task 1
Tosls 2
Method
Diameter
Diameter
DespLiFFT
100
80.0
0.1
300
60.0
0.3
Salieney
7000
Grad-CAM
100
80.0
0.3
100
300
0.2
LIME
80.0
100
80.0
Shmpley
100
50.0
0.3
300
60.0
0.3
Tegor x Grrniion
100
80.0
0.1
300
60.0
0.3
Crad-CAM
80.0
100
800
Table 4: Chosen EvoAttack parameters for attacking the considered XAI methods.
different XAI methods. Specifically, the results indicate that XAI methods that produce similar granularity in attribution maps are effectively attacked with comparable parameter setups. For example, when targeting highly gramular XAI methods of DeepLIFT, Input Gradient, and SHAPLEY (sce Figure 2 the proposed method achieves superior performance using smaller diameter circles. Conversely, XAI methods Grad-CAM, Brad-CAM+HH and SHAPLEY, which highlight broader regions rather than individual pixels, are more susceptible to larger circular perturbations. This behaviour stems from altering the perturbations' gramularity by changing circle size, with smaller circles constructing perturbations more closely resembling pixel-level perturbations.
A surprising result from the ablation study was the performance of parameter configurations when targeting the Saliency XAI method. Unlike other granular methods, the Saliency method proved more vulnerable to medium to large circle perturbations, while remaining robust against smaller circles. This difference might be attributed to its level of granularity. Whereas SHAPLEY and DeepLIFTT produce sparse maps emphasizing specific pixels, Saliency maps highlight broader regions. Although the Input x Gradient method also emphasizes broader areas, its multiplication with the input image may also have an impact, resulting in being vulnerable to perturbations constructed from smaller circles.
These results underscore the advantages of the proposed attack method. By adjusting the size of the circular shapes, EvoAttack effectively manages the trade-off related to XAI gramularity when targeting explanation mothods-and an aspect that cxisting strategics lack. In conclusion, we recommend adopting the optimal configurations detailed in Table f for the XAI methods considered in this study. We provide the performance across all configurations in Figure 7 within the Appendix.
5 Conclusion, Limitations, and Future Work
This paper introduces a novel adversarial attack designed to attack computer vision XAI methods. Unlike existing approaches that modify each pixel of the benign image, our method constructs adversarial perturbations by concatenating RGB-valued circular shapes. We optimize the parameters of these shapes using a levollavvlu strategy, widely used optimization heuristic in evolutionary computation. To enhance the attack's efficacy, we conducted an ablation study assessing the influence of various parameters EvoAttack's performance across several XAI methods. The results demonstrate that larger circles effectively manipulate less granular XAT methods like Grad-CAM, Grad-CAM++, and LIME, whereas smaller circles yield better results against more granular XAI methods such as DeepLIFT and Input x Gradient. Compared to state-of-the-art attacks, the proposed method consistently outperforms them in distorting XAI attributions maps across all attack setups, showcasing its efficiency and effectiveness.
Conclusion:
Then, we leverage the EvoAttack method evaluate and rank the robustness of XAI mothods. Given the complexity of assessing XAI methods using both PCC and constraint metrics, we employed the EvoAttack domination relation for each task to rank the resistance of XAI methods with respect to each attacked image. By averaging the rank of each XAI method across all attacked images and underlying classifiers, we formulate a comprehensive ranking. This approach allowed us to incorporate all data regarding XAI distortion alongside constraint satisfaction, culminating in a unified ranking table for each dataset.
To counter the proposed attack, we developed an adversarial training procodure that incorporates random Att perturbations into the training process. Attacking adversarially trained models revealed enhanced robustness in most XAI methods; however, we noted decrease in classifier robustness the
14

---

Published in Transactions on Machine Learning Research (10/2025)
HAM10000 dataset, accompanied by improved XAI robustness. In the Task 2 attack scenario, adversarially trained classifiers decreased EvoAttack's effectiveness in redirecting the DNN classifier towards correct predictions, suggesting that while the classifier became more susceptible to random EvoAttack perturbations, it also developed a stronger resistance to subsequent manipulations. This study underscores the potential of adversarial training as a crucial stralegy for defending against attacks, addition to the utility of considering both classifier and XAI performance when evaluating the robustness of human-in-the-loop systems. Nevertheless, our study highlights the need for further exploration, marking this as a vital area for future research.
This study focuses on evaluating and enhancing the robustness of XAI techniques, with an emphasis on medical imaging datasets. While the proposed method demonstrates promising results, there are limitations and avenues for future research. Firstly, the paper-prrameters were optimized using basic grid-scarch approach, aimed at analysing the impact of varying paramcter values. In future work, more advanced hyper-parameter optimization frameworks, such as Bayesian Optimization methods assisted by
Limitations and Future Work:
models, should be explored Snoek et al. (012): Zhang al. (2023
Additionally, this study employed a predefined Lo constraint e based on existing recommendations. Exploring minimum-norm attacks, which identify the minimal e value necessary to compromise an AI system could provide deeper insights into the XAI and classifier vulnerabilities Williams & Li (202a Moreover, while this research examined perturbations that modify all pixels, future studies should consider alternative perturbations, such as sparse attacks, where only a limited number of pixels are altered.
Finally, this study assumes access to both classifier output probabilities and attribution maps. In practico, such access may be restricted. Evaluating robustness under these conditions could be guided by decision-only or transfer-based attack paradigms Ilyas et al. 2018); Papernot et al. (2016); Guo et al. (2119:; Cheng et al. (2019), enabling the development of novel robustness assessments tailored to explainable AT systems.
Our experiments highlight the potential of adversarial training in mitigating the impact of EvoAttack on both classifier and XAI robustness. At the same time, they reveal EvoAttack's ability to manipulate explanations to appear cither consistent or inconsistent, depending on the task, while still satisfying the respective constraints. Future work should therefore explore more sophisticated adversarial training strategies aimed at strengthening the robustness of both DNN classifiers and XAT methods. One promising direction ion is to replace random perturbations with targeted E'voAttack perturbations during adversarial training, even under limited query budgets, to expose models to more harmful perturbations and improve resilience.
To encourage further research in this domain, we will publicly release our implementation, datasets, and evaluation scripts upon acceptance of the paper.
Broader Impact Statement
In this work, we introduce novel adversarial attack method against explainable AI (XAI) techniques that can account of the varying gramularitiss in explanations, well reducing the dimension of the search space.
We apply our proposed attack to the robustness ranking of various XAI methods across three different medical image datasets. This study underscores the necessity of evaluating both classifier and XAI system robustness.
Within the healthcare domain, our research demonstrates two significant risks: first, the possibility of medical professionals trusting incorrect AI diagnoses due to soemingly plausible explanations, potentially endangering patient safety; and second, the risk of decreasing trust between healthcare practitioners and Al systems due to distorted explanations of accurate diagnoses, which could slow down the diagnostic process by necessitating additional human evaluations. To address these risks, we explore different mitigation strategies, such as adversarial training, which showed promise in enhancing the resilience of XAI systems against adversarial threats.
We hope our work will lead to further research into adversarial training stratogies and encourage practitioners to rigorously test the robustness of XAI systems before deployment. Our ultimate goal is to advance the safe and effective integration of AT in critical domains like healthcare.
15

---

Published in Transactions on Machine Learning Research (10/2025)
References
David Alvarez-Melis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR, 2018.
Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, and Mani B. Srivastava. Genattack: Practical black-box allacks with gradient-free optimization. CoRR, 2018.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient box adversarial attack via random search. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII. Springer, 2020.
Hubert Baniecki and Przemyslaw Biecek. Adversarial attacks and defenses in explainable artificial intelligence: A survey. Inf. Fusion, 2024.
Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XII, Lecture Notes in Computer Science, 2018.
Moritz Böhle, Navdeeppal Singh, Mario Fritz, and Bernt Schiele. B-cos alignment for inherently interpretable cnns and vision transformers. IEEE Trans. Pattern Anal. Mach. Intell., 2024.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Ahmad Chaddad, Jihao Peng, Jian Xu, and Ahmed Bouridane. Survey of explainable Al techniques in hcalthcare. Sensors, 23(2):634, 2023.
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. Grad-cam Generalized gradient-based visual explanations for deep convolutional networks. CoRR, 2017.
Jinyin Chen, Mengmeng Su, Shijing Shen, Hui Xiong, and Haibin Zheng. POBA-GA: perturbation optimized black-box adversarial attacks via genetic algorithm. Comput. Secur., 2019.
Pin-Yu Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: zeroth order optimization based black-bor box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec@CCS 2017, Dallas, TX, USA, November 3, 2017, 2017.
Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial atlacks with a transfer-based prior. Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
Bronya Roni Chernyak, Bhiksha Raj, Tamir Hazan, and Joseph Keshet. Constant random perturbations provide ad adversarial robustness with minimal effect accuracy. CoRR, 2021.
Carlos A. Coello Cocllo and Efrén Mezura-Montes. Handling constraints in genetic algorithms using dominance-basse tournaments. In Adaptive Computing in Design and Manafacture V, 2002.
Jeremy Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning, 1CML, 2019.
Kalyanmoy Deb. Multi objectvb optimization using evolutionary algorithms. Wiley-Interscience series in systems and optimization. Wiley, 2001.
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-IT. IEFE Trans. Fvol. Comput., 2002.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Dharmesh Dhabliya, Swati Saxena, Jambi Ratna Raja Kumar, Dinesh Kumar Pandey, NV Balaji, and X Mercilin Raajini. Exposing the financial impact of ai-driven data analytics: A cost-benefit analysis. In World Conference on Communication & Computing (WCONF), 2024.
Ann-Kathin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems 32, 2019.
Junhao Dong, Junxi Chen, Xiaohua Xie, Jianhuang Lai, and Hao Chen. Survey on adversarial attack and defense for medical image analysis: Methods and challenges methods and challenges. ACM Comput. Surv., 2025.
Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O'Brien, Stuart Schieber, James Waldo, David Weinberger, and Alexandra Wood. Accountability of AI under the law: The role of explanation. CoRR, 2017.
Filip Karlo Dosilovic, Mario Breic, and Nikica Hlupic. Explainable artificial intelligence: A survey, In 41st International Convention on Information and Communication Technology, Electronics and Microelectronics, MIPRO 2018, Opatija, Croatia, May 21-25, 2018. IEEE, 2018.
Tarik Eltaeib and Ausif Mahmmood. Large-scale evolutionary oplimization using multi-layer differential evolution. In 2018 9th JEFE Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), 2018
Julia Garbaruk, Doina Logofatu, Costin Badica, and Florin Leon. Digital image evolution of artwork without human evaluation using the example of the evolving mona lisa problem. Vietnam Journal of Computer Science, 2022.
Amirata Ghorbani, Abubakar Abid, and James Y. Zou. Interpretation of neural networks fragile. In The Third AAAI Conference Artificial Intelligence, AAAI, 2019.
Yiwen Guo, Ziang Yan, and Changshui Zhang. Subspace attack: Exploiting promising subspaces for query- efficient black-box attacks. In Advances in Neural Information Processing Systems 32: Annual Conference
on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
Ahmed Hamada. Br35H :: Brain Tumor Detection 2020.
ahmedhamadaO/brain coumor-detection 2020. Accessed: 2024-10-15
Jinkui Hao, William R Kwapong, Ting Shen, Huazhu Fu, Yanwu Xu, Qinkang Lu, Shouyue Liu, Jiong Zhang, Yonghuai Liu, Yifan Zhao, et al. Early detection of dementia through retinal imaging and worthw ai. upj Digital Medicine, 2024.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems 32, 2019.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang- Chich Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenetv3. In International Conference on Computer Vision, ICCV, 2019.
Wei Huang, Xingyu Zhao, Gaojie Jin, and Xiaowei Huang. SAFARI: versatile and efficient evaluations for robustness of interpretability. In IEEE/CVE International Conference on Computer Vision, ICCV, 2023.
Zhichao Huang and Tong Zhang. Black-box adversarial attack with transferable model-based embedding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020, 2020.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black- box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning, JCML 2018, Stockholmsmsssan, Stockholm, Sweden, July 10- 10-15, 2018, Proceedings of Machine Learning Research. PMIRR 2018.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Fekhr Eddine Keddous, Nadiya Shvai, Arcadi Llanza, and Amir Nakib. Inference acceleration of deep learning classifiers based on RNN. In TEEE International Conference on Image Processing, TCIP. IEEE, 2023.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un)reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Represenlations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Alex Krizhevsky, Nly Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep convolutional neural networks. In Advonces in Neural Information Processing Systems, 2012.
Aditya Kuppa and Nhicn-An Le-Khac. Black box attacks on explainable artificial intelligence(xai) methods in cyber security. In nnernational Joint Conference on Neurol Networks, IJCNN. IEEE, 2020.
Nicholas Lambert, William H. Latham, and Frederic Fol Leymaric. The emergence and growth of evolutionary art: 1980-1993. In International Conference Compuler Gruphics and Interactive Techniques, SIGGRAPH 2013, Anaheim, CA, USA, July 21-25, 2013, Art Gallery, 2013.
Nan Li, Lianbo Ma, Guo Yu, Bing Xue, Mengjie Zhang, and Yaochu Jin. Survey on evolutionary deep learning: Principles, algorithms, applications, and open issues. ACM Comput. Surv., 2024.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advonces in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. CoRR, 2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference Computer Vision and Pattern Recognition, 2016.
Hemanth Nadipineni. Method to classify skin lesions using dermoscopic images. CoRR, 2020.
Daniel Omeiza, Helena Webb, Marina Jirotka, and Lars Kunze. Explanations in autonomous driving: A survey. IEEE Trans. Intell. Transp. Syst., 2022.
Nicolas Papernot, Patrick D. McDanicl, and Ian J. Goodfellow. Transforability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca. Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32. 2019.
Kewen Peng and Tim Menzies. Documenting evidence of a reuse of "why should trust you?': predictions of any classifier.. In ESEC/FSH 29th ACM Joint European Sofhware Engineering Conference and Symposium on the Foundations of Software Engineering, 2021.
explaining the
Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against query-basod black- box attacks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 6-14, 2021, virtual, 2021.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Hao Qiu, Leonardo Lucio Custode, and Giovanni lacca. Black-box adversarial attacks using evolution strategies. In GECCO '11 Genelic and Exolutionary Compulation Conference, Companion Volume, Lille, France, July 10-14, 2021, 2021.
Thomas Quinn, Manisha Senadeera, Stephan Jacobs, Simon Coghlan, and Vuong Le. Trust and medical AI: the challenges we face and the expertise neoded to overcome them. J. Am. Medical Informatics Assoc., 28, 2021.
Nitin Rane, Saurabh Choudhary, and Jayesh Rane. Explainable artificial intelligence (xai) in healthcare: Interpretable models for clinical decision support. SSRN, 2023.
Rikard Rosenbacke, Asa Melhus, Martin McKee, and David Stuckler. How explainable artificial intelligence can increase or decrease clinicians' trust in ai applications in health care: Systematic review. JMIR AI, 2024a.
Rikard Rosenbacke, Asa Melhus, Martin McKee, and David Stuckler. How explainable artificial intelligence can increase or decrease clinicians' trust in ai applications in health care: Systematic review. JMIR AI, 2024b.
Andrei A. Rusu, Dan Andrei Calian, Sven Gowal, and Raia Hadsell. Hindering adversarial attacks with implicit neural representations. In International Conference on Machine Learning, 1CML, 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 94th International Conference on Machine Learning, ICML. PMLR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, ICLR, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, 2014.
Christopher C. Skiscim and Bruce L. Golden. Optimization by simulated anncaling: A preliminary com- putational study for the TSP. In Proceedings of the 15th conference on Winter simulation, WSC 1983, Arlington, VA, USA, December 12-14, 1983, 1983.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, 2012.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
Rainer Storn and Kenneth V. Price. Differential evolution A simple and efficient heuristic for global optimization over continuous spaces. 1997.
Akshayvarun Subramanya, Vipin Pillai, and Hamed Pirsiavash. Fooling network interpretation in image classification. In 2019 IEEE/CVF International Conference Computer Vision, ICCV, 2019.
Anas M. Tahir, Muhammad Enamul Hoque Chowdhury, Amith Khandakar, Tawsifur Rahman, Yazan Qiblawey, Uzair Khurshid, Serkan Kiranyaz, Nabil Ibtchaz, M. Sohel Rahman, Somaya Al-Máadoed, Sakib Mahmud, Maymouna Ezeddin, Khaled Hameed, and Tahir Hamid. COVID-19 infection localization and severity grading from chest x-ray images. Compul. Biol. Medicine, 2021.
19

---

Published in Transactions on Machine Learning Research (10/2025)
Snir Vitrack Tamam, Raz Lapid, and Moshe Sipper. Foiling explanations in deep neural networks. Trans. Mach. Learn. Res., 2023.
Yingtao Tian and David Ha. Modern evolution strategies for creativity: Fitting concrete images and abstract concepts. In Artificial Intelligence in Music, Sound, Art and Design 1 11th International Conference, EsoMUSART 2022, Held Part of EvoStar 2022, Madrid, Spain, April 20-22, 2022, Proceedings, Lecture Notes in Computer Science. Springer, 2022.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions. CoRR, 2018.
Chun-Chen Tu, Pai-Shun Ting, Pin-Yu Yu Chen, Sijia Liu, Huan Zhang, Jinseng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-bast based zeroth order optimization method for attacking black-box neural networks. In The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty- Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Arlificial Intelligence, EAAT 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, 2019.
Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aäron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässon, Stockholm, Sweden, July 10-15, 2018, Procoedings of Machine Learning Research, 2018.
Bas H. M. van der Velden, Hugo J. Kuijf, Kenneth G. A. Gilhuijs, and Max A. Viergever. Explainable artificial intelligence (XAT) in deep learning-basee medical image analysis. Medical Image Anal., 2022.
Jiakai Wang, Donghua Wang, Jin Hu, Siyang Wu, Tingsong Jiang, Wen Yao, Aishan Liu, and Xianglong Liu.
Adversarial examples in the physical world: A survey. CoRR, 2023.
Shunxin Wang, Raymond Veldhuis, Christoph Brune, and Nicola Strisciuglio. survey on the robustness of computer vision models against common corruptions, 2024.
Wentao Wang, Han Xu, Xiaorui Liu, Yaxin Li, Bhavani Thuraisingham, and Jiliang Tang. Imbalanced adversarial training with reweighting. In IEEE International Conference on Data Mining, ICDM. IEEE, 2022.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. J. Mach. Learn. Res., 2014.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics. Springer, 1992.
Phoenix Williams and Ke Li. Camopatch: An evolutionary strategy for generating camoflauged adversarial patches. In Advonces Neural Information Processing Systems, 2023a.
Phoenix Neale Williams and Ke Li. Black-box sparse adversarial attack via multi-objective optimisation CVPR proceedings. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023b.
Phoenix Neale Williams, Ke Li, and Geyong Min. Large-scale evolutionary optimization via multi-task random grouping. In 2021 IEEE International Conference on Systems, Man, and Cybernetics, SMC, 2021.
Phoenix Neale Williams, Ke Li, and Geyong Min. Sparse adversarial attack via bi-objective optimization. In Evolutionary Multi- Criterion Optimization 12th International Conference, EMO, 2023.
Jing Wu, Mingyi Zhou, Ce Zhu, Yipeng Liu, Mehrtash Harandi, and Li Li. Performance evaluation of adversarial attacks: Discrepancies and solutions. CoRR, 2021.
Hao Yu, Ke Liang, Dayu Hu, Wenxuan Tu, Chuan Ma, Sihang Zhou, and Xinwang Liu. Gzoo: Black-box node injection attack on graph neural networks via zeroth-order optimization. IEEE Transactions on Knowledge and Data Fngineering, 2025.
20

---

Published in Transactions on Machine Learning Research (10/2025)
Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. CoRR, 2023.
Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In 29th USENIX Security Symposium, 2020.
21


Reviews:
[{"review_id": "bldreUgQeP", "reviewer": "Reviewer_jqnL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes EvoAttack, a black-box adversarial attack towards Explainable AI (XAI) in the context of medical imaging. The authors use the semi-transparent, RGB-valued circles to create perturbations, which largely reduce the search space of attacks, thus leading to higher attack effects with fewer query numbers. Finally, the paper discusses the vulnerabilities of different XAI methods based on the attack results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides comprehensive and extensive experiments to support each of the claims. The experiments are conducted across three different medical imaging datasets, including HAM10000, Br35h, and the COVID-QU-Ex dataset; seven XAI methods; and three different classifier networks, including MobileNet, AlexNet, and VGG. The experiments are repeated over 10 different random seeds and are reported with the final mean and variance.", "audience_yn": "Yes", "audience_explanation": "The findings of the paper can be valuable to the community for a better understanding of the robustness of XAI methods in medical imaging. The discussion in the adversarial training can benefit future research, which tends to improve the robustness and reliability of the XAI methods.", "requested_changes": "Despite the strength mentioned above, I recommend addressing these issues before I make my recommendation for acceptance.\n\n- Lots of typos. I find the paper has a lot of typos, repetition, and even incompleteness. For example, in the last second paragraph on page 4, the sentence is not complete. In Equation 1, the $\\delta$ is not consistently bolded. There is an extra comma at the end of page 5. There is repetition in the paragraph above Section 4.2. Please carefully review the whole paper again and fix all the typos.\n- Attack effectiveness versus query number. It's interesting to see the comparison between EvoAttack and other black-box attack approaches under different query numbers. For example, the x-axis represents the query number, and the y-axis represents the attack effectiveness.", "broader_impact_concerns": "N/A", "additional_comments": "Please see the requested changes.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1aLTKqKu78", "reviewer": "Reviewer_62Tr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript examines the vulnerability of XAI methods in computer vision. Recent studies reveal these explanations can be manipulated by adversarial perturbations. Existing attacks, particularly white-box methods, lack generalizability, while black-box approaches often rely on costly meta-heuristics with high query demands. Moreover, current techniques overlook the varying granularity of XAI methods, performing poorly against region-based explanations such as Grad-CAM. To address these gaps, the authors propose a shape-based black-box attack inspired by computational art. By optimizing semi-transparent RGB shapes with evolutionary strategies, the method reduces the search space, operates efficiently under limited queries, and adapts to different explanation granularities. Evaluations demonstrate its effectiveness across diverse XAI methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The method is evaluated across three diverse medical imaging datasets (HAM10000, Br35h, COVID-QU-Ex), using multiple widely adopted classifiers (MobileNet, AlexNet, VGG-16) and seven explanation methods (e.g., Grad-CAM, SHAPLEY, Saliency).\n- Extensive results are presented, including constraint satisfaction rates and PCC scores, which clearly demonstrate EvoAttack’s superior performance compared to state-of-the-art methods such as NES and SAFARI. Statistical validation with the Wilcoxon signed-rank test further confirms the significance of improvements.\n- Visual comparisons of adversarial images and attribution maps (Figures 3, 4, 7) convincingly illustrate how EvoAttack achieves either subtle misclassification with preserved explanations or distorted explanations with preserved classifications.", "audience_yn": "Yes", "audience_explanation": "The work directly addresses the intersection of XAI, adversarial robustness, and evolutionary optimization, all of which are active research areas in the machine learning community. By proposing a novel and query-efficient black-box attack EvoAttack that effectively compromises multiple explanation methods, the paper contributes to both the theory and practice of XAI robustness evaluation.", "requested_changes": "- The experiments are conducted solely on medical imaging datasets (HAM10000, Br35h, COVID-QU-Ex). Although this domain is highly relevant and safety-critical, it is not clear whether the proposed attack generalizes to other computer vision tasks, such as natural images or large-scale benchmarks (e.g., ImageNet). Broader evaluation would strengthen the claim of general applicability.\n- The comparisons focus primarily on NES and SAFARI. While these are appropriate black-box adversarial baselines, the exclusion of other query-efficient or evolutionary optimization attacks (e.g., Square Attack, genetic search methods) may leave some doubt about whether the performance gains are universally superior.\n- The attack assumes access to both the DNN predictions and the outputs of the XAI method. In practice, this dual access may not always be available in real-world settings, particularly in sensitive domains like healthcare. Clarifying the threat model and its practical feasibility would improve the paper’s impact.\n- The exploration of defenses is somewhat limited. The study considers adversarial training, which is a natural baseline, but does not evaluate other plausible defense strategies such as noise injection, explanation regularization, or randomized smoothing. This limits the depth of the defense discussion.", "broader_impact_concerns": "Despite its close connection to the black-box optimization setting, the paper provides very little discussion of zeroth-order optimization techniques. These methods are widely studied for query-limited adversarial attacks, and it would be valuable to clarify how the proposed shape-based evolutionary strategy differs from, or improves upon, established zeroth-order approaches. Without such discussion, readers may find it difficult to fully situate this work in the broader landscape of black-box adversarial optimization. For example, prior works such as Yu et al. (2025, IEEE TKDE) on GZOO and Chen et al. (2017, AISec) on ZOO provide relevant baselines and conceptual context.\n\n1. Yu et al. GZOO: Black-Box Node Injection Attack on Graph Neural Networks via Zeroth-Order Optimization. IEEE TKDE 2025.\n2. Chen et al. ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models. AISec 2017.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9tPZe9s309", "reviewer": "Reviewer_3pRH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a black-box adversarial attack method validated on explainability methods for medical imaging applications. Unlike related approaches, the proposed method, EvoAttack, distorts images using semi-transparent circles with parameters determined by an evolutionary algorithm, requiring a much smaller computational budget. EvoAttack consistently outperformed two other black-box adversarial methods, NES and SAFARI, on two tasks across a variety of explainability methods and datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Generally yes, but experiments are limited to only (a) one choice of compute budget, (b) two baseline black-box attack methods, and (c) one problem domain (medical imaging). Including additional evidence addressing some or all of these would strengthen the claims made in the paper.", "audience_yn": "Yes", "audience_explanation": "This submission would appeal to researchers who work on explainability and robustness of deep image recognition models.", "requested_changes": "**Main feedback**\n- Include performance metrics and some measure of speed/throughput across various budgets. It remains unclear whether the fixed budget in these experiments favors the proposed method. I’m aware that this is an advantage to EvoAttack regardless, but it would be useful to know how these black-box methods perform across various budgets\n- If possible, include other relevant black-box attack methods or explain why they cannot be included\n- If possible, include experiments on a natural image dataset. This is not strictly necessary, but would certainly improve the paper \n\n**Minor feedback**\n- Incorrect in-text citations in several places. E.g., “Ghorbani et al.Ghorbani et al. (2019)”. I believe the authors need to use \\pcite for a proper parenthetical citation (i.e., when referencing a work such that the reference is not part of the sentence).\n- Figure 5: The reader does not yet know the term “(1+1)-ES structure” at this stage in the paper. I would either remove this from the caption or briefly define it.\n- Remove extra comma after Eq 1\n- Some discussion or quantification of the computational demands of your method would be appreciated.\n- As a reader not familiar with this space, is attack optimization done *per image*? What is a typical compute budget? Some brief discussion would be helpful\n- “Figure 8” should read “Table 8”\n- Remove extra period at end of section 4\n- At the end of the first limitations paragraph, there are dangling citations that are their own sentence. Unclear where these belong", "broader_impact_concerns": "I feel that the Broader Impacts statement is adequate.", "additional_comments": "**Strengths**:\n- The paper is very well-written and clearly organized. Visualizations help ease the reading experience, particularly for those like myself who are not experts in this area\n- Prior work is thoroughly explored and used to contextualize both the problem setting and the proposed method\n- The proposed method appears to be novel, uniquely drawing upon the computational art literature to provide a more efficient approach\n- Experiments are thorough and described in detail. Multiple datasets and explainability methods are used to validate EvoAttack\n- Given the fixed computational budget, EvoAttack considerably outperforms existing black-box attack methods in virtually all settings\n\n**Weaknesses**:\n- While the quantitative results are compelling, I would like to see performance across a range of “query”/computational budgets. E.g., perhaps EvoAttack only demonstrates such a margin of superiority in the low compute budget regime chosen; does this advantage disappear with larger budgets? As someone who is not deeply familiar with this field, I would be interested to see some practical comparison of time taken per attack (if applicable) across various compute budgets as well.\n- Experiments only compare to 2 baseline methods and one problem domain (medical imaging). What about other black-box attack methods such as those listed in Table 2 here [1]? I appreciate that experiments were comprehensive, but the generalizability of the method could be strengthened with validation in domains other than medical imaging as well.\n\n**References**\n\n[1] Baniecki, Hubert, and Przemyslaw Biecek. \"Adversarial attacks and defenses in explainable artificial intelligence: A survey.\" Information Fusion 107 (2024): 102303.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)