Foiling Explanations in Deep Neural Networks
Authors: Snir Vitrack Tamam
Raz Lapid
Moshe Sipper
Venue: Accepted by TMLR ()

Abstract: Deep neural networks (DNNs) have greatly impacted numerous fields over the past decade. Yet despite exhibiting superb performance over many problems, their black-box nature still poses a significant challenge with respect to explainability. Indeed, explainable artificial intelligence (XAI) is crucial in several fields, wherein the answer alone---sans a reasoning of how said answer was derived---is of little value. This paper uncovers a troubling property of explanation methods for image-based DNNs: by making small visual changes to the input image---hardly influencing the network's output---we demonstrate how explanations may be arbitrarily manipulated through the use of evolution strategies. Our novel algorithm, AttaXAI, a model-and-data XAI-agnostic, adversarial attack on XAI algorithms, only requires access to the output logits of a classifier and to the explanation map; these weak assumptions render our approach highly useful where real-world models and data are concerned. We compare our method's performance on two benchmark datasets---CIFAR100 and ImageNet---using four different pretrained deep-learning models: VGG16-CIFAR100, VGG16-ImageNet, MobileNet-CIFAR100, and Inception-v3-ImageNet. We find that the XAI methods can be manipulated without the use of gradients or other model internals. Our novel algorithm is successfully able to manipulate an image in a manner imperceptible to the human eye, such that the XAI method outputs a specific explanation map. To our knowledge, this is the first such method in a black-box setting, and we believe it has significant value where explainability is desired, required, or legally mandatory.

Published in Transactions on Machine Learning Rescarch (08/2023)
Foiling Explanations in Deep Neural Networks
Snir Vitrack Tamam"
snirut&gmail..com com
Deportment of Computer Science
Ben- Gurion University, Ismall
Raz Lapid
razla razla oc.il
Deportment of Computer Science
University, Israel & DeepKeep, Israel
Moshe Sipper
sipperObgu.oc oc.il
Department of Computer Science
Ben- Gurion University, Isruel
Reviewed on OpenReview: https: // openreview. net/ forum? id= wuLQMHtyLk
Abstract
Deep neural networks (DNNs) have greatly impacted numerous fields over the past decade. Yet despite exhibiting superb performance over many problems, their black- box nature still poses a significant challenge with respect explainability. Indeed, explainable artificial intelligence (XAT) is crucial in several fields, wherein the alonc-sans reasoning of how said answer was derived-is of little value. This paper uncovers troubling properly of explanation methods for image-based DNNs: by making small visual changes to the imput image-hardlyy influencing the network's output-we demonstrate how explanations may be arbitrarily manipulated through the use of evolution strategies. Our novel algorithm,, AttaXAI, a model-and- data XAT-agnostic, adversarial attack on XAT algorithms, only requires access to the output logits of classifier and to the explanation map; these weak assumptions render our approach highly useful where real-world models and data are concerned. We compare our method's performance on two benchmark datasets CIFAR100 and ImageNet-using four different pretrained deep-learning models: VGG16-CIFARIUE VGG16-ImageNet, MobillNeet-CIFARRIO and Inception-v3-ImageeNe the XAI methods can manipulated without the use of gradients or other model internals. AttaXAT successfully manipulates an image such that several XAI meth- ods output specific explanation map. To our knowledge, this is the first such method in black-box setting, and we believe it has significant. value where ex- plainability is desired, required, legally mandatory. The code is available https://github. cwwccooo//aiaa cooon/aalaa/ooltti Expariiionpppaannion-eieeee
We find that
Mesep--eurrll-ettaeee Keywords: deep learning, computer vision, adversarial attack, evolutionary algo- rithm, explainable artificial intelligence
1 Introduction
Recent research has revealed that deep learning- image-classiffattion systems are vulnerable to adversarial instances, which designed to deccive algorithms by introducing perturbations to benign images (Carlini & Wagner, 2017; Madry el al., 2017; Xu et al., 2018; Goodfellow et al., 2014; Croce & Hein, 2020). A variety of strategies have been developed to generate adversarial instances,
*Equal contribution

---

Published in Transactions on Machine Learning Rescarch (08/2023)
and they fall under two broad categorics, differing the underlying threat model: white-box attacks (Moosavi-Dezfooli al., 2016; Kurakin et al., 2018) and black-box attacks (Chen et al., 2017; Lapid et al., 2022).
In white box attack, the attacker has access to the model's parameters, including weights, gradients, etc'. In a black-box attack, the attacker has limited information or no information at all; the attacker generates adversarial instances using either a different model, a model's raw output (also called logits), or no model at all, the goal being for the result transfer to the target model (Tramèr et al., 2017; Inkawhich al., 2019).
In order to render model more interpretable, various explainable algorithms have been conceived. Van Lent et al. (2004) coined the term Explainable Artificial Intelligence (XAI), which refers to Al systems that "can explain their behavior either during execution or after the fact". In-depth research into XAT methods has been sparked by the success of Machine Learning (MI.) systems, particularly Deep Learning (DL), in a variety of domains, and the difficulty in intuitively understanding the outputs of complex models, namely, how did a DL model arrive at a specific decision for a given input.
Explanation techniques have drawn increased interest in recent years due to their potential to reveal hidden properties of deep neural networks (Došilovié et al., 2018). For safety-critical applications, interpretability is essential, and sometimes even legally required.
The importance assigned to each input feature for the overall classification result may be observed through explanation maps, which can be used to offer explanations. Such maps can be used to creale defenses and detectors for adversarial attacks (Walia et al., 2022; Fidel et al., 2020; Kao al., 2022). Figures 1 and 2 show examples of explanation maps, generated by five different methods discussed in Section 2.
In this paper, we show that these explanation maps can be transformed into any target map, using only the maps and the network's output probability vector. This is accomplished by adding a perturbation to the input that is usually unnoticeable to the human eye. This perturbation has minimal effect on the neural network's output, therefore, in addition to the classification outcome, the probability vector of all classes remains virtually identical.
Our contribution. As we mentioned earlier, recent studies have shown that adversarial attacks may be used to underminc DNN predictions (Goodfellow et al., 2014; Papernot et al., 2016b; Carlini & Wagner, 2017; Lapid et al., 2022). There are several more papers regarding XAI attacks, which have also shown success on manipulating XAT, but to our knowledge, they all rely on access to the neural network's gradient, which is usually not available in real-world scenarios (Ghorbani et al., 2019; Dombrowski et al., 2019).
Herein, we propose a black-box algorithm, AttaXAI, which enables manipulation of an image through a barely noticeable perturbation, without the use of any model internals, such that the explanation fits any given target explanation. Further, the robustness of the XAI techniques are tested as well. We study AttaXAl's efficiency on 2 benchmark datasets, 4 different models, and 5 different XAI methods.
The next scction presents related work. Scction 3 presents AttaXAI, followed by experiments and results in Section 4. In Section we analyze our results, In Section 6 we make discussion followed by concluding remarks in Section 7.
2 Related Work
The ability of explanation maps to detect even the smallest visual changes was shown by Ghorbani et al. (2019), where they perturbed a given image, which caused the explanatory map to change, without any specific target.

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Original
LRP
Deep Lift
Gradient
Grad x Iuput
G-Backprop
Figure Explanation maps for 5 images using 5 different explanation methods. Dataset: ImageNet. Model: VGG16.
Kuppa Le-Khac (2020) designed a black box attack to examine the security aspects of the gradient- based XAI approach, including consistency, accuracy, and confidence, using tabular datasets.
Zhang et al. (2020) demonstrated a class of white-box attacks that provide adversarial inputs, which deceive both the interpretation models and the deep-learning models. They studied their method using four different explanation algorithms.
Xu et al. (2018) demonstrated that a subtle adversarial perturbation intended to mislead classifiers might cause significant change a class-specific network interpretability map.
The goal of Dombrowski et al. (2019) was precisely replicate given target map using gradient descent with respect to the input image. Although this work showed an intriguing phenomenon, it is less-realistic scenario, since the attacker has full access to the targeted model.
We aimed to veer towards a more-realistic scenario and show that we can achieve similar results using no information about the model besides the probability output vector and the explanation map. To our knowledge, our work the first introduce a black-box attack on XAT gradient-based methods in the domain of image classification.
We will employ the following explanation techniques in this paper:
1. Gradient: Utilizing the saliency map, g(x) = af (x), one may measure how small perturba- 8h
tions each pixel alter the prediction of the model, f(x) (Simonyan et al., 2013).

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Original
LRP
Deep Lift
Gradient
Grad x Iuput
G-Backprop
Figure 2: Explanation maps for images using 5 different explanation methods. Dataset: CIFAR100. Model: VGG16.
The explanation map is calculated by multiplying the input by the of partial derivatives of the output with regard the input, g(x) Oz (z) (Shrikumar et al., 2016).
Gradient x Input:
3. Guided Backpropagatton: A variant of the Gradient explanation, where the gradient's negative components are zeroed while backpropagating through the non- noneriieasssies of the model (Springenberg et al., 2014).
Layer-wise Relevance Propagation (LRP):
With this technique, pixel importance prop- agates backwards across the network from top bottom (Bach et al., 2015; Montavon et al., 2019). The general propagation rule is the following:
R₁ = Σ asp(W)k) c+ Eo,t asp(Wgg)
R
(1)
where and are neurons of any two consccutive layers, RR, Ry the relevance maps of layers & and j, respectively, is function that transforms the weights, and is small positive increment.
In order to propagate relevance scores to the input layer (image pixels), the method applics an alternate propagation rule that properly handles pixel values received as input:

---

Published in Transactions on Machine Learning Rescarch (08/2023)
R Σ a, L.u.y - 1, Rgy
(2)
Ei a.wii - Lwiiy hiw
where 4, and h2 are the lower and upper bounds of pixel values.
5. Deep Learning Important FcaTures (DeepLIFT) compares a neuron's activation to its "reference activation", which then calculates contribution scores based on the difference. DeepLIFT has the potential to separately take into account positive and negative contribu- tions, which might help to identify dependencies that other methods might have overlooked (Shrikumar et al., 2017).
3 AttaXAI
This section presents our algorithm, AttaXAI, discussing first evolution strategies, and then delving into the algorithmic details.
3.1 Evolution Strategies
Our algorithm is based Evolution Strategies (ES); family of heuristic search techniques that draw their inspiration from natural evolution (Beyer & Schwefel, 2002; Hansen et al., 2015). Each iteration (aka generation) involves perturbing (through mutation) population of vectors (genotypes) and assessing their objective function value (fitness value). The population of the next generation is created by combining the vectors with the highest fitness values, a process that. repeated until a stopping condition is met.
AttaXAI belongs to the class of Natural Evolution Strategies (NES) (Wierstra et al., 2014; Glas- machers et al., 2010), which includes several algorithms in the ES class that differ in the way they represent the population, and in their mutation and recombination operators. With NES, the popu- lation is sampled from a distribution Ty., which evolves through multiple iterations (gencrations); we denote the population samples by Z, Through stochastic gradient descent, NES attempls to maximize the population's 's average fitness, Ez [f()), given a fitness function, f(-)-
A version of NES we found particularly useful for our case was used to solve common reinforcement
learning (RL) problems (Salimans et al., 2017).
We used Evolution strategies (ES) in work for the following reasons:
Global optima scarch. ES are effective at finding global optima in complex, high- dimensional search spaces, like those we encounter when trying to generate adversarial exam- ples. ES can often bypass local optima that would trap traditional gradient-baae methods. This ability is crucial for our work, where we assumed black-box threat model.
Parallelizable computation. ES are highly parallelizable, allowing us to to simultaneously compute multiple fitness values. This is significant advantage given our high computational requirements; using ES allows us to significantly reduce run times.
No Need for backpropagation. ES do not require gradient information and hence eschew backpropagation, which is advantagcous in scenarios where the models do not have casily computable gradients or are black-box models.
Search gradients. The core idea of NES is to use search gradients to update the parameters of the search distribution (Wierstra et al., 2014). The search gradient can be defined as the gradient of the expected fitness: Denoting by n a distribution with parameters v, x<2[²) is the probability density

---

Published in Transactions on Machine Learning Rescarch (08/2023)
function of given sample 2. With f(z) denoting the fitness of sample 2, the expected fitness under the search distribution can be written as:
= [[(=)] [())m(=WW)))
(3)
The gradient with respect to the distribution parameters can be expressed as:
VaJ(0) - Va / (())m(2(0)d=
= / f(=)
= f(=)Vom(=20)) T(210) dz T(2(0)
/ [I(()VVo (o0))
= [ell((=)Vglogg<((0))
From these results we can approximate the gradient with Monte Carlo (Metropolis Ulam, 1949) samples Ø1, 22; ... Zx:
VoJ (zx) log
our experiment we sampled from a Gaussian distribution for calculating the search gradients.
Latin Hypercube Sampling (LHS). LHS form of stratified sampling scheme, which improves the coverage of the sampling space. It is done by dividing given cumulative distribution function into M non-overlapping intervals of equal y-axis length, and randomly choosing one value from each interval to obtain M samples. It ensures that each interval contains the same number of samples, thus producing good uniformity and symmotry (Wang et al., 2022). We used both LHS and standard sampling in our experimental setup.
3.2 Algorithm
AttaXAI is an evolutionary algorithm (EA), which explores a space of images for adversarial instances that fool a given explanation method. This space of images determined by a given input image, a model, and a loss function. The algorithm generates a perturbation for the given input image such that it fools the explanation method.
More formally, consider neural network, f: gh.... + K which classifies given image, Rhine, where h, w, are the image's height, width, and channel count, respectively, to one of K predetermined calegories, with the predicted class given by k = arg max, f(x); The explanation map, which is represented by the function, g: gd.... - gh.,, links each image to an explanation map, of the same height and width, where each coordinate specifies the influence of each pixel on the network's output.
AttaXAI explores the space of images through evolution, ultimately producing adversarial image; it does so by continually updating Gaussian probability distribution, used to sample the space of images. By continually improving this distribution the search improves.
We begin by sampling perturbations from an isotropic normal distribution o Then we add them to the original image, x, and feed them to the model. By doing so, we can approximate

---

Published in Transactions on Machine Learning Rescarch (08/2023)
the gradient of the expected fitness function. With an approximation of the gradient hand we can advance in that direction by updating the search distribution parameters. schematic of our algorithm is shown in Figure 3, with full pseudocode provided in Algorithm 1.
Update proposed distribution parameters
N(0,0²1)
1(
f()
0.1
0.1
0.05
0.07
Objective valuue
0.33
0.04
0.0
Probability vector
Model
Stochastic samples
Explanation map
L
Feature
Feature
g(x)
Figure 3: Schematic of proposed algorithm. Individual perturbations are sampled from the popula- tion's distribution NNO,, and feed into the model (Feature and Feature are image features, e.g., two pixel values; in reality the dimensionality is much higher). Then, the fitness function, i.c. the loss, is calculated using the output probability vectors and the explanation maps to approximate the gradient and update the distribution parameters.
3.3 Fitness Function
Given image, Rhme specific explanation method, g: R..e.. Rh."" target image, Ttargets and target explanation map, g g(aterget), seek adversarial perturbation, d Rh..,, such that the following properties of the adversarial instance, Fadv + 5, hold:
1. The network's prediction remains almost constant, i.e, f(x) f(Iadv).
2. The explanation vector of Tadv is close to the target explanation map, g(Ftarget); i.e.
(xade) R3 g(ttarget).
The adversarial instance, Fade; is close the original image, æ, i.e., x Fade-
We achieve such perturbations by optimizing the following fitness function of the evolutionary algo- rithm:
C = allg((aaa)) g( + ()] 1----se f())]]
(5)
The first term ensures that the altered explanation map, ((Fddy), close the target explanation map, g(Ztarget); the second term pushes the network to produce the same output probability vector. The hyperparameters,
Rt, determine the respective weightings of the fitness components.
In order to use our approach, we only need the output probability vector, f(ravv), and the target explanation map, ((targee)) Unlike white-box methods, we do not presuppose anything about the targeted model, its architecture, dataset, or training process. This makes our approach more realistic.
Minimizing the fitness value the ultimate objective. Essentially, the value better the proper class's logit remains the same and the explanation map looks similar to the targeted explanation map:
argmin L = allg((addv) - (( Ill (Zadv) - f(x))
(6)

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Algorithm AttaXAI
Input:
x original image
f original image's e labe]
Sezp! f- target explanation map
Spred logits values æ used
model model
G maximum number of geuerations
À population size
6 initial standard deviation value
Q explanation loss weight
8 prediction weight
9- learning rate
+ standard deviation learning rate
Output:
adversarial image
# Main loop
1: Ø
2:
for 1,2...., do
3: for 1,2, do
draw sample 2 ~ N (x. o²I) at used perturb an image
5: evaluate fitness f(xk) FITNESS(*E)
calculate log-derivative V- Volog d²)
7:
calculate log-derivative V. No (a o²) V-J= + log o²)
8:
V.J= = E (axV Nog
+ nv V-J
11:
= V.J
12:
function (INUSS(() tezpi = (x.
13:
14:
Spred = model(=)
eaplioss ||Zeppl Kexp!
16:
prediozs Epred Spred
17:
explioes predines
4 Experiments and Results
Assessing the algorithm over particular configuration of model, dataset, and explanation technique, involves running it over 100 pairs of randomly selected images. We used 2 datasets: CIFAR100 and ImageNet (Deng et al., 2009). For CIFAR100 we used the VGG16 (Simonyan & Zisserman, 2014) and MobileNet (Howard et al., 2017) models, and for ImageNet we used VGG16 and Inception (Szegody et al., 2015); the models are pretrained. For ImageNet, VGG16 has an accuracy of 73.3% and Inception-v3 has accuracy of 78.8%. For CIFAR100, VGG16 has accuracy of 72.9% and MobileNet has accuracy of 69.0% (these are top-1 accuracy values; for ImageNet, top5 accuracy values are: VGG16 - 91.5%, Inception-v3 - 94.4%, and for CIFAR100, VGG16 91.2%, MobileNet 91.0%). We chose these models because they are commonly used in the Computer Vision community for many downstream tasks (Haque et al., 2019; Bhatia et al., 2019; Ning et al., 2017; Younis et al., 2020; Venkateswarlu et al., 2020).
The experimental setup summarized in Algorithm 2: Choose 100 random image pairs from the given dataset. For each image pair compute a target explanation map, ((taagget), for of the two images. With budget of 50, 000 queries to the model, Algorithm 1 perturbs the second image, aiming to replicate the desired g(Ztarget). We assume the model outputs both the output probability vector and the explanation map per each query to the model-whiee is realistic scenario nowadays, with XAI algorithms being part real-world applications (Payrovnaziri et al., 2020; Giuste et al., 2022; Tjoa Guan, 2020).

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Algorithm Experimental setup (per dataset and model)
Input:
dataset < dataset to be used
model + model to be used
G f- maximum number of generations
- population size
initial standard deviation value
a explanation-loss weight
8 +- prediction-loss weight
Output:
Performance scores
1: for i i 1 to 100 do
2: Randomly choose a pair of images x and Vtaryet from dataset
3: Generate Zadv by running Algorithm 1, with x and Starget (and all other imput parameters)
Save performance statistics
In order to balance between the two contradicting terms in Equation 5, we chose hyperparamelers that empirically proved to work, in terms of forcing the optimization process to find solution that satisfies the two objectives, following the work done in (Dombrowski et al., 2019): for ImageNet, and a = 1e7, 3 = 1e6 for CIFAR100. After every generation the learning rate was decreased through multiplication by factor of 0.999. We tested drawing the population samples, both independent and identically distributed (iid) and through Latin hypercube sampling (LHS). The generation of the explanations was achieved by using the repository Captum (Kokhlikyan et al., 2020), unified and generic model interpretability library for Pylorrc.
1el1, = le6
Figures 4 through 7 shows samples of our results. Specifically, Figure 4 shows AttaXAl-generatee atlacks for images from ImageNet using the VGG16 model, against each of the 5 explanation methods: LRP, Deep Lift, Gradient, Gradient x Input, Boaddedd-aakkpopagatiiee generated attacks for images from ImageNet using the Inception model; Figure shows AttaXAI- generated attacks for images from CIFAR100 using the VGG16 model; and Figure 7 shows AtlaXA-- generated attacks for images from CIFAR100 using the MobileNet model.
Figure 5 shows AttaXAI-
Note that our primary objective has been achieved: having generated an adversarial image (æadv); virtually identical to the original (x), the explanation (g) of the adversarial image (Kade) is now, incorrectly, that of the target image (Ftarget)-essentilllyy essentially, the two rightmost columns of Figures 4-7 are identical; furthermore, the class prediction remains the same, ie., arg max, f(x); = arg max, f (Xodu)s.
5 Conclusions
We examined the multitude runs in-depth, producing several graphs, which are provided in full in the extensive Appendix. We present the mean of the losses for various query budgets Table 1.
Below, we summarize several observations we made:
Our algorithm was successful in that f(radv) ze f(x) for all Fads generated, and when applying arg max the original label remained unchanged.
For most hyperparameter values examined, our approach converges for ImageNet using VGG, reaching ~ 1e 10 MSE loss between g(Eade) and g(Ztarget), for every XAI except Guided Backpropagation which was found to be more robust then other techniques in this configu- ration, x10 more robust.

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Etarget
Bade
g(r)
g g(aangge))
g (Iade)
LRP
Deep Lift
Gradient
Grad Input
G-Backprop
Figure 4: Attacks generated by AttaXAI. Dataset: ImageNet. Model: VGG16. Shown for the 5 explanation methods, described in the text: LRP, Deep Lift, Gradient, Gradient x Input, Guided Backpropagation (denoted G-Backprop in the figure). Note that our primary objective has been achieved: having generated an adversarial image (Fadv); virtually identical to the original (1), the explanation (g) of the adversarial image (Iade) is now, incorrectly, that of the target image (Itorget); essentially, the two rightmost columns are identical.
For all the experiments we wilnessed that the Gradient XAI method showed the smallest mean squared error (MSE) between g(aadv) and g( i.e., it was the least robust. The larger the MSE between ((Fadz) and ((zrarget) the better the explanation algorithm can handle our perturbed image.
For VGG16 (Figures and 14), Gradient XAI showed the smallest median MSE between g(aadv) and g(atarget). while Guided Backpropogation showed the most. This means that using Gradient XAT's output as explanation incurs the greatest risk, while using Guided Backpropogation's output explanation incurs the smallest risk.
For Inception (Figure 11), Gradient XAT and Guided Backpropogation exhibited the smallest. modian MSE, while LRP, Gradient Input, and Deep Lift displayed similar results.
For the MobileNet (Figure 17), Gradient XAI exhibited the smallest MSE, while Guided Backpropogation showed the largest MSE rendereng it more robust than other techniques in this configuration.
10

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Etarget
Bado
g(r)
g g(aang))))
g (Iade) )
LRP
Deep Lift
Gradient
Grad Input
G-Backprop
Figure 5: Attacks generated by AttaXAI. Dataset: ImageNet. Model: Inception.
Table 1: Different evaluations of our algorithm as function of number of queries to model. Best (lowest) per experiment boldfaced.
Model
Loss
10k
20k
30k
40k
50k:
Input
1.0e-2
1.8e-2
2.3e-2
2.8e-2
3.4e-2
VGG16-CIFAR100
Explanation
1.1c-6
9.9c-7
9.1c-7
8.6c-7
8.2c-7
Output
3.6e-4
2.0e-4
1.1e-4
7.3e-5
4.1e-5
Input
1.0e-2
1.7e-2
2.3e-2
2.8e-2
3.4e-2
MobileNet-CIFAR100
Explanation
1.4c-6
1.20-6
1.1c-6
1.1c-6
1.0c-6
Output
7.4e-5
3.1e-5
1.9e-5
1.4e-5
1.3e-5
Input
1.0c-2
1.7o-2
2.20-2
2.70-2
3.3e-2
VGG16-ImageNet
Explanation
1.2e-9
1.0e-9
9.0e-10
8.3e-10
7.9e-10
Output
1.0e-5
7.5e-6
6.2e-6
5.4e-6
5.6e-6
Input
1.0c-2
1.7e-2
2.2c-2
2.70-2
3.30-7
Inception-v3-ImaagNNee
Explanation
7.5e-10
6.5e-10
6.1e-10
5.8e-10
5.6e-10
Output
1.9c-5
1.3e-5
1.1e-5
9.5e-6
1.0e-5
MobileNet is more robust than VGG16 in that it attains higher MSE scores irrespective of the XAT method used. We surmise that this is due to the larger number parameters in VGG16.
11

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Daarget
Bade
g(r)
g (aunggss
g (Zack) )
LRP
Deep Lift
Gradient
Grad x Input
G-Backprop
Figure 6: Attacks generated by AttaXAL. Dataset: CIFAR100. Model: VGG16.
The query budget was 50,000 for all experiments. In many runs the distance between the target explanation and the adversarial explanation reached a plateau after roughly 25,000 queries.
6 Discussion
Broader impact. Our research aims evidentiate vulnerabilities of explainable artificial intelligence (XAI) in adversarial attacks which is of paramount. importance for wide array of domains.
For example, in healtheare, XAI is increasingly used in decision-making processes, such as discase diagnosis and treatment recommendations (Zhang et al., 2022; Muncer & Rasool, 2022). If these systems are susceptible to adversarial attacks, the consequences could be potentially life-threatenigg
the financial sector XAI systems are used for tasks such as fraud detection and credit-risk assessment (Cirqueira al., 2021: Psychoula et al., 2021). Vulnerabilities could lead to substantial financial losses.
In cybersccurity, XAI is used to detect anomalies and potential threats, and weaknesses could un- dermine the entire security infrastructure of organization (Srivastava et al., 2022; Capuano et al., 2022).
Thus, we believe our research has broad implications, as it aims to fost the development of robust, secure XAI systems, which retain reliability under adversarial conditions.
12

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Dtarget
Bade
g(r)
g g(aang))))
g (aade)
LRP
Deep Lift
Gradient
Grad x Input
G-Backprop
Figure 7: Attacks generated by AttaXAI. Dataset: CIFAR100. Model: MobileNet.
Potential real-world value. Our study offers several real-world benefits. First, by identifying the potential weaknesses in XAI systems, we provide crucial insights for organizations to take necessary precautions and develop couttermesurres against adversarial attacks. This can help prevent extensive damage, both in terms of financial losses and trust erosion, which may result from successful attack.
Additionally, our research might prompt organizations to invest more in Al security, thereby promoting the development of more resilient XAI systems. Policymakers can also utilize our research findings to create or amend legislation surrounding Al and cybersecurity. As Al technologies become increasingly integrated into our daily lives, establishing proper regulatory frameworks crucial to ensure their safe, ethical, and responsible use. Our research provides empirical evidence that guide these legislative efforts, contributing to more-secure digital society.
Query cost and adversarial effectiveness. The relationship between query cost and adversarial effectiveness is critical in assessing atlacks against deep neural networks, particularly for manipulating XAI maps. This paper's proposed black-bor box attack leverages a query budget of 50, 000, striking a balance between computational demand and attack success. While increasing the query budget may offer more precision, this would be resource-intensive. Conversely, a lower query budget may be less effective but more computationally efficient. The of 50, 000 queries in our attack demonstrrtes vulnerability in deep learning models that can be exploited without exorbitant computational costs, revealing potential security issues and improvement areas. Future research could focus on techniques to lower query cost while maintaining adversarial effectiveness, as well as methods to bolster model resilience against such attacks. This study, therefore, offers valuable insights into the tradc-off between query cost and adversarial effectiveness.
13

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Future directions. Our rescarch also paves the way for future studies on adversarial attacks. For one, while our study focuses on certain types of XAI techniques, future research could explore the susceptibility of other XAI techniques to adversarial attacks, broadening understanding of this critical security issue.
Further, we highlight the need for additional research on the development of more-advanced defensive mechanisms against adversarial attacks. As adversaries continually evolve their tactics, it is imperative that our defenses evolve as well.
Moreover, our research underscores the importance of considering the socio-ethical implications of adversarial attacks. As Al becomes more pervasive, attacks against these systems have the potential to cause widespread disruption and harm. Thus, future research should also consider the societal and ethical consequences of these attacks, informing policies and practices aimed at mitigating these potential harms.
In summary, the broader impact and potential real-world values of our study are substantial, ex- tending across various domains, ranging from healthcare to finance, to cybersocurity. By offering a foundation for future research and by suggesting practical strategies to mitigate adversarial attacks, our study contributes the ongoing effort to secure AI systems and ensure their responsible use.
7 Concluding Remarks
Recently, practitioners have started to use explanation approaches more frequently. We demonstrated how focused, undetectable modifications to the input data can result in arbitrary and significant adjustments to the explanation map. We showed that explanation maps of several known explanation algorithms may be modified at will. Importantly, this is feasible with a black-box approach, while maintaining the output of the model. We tested AttaXAI against the ImageNet and CIFAR100 datasets using& 4 different network models.
It is obvious that neural networks operate quite differently from humans, capturing fundamentally distinct properties. In addition, further work is required in the XAI domain to make XAI algorithms that are more reliable.
This work has shown that explanations are casily foiled- any recourse to internal information-raisiie questions regarding XAI-based defenses and detectors.
This work has also investigated the robustness of various XAI methods, revealing that Gradient XAI is the least robust XAI method and Guided Backpropagation is the most robust one.
Future suggestions. In our study we examined how to attack a model's (XAT) explanation for a given input, prodiction, and XAI mothod. Some questions still remain:
way predict whether a XAI attack will be successful, and how many queries will be needed.
better metric for a successful XAI attack, since in our results we observed that smaller L2 distance does not necessarily translate a more "convincing" attack.
Find a way to climinate the need for model feedback, i.c., go fully black-box. Applying XAI attacks via transferability (Papernot et al., 2016a; Xie et al., 2019; Wang et al., 2021) might be way move forward.
When developing new XAI methods find ways to render them more robust to adversarial attacks.
14

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Acknowledgement
We thank the reviewers for fruitful discussion. This research was partially supported by the Israeli Innovation Authority through the Trust.Al consortium.
15

---

Published in Transactions on Machine Learning Rescarch (08/2023)
References
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One, 10(7):e0130140, 2015.
HlansGGeorg Beyer and Hans-Paul Schwefel. Evolution strategies a comprehensive introduction. Natural Computing, 111:33-52, 2002.
Yajurv Bhatia, Aman Bajpayee, Decpanshu Raghuvanshi, and Himanshu Mittal. Image captioning using Google's Inception-Resnete and recurrent neural network. In 2019 Twelfth International Conference on Contemporary Computing (IC3), pp. 1-6. IEEE, 2019.
Nicola Capuano, Giuseppe Fenza, Vincenzo Loia, and Claudio Stanzione. Explainable artificial intel- ligence in cybersecurity: A survey, IEEE Access, 10:93575 93600, 2022.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box atlacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26, 2017.
Douglas Cirqueira, Markus Helfert, and Marija Bezbradica. Towards design principles for user-centric explainable ai in frand detection. In Artificial Intelligence in HCT: Second International Conference, AI-HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24-29, 2021, Proceedings, pp. 21-40. Springer, 2021.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, pp. 2206-2216 PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Potlern Recognition, pp. 248-25 255. IEEE, 2009.
Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermaan, Claus-Rober Müller, and Pan Kessel. Explanations can be manipulated and geometry is blame. Advances in Neural Information Processing Systems, 32, 2019.
Filip Karlo Došilovié, Mario Brčić, and Nikica Hlupić. Explainable artificial intelligence: A survey. In 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectvonccs (MIPRO), pp. 0210 0215. IEEE, 2018
Gil Fidel, Ron Bitton, and Asaf Shabtai. When explainability meets adversarial learning; Detecting adversarial examples using shap signatures. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1 8. IEEE, 2020.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Felipe Giuste, Wenqi Shi, Yuanda Zhu, Tarun Naren, Monica Isgut, Ying Sha, Li Tong, Mitali Gupte, and May D Wang. Explainable artificial intelligence methods in combating pandemics: A systematic review. IEEE Reviews in Biomedical Engineering, 2022.
Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and Jürgen Schmidhuber. Exponential nat- ural evolution strategies. In Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, pp. 393 400, 2010.
16

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Nikolaus Hansen, Dirk V Arnold, and Anne Auger. Evolution stratogics. In Springer Handbook of Computational Intelligence, pp. 871-898. Springer, 2015.
Md Foysal Haque, Youn Lim, and Dae-Seong Kang. Object detection based on vgg with resnet network. In 2019 International Conference on Electronics, Information, and Communication (ICEIC), pp. 1-3. TEEE, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.00. 1eo.17044.4.... 2017.
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable vevesraiall examples. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Ching Yu Kao, Junhao Chen, Karla Markert, and Konstantin Böttinger. Rectifying adversarial inputs using xai techniques. In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 573- 577. IEEE, 2022.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardese Captum: A unified and generie model interpretability library for pytorch, 2020.
Aditya Kuppa and Nhien-An Te-Khac. Black box attacks on explainable artificial intelligence (xai) methods in cyber security. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Alexcy Kurakin, Ian J Goodfellow, and Samy Bongio. Adversarial examples in the physical world. In Artificial Intelligence Safely and Securily, pp. 99 112. Chapman and Hall/CRC, 2018.
Raz Lapid, Zvika Haramaty, and Moshe Sipper. An evolutionary, gradient-free, query-efficient, black- box algorithm for generating adversarial instances in deep convolutional neural networks. Algo- rithms, 15(11), 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. arXiv preprint ar Xrviv:1706. 106:1006606083, 2017.
Nicholas Metropolis and Stanislaw Ulam. The Monte Carlo method. Journal of the American Statis- lical Association, 44(247):335 341, 1949.
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: an overview. Explainable Al: interpreting, explaining and visualizing deep learning, pp. 193- 209, 2019.
Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-255 25822, 2016.
Salman Muncer and Malik Atta Rasool. Aa systematic review: Explainable artificial intelligence (xai) based disease prediction. International Journal of Advanced Sciences and Computing, 1(1):
1 6, 2022.
Chengcheng Ning, Huajun Zhou, Yan Song, and Jinhui Tang. Inception single shot multibox detector for object detection. In 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 549 554. IEEE, 2017.
17

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint uxXi:1605.00227 2016a.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroSEP), pp. 372-387. IEEE, 2016b.
Seyedch Neelufar Payrovnaziri, Zhaoyi Chen, Pablo Rengifo-Moreno, Tim Miller, Jiang Bian, Jonathan Chen, Xiuwen Liu, and Zhe He. Explainable artificial intelligence models using real- world electronic health record data: systematic scoping review. Journal of the American Medical Informatics Association, 27 2020.
titcolas. Explainable machine learning for fraud detection. Computer, 54(10):49 59, 2021. Ismini Psychoula, Andreas Gutmann, Pradip Mainali, Sharon H Lee, Paul Dunphy, and Fabien Pe-
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv:1003.0386, 2017.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. orXiv preprint Xrx:::6605. 01713, 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3143- 3153. PMLR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1ixi:1109.155 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint
2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXio preprint 141.66200000 2014.
Gautam Srivastava, Rutvij H Jhaveri, Sweta Bhattacharya, Sharnil Pandya, Praveen Kumar Reddy Maddikunta, Gokul Yenduri, Jon G Hall, Mamoun Alazab, Thippa Reddy Gadekallu, al. Xai for cybersecurity: state of the art, challenges, open issues and future directions. arXiv preprind arXiv:2206. 03585, 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 9, 2015.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE Transactions Neural Networks and Learning Systems, 32(11):4793 4813, 2020.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv 1ee:1704.000 714.03455) 2017.
Michael Van Lent, William Fisher, and Michael Mancuso. An explainable artificial intelligence system for small-unit tactical behavior. In Proceedings of the National Conference on Artificial Intelligence, pp. 900-907. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2004.
Isumuri B Venkateswarlu, Jagadeesh Kakarla, and Shree Prakash. Face mask detection using mo- bilenet and global pooling block. In 2020 IEEE 4th Conference on Information & Communication Technology (CICT), pp. 5. IEEE, 2020.
18

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Savita Walia, Krishan Kumar, Saurabh Agarwal, and Hyunsung Kim. Using xai for deep learning- based image manipulation detection with shapley additive explanation. Symmetry, 14(8):1611, 2022.
Dan Wang, Jiayu Lin, and Yuan-Gen Wang. Query-efficient adversarial allack based on latin hyper- cube sampling. arXiv preprint ar 2t::2227.02009 2022.
Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transferability adversarial attacks In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 16158 16167, 2021.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15(1):949-998 980, 2014.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Im- proving transferability of adversarial examples with imput diversity. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 2730 2739, 2019.
Kaidi Xu, Sijia Liu, Pu Zhao, Pin-Yu Chen, Huan Zhang, Quanfu Fan, Deniz Erdogmus, Yanzhi Wang, and Xue Lin. Structured adversarial attack: Towards general implementation and better interpretability. arXiv preprint orXiv: 01664, 2018.
Ayesha Younis, Li Shixin, Shelembi Jn, and Zhang Hai. Real-time object detection using pro-trained deep learning models MobileNet-SSD. Proceedings of 2020 the 6th International Conference on Computing and Data Engineering, pp. 48, 2020.
Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In 29th {USENIX) Security Symposium (UUSENIX) Security 20), 2020.
Yiming Zhang, Ying Weng, and Jonathan Lund, Applications of explainable artificial intelligence in diagnosis and surgery. Diagnostics, 12(2):237, 2022.
19

---

Published in Transactions on Machine Learning Rescarch (08/2023)
Appendix
The following figures provide our full qualitative results. Hyperparameters: pop population size of evolutionary algorithm; Ir - learning rate of gradient approximation step for updating the attack; LS use of Latin sampling or regular sampling.
Figure 8: Similarity in terms of MSE between the target explanation map, g ((Taarget); and the best final adversarial explanation map, (Zodv), for different hyperparameter configurations (dataset: ImageNet, model: VGG16).
Figure 9: Similarity in terms of MSE between the input image, 2, and the best final adversar- ial, Tadu: for diffcrent hyperparameter configurations (dataset: ImageNet, model: VGG16).
Figure 10: MSE loss value function of cvolutionary generation for 8 different hyperparam- eler configurations (datasel: ImageNet, model: VGG16).
Figure 11: Similarity terms of MSE between the target explanation map, ((atarget). ) and the best final adversarial explanation map, g(Vade), for different hyperparameter configurations (dataset: ImageNet, model: Inception).
Figure 12: MSE loss values for the imput image versus the chosen adversarial image for 8 different hyperparameter configurations (dataset: ImageNet, model: Inception).
Figure 13: MSE loss value as function of evolutionary generation for 8 different hyperparam- eter configurations (dataset: ImageNet, model: Inception).
Figure 14: Similarity in terms of MSE between the target explanation map, (( and the best final adversarial explanation map, (xade), for 8 different hyperparameter configurations (dataset: CIFAR100, model: VGG16).
Figure 15: MSE loss value for input image versus chosen adversarial image for 8 different hyperparameter configurations (dataset: CIFAR100, model: VGG16).
Figure 16: MSE loss value as function of evolutionary generation for 8 different hyperparam- eter configurations (dataset: CIFAR100, model: VGG16).
Figure 17: Similarity in terms of MSE between the target explanation map, ((target)) and the best final adversarial explanation map, (Ladv), for 8 different hyperparameter configurations (dataset: CIFAR100, model: MobileNet).
Figure 18: MSE loss value for imput image versus chosen adversarial image for 8 different hyperparameter configurations (dataset: CIFAR100, model: MobileNet).
Figure 19: MSE loss value as function of evolutionary generation for 8 different hyperparam- eter configurations (dataset: CIFAR100, model: MobileNet).
20

---

Published in Transactions Machine Learning Rescarch (08/2023)
1e-9
pop=125, lr=0.0125,
pop=125, Ir=0.00625,
MSE
10
9. 40
300-30
7,69e-10
206-10
228-00
231--10
800-11
4.67e-10
irp
guided backprop gradient
grad times input deep, lift
trp
guided backprop
gradient
grad times input
deep_ lift
le-9
1r=0=125 LS=False
popo==12 1r=0.00625, LS=False
120-00
MSE
30
74
43e-10
7.76e10
8116-110
8.06e-10 10
428e-10
4.70k110
guided backprop gradient
grad times deep lift
times_input input
guided backprop
gradient
grad times input
jinestinput
deep lift
1e-9
pop==80 1r=0. LS=True
p_pop=80 Ir=0.00625, LS=True
2+
1196-09
MSE
7.31e-10
7226--10
140-10
976-110
8 06
8 Se33
11---1
4.66-11 10
hrp
guided backprop
gradient
grad times input
deep lift
trp
guided guided_backnng backprop
gradient
grad times, input deep
pop=800 0r=050 LS=false
n_pop=80, In= 0.255 LS=False
1158-09
MSE
7.02-100
18B--10
8.35-100
7.16e
6.90.10
222
4,60-110
4,226-100
krp
guided backprop gradient
grad times deep. lift
times_input input
Irp
guided backprop
gradient
grad times, input
deep lift
Figure 8: Similarity in terms of MSE between larget explanation map, g(Starget), and best final adversarial explanation map, g(Zadv), for 8 different hyperparameter configuratonss. Dataset: Ima- geNet. Model: VGG16. The Gradient XAI method is the most susceptible to attacks while Guided backpropogation is the hardest to attack; Deep Lift, LRP, and Gradient x Input are similar.
21

---

Published in Transactions on Machine Learning Rescarch (08/2023)
pop 125, 0r02,5,,, LS=True True
popppppppp00p 125, 0.0005,, LS True
0.06
A.20e-02
496-002
A.5e-002
A.3e-022
42256-02 02
MSE
0.04
0.02
1078-02
1446-12
1068-02
107e-02
05
Irp
guided backprop gradient
grad times input
deep_lift
Irp
guided backprop
gradient
grad times Input
deep lift
pop 125, Ir- 0 0125, LS
125, Ir- 0.00625, LS False
0.06
4236-02
48t--0 02
A06e-0p 02
A166-02
4228e-02
MSE
0.04
0.02-
1076-02
144-02
2-76-02
1076-02
deep lift 06a 02
trp
guided backprop
gradient
grad times input
deep_lf _efe_
Irp
guided backprop
gradient
grad times input
pop=80,
00025, LS=True
po 80, 0==0000 LS=True
0.06
396-02
A996-02
A550002
5300-02
5,36e-02
MSF
0.04
0.024
1.366-02
4346-02
1.34--02
1.34--02
02
guided backprop gradient
grad times input
deep lift
Irp
guided backprop
gradient
grad times input
deep. lift
n_pop=80, 0=0000025 LS=false
pop=80,
0.0125. LS=false
0.06-
5.376-02
5.71e-01
5.3te-02
5.35-002
0,37
MSF
0.04
0.024
1.33e-02
1436-02
0316-02
136-00 02
1244-22
guided backprop gradient
grad times input
deep lift
trp
guided backprop gradient
grad times input deep lift
Figure 9: Similarity in lerms of MSE between input image, x, and best final adversarial, aades for 8 different hyperparameter configurations. Dataset: ImageNet. Model: VGG16. A higher learning rate and a smaller population size (i.e., more gradient steps) contribute to the perturbation of the image. The sampling method has no effect.
22

---

Published in Transactions Machine Learning Rescarch (08/2023)
1e-9 p.pop=12,, 1r=250 True
pop=125, 1r=0.00625, S-True
deep_ lift
gradient
Irp
MSE
grad times jnnput
quided_backppee
1-
0
100
200
300
400
100
200
300
400
Generation
Generation
1e-9 n. p0popp 125, Ir- 0.0125, LS-False
n. 1o.pop=122 125, 0r=0.00 False
3-
MSE
2-
1-
0
100
200
300
400
0
100
200 Generation
300
400
Generation
1e-9
n_pop= 80, or-011 LS-True
n_pop 80, Ir 0r=0.00 LS True
3-
MSE
1-
0
100
200
300
400
500
600
0
100
200
300
500
600
Generation
Generation
1e-9
n_pop=80, lr=0.00625, LS=False
n_pop=80, 1r=250,,5, LS=False
3-
MSE
2-
0
100
200
300
400
500
600
0
100
200
300
400
560
600
Generation
Generation
MSE loss value function of evolutionary generation for 8 different hyperparameter configurations. Datasct: ImageNet. Model: VGG16.
Figure 10:
23

---

Published in Transactions on Machine Learning Rescarch (08/2023)
1e-9
pop=80. Ir=0.0125, LS=False
pop=80. Ir=000
MSE
1.0-
68
633-110
97
663--100
6.b--10
10 10
31e-10
10
3,00-10
362e-10
0.5+
irp
guided backprop gradient
grad times input deep lift
trp
guided backprop
gradient
grad times input
deep_ lift
1e-9
n p_pop=125, 1r=0.0125, LS=True
n_pop=125, 1r=0.0 LS=False
MSE
1.0-
6.80--10
26
5.99--11
6.67-100
300-10
San 1d
346--10
364e-10
2655-10
600e-10
0.5*
guided backprop
gradient
grad times deep lift
times_input input
guided backprop
gradient
grad times deep lift
Lines_inuut input
1e-9
p_pop=125, 1r=0.00625, LS=True
n_pop=125, Ir=0.00625 LS=false
1.30p-10 10
7-326-10
6:43--10
73
MSE
3,760110
888-10
Irp
guided backprop
gradient
grad times input
deep lift
trp
guided guided_backnng backprop
gradient
grad times, input
deep
pop==80 f==0.0625, Tr=
pop==88 Ir=0.00625, FS=le
1.0-
1225-10
6.58-11
6.5p-10
1133-10
6
2
MSE
3.71-10
78e-10
3
3.776-10
0.5
Irp
guided backprop
gradient
grad times deep. lift
times_input input
Irp
guided backprop
gradient
grad times, input
deep lift
Figure 11: Similarity in terms of MSE between target explanation map, g(Starget), and best. final adversarial explanation map, g(Zado), for 8 different hyperparameter configuratinss. Dataset: Ima- geNet. Model: Inception. The Gradient XAI and Guided backpropogation methods are the most susceptible to attacks while Deep Lift, LRP, and Gradient x Input are less susceptible.
24

---

Published in Transactions on Machine Learning Rescarch (08/2023)
pop=80, 1r=0.0125, LS=False
pop=80, 1r=0.0125, LS=True
0.06
$.56.07
$12600
5.14--22
5100
$186-02
5256-07
5.
226-62
5 oz
220
MSE
0.04
0.02-
Irp
guided backprop
gradient
grad times input
deep_ liftt
Irp
guided backprop
gradient
times_inpul input deep, lift
grad times
pop=125, 1r02.0 LS=True
n pop=125, Ir-0.0 LS=false
0.06
4.12e002
0966-02
024--02
11le02
4.1e-OZ
4.76-02
401e-07
4076-0
407e-02
MSE
0.04
0.02
Irp
guided backprop
gradient
grad times input
times_input
deep_lift lift
Irp
guided backprop gradient
grad times input deep lift
n 1op= 0 0.6255.. LS=True
pop=125, 1r= LS=false
MSE
0.04
0.02-
1.04-02
02
144e-02
1:44-00
102e-022
033-00
105e-00
04
Irp
guided backprop
gradient
grad times input
fimes_inptt deep lift
Irp
guided backprop gradient
grad times input
deep lift
80
1r=0.0062) True
pop=80, 1r=0.00625, FS=la
0.06-
MSE
0.02
322-002
31e-02
292--02
286-02
31e-02
326-00
31e02
33e-02
326-07
guided backprop
gradient
grad times Input deep lift
guided _ backprop gradient
grad times input deep lift
Figure 12: MSE loss value for input image versus chosen adversarial image for 8 different hyperpa- rameter configurations. Dataset: ImageNet. Model: Inception. A higher learning rate and smaller population size (i.c., more gradient steps) contribute to the perturbation of the image. The sampling method has no effect.
25

---

Published in Transactions Machine Learning Rescarch (08/2023)
1e-9 pop=880 1r=0.0125, LS=False
pop=80. Ir=0.0 True
deep_liff _lfi
grad times_inple
1.25
gradient
MSE
1.00-
guided_backpra backprop
0.75-
Irp
0.50-
200
400
600
200
400
600
Generation
Generation
1e-9 n_pop= 125, 1r00.0 True
n_ n__ppp= 125, 0.0125, False
1.25
MSE
1.00-
0.75-
0.50-
100
200
300
400
100
200 Generation
300
400
Generation
1e-9 n_pop 125, lr-00, LS True
n_pop 125, Ir- 0.00600 LS False
1.25-
MSE
1.00-
0.75-
0.50-
100
200
300
400
100
200
300
400
Generation
Generation
1e-9 n n_pop=80, 80, Ir=0000 LS=True
80 Ir=0.00625, Fa=Fal
1.25-
MSE
1.00-
0.75
0.50-
2ô0
400
600
200
400
600
Generation
Generation
Figure 13: MSE loss value function of evolutionary generation for 8 different hyperparameter configurations. Datasct: ImageNet. Model: Incoption.
26

---

Published in Transactions on Machine Learning Rescarch (08/2023)
1e-6
p.popp15, Ir=0.00
pop=125, 1=0.00625,
MSE
07
L.30-00 06
8776687
650-077
9
77 07
Ne-00
55
536-07
290-99
Irp
guided backprop
gradient
grad times input
deep lift
guided, backprop
deep lift n_pop=125, 01125 LS=False
gradient
grad times input
1e-6
1_pop=125, 0r-2.5, LS=True
3-
MSE
7.476-07
1.
4313--07
7.360-07
8A5eat
7.3
Lame- 96
01
3:3--077
hrp
guided backprop
grad times dimesinnpt input deep lift n_pop=80, 1r=0.0 LS=True
gradient
guided, backprop gradient
grad times input
deep lift
1e-6
p_pop=80, lr=0000 LS=false
MSE
24
70
1.00-00
7003007
7950-77
7:20e-07
D4 06
441
1:30-07 07
Lasee or
Irp
guided backprop
gradient
grad times input
deep lift
Irp
guided, backprop quiked_backprop
gradient
grad times input
deep lift
pop=880 1-=0 0-025, LS=false
pop=88, f==000622 LS=
MSE
2-
70
1.05-066
2
6.95e-07
0.00-07
7.28
Th 06
4.3--07
7.45e-07
14
hrp
guided backprop gradient
grad times input deep lift
trp
guided backprop
gradient
grad times times_input input
deep_ lift
Figure 14: Similarity in terms of MSE between target explanation map, g(Starget), and best. final adversarial explanation map, g(Zadv), for 8 different hyperparameter configurations. Dataset: CI- FAR100. Model: VGG16. The Gradient XAI method is the most susceptible to attacks while Guided backpropogation and Deep Lift are the hardest to attack; LRP and Gradient x Input are similar.
27

---

Published in Transactions on Machine Learning Rescarch (08/2023)
pop=125, 0=0.00 LS=True
pop=125, Ir=0.00625. LS=False
0.06-
MSE
0.04
0.02-
02
33e-00
1056-02
10le02
1le-02
1.07-022
336447
1046-02
1.44-02
13
Irp
guided backprop gradient
grad times input
deep lift
lrp
guided backprop
gradient
grad times input
deep, lift
n pop=125, 1r=0.0 LS=True
n pop=125, 1=-0.0 LS=false
0.06
4376-02
736-022
82
#00e-02
3.766-02
T334402
333322
394e-02 02
4114
MSE
0.04
0.02
Irp
guided backprop
gradient
grad times input
times_input
deep_lift lift
Irp
guided backprop gradient
grad times input deep lift
pop=80,
00025,, LS=True
n pop=80, 000255. LS=False
5216-02
5888-02
8.08-00 02
02
S.de-02
0.06
MSE
0.04
1.34--02
130--02
134e-02
138e02
0.02
Irp
guided backprop gradient
grad times input
dimes_input deep, lift
Irp
guided backprop gradient
grad times input
deep_ lift
p0 Ir= 0.0125, FS=False
pop=80, Ir= 0.00625, LS LS= True
5.02e002
5:126-02
5.15e-02
02
5 02
0.06
MSE
0.04
1.34-022
02
336802
3346-02
Irp
guided backprop
gradient
grad times input deep lift
guided backprop gradient
guided_backprop
grad times input deep lift
Figure 15: MSE loss value for input image versus chosen adversarial image for 8 different hyperpa- rameter configurations. Dataset: CIFAR100. Model: VGG16. A higher learning rate and smaller population size (i.c., more gradient steps) contribute to the perturbation of the image. The sampling method has no effect.
28

---

Published in Transactions Machine Learning Rescarch (08/2023)
1e-6 m.pppp=15, Ir=0.00625, True
pop=125, 1r=0.00625 LS=False
deep_liff _lfi
4-
grad times_inple
3-
gradient
MSE
guided_backprag backprop
2-
Irp
1-
0
100
200
300
400
100
200
300
400
Generation
Generation
1e-6 n. n_popp 125, 0.0125, True
n. popppop-lle 125, Ir 0.0125, LS-False
4-
3-
22
1-
0
100
200
300
400
0
100
200 Generation
300
400
Generation
1e
6
pop.pp-oll 80, or-0.1155 LS-True
n_pop= 80, tr-0.0065,, LS - False
3-
2-
0
100
200
300
400
500
600
0
100
200
500
600
Generation
Generation
1e-6
80 1r=0.0,55, FS=False
B0, Ir=0000 LS=True
MSE
0
100
200
300
400
500
600
0
100
200
300
400
560
600
Generation
Generation
MSE loss value function of evolutionary generation for 8 different hyperparameter configurations. Datasct: CIFARI00. Model: VGG16.
Figure 16:
29

---

Published in Transactions on Machine Learning Rescarch (08/2023)
1e-6
pop=p25, Ir= LS=False
n. pop=125, l==0.0
MSE
102e-06
46
6222-077
956-07
360-07
971e07
466-06
621e077
98
9117647
trp
guided backprop
gradient
grad times limes_inpu input
deep lift
guided, backprop
gradient
grad times input
deep lift
1e-6
nopo-=00, lr=0.00625, LS=True
1_pop=80, 1r=0.00625, LS=False
1. 06
MSE
2
9.726-07
961e-77
99
8.8 07
07
6.198-77
503=-07
trp
guided backprop
gradient
grad times imee_input input
deep lift
guided, backprop
gradient
grad times input
deep. lift
1e-6
pop==125, Ir= 00625 LS=False
pop==12, 1r=0.00625, LS=True
1.61e-06
MSE
2-
9 95ge 07
9 908-07
7Be-07
100e 06
9.00-07 07
6.366-07
6
trp
guided backprop
gradient
grad times input
deep lift
trp
guided, backprop quided_backkpro
gradient
grad times input
deep
pop=80, 0r=050 LS=True
pop==80 Ir=0 0.025, LS=False
MSE
9.71e-07
380-08
9.74477
T 966
887e-77
428-08
6.118-07
6.190-0
hrp
guided backprop
gradient
grad times input
deep lift
trp
guided backprop
gradient
grad times timessiimut input
deep_ lift
Figure 17: Similarity in terms of MSE between target explanation map, g(starget), and best. final adversarial explanation map, g(Zadv), for 8 different hyperparameter configurations. Dataset: CI- FAR100. Model: MobileNet. The Gradient XAI method is the most susceptible to attacks while Guided backpropogation is the hardest to attack; Deep Lift, LRP, and Gradient x Input are similar.
30

---

Published in Transactions on Machine Learning Rescarch (08/2023)
pop=125. 1r000,,,, FS=false
pop=125, Ir=0.0125, LS=True
0.06-
3836-02
436--02
7.76-02
Jale-02
4.0e 02
3,786-02
423e-02
1726-02
35ze-12
MSE
0.02
Irp
guided backprop
gradient
grad times input
deep_ lift
Irp
guided backprop
gradient
grad times _npput
deep lift
n pop=80, 00625, LS=True
n pop=80, Ir=0.00625, LS=false
0.06-
MSE
0.04
0.02-
3444-02
4.46-02 02
131e-z
328-02
416--22
1.326-02
1484-02
Tie-42
02
Irp
guided backprop
gradient
grad times, input
deep deep_lift lift
Irp
guided backprop gradient
grad times input
deep lift
pop=125, 000255,, LS=false
pop=125, 0r-2.0 LS=True
0.06-
MSE
0.04
0.02-
001e-02
15e-02
91e-03
001602
100-022
1226-02
01e-02
Go0e
Irp
guided backprop
gradient
grad times input
deep lift
Irp
guided backprop
gradient
grad times input
deep lift
po Ir= 0. 0125, LS=True
pop=80, 1r=0.5, LS=
0.06
501e-2
5526-02
d 90e--22
02
$333--02 02
5.136-02
547642
433002
5.0e--02
02
MSE
0.04
0.02-
Irp
guided backprop
gradient
grad times input deep lift
guided backprop
_ gradient
grad times input deep lift
Figure 18: MSE loss value for input image chosen adversarial image for different hyperparam- eter configurations. Dataset: CIFAR100. Model: MobileNet. A higher learning rate and smaller population size (i.c., more gradient steps) contribute to the perturbation of the image. The sampling method has no effect.
31

---

Published in Transactions Machine Learning Rescarch (08/2023)
1e-6 n. 2_pop=125, 1r-0.0 LS=False
pap=125. 1r=0.0125, LS=True
deep_liff _Jfllftt
3-
grad times_inpue
gradient
MSE
guided_backprog
Irp
1-
0
100
200
300
400
100
200
300
400
Generation
Generation
1e-6 n. 1l.pppp8..
0.00625 LS True
n_ poppop8000 80, 0r00.5, LSlFe
3-
MSE
2-
0
100
200
300
400
500
600
0
160
200
300
400
500
600
Generation
Generation
1e 6 o_pop m 125, Ir- 0. 0.00625, False
n_pop 125, - 0.00625, LS True
3-
2-
1-
100
200
300
400
100
200 Generation
300
400
Generation
1e-6
1_pop=80, 0r=25 LS=True
n_pop=80, 1r=25,,, LS=False
3-
MSE
2-
1-
0
100
200
300
400
500
600
0
100
200
300
400
560
600
Generation
Generation
MSE loss value function of evolutionary generation for 8 different hyperparameter configurations. Datasct: CIFARI00. Model: MobileNet.
Figure 19:
32


Reviews:
[{"review_id": "peuqwQ6hhC", "reviewer": "Reviewer_PXqs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n  - The idea of constructing adversarial examples with the goal of foiling not a classifier but an explanation is an intriguing one. As far as I am aware most prior work on this topic had focused on leveraging adversarial examples with respect to to an image's classification to provide insight into the interpretability of the model. The construction of adversarial examples with respect to the explanation seems to suggest that these methods might not be as robust as previously thought, which is certainly of interest to the broader community.\n\n  - The paper provides several nice visualizations of the adversarial examples found, which gives qualitative insight into the method's strengths and limitations, and covers several different methods of generating model explanations.\n\nWeaknesses:\n\n  - The adversarial examples found are visually perceptible perturbations in several instances, in contrast to the paper's claims that the perturbations are imperceptible. The perturbed photos look noticeably more grainy and often have odd patterns or small patches of colour. This suggests the method may not be finding 'true' adversarial examples in many cases, as it requires too large of a perturbation budget in order to attain the desired explanation output.\n\n  - The main paper lacks quantitative results. How often is the method successful across different network architectures / explanation methods at a given budget? How does the perturbation magnitude trade off with the quality of the adversarial example? How does the computational budget? These questions cannot be answered by a handful of pre-selected examples, and are necessary to validate the authors' claims about the method. While the authors include several figures in the appendix, a concise summary should be included in the main body of the paper.\n\n  -  The method is very sample-expensive ( using 50K model queries per example). Part of this can be attributed to the parameterization of the problem that allows for the use of the reinforce/reparameterization trick. This begs the question of why this particular approach was chosen, and I didn't notice a compelling justification for this approach in the paper. In principle, it seems as though using a unimodal gaussian distribution shouldn't offer much benefit over a greedy method that would require fewer model queries per generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for producing image perturbations which change the output of saliency maps towards arbitrary values. The approach is evaluated on several interpetability methods and image datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please update the claims about imperceptible perturbations to more accurately reflect the (frequently clearly perceptible) examples shown in the paper.\n\n2. The authors should include a comprehensive summary of their quantitative results on the success rate of their method to obtain an approximation of a given explanation at a given budget for various degrees of approximation. \n\n3. The choice of method would benefit from further justification as to why this particular form of population based search is needed.\n\n\nErrata: please make appropriate use of \\citep and \\citet. The sudden appearance of names outside of the citation's parentheses is confusing when the citation is used in passing.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z97avbcmKL", "reviewer": "Reviewer_Yi5a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The approach is a black-box attack and does not rely on model parameters. \n\n2. The presentation of the approach is detailed and well-structured.\n\nWeakness:\n\n1. The paper does not have discussions on broader impact and potential real-world values of its study on adversarial attacks against XAIs. \n\n2. There's no quantitative evaluation of their approach in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposed an adversarial attach against explainable AI algorithm based on evolution strategy algorithm and Monte Carlo sampling to estimate the gradient of a fitness function. The attach method only relies on the classification logits and the explanation map. It demonstrates good results on a wide range of experiment settings, across different models (VGG, MobileNet) and different datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper should include discussions on broader impact and potential real-world values of its study on adversarial attacks against XAIs, which could help better motivate their study.\n", "broader_impact_concerns": "Since the work studies adversarial attacks on AI models, the paper should discuss what real-world threats the proposed approach could pose for existing XAI approaches and the values of their study in terms of dealing with such threat.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hWenf3JmHy", "reviewer": "Reviewer_rAzr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The algorithm is model-agnostic and data-agnostic, making it practical in real-world scenarios.\n2. The authors show the effectiveness of the framework on two benchmark datasets using four different pretrained deep-learning models.\n3. The paper is well-written and provides a clear storyline that motivates the proposed approach.\n\n**Weaknesses**\n1. The success of the black-box attack on explanation maps will require a sufficient number of queries to generate effective adversaries. It will be constructive if the paper elaborates more details on the query cost (e.g., time of inference/API call) and discuss how adversarial effectiveness is affected by the number of queries.\n2. Authors choose to use NES for the black-box optimization. It will be validating the framework if authors can do ablation study on different types of black-box optimization.\n3. The paper shall provide more details on how the crafted loss can balance two objectives: maintaining the model prediction while manipulating the explanation map. \n4. Authors should analyze the perturbation budget (i.e., the numerical range of noise) imposed on the image. A too large budget will affect the image quality and make it hard to pass user observation. The paper should evaluate how the perturbation affects the image quality and perform analysis on the relation of perturbation budget with attack success rate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a new framework, AttaXAI, to add pixel-space perturbation on the input image (which maintains a consistent classifier output) to mutate explanation maps for DNN classifiers. The algorithm uses Natural Evolution Strategies (NES) which is model-agnostic: only requires access to (1) the model output logits and (2) the explanation map. The authors tested AttaXAI against the ImageNet and CIFAR100 datasets to show the effectiveness of the framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my concerns stated in the weakness section.", "broader_impact_concerns": "Given that the pixel-space perturbation is hard to apply to real-world images, the authors should discuss how the proposed attack will have potential real-world applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is interesting and they all voted for acceptance. They raised several issues, please see the list below:
1. Please update the claims about imperceptible perturbations to more accurately reflect the (frequently clearly perceptible) examples shown in the paper.
2. The choice of method would benefit from further justification as to why this particular form of population-based search is needed.
3. Please make appropriate use of \citep and \citet. The sudden appearance of names outside of the citation's parentheses is confusing when the citation is used in passing.
4. The success of the black-box attack on explanation maps will require a sufficient number of queries to generate effective adversaries. It will be constructive if the paper elaborates more details on the query cost (e.g., time of inference/API call) and discusses how adversarial effectiveness is affected by the number of queries.
5. The paper shall provide more details on how the crafted loss can balance two objectives: maintaining the model prediction while manipulating the explanation map.
6. Authors should analyze the perturbation budget (i.e., the numerical range of noise) imposed on the image. A too-large budget will affect the image quality and make it hard to pass user observation. The paper should evaluate how the perturbation affects the image quality and perform an analysis of the relation of the perturbation budget with the attack success rate.

If any of these points are already addressed in the paper, please comment on them.

Decision: Accept (poster)