Using Explainabilty to Detect Adversarial Attacks
Authors: Ohad Amosy and Gal Chechik
Venue:  (2020)
Keywords: adversarial
detection
explainability

Abstract: Deep learning models are often sensitive to adversarial attacks, where carefully-designed input samples can cause the system to produce incorrect decisions. Here we focus on the problem of detecting attacks, rather than robust classification, since detecting that an attack occurs may be even more important than avoiding misclassification. We build on advances in explainability, where activity-map-like explanations are used to justify and validate decisions, by highlighting features that are involved with a classification decision. The key observation is that it is hard to create explanations for incorrect decisions.  We propose EXAID, a novel attack-detection approach, which uses model explainability to identify images whose explanations are inconsistent with the predicted class. Specifically, we use SHAP, which uses Shapley values in the space of the input image, to identify which input features contribute to a class decision. Interestingly, this approach does not require to modify the attacked model, and it can be applied without modelling a specific attack. It can therefore be applied successfully to detect unfamiliar attacks, that were unknown at the time the detection model was designed. We evaluate EXAID on two benchmark datasets CIFAR-10 and SVHN, and against three leading attack techniques, FGSM, PGD and C&W. We find that EXAID improves over the SoTA detection methods by a large margin across a wide range of noise levels, improving detection from 70% to over 90% for small perturbations.

Under review as a conference paper at ICLR 2019
USING EXPLAINABILTY TO DETECT ADVERSARIAL ATTACKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep learning models are often sensitive to adversarial attacks, where carefully-desineed input samples can cause the system to produce incor- rect decisions. Here we focus on the problem of detecting attacks, rather than robust classification, since detecting that an attack occurs may be even more important than avoiding misclassification. We build on ad- vances in explainability, where activity-map-like explanations are used justify and validate decisions, by highlighting features that are involved with a classification decision. The key observation that it hard to create explanations for incorrect decisions.
We propose EXAID, novel attack-detection approach, which uses model explainability to identify images whose explanations are inconsistent with the predicted class. Specifically, we use SHAP, which uses Shapley values in the space of the input image, to identify which input features con- tribute to class decision. Interestingly, this approach does not require to modify the attacked model, and it can be applied without modelling a specific attack. can therefore be applied successfully to detect unfa- miliar attacks, that were unknown at the time the detection model was designed. We evaluate EXAID on two benchmark datasets CIFAR-1o and SVHN, and against three leading attack techniques, FGSM, PGD and C&W. We find that EXAID improves over the SoTA detection methods by large margin across wide range of noise levels, improving detection from ~ 70% to over 90% for small perturbations.
INTRODUCTION
Machine learning systems can be tricked to make incorrect decisions, when presented with samples that were slightly perturbed, but in special, adversarial ways (Szegedy et al.) 2013). This sensitivity, by now widely studied, can hurt networks regardless of the ap- plication domain, and can be applied without knowledge of the model (Papernot et al., 2017 Detecting such adversarial attacks is currently key problem in machine learning.
To motivate our approach, consider how most conferences decide on which papers get accepted for publication. Human classifiers, known reviewers, make classification de- cisions, but unfortunately these are notoriously noisy. To verify that their decision are sensible, reviewers are also asked to explain and justify their decision. Then, second classifier, known as an area-chair or an editor, examines the classification, together with the explanation and the paper itself, to verify that the explanation supports the decision. If the justification is not valid, the review may be discounted or ignored.
In this paper, we build on a similar intuition: Explaining a decision can reduce misclas- sification. Clearly, the analogy is not perfect, since unlike human reviewers, for deep models we do not have trustworthy methods to provide high level semantic explanation of decisions. Instead, we study below the effect of using the wider concept of explana- tion on detecting incorrect decisions, and in particular given adversarial samples that are designed to confuse classifier. The key idea is that different classes have different plaining features, and that by probing explanations, one can detect classification decisions that are inconsistent with the explanation. For example, if an image is classified as dog,

---

Under review as a conference paper at ICLR 2019
but has an explanation that gives high weight to striped pattern, it is more likely that the classification is incorrect.
We focus here on the problem of detecting adversarial samples, rather than developing system that provides robust classifications under adversarial attacks. This is because in many cases we are interested to detect that an attack occurs, even if we cannot automati- cally correct the decision.
The key idea in detecting adversarial attacks, is to identify cases where the network be- haves differently than when presented with untainted inputs, and previous methods fo- cused various different aspects of the network to recognize such different behaviours
Lee et al. (2018); Ma et al. (2018); Liang et al. [2018); Roth et al. (2019); Dong et al. 2019); Katzir & Elovici (2018); Xu et al. (2017). To detect these differences, here we build on recent work in explainability Lundberg & Leel (2017b). The key intuition is that explain- ability algorithms are designed to point to input features that are the reason for making decision. Even though leading explainability methods are still mostly based high-order correlations and not necessarily identify purely causal features, they often yield features that people identify as causal (Lundberg & Lee, 2017a). Explainability therefore operates directly against the aim of adversarial methods, which perturb images in directions that are not causal for class. The result is that detection methods based explainability holds the promise to work particularly well with adversarial perturbations that lead to nonsensical classification decisions.
There is second major reason why using explainable features for adversarial detection is promising. Explainable features are designed to explain the classification decision of a classifier trained on non-modified (normal) data. As a result, they are independent of any specific adversarial attack. Some previous methods are based on learning the statistical abnormalities of the added perturbation. This makes them sensitive to the specific perturbation characteristics, which change from one attack method another, or with change of hyperparameters. Instead, explainability models can be agnostic of the particular perturbation method.
The challenge in detecting adversarial attacks becomes more severe when the perturba- tions of the input samples are small. Techniques like C&W Carlini & Wagner (2017b) can adaptively select the noise level for given input, to reach the smallest perturbation that causes incorrect classification. It is therefore particularly important to design detection methods that can operate in the regime of small perturbations. Explanation-based detec- tion is inherently less sensitive to the magnitude of the perturbation, because it focuses those input features that explain decision for given class.
In this paper we describe an EXAID -Ex---ei-then-Deteee) an explanation-based method to detect adversarial attacks. It is designed to capture low-noise perturbations from un- known attacks, by building an explanation model per-class that can be trained without access to any adversarial samples.
Our novel contributions are as follows: We describe a new approach to detect adver- sarial attacks using explainability techniques. We study the effect of negative sampling techniques to train such detectors. We also study the robustness of this approach in the regime of low-noise (small perturbations). Finally, we show that the new detection pro- vides state-of-the-art defense against the three leading attacks (FGSM, PGD, CW) both for known attacks and in the setting of detecting unfamiliar attacks.
RELATED WORK
EXPLAINABLE AL.
Several methods have been recently proposed to address black box decisions of Al sys- tems. LIME RRbeiro et al.) (0016) is based on locally approximating the model around given prediction with simple interpretable model (e.g. decision tree). DeepLIFT (Shrikumar et al. 2017) uses a modified version of back propagation to compute the con- tribution of each input feature to the output. SHAP (Lundberg & Lee 2017a) approximates

---

Under review as a conference paper at ICLR 2019
Owl
Figure 1: Illustration of EXAID. First, an image is classified by a standard image clas- sification system like ResNett Then, an explanation is created based on the image, the network activations and the network output. Finally, a detector checks if the generated explanation is consistent with the predicted label. (a) An image of an owl correctly classified, and the produced explanation is consistent with the label "owl". (b) An image of peacock perturbed and used as an attack. is falsely classified as an owl, and is detected as adversarial because its explanation is inconsistent with the predicted label.
the Shapley values of input features, which were derived in cooperative game theory to distribute the total gains to the players. Specifically, for explaining deep neural networks, SHAP use variant of DeepLIFT as approximation for Shapley values.
ADVERSARIAL ATTACKS
The literature on adversarial attacks vast. We focus here on three high-performing adversarial attacks which are relevant for the experiments. Each of the three represents group of attacks that share the same main idea.
Fast Gradient Sign Method (FGSM). This attack by (Goodfellow et al. 2014) creates perturbation by "moving" an example one step in the direction of the gradient. Let c be the true class of x and x,C, be the loss function used to train our deep neural network C. The perturbation
computed as sign of the model's loss function gradient Ax =x=c E * sign( ( (C, c)), where ranging from 0.0 to 1.0. The parameter e controls the magnitude perturbation and can be thought as the noise-level the adversarial sample.
Projected Gradient Decent (PGD). Madry et al. (2017) suggested to improve FGSM, in the following way. One can interpret FGSM as a one-step scheme for maximizing the inner part of the saddle point formulation. A more powerful adversary will be multi-step variant, which essentially applies projected gradient descent on the negative loss function x = x + * sign (C, while x² =
Carlini and Wagner (C&W). Carlini & Wagner (2017b) employed an optimization algo- rithm to seek the smallest perturbation that enables an adversarial example to fool the classifier. As showed in (Carlini & Wagner, 2017a), this attack is considered be one of the most powerful attacks, and theretore is common baseline.
When designing attacks, previous studies took into account various factors: the probabil- ity that the attack is successful, the effect on the appearance of perturbed image, and the time takes run the attack. The above three methods prioritize these aspects dif- ferently, reaching different tradeoff operating points. Specifically, FGSM is usually faster and the C&W attack yields less-visible perturbation of the input images.
DETECTING ADVERSARIAL ATTACKS
Several previous techniques have been proposed to detect adversarial examples. Liang et al. (2018) measured the effect of quantization and smoothing of the image on the net- work classification, both parameterized as function of image entropy. Similarly, Xu et al. (2017) suggested to reduce the degrees of freedom of the input space by applying transformations like quantization and smoothing, and then compare model predication before and after each transformation. Ma et al. (2018) measures the characteristics of the region surrounding reference example. Lee et al. (2018) models the distribution of ac-

---

Under review as a conference paper at ICLR 2019
tivation at the hidden layers of the classifier, using Gaussian mixture model, and feeds the likelihood to a classifier. Katzir & Elovici (2018) models the changes in the labels of k-nn for each activation layer in the base model. Pang et al. (2018) propose using new loss in training, which encourages the neural network to learn latent representations that better distinguish adversarial examples from normal ones. RRoth et al. (2019) models the statistical robustness of log-odds to perturbations, for normal and adversarial examples. Generally speaking, these methods assume that adversarial examples differ intrinsically from natural images, either in the sample space or because the perturbation affects prop- agation of activity in the neural network. Some of those methods require modifying the base model. Very recently, (Fidel et al. 2019) described an explanation-based approach to detection, related to the current paper:
EXAID: EXPLAIN THEN DETECT
EXAID consists of two components: (1) Explain. Create per-class explanations for both correct predictions and incorrect ones. (2) Detect. Train binary classifier to decide if an explanation is consistent with the class decision. These two components are schematically shown in Figure
3.1 EXPLAIN
The first step in EXAID implements an explanation model. Given a pretrained classifier that may be attacked, we used an explainability model to extract explanations for every sample classified by the model. The explanation model can take as input the raw input image, as well as the whole base model architecture and weights, and produces an expla- nation in the terms of the input features. Formally it is a function that maps a sample and classifier, and its prediction into explanation space & (x, fiheta (x)) + R", where fiheta classification model producing a predicted label y = output
Since our goal is to learn which explanations are typical for each class, we collected both positive explanations- applying an explanation model to a correct prediction the network, and negative explanations corresponding to incorrect predictions of the model.
SAMPLING EXPLANATIONS
Creating positive explanations Exi,,y) is usually straight forward, one simply applies the explanation model on each sample that was correctly classified fx,) = y,. More care should be given to creating negative explanations. We consider three types of negative explanations: wrong negatives, adversarial negatives and other-class negatives.
First, one may collect samples (x, where the model made an incorrect decision fxx) # y;, and collect their explanations (x;,()))) fy;)). We name these wrong negatives. For models that well trained, the number of these explanations is small. Furthermore, not all classes are confused by other classes, and only some classes may lead to explanations of some other classes.
Second, one can employ an adversarial attack on the training data and collect negative explanations of adversarially perturbed samples. We name them adversarial negatives. As with wrong negatives, these explanations correspond to cases where the model made an incorrect decision, but unlike wrong negatives the explanations may have a different distribution, because the input was designed to confuse the network. Even if the specific type of adversarial attack is not known, these samples may be useful because they are based on fooled decisions and may reflect typical patterns of adversarial examples. How- ever, training against an incorrect attack may cause overfitting to a specific type of attack and hurt detection accuracy.
Third, for every labeled sample (x, y:), we produce explanations E(x;, y) for all incorrect classes y V, * yi. For example, for car image correctly classified as car, we produce explanations for classes like dogs and cats. These explanations are used as other-class negatives for the correct class y;:

---

Under review as a conference paper at ICLR 2019
THE EXPLAINABILITY MODEL
As an explainable Al approach we used SHAP deep explainer. As shown in (Lundberg & Leee 2017b) SHAP considered a leading explainer, providing explanations that have stronger agreement with human explanations than other methods. We therefore believe is likely to capture the "correct" features by which people make labeling decisions. In addition, Lundberg & Lee| (2017b) has shown that SHAP is the only explainer that has both local accuracy and consistency, which are desirable properties.
3.2 DETECT
Given set of positive and negative explanations per class, train deep binary classifier per class, to detect explanations that are inconsistent with model predictions. Note that in this settings, it is natural to train detector as a binary multiclass multi-label classifier, and not as a multiclass classifier, because we wish to condition the decision on the prediction of the image classifier.
When training the detector, one may consider two learning setups, aiming to protect against unknowwn-attacss, or against familiar attacks. It appears if defending against known attack would be an easier task, because one may learn the properties the attack. Unfortunately, since new attacks can be easily designed, it is highly desirable to devise generic defenses.
We address this topic by controlling the data that is used for training the detector. Specif- ically, we consider two variants of EXAID.
EXAID familiar. During training, the binary detector is presented with adversarial nega- tives. It can therefore learn a distribution of explanations resulting from a specific adver- sarial attack. Specifically, we trained using high-noise FGSM.
EXAID unknown. The binary detector is not presented with any adversarial negatives during training. The only negative explanation the classifier trained on are other-class negatives and wrong negatives.
Below we tested both variants on the known attack (FGSM) and on unfamiliar attacks (PGD, C&W).
4 EXPERIMENTAL RESULTS
We evaluated EXAID on two benchmark data sets, in the task of attack detection. Our code will be available at htttps://githiie com httpc///aonoooooseaaoooo////////
DATASETS
We evaluated EXAID on two standard benchmarks: CIFAR1o (Kiizhevsky al.) and SVHN (Netzer et al. 2011). As Carlini & Wagner (2077)) showed, MNIST is not a good dataset for evaluating adversarial detences. This is probably due the fact that it is low-dimension dataset, making it easier to detect changes an attacker made to the image. [arlinii Wagner 2017a) show their results on CIFAR-10. In order to show the validity of our results on more than one dataset, we also used SVHN that has similar complexity. The CIFAR-10 dataset consists of 60,000 32X32 colour images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. The SVHN dataset is obtained from house numbers in Google Street View images. It consists of more than 600,000 32x32 colour images in 10 classes. While similar in flavor to MNIST, it comes from a significantly more diverse distribution. We used the 73,257 digits provided for training and the 26,032 digits for testing.

---

Under review as a conference paper at ICLR 2019
IMPLEMENTATION DETAILS
For both CI and SVHN we used pretrained Resnet34 as a base model. To train the EXAID detector extract positive explanation, wrong negative and other-class negative from natural images as described in algorithm [1] The EXAlD-unknown model was trained on those explanations. To train EXAID-familiar we extracted adversarial negative using FGSM attack with a noise level of c 0.1.
Algorithm Create positive and negative explanation, and train bad explanation detector for each class
Input: F Trained model, (X, Y) Dataset of labeled samples
2: Initialize: positives +- array of empty sets, negatives + array of empty sets
3: for (x, E (X do
4: 1 F(x)
5:
explanation = SHAP(F,x) if - y then
6:
positives 9 append (xppaatttitaaaoion(())
7:
8:
for i - 1, 1..1l do
negatives [appen((expllnniiinnn [i] // Collect other-class negative end for
9:
11: else
// Collect wrong negatives
13: end if
14: end for
15: for = 1..1t
16: C Tran(posstivesil,eegttteee))))))
// Train classifier for class
17: end for
As described, we used SHAP as an explainability model (Lundberg & Lee 2017a). The original implementation SHAP runs on CPU and therefore suitable for use only on small number of samples, rather than entire datasets. To enable us to run SHAP on large number of examples we modified the original implementation to run on GPU. The new version is available in our Git repository.
DEFENSE BASELINES
We compared EXAID with three recently-published adversarial detection baselines, and two new variants of these baselines.
(1) ANR (Liang et al. 2018). A method based on measuring the effect of quantization and smoothing of the image on the network classification, both are parameterized as function of the image entropy. We used the implementation provided by the authors. Since ANR not tested in the original paper on CIFAR-10 and SVHN as done here, tuned the hyperparameters of their method using hyperopt (Bergstra et al., 2013).
(2) Mahanalobis (Lee al., 2018). This approach models the distribution of activations in the hidden layers of the classifier, as obtained in response to natural (unperturbed) sam- ples, using a Gaussian mixture model. Given a set of likelihood scores from the GMM, a classifier is trained to determine a set of activations is obtained in response to an ad- versarial example or natural one. That classifier trained on adversarial examples. We used the implementation provided by the authors, and as the original paper, we trained the classifier with adversarial examples crafted by FGSM.
(3) Mahanalobis Unsupervised. We modified the method of (Lee et al.) 2018) to reach attack-agnostic baseline as follows. Instead training an attack-dependent discriminator adversarial samples, estimated the likelihood of set of network activations the product of likelihoods of all layers.
(4) LID (M et al., 2018). LID measures the characteristics of the region surrounding reference example, and give it a likelihood score. This is done separately for each repre- sentation of the example, in the classifier's hidden layers. As in Mahalanobis, classifier

---

Under review as a conference paper at ICLR 2019
is trained to determine if set of activations is obtained in response to adversarial example or a natural one. We used the implementation from (Lee et al. 2018), and trained the classifier with adversarial examples crafted by FGSM.
(5) Unsupervised LID. As for Mahalnobis, we test an unsupervised version of LID, based the product of likelihoods of individual layers, without training classifier.
(a) CIFAR-10 FGSM
(b) SVHN FGSM
1.0
1.0
0.8
0.8
auc
auc
0.6
0.6
0.4
0.4
10-3
10-2
10-1
10-3
10-2
10
noise level
noise level
(c) CIFAR-10 PGD
(d) SVHN PGD
1.0
1.0
0.8
0.8
auc
auc
0.6
0.6
0.4
0.4
10
10
10
10
10
10
noise level
noise level
(e) CI C&W
(f) SVHN C&W
1.00
1.00
auc
0.75
DuC
0.75
0.50
0.50
10-4
10 -0
10 -2
10
10 10-3
10
noise level
noise level
Unsupervised Ud
Original Lid
Unsupervised Mahanalobis
Original Mahanalobis
ANR
EXAID unknown (ours)
FXAID familiir (ours)
Figure Defense methods comparison. Each sub-figure compares EXAID variants to five baselines. (a,b) defend against FGSM, for CIFAR-10 and SVHN. (c,d) same against PGD (d,e) Same against C&W. EXAID outperforms all baselines in low-noise scenarios, and is comparable in the high-noise regime.

---

Under review as a conference paper at ICLR 2019
4.1 EXPERIMENTAL SETUP
We test the detection models against oblivious adversaries, an attack scenario in which attacker has full knowledge about the model (white box attack), but is not aware of the existence of the defense model. We believe this is a relevant scenario, since in the real world, most attackers will not have direct access the attacked model and its defense. In this case, the attacker will be forced to use black box attack. However, as (Papernot et al., 2017) showed, adversarial examples are transferable between models. Given transterability attacking black box model is not marginally harder than white box. Because of that, we baseline our model against a white box attack as which is harder to detect. This is not the case when the model is defended, since (Li et al. 2019) shows that the transferability of adversarial examples works well between vanilla neural networks, but fail to transfer between defended neural networks.
We believe that the magnitude of perturbation used by an attack a major factor that determines the success of adversarial detection methods. There is still clear protocol in the literature about comparing attacks and detections depending on this factor, and different reported experiments use different values. We therefore repeated all experiments for a wide range of noise levels and report performance across that wide range.
4.2 ATTACKS
We used three attack methods to test EXAID: (1) One step gradient attack (FGSM) Good- fellow et al. 2014), (2) Iterative projected gradient (PGD) (Madry et al., 2017) and the
Carlini and Wagner attack, which uses optimization to add as small as possible perturba- tion (C&W) (Carlini & Wagner 2017b). All attacks
implemented using Advertorch (Ding et al.) 2019). Opposed to other defense methods benchmarks, we examined the effect noise-level on range of three orders of magnitude.
RESULTS
The results for all detection methods are shown in figure 2 EXAID significantly outper- forms the other methods when the noise level is small (small perturbations), and with attack methods that use adaptive noise levels (C&W). Typically, the AUC is increased from 70% to over 90%.
LID and Mahalanobis both perform well in high noise scenarios, and slightly outperform EXAID SVHN in these scenarios. However, when the noise level decreases LID and Mahalanobis performance suffers drastically, while EXAID's remains high. Interestingly, our unsupervised variant of LID, performs at least as well, and sometimes better, than the original LID. This may be because LID was trained with FGSM samples and may de- teriorate in cross-attack scenarios. These findings show the importance of benchmarking defense models against wide range of noise levels.
6 CONCLUSION
In this paper we proposed EXAID, novel attack-detection approach, which uses model explainability to identify images whose explanations are inconsistent with the predicted class. Our method outperforms previous state-of-the-art methods, for three attack meth- ods, and many noise-levels. We demonstrated that the attack noise level has a major impact on previous defense methods. We hope this will encourage the research commu- nity to evaluate future defense methods on a large range of noise-levels.
8

---

Under review as a conference paper at ICLR 2019
REFERENCES
James Bergstra, Dan Yamins, and David D Cox. Hyperopt: A python library for opti- mizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python in science conference, pp. 13-20. Citeseer, 2013.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: By- passing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, PP- 3-14. ACM, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural net- works. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017b.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch vo.1: An adversarial robustness toolbox based on pytorch. arXiv preprint ar 8rXi0:1902.07623, 2019.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable ad- versarial examples by manslation-innvriintt attacks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 4312-4321, 2019.
Gil Fidel, Ron Bitton, and Asaf Shabtai. When explainability meets adversarial learning: Detecting adversarial examples using shap signatures. arXiv preprint ar Xivu:1999.03418. 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arx preprint arXit:1412.6572, 2014.
Ziv Katzir and Yuval Elovici. Detecting adversarial perturbations through spatial behavior in activation spaces. Xrviv preprint ar arXio:8811.00043 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-1o (canadian institute for ad- html
vanced research). URL http::/www. http://wwwc.ttrrooo.eeee//////2/ wttp:/tto....oorrreeeeeeeea
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Adoances in Neural Information Processing Systemss pp. 7167-7177, 2018
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: the distributions of adversarial examples for an improved black-box attack on deep neural networks. arXiv preprint ar (XXi:19055000444 2019.
Learning
Bin Liang, Hongcheng Li, Miaoqiang Su, Xirong Li, Wenchang Shi, and XiaoFeng Wang- Detecting adversarial image examples in deep neural networks with adaptive noise reduction. IEEE Transactions on Dependable and Secure Computing, 2018.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model pre- dictions. In 1. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems Systems 30, pp. 4765-4774- Curran Associates, Inc., 2017a. URL prdepppeciingoolleeeeeeeeeiittoo 7062 a a-w---iineeeeeeee aapppoaapadd---epccccc------ to
www.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017b.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael Houle, and James Bailey. Characterizing adver- sarial subspaces using local intrinsic dimensionality. arXiv preprint ar arXiv:1801.02613, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arxiv preprint 77X1001706.06083, 2017.

---

Under review as a conference paper at ICLR 2019
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of ad- versarial examples. In Advances in Neural Information Processing Systems, pp. 4579-4589, 2018.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference computer and communications security, pp. 506-519. ACM, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Ex- plaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD inter- national conference on knowledge discovery and data mining, pp. 1135-1144- ACM, 2016.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. arXiv preprint arXiv:1902.0481 2019.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Con- ference on Machine Learning- Volume 70, Pp. 3145-3153. JMLR. org, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ar ariiv preprint arXie:1322.6199, 2013.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial exam- ples in deep neural networks. arv preprint XxX::1704.001553 2017.
10


Reviews:
[{"review_id": "r1xUKwB0FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests a method for detecting adversarial attacks known as EXAID, which leverages deep learning explainability techniques to detect adversarial examples. The method works by looking at the prediction made by the classifier as well as the output of the explainability method, and labelling the input as an adversarial example if the predicted class is inconsistent with the model explanation. EXAID uses Shapley values as the explanation technique, and is shown to successfully detect many standard first-order attacks.\n\nThough method is well-presented and the evaluation is substantial, the threat model of the oblivious adversary is unconvincing. The paper makes the argument that oblivious adversaries are more prevalent in the real world, but several works [1,2,3,etc.] have shown that with only query access to input-label pairs from a deep learning-based system, it is possible to construct black-box adversarial attacks. Thus, it is unclear why an attacker cannot just treat the detection mechanism as part of this black box, and mount a successful query-based attack. \n\nThough I recognize that the task of detection is separate from the task of robust classification, in both cases the defender should at least operate in the case where the attacker has input-output access to the end-to-end system (including whatever detection mechanisms are present). In particular, it seems impossible to \"hide\" a detector from an end user (when the method detects an adversarial example, it will alert the user somehow that the input was rejected), and so the user will be able to use this information to fool the system. The authors should investigate the white-box accuracy of their detection system, or at the very least try black-box attacks against the detector. For this reason I do not recommend acceptance for the paper at this time.\n\n[1] https://arxiv.org/abs/1804.08598\n[2] https://arxiv.org/abs/1807.04457\n[3] https://arxiv.org/abs/1712.04248", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skxtu9afqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose an explanation-based adversarial example detection algorithm. The main idea is to train a discriminator to detect whether the explanatory saliency map is consistent with the input. Experiments have been conducted on CIFAR10 and SVHN to validate the method.\n\nComments:\n+ The idea is straightforward and easy to follow.\n\n- The use of SHAP as the only explanation method is not well explained. There are a plenty of works on visual explanation methods, such as guided-backprop[1], excitation-backprop[2], integrated gradient[3], Grad-CAM[4], real-time saliency[5] and so on. And based on my expertise, SHAP cannot generate the most accurate saliency among these methods.  If the proposed framework is general, why not to conduct ablation study on the different choice of explainer?\n\n- Doubts on the effectiveness of the proposed method. According to former works[6, 7], explanatory saliency methods  are vulnerable and unreliable with respect to input perturbations. But in this paper, the authors assume that the explanation saliency map for normal examples are perfectly correct and used as positive instances for training the discriminator. I think they only focus on target attack, in which the attacking target label is semantically distinct from the original label, and the resulting saliency map distribution is very different from the correct one. However, considering a tabby cat image is perturbed to become tiger cat, since two classes are very close, the resulting saliency maps should be similar and the detector may fail to detect the adversarial example. Therefore, I encourage the authors to provide more results on this challenging scenario (for example, conduct un-target attack on imagenet dataset).\n\n- The reported results in Figure 2(e) is abnormal. First, the blue line (authors' method) is very close to AUC=1.0 across different noise levels, which means that the detector can perfectly classify all the adversarial examples in all the situation. Second, the reported values of other methods are not correct. For example, the black line (original Mahalanobis) is below AUC=0.5 across all the noise level. However, in the Table3 ResNet-CIFAR10 row of its original paper[8], the reported AUC under C&W attack is 95.84, which is much larger than those shown in the figure. Therefore, I think the comparison is invalid. Similar problems also appear in Figure 2(f).\n\n[1] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. (2015). Striving for simplicity: The all convolutional net. In ICLR (workshop track).\n[2] J. Zhang, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff. (2016). Top-down neural attention by excitation backprop. In ECCV.\n[3] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. In ICML. \n[4] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV.\n[5] Dabkowski, P., & Gal, Y. (2017). Real time image saliency for black box classifiers. In NeurIPS.\n[6] Kindermans, P. J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., ... & Kim, B. (2019). The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (pp. 267-280). Springer, Cham.\n[7] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In NeurIPS\n[8] Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeuJ1NX9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to check whether a model is under attack by using state of the art explainability model, SHAP. They evaluated their technique using CIFAR-10 and SVHN w.r.t. 5 baseline techniques. They showed their method outperforms all the other baselines with a significant margin.\n\n+ Overall I think the paper made a valuable contribution to the adversarial ML literature. Using explainability to detect the presence of adversarial attacks seems like a nice intuitive idea and the results show that it indeed works. \n\n-\tHowever, the contribution of the paper is rather incremental. They just used SHAPE to adversarial and negative examples. I do not see any insight while explaining the results. \n\n-\tWhy under PGD and FGSM attack under higher noise, the proposed technique is similar or slightly worse than Lid and Mohalonabis baselines?\n\n-\tI would also like to see how these results hold good for a complicated dataset like ImageNet\n\n\t\t\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bke1qCDAtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A Simple method to detect adversarial examples, but needs more work.\n\n#Summary:\nThe paper proposed a method that utilizes the model’s explainability to detect adversarial images whose explanations that are not consistent with the predicted class.  The explainability is generated by SHAP, which uses Shapley values to identify relative contributions of each input to a class decision. It designs two detection methods: EXAID familiar, which is aimed to detect the known attacks and EXAID unknown, which is against unknown attacks. Both of the two methods are evaluated on perturbed test data which are generated by FGSM, PGD and CW attack with perturbations of different magnitudes. Qualitative results also show that the proposed method can effectively detect adversaries, especially when the perturbation is relatively small.\n\n#Strength\nThe method is easy to implement and using the idea of interpretation for detecting adversarial examples seems interesting.\n\nGood results are demonstrated compared with other comparators.\n\n#Weakness\nThe idea of this paper is based on the interpretation method of DNN. However, it has been shown that these interpretation methods are not reliable and easy to be manipulated [1][2]. Therefore, although the method is simple to design, it also brings other security concerns.\nUnfortunately, the paper does not address these issues. In addition, the comparators listed in the experiments are not state-of-art or common baselines. It is either not clear why authors modified the existing method and develop their own “unsupervised” version. \nIn the experiments, many details are omitted. For example, how is the “noise level” defined? Are they based on L1, L2 or L-inf perturbation? For PGD attack, how many iterations does the generation run and what is the step size? How many effective adversarial examples are generated for training and testing? And all the experiments are conducted in a relatively small dataset, it is also suggested to do experiments on large datasets, e.g. Imagenet.\nIn the evaluation part, it looks strange to me why the EXAID familiar performs worse than EXAID unknown in evaluating FGSM attack on SVHN since the EXAID familiar is trained using FGSM attack.\n\n#Presentation\nI think the authors used a wrong template to generate the article. The font looks strange and the headnote indicates it is prepared for ICLR2020. The paper contains many typos and even the title contains a misspelling. Poor coverage of citations. There are more works for detecting adversarial examples that are published, e.g. [3][4][5]. On the other hand, the paper does not have the literature review for work related to the model interpretation.\n\nOverall, I think the paper is not good enough for publication at ICLR.\n[1] Dombrowski, Ann-Kathrin, et al. \"Explanations can be manipulated and geometry is to blame.\" arXiv preprint arXiv:1906.07983 (2019).\n[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. \"Interpretation of neural networks is fragile.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n[3] Meng, Dongyu, and Hao Chen. \"Magnet: a two-pronged defense against adversarial examples.\" In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135-147. ACM, 2017.\n[4] Liao, Fangzhou, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. \"Defense against adversarial attacks using high-level representation guided denoiser.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787. 2018.\n[5] Ma, Shiqing, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. \"NIC: Detecting Adversarial Samples with Neural Network Invariant Checking.\" In NDSS. 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes EXAID, a method to detect adversarial attacks by building on the advances in explainability (particularly SHAP), where activity-map-like explanations are used to justify and validate decisions. Though it may have some valuable ideas, the execution is not satisfying, with various issues raised in comments. No rebuttal was provided.
Decision: Reject