Model-based Saliency for the Detection of Adversarial Examples
Authors: Lisa Schut
Yarin Gal
Venue:  (2020)
Keywords: Adversarial Examples
Defense
Model-based Saliency

Abstract: Adversarial perturbations cause a shift in the salient features of an image, which may result in a misclassification. We demonstrate that gradient-based saliency approaches are unable to capture this shift, and develop a new defense which detects adversarial examples based on learnt saliency models instead. We study two approaches: a CNN trained to distinguish between natural and adversarial images using the saliency masks produced by our learnt saliency model, and a CNN trained on the salient pixels themselves as its input. On MNIST, CIFAR-10 and ASSIRA, our defenses are able to detect various adversarial attacks, including strong attacks such as C&W and DeepFool, contrary to gradient-based saliency and detectors which rely on the input image. The latter are unable to detect adversarial images when the L_2- and L_infinity- norms of the perturbations are too small. Lastly, we find that the salient pixel based detector improves on saliency map based detectors as it is more robust to white-box attacks.

Under review as a conference paper at ICLR 2020
MODEL-BASED SALIENCY FOR THE DETECTION OF ADVERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial perturbations cause shift in the salient features of an image, which often results in misclassification. Previous work has suggested that these salient features could be used defense, arguing that with saliency tools could successfully detect adversarial examples. While the idea itself is appealing, we show that prior work which used gradient-based saliency tools is ineffective adversarial defense - it fails to beat a simple baseline which uses the same model but with the saliency map removed. To remedy this, we demonstrate that learnt saliency models can capture the shifts in saliency due to adversarial perturbations, while also having a low computational cost. This allows saliency models be used effectively as real-time defense. Further, using the learnt saliency model, we propose a novel defense: CNN that distinguishes between adversarial images and natural images using salient pixels as its input. On MNIST, CIFAR-10 10, and ASSIRA, our defense improves on using the saliency map alone, and can detect various ad adversarial attacks. Lastly, we show that even when trained weak de- fenses, we can detect adversarial images generated by strong attacks such as C& and DeepFool.
INTRODUCTION
Adversarial examples highlight a crucial difference between human vision and computer image pro- cessing. Often computers fail to understand the relevant characteristics of an image for classification (Ribeiro et al., 2016) or fail generalize locally, i.e., misclassify examples close the training data SSzeedy et al. (2013). Attacks exploit this property by altering pixels the classifier heavily relies on an-pixxe pixels which are irrelevant humans for object recognition. As a consequence, adversarial perturbations fool classifiers while the correct class remains clear to humans.
Saliency maps identify the pixels an image classifier uses for its prediction; as such, they can be used as tool to understand why classifier is fooled. Building on this concept, researchers have shown qualitatively that adversarial perturbations cause a shift the saliency of classifiers (Fong & Vedaldi 2017; Gu & Tresp 2019). Figure shows examples of natural image and corresponding adversanal images, each above their respective saliency maps. The saliency maps corresponding to adversarial images show perceptible differences to that of the original image, even though adversar- ial images themselves often seem unperturbed. For the original image, the saliency map shows that the classifier focuses on the four (and couple of random pixels the left). We observe that for the adversarial images, the classifier starts focusing more on irrelevant aspects of the left side of the image.
There is ample research into different techniques for finding saliency maps (see e.g. Zeiler & Fergus 2014 'Springenberg al., 2014, Bach al., 2013) Ribeiro al., 2016 Shrikumar al., 2017
Selvaraju et al. 2017;] ZZintgraf et al. 2017 Fong & Vedaldi 2017). However, pot all saliency maps are equally informative (Fong Vedaldi 2017). For example, the Jacobiar can be used to determine the saliency of pixel in the classification of the image (Papernot et al. 201 Zhang jet al., 2018). As the Jacobian often used to generate adversarial examples, intuitively, we expect that it can be used effectively to detect adversarial perturbations. /Zhang et al. (20T8) propose defense to this effect: they determine whether an input is adversarial, given the Jacobian-base
i... the forward derivative of the classifier with respect the input image

---

Under review as a conference paper at ICLR 2020
Original
FGSM
BIM
MIM
JSMA
C& W
DF
4
9
9
9
4
9
9
4
Figure 1: Example of the saliency maps of natural and adversarial images for the MNIST dataset. The top is the input image and the bottom shows the corresponding saliency map. In the second row, lighter colours correspond to higher saliency (black corresponds to saliency of O. the lowest possible value). The classifier predicts (from left to right) the images as: 4, 9, 9 8, 9, 9. Note the stark difference between the saliency masks of the original image and those of the adversarial examples.
saliency map concatenated with the image. However, as shown qualitatively by Gu & 'resp 2019), gradients are not always able to capture differences between adversarial images and natural images (for an example see Figures 7 and g in Appendix D) Here we inspect the proposed Jacobian-based approach and show that only the concatenated input affects the technique's performance in detecting adversarial examples, with the Jacobian having no effect.
While gradients may not be informative for detection, saliency should be effective tool for detect- ing adversarial images. In our analysis, we use more powerful model-based saliency techniques and show that the magnitude of the shift of the saliency map due adversarial perturbations often ex- ceeds the L2 distance between the saliency maps of different natural images. Building on this result, we consider two different possible effects adversarial perturbations might have on the classifier:
1. They might cause the classifier to focus the wrong pixel locations
2. They might change the pixel values of salient pixels
Based on these hypotheses, we employ two CNN classifier architectures to detect adversarial im- ages. Claim (1) can be captured by shifts in saliency maps, as previously considered by Fong & Vedaldi (017)). In this work, we extend on their analysis by proving the defensive capability of our model-based saliency against difficult black-box attacks, such C&W and DeepFoo as well as white-box adversarial attacks. By considering claim (2), we demonstrate that incorporating pixel values improves the performance of the classifier when shifts in saliency maps do not suffice to cap- ture adversarial perturbations. We also show that our salient-pixel based defense generalizes well (detecting stronger attacks when trained on weaker attacks) and is more robust than the saliency map defense against white-box attacks. Lastly, demonstrate that saliency can be used detect adversarial examples generated by small perturbations, contrary to other defenses, which exhibit threshold behavior: i.e., when the adversarial perturbation is too small, other defenses (specifically Gong et al. 2017 Zhang et al. 2018) are unable to detect the adversarial images.
2 SALIENCY AND ADVERSARIAL EXAMPLES
Saliency maps and adversarial perturbations have similar mathematical formulations and deriva- tions. Both are computed by investigating the relation between the values of pixels and the classi- fication score. Adversarial examples are found by deriving the minimal perturbations required to change the classification of an image. Saliency computed by finding the pixels used by the model
Simillarly, Fong Vedaldi (011)) show that gradient- hased heat maps are less effective than other saliency methods in detecting adversarial perturbations generated using BIM (Kurakin et al., 2016).
"Their main contribution is that saliency maps generated by different techniques
not equally effective in
capturing changes due to adversarial perturbations (produced using BIM (Kurakin et al. (0166).
"These attacks generate smaller T.2 perturbations, making them more difficult to detect. The perturbation
size used by Fong & Vedaldi (207)) can likely still be detected by simple classifier that trains on images.
2

---

Under review as a conference paper at ICLR 2020
to determine the class of an object (Simonyan al. 2013). Saliency maps be found by consider- ing the smallest part of an image that is sufficient for a correct classification, known as the smallest sufficient region (SSR), or whose removal is sufficient for an incorrect classification, known as the smallest destroying region (SDR) (Dabkowski & Gal 2017 Fong & Vedaldi 2017). Observe that the latter definition of saliency is very close that of adversarial examples.
Mathematically, both saliency maps and adversarial perturbations can be derived a similar fashion. Consider ad examples. The general formulation of an adversarial attack can be summarized as follows:
min///rlla s.t. f(x (ie+r)= +r) # TEEEE E [0. 1)
(1)
where x is the natural image, r is the adversarial perturbation, y the correct class, and y is an incorrect class. Due to the non-linearity of NNs, solving the above problem requires non-linear optimization. Therefore, in practice several different approaches to solving the above formulation
have been implemented. For example, Goodfellow et al. 20144) set r esign( 6f(w)) Similarly,
saliency can be computed using the forward derivative kf(m) 2018).
Papernot et al. 2016b Zhang et al,
6a
Previous research has already started investigating the relation between saliency and adversarial examples. This includes:
Using saliency to attack Researchers have devised adversarial attacks that use saliency (Papemot al. 2016b) Yu al.] (018). The key idea is to use saliency determine the pixel that is most sensitive to perturbation iteratively. The main benefit is that fewer pixels are perturbed often perturbing as few as 4% of the pixels suffices to change the classification of the image (Papernot
et al. 2016b).
Using saliency to defend Fong & Vedaldi (0177) introduce a method that detects adversarial per- turbations by using mapt-map visualizations of the predicted class. However, in their analysis, they only use BIM (Kurakin et al. (0166), which is easily detected. Further, Zhang et al.) (2018) hypoth- esize that there a mismatch between the saliency of classification model and the adversarial example. They propose defense against adversarial attacks by training a classifier on images con- catenated with their saliency map, which is computed by calculating the Jacobian of the classifier with respect to the image x, ie., $g (x)) Zhang et al. (2018) find that their method obtains high accuracy (often near 100%) when detecting adversarial images generated by FGSM, MIM, and C&W attacks MNIST, CIFAR-10 10, and 10-ImageNet. However, Gu & Tresp) (2019) contradict these results, and demonstrate that the gradients show imperceptible differences due to adversarial perturbations (see Figures and in Appendix DJ.
Adversarial robustness and interpretability of models Fong & Vedaldi (2017) and Gu & Tresp 2019 show that saliency maps can be used to explain adversary classifications. Both highlight an important trend: not all techniques used compute saliency maps show shifts in saliency maps due to adversarial perturbations. Further, [Tsipras et al. (2018) shows that more robust models have more interpretable saliency masks. [Etmann al. (2019) quantify the relation by investigating the alignment between the saliency map and the input image.
3 MODEL-BASED SALIENCY FOR THE DETECTION OF ADVERSARIAL EXAMPLES
In this section, we explain how we construct and evaluate sabiency-based adversarial example detectors.
Work done Gu & Tresp (2019) was in parallel.
3

---

Under review as a conference paper at ICLR 2020
3.1
CLASSIFIER
We train convolutional neural network image classifier, which we target with black-box attacks; the architectures are summarized in Appendix[A] We use cross-entropy loss and optimize the parameters using Adam (Kingma & Ba 2014) with the standard hyperparameters settings.
3.2 BLACK-BOL ATTACKS
In our analysis, we consider six different adversarial attacks: Fast Gradient Sign Method (FGSM,,
Basic Iterative Method (BIM), Momentum Iterative Method (MIM), L2 Carlini & Wagner (2017b) (C&W), Jacobian-boee based Saliency Map Approach (JSMA) and DeepFool (DF) (for each attack see
Goodfellow al. 2014 Kurakin al.
2016 Dong et al. 2018: Carlini & Wagner 2017b; Papernot
et 2016b) Moosavi-Dezfooll et al.
2016, respectively). We use the implementation as provided
in cleverhans (Papernot al. 2016a). The hhper- are summarized in Appendix B
3.3 MODEL-BASED SALIENCY
To generate saliency masks, we adapt the method used by Dabkowski & Gal (2017). Our rea- son is twofold: the technique computes high-quality saliency masks at a low computational cost. Dabkowski & Gal (2017) employ U-Net with novel loss function that targets SDR, SSR, mask sparsity, and mask smoothness. We adapt the original loss function to omit the total variational term, as mask smoothness not required in our analysis.
Let f. = (a, fz(z)) denote the generated map. First, the map average (Vsss) is used to ensure that the area of the map is small. Second, (og(fo(((x, fa))) is included to ensure that the salient pixels suffice to identify the correct class. Finally, fo(2(x,) - fa)) is included to ensure that the classifier can no longer recognize the class if the saliency map is removed. Therefore, our saliency loss function is:
L(f.) AAV((f) log(ff fa)) + (01,1(((111)))))
(2)
where fe is the softmax probability of the class c, Φ(x, fa) applies mask f. to image and A; 2 0 are paraee-s
We adapt the PyTorch implementation provided by Dabkowski & Gal 2017 and train the saliency model on standard, non-adversarial images only. For evaluation, we use the same saliency model for both natural and adversarial images. When generating the saliency maps for our images, the predicted classification for feature selection to prevent an information leak (which would occur if we use the true label).
3.4 DEFENSE
Our hypothesis is that if an image is adversarial, the classifier likely focuses on the wrong aspects or the pixels on which it focuses are misleading (due to the perturbed color or intensity) when classifying an image adversarial. We consider two different cases by building classifiers for (1) saliency maps and (2) salient pixels. For both classifiers, we use the same architecture (and hyper- parameters) as for the black-box image classifiers (as summarized in Appendix Ap.
3.4.1 SALIRNCY MAP: "THE CLASSIFIER FOCUSES ON THE WRONG PIXEL LOCATIONS"
We build detector based on the saliency maps of images as follows. First, we train a classifier and generate adversarial images for every natural image the training dataset. Then we generate the saliency maps for the clean data {X)) and adversarial images (Xadv)) }. We build binary detector for the saliency maps, which predicts whether the corresponding image is adversarial or natural. We abbreviate this defense as SMD (Saliency Map Defense). We do not concatenate the saliency maps to the input image.
TThe code is available http://gthccccom/aaaaaaaaaiiiiat

---

Under review as a conference paper at ICLR 2020
3.4.2 SALIENT PIXEL: "THE PIXEL VALUES ARE WRONG"
We construct a second classifier for the salient pixels. We follow the same sleps as outlined in the previous section, aside from the final step. We define the salient pixels as fz(x) where is the image, (z) the saliency map corresponding to and denotes the element-wise product. We abbreviate this defense as SPD (Salient Pixel Defense). Similarly to SMD, we do not concatenate the saliency maps to the input image.
3.5 BENCHMARKS AND BASELINES
To benchmark our results, consider two baselines. First, train baseline classifier that classi- fies imput as adversarial or natural based on the images alone. This allows us to evaluate the added benefit of using saliency maps. This method was implemented by Gong et al. (2017). We abbreviate this defense as ID (Image Defense).
Second, we compare our defense method with the saliency- defense of Zhang et (2018) (see Section 2. We abbreviate this defense JSD, for Jacobian- based Saliency map Defense. In our implementation, we adapt the method of Zhang et al. (2018): we find that if we use fa( Va (xx as the saliency map it leads to underflow, resulting in zero matrix. Therefore, instead we take the derivative with respect to the logits, i.e. [x(x) (x))
JSD is mathematically related to the other defenses. First, it is more general compared to ID: the filters of JSD can learn to ignore the Jacobian-based saliency, in which case the two methods are equivalent. Further, JSD is similar to SMD, as the filters can learn to ignore the image input. In this case, the only difference between JSD and SMD is that they use different techniques to derive saliency. However, JSD differs from SPD, CNN filters cannot multiply one channel by another.
3.6 EVALUATION WITH BLACK-BOX ATTACKS
We follow the evaluation protocol Zhang et al. (2018) and train each defense to detect adversarial images generated by a specific attack, thereby generating six different detection models (one for each black-box attack). To generate the training data, we generate one adversarial example for every clean image. The training data becomes [X, Xadel. where X denotes the clean data and Xade denotes the adversarial data, and the labels are [1n, O,.] 1n and O,, are one- and zero-vectors of length n, respectively. We use the same training procedure and models, as summarized in Appendix A and report the accuracy of the classifiers the test dataset.
We compare the performance of the models on MNIST, CIFAR-10 and ASSIRA (see Burges & Cortes, 1998 Krizhevsky al., 2009 Elson al. 2007 respectively). In addition to the two frequently used benchmarks, we consider the ASSIRA cats and dogs datasef] as it contains high- quality images but is less computationally expensive than ImageNet Further details on the datasets can be found in Appendix A
3.7 WHITE-BOX ATTACK
Many defenses hold up against black box attacks but often are unable to defend against white-bon box attacks (Carlini & Wagner 2017a). For this reason, generate white-box attacks tailored to the defense strategy. Our white-box attacks are iterative gradient-basee based attacks, which target both the classifier and the defense. Inspired by FGSM, we can target the classifier f as
Eadv Clip(æ + esign( V fI
(3)
and the defense d as
Tado = Clip(æ + esign V Vd((zdd)))
(4)
where Clip clips the pixels to the pre-defined maximum range. Using the above idea, we iterate between Equations 3 and to generate the white-box attack for ID (the defense based on image
with the images scaled 112 112 for computational reasons.
"In particular, JSMA would have required substantial time; other researchers have omitted this attack on ImageNet due to the high computational cost that results from the high resolution of the images (Carlini & Wagner 20175).
5

---

Under review as a conference paper at ICLR 2020
classification).
We propose similar white-bos box attacks for the other defenses, shown in Appendix
C We limit the number of iterations T to 5, as we find to be adequate to generate a sufficiently strong attack and further increasing T does not improve the performance.
Our method similar to that of Metzen et al. (017)). They propose finding adversarial examples as: Tudo = (ad + (asiin)( V (Irdo)) + (1 s)siig(( Vd(zade))),
(5)
where in our case a 0.5. The key difference is that we iterate between Equations 3 and 4 rather than applying 3 and 4 simultancously. We find that this is more effective at targeting the defense, which is more difficult to fool than the original classifier.
DEFENSE RESULTS
We start by assessing the shift in saliency maps generated by adversarial perturbations and then present the efficacy of the detector against different adversarial attacks. Details, such as attack success rate, can be found in Appendix
4.1 L2 DISTANCES BETWEEN SALIENCY MAPS
We start by quantifying the shift in saliency maps due to adversarial perturbations; we compute the L2 distance between saliency maps natural image and its corresponding adversarial image. baseline, compare these values with the L2 distance between two different natural images. These statistics are summarized in Table
For CIFAR-10 10 and ASSIRA, the L2-norm between the saliency maps of natural image and its cor- responding adversarial image is comparable to or larger than the L2 distance between two different natural images. Using a Mann- Mann-Whitney U-test, prove quantitatively that the shift is significant for most adversarial attacks on CIFAR-I0 and ASSIRA images. This suggests that our saliency-based method is effective way of capturing adversarial perturbations.
Table 1: Lg distance between (1) saliency maps of different images (row labelled Different Images) and (2) the saliency maps of natural images and the adversarial image (generated by the type of attack The p-value
specified the row). The entries correspond derived using the Mann Whitney U-test, where we test whether the sample of L-2 distances between a natural and adversarial image is from the same distribution as different images. We use a non- parametric test avoid assuming normality of the data.
STATISTIC
MEAN (109 11000111000)
STD (10"/10 (10"/10-1/10")
P-VALUE
DIFFERENT IMAGES
0.13/0.75/0.26
0.03/0.28/0.03
FGSM
0.09/0.83/00.20
0.03/0.27 /0.03
0.00/0.00/0.00
BIM
0.08/0.76/0.29
0.03/0.27/0.03
0.00/0.31/0.00
MIM
0.08/0.76/0.22
0.04/0.27/0.03
0.00/0.39/0.00
C& W
80//0.
0,04/0.27/0.03
0.00/0.00/0.000
0000000
DF
0.07 /07/0.8//0.00
0.03/0.27/0.03
0.00/0.00/0.00
JSMA
0. 0.10/0.88/0.29
2.03/0.27/0.03
0.00/0.00/0.000
4.2 BLACK-BOX DEFENSE
Figure Summmrrizes the performance of the defense models trained on single adversarial attack on different adversarial attacks; the values and standard deviations can found in Appendix G The over- all performance of the model-based saliency defense suggests that saliency can used determine whether an image is adversarial.
Salient Pixel Defense outperforms Saliency Map Defense Overall, SPD (shown in blue) out- performs the other defenses, suggesting that the salient pixels provide useful information to the detector. Further, our defense generalizes well: even when trained on a weaker attack, SPD is able to detect stronger attacks. Both baseline methods, ID and JSD, only generalize well when trained on a stronger attack. When trained on a weaker attack, they are not able to detect stronger adversarial attacks.

---

Under review as a conference paper at ICLR 2020
FGSM
BIM
MIM
C&W W
DF
JSMA
MNIST
CIFAR-
ASSIRA
Figure 2: From left right: performance of SPD, SMD, ID and JSD in detecting adversarial amples when trained on FGSM, BIM, MIM, DE, CW and JSMA. Rows correspond to MNIST, CIFAR-10 and ASSIRA, respectively.
Jacobian-based Saliency is Uninformative The method introduced by Zhang et al. (2018) does not perform well the (Jacobian) saliency mask defense does not improve fense congruent observation was made qualitatively by Gu & Tresp 2019), who show that changes in gradient due to adversarial perturbations are imperceptible (for an example, see Figures and in Appendix DD. Our quantitative results build on their observation, showing that not all saliency methods show shifts due adversarial perturbations.
mnage-based de-
Worse generalization on JSMA We observe drop in performance of the models when detecting JSMA, likely because JSMA is an Lo-norm attack, which generates a different type of adversarial examples. This may suggest that defenses trained on specific norm, only generalize well other attacks generated by a norm that produces similar perturbations. FGSM, BIM, and MIM are L norm attacks, and &W and DF are Tn-m attacks. Both generate perturbations that are spread out over the entire image, contrary to Lo norm attacks, which changes few pixels using larger perturbations.
Threshold Behavior Both and JSD exhibit threshold behavior: they are unable to detect adver- sarial examples if the perturbation size is below given threshold. For example, see the performance of both defenses on the ASSIRA dataset. There is a strong correlation between detection accuracy and perturbation size, as measured by Loo and L2 (see Table 2). ID is able detect all adversar- ial images for which the perturbation size is either L2 > 0.027 or Loo > 0.50, such as FGSM and JSMA However, the perturbations are much smaller for DF and CW, making these attacks harder to detect. The threshold appears to occur around L2 = 0.025, as ID can sometimes detect the FGSM perturbations, generated with this size. This observation in line with the results of Gong et al. (2017). who find that ID is highly efficient at detecting adversarial images with perturbations e e 2 0.03 but unable to detect adversarial perturbations generated using = 0.01 (using FGSM for images scaled between 0 and 1), obtaining an accuracy of 50.0% in the latter case IT
We observe that JSD performs similarly, although sometimes worse, compared to ID. Theoretically, the parameter space of ID is subset of the parameters of JSD. The additional input (the Jacobian) makes the model more difficult to train. Therefore, difference in results can be attributed training: model more difficult train due to the increased number parameters and does not learn to ignore the additional input.
10FSSM known generate large perturbations. The perturbations for JSMA relatively large the attack minimizes the To norm, thereby perturbing as few pixels as possible, but by large amount.
"Our perturhations for FGSM are larger than 0.01 to ensure that FGSM is sufficienlly strong (see Appendix for summary of the attack success rates).

---

Under review as a conference paper at ICLR 2020
Table 2:
Perturbation Sizes of Different Attacks. 10/ASSIRA. Entries for which ||rl|2 2 0.025 and |||] perturbation. m is the number of pixels in an image.
Entries correspond to MNIST/CIEAR.
0.5 are in bold, where is the adversarial
ATTACK
PERTURBATION RATE
PERTURBATION SIZE
(La/m)
L2
FGSM BIM
0.61/0.91/0.99
0.15/0.19/0.025
020/0.20/0.08
00.05
061/0..2/0.88
/0.013
0.15/0.15/0.02/
MIM
0.62/0.91/0.. /0.93
0.10/0.11 /0.019
0.15/0.15/0.023
C W
0.99/1.00/1.00
/0.0017
0.32/0.013/0.007
DF
1.61/0.830..9
0.062/0.0019/0.00m
0.45/0.013/00044
JSMA
0.13/0.03/0.03
0.30//.0039/..13
1.00/0.77/0.50
4.3 WHITE-BOX DEFENSE
Table 3 summarizes the performance of different defenses against our white-box attack. Our white- box methods highly effective in fooling the classifier well as the defenses for MNIST and ASSIRA, as shown by the before adversarial training results. The white-box attack is unable to fool the detector for CIFAR-10 successfully.
Next, we perform advensarial training: we iteratively train the detectors against the white-box attack and allow the white-box attack access the defense. The white-box attack no longer cessfully defeats SPD, which becomes more robust against the attack, whereas SMD is not able to become robust against the white-box attack.
Table 3: Performance of the different defenses against white-box attacks. Classifier accuracy refers to the accuracy of the image classification. Defense accuracy refers to the accuracy of the adver- sarial image detector (i.e. the ability to distinguish adversarial images from natural images). The entries correspond to standard deviations.
and the values in the parentheses below denote
MODEL
DEFENSE
SMD
SPD
Before Adversarial Training
CLASSIFIER
0.0/0.00/0.03
0.0/0.00/0.13
(0.00)00.0000000)
(0 10.00//0 00)(000))
DETECTOR
0.05/0.50/0.35
0.18 0.18/0.95/0.33
(0.02)K..07)K0006
(0. 10.110000.11000000
After Adversarial Training
CLASSIFIER
0.03/0.00/0.02
0.00/0.0070.04
(0.03Y0000000000
(0.
DETECTOR
0.25/00.44/0.00
0.79/0.93/0.77
(0.20)(0000)K00007) (0.00)(0.01>X0000)
5 CONCLUSION
In our analysis, we ascertain that the saliency maps of adversarial images differ from those of nat- ural images. Further, we show that salient pixel based defenses perform better than a saliency map defense. When trained on single black-box attack, our method is able to detect adversarial pertur- bations generated by different and stronger attacks.
We show that gradients are unable to capture shifts in saliency due to adversarial perturbations and present an alternative adversarial defense using learnt saliency models that is effective against both further establish
black-box and white-box attacks. Building the work of Gong et al. (017)), the notion threshold behavior, showing that the trend depends on the L2 and Loo- norms of the perturbations and therefore also prevails when using other methods (JSD) and across different attacks.
Future work could further investigate the performance of the defense in different applications. For example, as our method runs real-time, could be used to detect adversarial perturbations in video counter recent attacks (Li al. 2018 Diang al. 2019
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Alaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, (0(7):00130140 2015.
Chris Burges and Corrinna Cortes. Mnist. 1998.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017b.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in Neural Information Processing Systems, pp. 6967-6976, 2017.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Jeremy Elson, John JD Douceur, Jon Howell, and Jared Saul. Asirra: captcha that exploits interest- aligned manual image categorization. 2007.
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schonlieb. On the nection between adversarial robustness and saliency map interpretability. arXiv preprint 140.0:7905.0 2019.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference Computer Vision, pp. 1429-3437, 2017.
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. arXiv preprint arXiv:1704. 04960, 2017.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 4rXiv:1412.6572, 2014.
Jindong Gu and Volker Tresp. Saliency methods for explaining adversarial attacks. arXiv preprint arXiv: 084113 2019.
Linxi Jiang, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. Black-box adversarial attacks on video recognition models. arXiv preprint arXiv:1904.05/81, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 44X.9.1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 2016.
Minghan Li, Qi Xie, Qian Zhao, Wei Wei, Shuhang Gu, Jing Tao, and Deyu Meng. Video rain streak removal by multiscale convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6644 6653, 2018
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv: 7702.04267 2017.
Seyed-Mohsen Moosavi- Deziooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- pufer vision and pattern recognition, pp. 2574-2582, 2016.

---

Under review as a conference paper at ICLR 2020
Nicolas Papernot, Ian Goodfellow, Ryan Sheatsley, Reuben Feinman, and Patrick McDaniel. clev- erhans v1.0.0: an adversarial machine learning library. arXiv preprint 1et:000100078, 2016a.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016b.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-1144, ACM, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference Computer Vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning- Volunee 70, pp. 3145-3153, JMLR. org, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networs:: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412:::412...8000 6806, 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.611 2013.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint IrXiv:180s 12152, 2018.
Fuxun Yu, Qide Dong, and Xiang Chen. Asp: fast adversarial attack example generation frame- work based adversarial saliency prediction. arXiv preprint arXiv: 100.0056, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833. Springer, 2014.
Chiliang Zhang, Zuochang Ye, Yan Wang, and Zhimou Yang. Detecting adversarial perturbations with saliency. In 2018 IEEE 3rd International Conference on Signal and Image Processing (IC- SIP), pp. 271-275 IEEE, 2018.
Luisa M Zintgraf. Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 172.v:1702.04595, 2017.
10

---

Under review as a conference paper at ICLR 2020
A ARCHITECTURES, HYPER-PARAMETERS AND DATA
Figure 3: ASSIRA, CIFAR-1O, and MNIST image classifier architecture and nyper-parameete The first entry corresponds to the first layer, and the table proceeds chronologically until the last layer. Parameters f, k, p, s and n represent the number filters, kernel size, pooling size, stride, number of filters, respectively. If stride is omitted, it is set 1. All classifiers have final softmax activation.
ASSIRA CATS AND DOGS
LAYER
PARAMETERS
2 X CONV LAYER
64, =
MAXPOOLING
p 2,8
2 X CONV LAYER
f = 128,
MAXPOOLING
p p= 2.8 2
CIFAR-10
2 X CONV LAYER
f = 256, k 3
LAYER
PARAMETERS
MAXPOOLING
p 2.5 = 2
2 X CONV LAYER
f = 2 & =
CONV LAYER
f 512,
2 X CONV LAYER
f = 32, k = 3
MAXPOOLING
p == 2, =
MAXPOOLING
p 2. = 2
FLATTENING LAYER
2 CONV LAYER
= 63, k
DENSE
256
X CONV LAYER
Í 63, k
DENSE
256
MAXPOOLING
P = 2,8 =
DENSE
n
2 X CONV LAYER
f = 128, & 3 128, A
2 X CONV LAYER
MNIST
MAXPOOLING
p 2, =
LAYER
PARAMETERS
FLATTENING LAYER
X CONV LAYER
64, =
DENSE
n 128
2 X CONV LAYER
= 128, A =
DENSE
n 10
2 X CONV LAYER
[ = 128, &
FLATTENING
DENSE
10
We apply drop-out before every dense layer. Using a validation set, we experimented with different drop-out rates between 0.3 and 0.7 and found that the rate 5 = 0.6 was optimal. We use ReLu activation for the penultimates layers and softmax activation for the final layer. We train the model for 10 epochs batches of size 50.
We compare the performance of models on MNIST, CIFAR-IO and ASSIRA (see Burges & Cortes, 1998 Krizhevsky et al. 2009 Elson al. 2007 respectively). For MNIST and CIFAR-IO. 10, we use the standard train and test splits, and for ASSIRA, we use 3, 000 images. We use 10% of the training data for the validation set, and re-train the full training dataset hyper-parameters were selected.
Further experimentation of ID and JSD architecture We further experiment with the architec- tures of ID and JSD to determine whether the observed performance was the result of the architec- ture. In particular, we considered the adjustments as summarized in Table A however, we found that the changes did not improve performance.
ARCHITECTURE EXPERIMENTATION
PARAMETER
VALUES/ VARIATION
DROPOUT
INCLUSIIO//XCCLUION
OPTIMIZATION
ADAM, SGD, RMS-PROPOGRATION
No. CONV LAYERS
3 9-9
No. HIDDEN UNITS
STARTING 32 64, AND DOUBLING REMAINING CONSTANT
No. DENSE LAYERS
1 1-3
POOLING LAYERS
INCLUSION/EXCLUSION
11

---

Under review as a conference paper at ICLR 2020
B ADVERSARIAL ATTACKS
In this section, we present the black-box adversarial attack hyper-parameters (see Figure 4. the success rates of the different adversarial attacks (see Table 4) and an example of adversarial image generated by the various black-box attacks (see Figure
Figure 4: Adversarial attack hyper- Syper--arameetrs.. For the saper-parameters not listed, the default values in cleverhans (Papernot et al. (0166) are used. E is the maximum perturbation allowed and is the maximum perturbation allowed in an iteration. We use different hyperparameters for the MNIST and CIFAR-10 to ensure the attack is sufficiently strong.
ASSIRA
MNIST and CIFAR-10
Attack
Norm
Hyper-parameter
Attack FGSM
Norm
Hyper-parameter
FGSM
L
0.025
0.20
BIM
Loo
0.025, 0.0025
BIM
Too
8 = 0.15, 8 = 0.015
MIM
Too0
E == 0.025, & = 0.0025
MIM
Too
8 == 0.15, 8; = 0.015
C & W
L2
binary search steps 3
C & W
L2
binary search steps =
DF
L2
max. iterations =
DF
L2
max. iterations 2
JSMA
N.A.
0.025
JSMA
N.A.
== 0.05
Table 4: Accuracy of Classifier on (1) natural images (corresponding to the row Baseline) and adversarial images (corresponding the remaining rows). Entries correspond to MNIST/CIFAR-
10ASSIRA.
ATTACK
ACCURACY
BASELINE
99.12/85 02 /92.06
FGSM
19.27/1 19.27/11.11/99.21
BIM
17.07/10.16/1
MIM
22.00/10.09/9.21
C & W
6.86/8.99/7.99
DF
0.91/8.80/7.0
JSMA
6055/.86/7.99
Original
FGSM
BIM
MIM
JSMA
C& W
DF
4
q
q
4
4
4
Figure 5: Example of an Adversarial Image for the MNIST dataset. From top to bottom: the top row is the set of images; the bottom row shows the size of the noise added. Gray indicates no change, whereas white indicates that the image has been made lighter, and black indicates that the image has been made darker. As MNIST images gray-scale low-resolution images, the adversarial perturbations are perceptible to the human eye. Nevertheless, the correct classification of the image still clearly 4. However, the classifier predicts (from left to right) the images 9, 9. 8, 9, 9. Further, we observe that the perturbations FGSM, BIM, and MIM are more visible than those of C & W and DF.
12

---

Under review as a conference paper at ICLR 2020
C WHITE-BOX ATTACKS PSEUDO-CODE
Algorithm White-box attack for JSD
1: Tado f 2
2: for = =0:7 ( do
3: for :
4:
Tadv does not fool the classifier then
5: Zadv t Tade + Esign V (Tades y))
6: end if
7:
if Bade does not fool the detector then
8:
Sado f V (Iadd)
9:
[Tados Sado] +- [Cadu Sady] + esign( (ad( Sadu v))
Tadv + (Tap(
11: end if
12: end for
13:
end for
14:
return Tade
Algorithm provides the white-box attack for JSD. As mentioned in Section 2 JSD concatenates the image with saliency map (computed as the Jacobian) and uses this as an input to the classifier. Algorithms 2 and 3 provide the white-box attacks for our defenses: SPD and SMD. The function f, corresponds to generating the saliency map using the method introduced by Dabkowski Gal (2117). Their method returns a two-dimensional saliency map. However, as the image is three dimensions, we expand the last dimension and stack the map to match the number of channels (n.) of the image. In doing so, assume that the saliency is constant along depth.
Algorithm White box attack for SPD
Algorithm 3 White-box attack for SMD
1: Tado <- T
1: Tude t-
2: for 1: do
2: for j = do
3: for = I do
3: for 1: T do
4:
Fade does not fool the classifier
4:
if Fado does not fool the classifier
then
then
5:
Fadu + Fado + Esign( V (Z 3)
5:
Tadv +- Fado esign( (osgenV/laaaae ("adoo y))
6:
end if
6: end if
7:
Sado f- [s(Fadv)
7:
Sadv (Zadv)
8:
sp((adv) Badv Sado
8:
if Sadv does not fool the detector then
9:
sp(Iodu) does not fool the detector then
9:
(ign(Vdd(Sute.. y))
10:
ne > 1, repeat r along the last
10:
r + esign( (ggg((s((((add))))
dimension until it matches ne
if no > 1, repeat along the last dimension until it matches n., 12: Fadv + Clip(Fade +))
11: Ladv + ClippIade + r)
12: end if
13: end for
13: end if
end for
end for
15: return Fado
15: end for
return Tadv
13

---

Under review as a conference paper at ICLR 2020
D SALIENCY MAPS
D.1 MODEL-BASED SALIENCY MAPS
Original
FGSM
BIM
MIM
JSMA
C&W W
DF
Figure 6: Example of the saliency maps of natural and adversarial images for the ASSIRA cats and dogs dataset. The top row shows the input image and the bottom shows the corresponding saliency map. In the saliency row, lighter colours correspond saliency of O, the lowest possible value). The classifier predicts dog for the original image and car for the adversarial images. Note the stark difference between the saliency masks of the original image and those of the adversarial examples.
higher saliency (black corresponds
D.2 GRADIENT-BASED SALIENCY MAPS
Original
FGSM
BIM
MIM
JSMA
C W
DF
4
q
4
4
4
4
4
Figure 7: Examples of gradient-based saliency maps for MNIST. The first row shows the natural and adversarial images and the second row shows their respective saliency maps. Although there are slight deviations between the saliency maps, if the maps were unlabelled, it would be unclear which maps belong to adversarial examples as opposed to the original image.
Original
FGSM
BIM
MIM
JSMA
C& W
DF
Figure 8: Examples of gradient- saliency maps for ASSIRA. The first row shows the natu- ral and adversarial images, and the second row shows their respective saliency maps. There are no perceptible differences between the saliency map of the original image and adversarial images generated using MIM, C& W. and DF. Further, we observe that for FGSM and JSMA, the gradients are all zero-valued. This is second drawback of using gradients- they are unstable and generate uninformative saliency maps due to underflow.
14

---

Under review as a conference paper at ICLR 2020
E L2 DISTANCES BETWEEN SALIENCY MAPS CORRESPONDING TO ADVERSARIAL IMAGES GENERATED BY DIFFERENT ATTACKS
Table 5: MNIST/ASSIRA: L2 distance between saliency maps generated by different adversarial
attacks
BIM
MIM
C &
DF
JSMA
FGSM
0.05/0.07 /0.01
0.04/0.07 /0.01
0.12/0.07/0.02
0.07/0.08/0.00
0.1T/0.07 /0.02
(0.033/0.03//0000)
(0.03)(0. 03)
(0.04)00033~00.00
10.03)00003И00000
(0.01)/0.03 (3..0.33)
BIM
0.02/0.03/0.00
0.12/0.05/0.0 00.01
0.01/0.05/0.02
0.11/0.07/0.00
(0.02)(0.02y(0.00)
(0.04)/(0033))) (3)00.00)))))
(0.03)10.033(0000)
(035)(0.033/0...
MIM
0.12/0.06/ 00.01
0.06/0.06/0.01
0.11/0.07 /0.01
00.04)(0.03)00000
(0.03)H0.03)K0000)
(0.03)/O. 1.03)00.03300.0 (O.
CW
0.11
0.13 /0.00
(0.05)((0.03)00000)
0.033/00033K00.00
DF
0.10/0.02/0.00
(0.04)/0.03)(0000)
F SINGLE BLACK- -ECCK-BOO BOX ADVERSARIAL ATTACK DETECTOR
Table 6 summarizes the accuracies the different defenses when training a single detector against combination of different types of black-bor attacks. All methods perform relatively similarly as when trained against a single defense, obtaining slightly worse performances than when trained against specific adversarial attack. This is useful in practice when it is unclear which adversarial attack is used.
Table 6: Performance of the different defenses against the combination of all black-box attacks. The entries are the accuracy of the defense in distinguishing natural and adversarial images for MNIST/CIEAR-IOAASSIAAA The values in parentheses below denote standard deviations.
DEFENSE
BASELINES
OUR METHODS
ID
JSD
SMD
SPD
1. 1.00 0..50
0.99/0 099/0058/00 /0.50
0.96 1.00/1.00 0.99/1.00/1.00
00.000/00166/00
(0.01)/(0.11)))
11000000
(0.03))0.000/0000)
(0.00)\0.00><00.0))
15

---

Under review as a conference paper at ICLR 2020
Table 7: The performance of the different defenses trained on a single black- box attack on detecting different black-box -oo attacks. The entries correspond to the accuracy the defenses ID/DSD/SPD/SMD) obtain in distinguishing adversarial examples from natural images. The values in parentheses below denote standard deviations.
0.74/0. 72 /2 00..0000.7700
(0 00.02 03)0 00 00)
0.72/0..c 68, /0. 0..22/0.72 52/0.72
(0.022 (0.02)C0.01)<C.. wwt. 02.0220000000 001)
0.72 /772/0.22 70 /0. 73/0.81
(0.03)((000) ((. ((0.2)(0000
0.74/0. 71 00.83/00.02
(0. (0.06 w10. (00.0550000000
(000
1.00 /00/0.. 70////0.///// 74/0.400000 2..084//...0 $4/0.88
w0w.000000000.000000000 00w.0000000000 00)
1.00 1.00/1.00 1/00//000///// /0.97/09/./// /0.97
(0 (0.00 00 w0w.0000000..0 .00)
(0 (0.02 00w.00033000000
JSMA
0.78 0 0.0.0.0000...... 00/00.00/0000 00.98
(0. (0.05X(0.083 03 03 00)
0. 0.83, /3/80/85/ .98/0. 08/0.73
(0.06 07) MCO.0.)))
0.81 /0 80/00000000 / 08.98/9//88/ /0.72
(0.04y/10.05) 03 (1.0)000000000 (O0.01
1.00 /1 .0 0.0.//9//////// 00..8/00000
/0.95
(0. (0.00W(0.cm 00 /OO 01)
1.00/ / .00 1.000.00000 00.95 (0.95/000 /0.91
(0.05 03w.0...300000.00 00.00/.3000000000000 00w.000.000000 00)
0. 50 / 00.000/0.85
(0 (0.00X00 00) ((0.0)
DF
http00002200000000 00)
0.74/0. 0.74/0.75//..55 /5..977/0.77
10.061/10-00 06/00 01000.00)000) 01)
0.72/0 /0. 72/0 / 96/666..66
(0.033/10003)) (tt/:/0003000000000000 00..000000000
1.00/1 1.000000//// /O. 99/99.95
(0. (0.000/0 00 00
1 1.00/ (0001.00 1.00000 /O 94/0.94
(0.03yr0o 00000300220000000 02 02 110.0.000000000 60)
11/0..8 0.89
(0. (0.0))oo. oo o I 00)
0.69 0.69/0.71 /1.0///....//0000
0.50 0.50/0.5m / 50 / //..0////.......
C w
ATTACK
1.00/1 1000/100///0 1.10000000... /0.00.900 92/ L.00
(0. (0.000iC. 000 02000.2200000 00)
1.00/1 100/1.000 10000.1110// /1000/0.0911
(0.00XCO 0OKO 00/00.01
1.00/1 00/10 1..000000// /0.0929
(0.00 (0.00KOO.C 00OO 01 WO 00)
1.00/1 / /0000.11110 / 0000 f0.85
(0.00)/C..))) (00.0000000000000000000 0000
1. 1.00/1 00/0000//////.. /0. 04 04.00/80.
(0 (0.00)<CO. 00)( 02X10 00)
0.52 (1.2//55//// /0. 73, 73//0.88 88
(0. (0.02) o1 vOO 02V(D.01)
MIM
BLACK- BOX ADVERSARIAL ATTACK RESULTS
wwt.://000..........o (2)00.2200000 00)
1.00/1 1.00/1.00/ .00 100.0.000000///. .0 /0.91
(0 (.,0XOO.OO (ODO, (0.00. (0)
1.00/ 000 /1/1../ 00 /0.92
(0.00) (0.00X(O.c)) (0..0000000000000000000.0. 00.00000000 X(O
1.00/1 10000.00////000 00 /0.98/ 0.87
(0. (0.00)(O..)) (O..00000.0000000000000. 0 00)
1.00/ 1. .0 /0.94/0000/// /0. 83
00.0000000000000 02 00)
0.51 0.51/0.51 /0.51 01.50/01.0..00.. /0.00/0..0..0./ 00.88
(0.0 01 IOO (0..0 02) 02)
1.00/ 1.00/.000 2/0//.000/.0000 000/0.91/1.00
(0. (0.00>(0.00)/))) 000.00000000000
(0.00 (0.00 ww..
BIM
(0. 00.00010.0000 000/0,0010000000 000 00)
1.00/1 /.//1./ 1.00 /000999
DOMOO 00)OO (0)(0.0))
1.00/1 00001.111 1.000.99 .22
(0. 10.00 00) 10.0000000000 00)
1. 1.00/1 1. 1.0/0.00000 99/0.87
DOY00 (O) (O)
1. 1.00/ 10001.00 100 10/000000000 95.888
(0 00000 DOM 00H 00)
0.73, /..6///.../.... /0 80/0.93
(0. 10X 05 (30(0.01)
1.00/1 1.00/0000000 100/0.94/1.00
00/0.99/008.
FGSM
MNIST
DEFENSE
FGSM
BIM
MIM
DF
CW
SAL
G.1
G
16

---

Under review as a conference paper at ICLR 2020
Table 8: The performance of the different defenses trained on a single black-bol box attack on detecting different black-bon box attacks. The entries correspond to the accuracy the defenses //DSD) obtain in distinguishing adversarial examples from natural images. The values in parentheses below denote standard deviations.
1.000099
(0 09 05) 00000.000000 (000))
0.90/6 1.90/0... 71 /1/1/..000 /1.00/111000/1 1.00
(00.0000000000 (0.00)
0.82 / S1 11/001000000000 (1.00.00
(0 (0.08wr0 wwt.:00000000000000 (0000)
0.50 /o. 50 10/550111000 1,00/10000000 11.00
(0. (0.00W00 DO W.O.0OOO..00 (.00)
0.50 /0.00 50/1 1.00/1.000 (1.00)
(0 000ro 00MOO WOW.000OOOOO (00)
1.00 /0. 99 /1.0/1.000 1.00/1.000 11.00
(0.00 (.00
(0. 10.10w0o. 020.00000000000000000
0.67 / /0/1.00/00000000
(0. (0.00yin w0
ISMA
0.50/0.50/1 .0000.99
(0. (0.00)CC (0.00)00.00)00 000 00010.011 01)
0.50/0.50/ /1 .00/1.00
(0.000)(D.))) (D) n00000000000 000)
0.50/0. 50 / 1.00/.000
(0.00)CC oowO 00000000000 00)
0. 0.50 /0.50/ 1.00 10001000
(0.00 (0.00) 0D00 00010 00)
/5000.50/1 .,00 /1.0000
(0.00)сO 00)0CCC0C00 00HO 0OHO 00)
0.50 /0.00/ 1 .0001.0000000000 1100
0000000000000 0.000
DF
0.50/O.50/.00 000/1.00/00 00.90
(0 10.00>C0 (0.00>0(0000))) 0000O 000000//0000 01)
0.50/0.50, /1 .000 1..000/100000 /1.00
(0.00)00 00000000000000000000000 0000000000000 (0.00)
0. 0.50/0.5 50/1. 1.00 1.00
(0.00>HC.00)) (0)(O (0)/00000000 00)
0.50/0. 50/1 1 .00 1100
(0.000 (0.00/C0, 000/ (0..000/.OOO (0)00)
0.50/0. 50 50/1..111. / 1.0011. 1.00
(0 00,00C 0D)(O. 00)( 00)
1.600/50//1100 (00) (0)
0.50/0 50 50/5.000000 .00 1100
C&W w
(0 (0.000/C..00))) (0.00000.00000000 (00.
ATTACK
(0.03y10 04) (00000000000000 (000))
1.00/ / .00 1 .,00/1 /1 .00
(0 (0.00yro. www.0000000000. (0..00))C.000 00)
1.00 /00/0..30 99/ 1. 1.00/1.000000 0000.0000 1.00
(0 (0.00W00 O00000000000// (ww0000000000.000 (0.000((0.000 00)
0.50 /O 50/ 10/5500//0/00//000 1.00/1 /1.00
(0 (0.00W10 00)(O. (0)/(. .00)
0.50 / 505 / (0001.000 11.00
(0 00w0o 000(O (0.000000000000 0D)
0.80 / 89/ 1 1.000111.00 11.00
(0. 18y0. 16) (6) (0).0000)).. 00)
0.55/0. 0.55 1//00//58100000/// 1.00/0090
MIM
50/0 00000.99
(0.00 (0.001/00 00 00 000/(000))
0.0000099//0 f1 1.001.0000 ..0
(0.00//1 00)00 (O)X0.00)
0. 0.62 /0. 00.6//111.. 11.00/1.00
(0.08)/10.1 (0.88/10.12///0 12 000(0.00)
0 5,50 00500 100/1.00
(0.00) 000 (00(X.000
0.50/0.5 /0.5 50/1 000/1.00
00.000 oo 09H(0.00)
0.52 /555/0555// / .00011000
(000000000000000000 00)
BIM
(0.03)000
(0. 10, 000(0 00)/(0 00)
1.00/1 1 .00/1. ..0 /1.00
(0.00XCCD... DOKOO 001/0.00) (0.00)
1.00/1 00 / 00/ 1.00/.0000 /1.00
(0.00><O.000 OOO( 001/00 00)
0.50 50/00///111// 1.00 11.00'1.000 1.00
(0)
0.50/0. 50 50/1 1.000111000 /1.00
C0)OC OOKO 00/100 00)
1.00/0 99 /1.00/000000000000 1100
w00//////000 n0)
1.00/1 10/1100/1.00
(0 (0.00/CO. 00K( 00//10000000
(0.01)000000 wwt.///..00..00..oooo..
FGSM
CIFAR-I 10
DEFENSE
FGSM
JSMA
BIM
MIM
DF
CW
G.2
17

---

Under review as a conference paper at ICLR 2020
Table 9: The performance of the different defenses trained on a single black-bol box attack on detecting different black-bon box attacks. The entries correspond to the accuracy the defenses //DD/) obtain in distinguishing adversarial examples from natural images. The values in parentheses below denote standard deviations.
(0 00 00) (0w.0ooooooooo (000
0.50 /0000. 50/5 5001..00 1.00/1 1//1..00 1.00
(0..00000.000 (0.00)
0.50/ / 50/ / 1110/100.0000 11.00
D. 0.50 /0. 505 10.50/11000 1,00/1.00000 1.000
(0. (0.00W00 DO (.0.0O0O..00 (.00)
0.50/ /0. 50 1.0011.00 (1.00)
(0 0owro 00MTO. OO0000000000 (00)
1.00/ 1.0 100000110000000 1.00/1.00 11.00
(0. (0.000yn..n) (tw. (0.00 (.00
(0 (0.00wr0 10 wwt.:/..........
0.50/0. 50/1.00/11
JSMA
(0. (0.000r0 w0w.0..0000000
0.50/0.50/1 1000001110 1.00
(0. (0.00)CC (0.00)00.00)00 00 00010 00)
0.50/0.50/ /1 000/1.00
(0.000)( (D) n0y000000000 000
0.50/0. 1.50/0.50// 50/ / 1.00/.00
(0.00)CC OOWO 00000.00000 00)
0. 0.50 0.50/0.500 0.50//0501111 / 1.00 1.001000
(0.00) (0.00) 0D 00010 00)
/5000.55/1 .,00 01.0000
(0.00)сO 0OHO 0OHO 00)
0.50 /0.00/ 1 ..001.0000000000000000000 1100
0000000000000 (0.00)
DF
0.50/0.50/.000 1////1.00000000 1.000
(0 10.00>CC (0.00>(0.00)))) 00 00)/0 00)
0.50/0.50, / .000 /1.00
(0.00)00 (00000 00000.000000
0. 0.50/0.5 50/1 1.00011.000 1.00
(0.00>CO. (O)(O (0)/00 00)
0.50/0. 50/1 1 .00 11.00
(0.00 10.00/CO. 00)/ (00.000/OOOO (0)00)
0.50/0. 50 50/1. / 1.00/1. 1.00
(0 10.00/CO OOMOO 00M( 00)
0.50/0 16000.50/1110 50/5.0000000.000 .00 11000
(00 (0)
C& w
(0 (0.000(C..0m) (0.00000.00000.00
ATTACK
0.50/0.50 10150010000//0 1.001.00000
(0 00 00) 00)0c0.00ykc (00.0000...0 00)
0.50/0 /0. 50/ 1 0,00 .00
(0..00))C.000 .0)
0.50/0 /0. 50/ 1. 1.00/1.000 0000000000 1.00
(0 (0.00W10 00 (0.00)0(.0000 00)
0.50 /O 50/ 10/5500///00/. 1.00/1. /1.00 .000
(0 10.00W00 00)(O. (0)/(. .0)
0.50 / 50/ / (00/1.00 11.00
(0 00w0O 000(O 00000.00000 00)
/1.00
(0 (0.00000.n)) no ( (0),000))))) 00)
0.50 /0 50/1 1 1.00011100000
MIM
(0 (0.00yro. w00.00000000000
4.50/0.50/1 50/0000////111111111111100000/01 11.00001100 1.01000
(0 00.00/100 00000000000000000000000000000000 000 000/(0.00) 00)
0.50/0.50/1 / .00/1.00 1..00
(0.00// D0) (O)(0.00)
0 0.50/0.5 50/0////////0 00/5500///0 1.00/1.00
(0.00)/10 ww.oo. 000(0.00)
0 1,50 1.50/0.50 0 5050 100/1.00
(0.00010 00 000(0.00)
0.50/0.5 /0. 50/1 00/11000
00vroo oo 09/(0.00)
0.50 /55/0050// 1 .0001.000 1.0
(ww..//0/0/.00000...... (00)(.00)) 00)
BIM
(0.00) (0.00//0..0000....o
0.66/0 166/0.66/ 060.66/1.000 1/00//000//000 0.00
(0.20 20.20>CO 200(0 00)/00 00)
0.50 0..00//5///// .000 1.00
(0.00X00... DO)(O 00//0.00) (0.00)
0.50/0.56 50 / 000 1000////000 11.00
(0.00)0O.000 (OO( 00)/00 00)
0.50/0. 50/1. 1.00 1.00/1.000 1.00
(0 10.00/Co. 00K( 00//00000000 (0)
0.50/0. 50 50/550011000 1,00/1 11.00
cO oOkO 001/00 00)
0.50/0.5 50 /1000 1100
w00///0//0000 n0)
FGSM
(0.00 (0.000((..om (0.00000.00000.00 (O)
G.3 ASSIRA
DEFENSE
FGSM
C& W
JSMA
BIM
MIM
DF
18


Reviews:
[{"review_id": "rklB0nk6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies methods for detecting adversarial examples using saliency maps. The authors propose using the method of Dabkowski and Gal (2017) to generate saliency maps and then train a classifier on these maps (or their dot product with the input image) to distinguish natural from adversarial examples. They perform experiments evaluating the white-box and black-box robustness of their detection scheme.\n\nFrom a technical perspective, the contribution of the paper is rather incremental. The detection of adversarial examples by training a classifier on saliency maps has already been studied in prior work. The only modification proposed in this work is using an (existing) alternative method for producing the saliency maps and utilizing the dot product of maps with images.\n\nFrom a conceptual perspective, the impact of detecting specific adversarial attacks is not clear. In a realistic setting, an adversary could use a very different attack or even utilize a different set of transformations (e.g. image rotations). Thus, in order to demonstrate the utility of their method in a black-box scenario, the authors would need to evaluate the defense in a variety of different scenarios. At the very least, they should consider generalization to difference attacks (e.g., train against FGSM and BIM, and test against DF).\n\nMoreover, the robustness against white-box adversaries is not sufficiently studied. Firstly, the robustness of the non-adversarially trained detector is suspiciously high. There is little reason to expect that a composition of two neural networks (the saliency map methods and the classifier) would be non-trivially robust. The authors should consider alternative attacks perhaps using more iterations with a smaller step size. Secondly, after adversarial training, only the robustness against the same attack is considered. In order to argue about white-box robustness, the authors would need to evaluate against a variety of diverse adversaries.\n\nOverall, the technical and conceptual contribution of this paper is insufficient for publication at ICLR, even ignoring the concerns about its experimental evaluation.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1luCy_aYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adversarial defense method that is a saliency-based adversarial example detector. The method is motivated by the well-known fact that saliency maps and adversarial perturbations are having similar mathematical formulations and derivations. By using model-based saliency maps rather than gradient-based ones, it seems to detect hard attacks with smaller perturbation size as well. As far as the authors mentioned, the proposed method is simply using different techniques to derive saliency maps compared to the previous methods.\n\nOverall, the intuition and motivation of this paper are from the previous works and the main contribution is to use another (powerful) saliency map extractor for learning an adversarial detector. Although the overall results are improved from the previous methods, the proposed method is lack of novelty. \n\n- For SMD (Saliency Map Defense), what is the reason that the input image is not used together? computational issue? performance degradation? \n- Is it possible to train a single detector that can handle all different adversarial attacks?\n- Would the distance between saliency maps from different attacks be small? How does the saliency map change under different attacks?\n- Have you tried any other powerful saliency maps other than Dabkowski & Gal (2017)?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyliC__PYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for training networks to detect adversarial examples and by virtue of doing so, providing defense against adversarial attacks. Two different approaches are examined, in which a saliency map is used in combination with the input as a mask. In one instance the saliency mask is based on a classifier used to distinguish \"normal\" from adversarial examples. In the other instance, the salient pixels themselves form the basis for defense. In both cases, the saliency map is combined with the image for training a CNN by way of an element-wise product.\nOverall, this presents a relatively simplistic way of deriving representations of saliency and combining these with inputs for training that builds robustness against white and black box attacks. At the same time, the empirical results presented reveal a considerable degree of success in providing a defense against such attacks. I find that this presents an interesting contribution to the literature addressing both adversarial attacks, and new notions on ways of characterizing saliency.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a method for detecting adversarial attacks using saliency maps.

Strengths:
-The experimental results are encouraging.

Weaknesses:
-The novelty is minor.
-Experimental validation of some claims (e.g. robustness to white-box attacks) is lacking.

These weaknesses were not sufficiently addressed in the discussion phase. AC agrees with the majority recommendation to reject.

Decision: Reject