Enhancing Transformation-Based Defenses Against Adversarial Attacks with a Distribution Classifier
Authors: Connie Kou
Hwee Kuan Lee
Ee-Chien Chang
Teck Khim Ng
Venue:  (2020)
Keywords: adversarial attack
transformation defenses
distribution classifier

Abstract: Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms. Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples. However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images. While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear. In this paper, we study the distribution of softmax induced by stochastic transformations. We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction. Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribution of softmax are similar to the distributions from the clean images. With these observations, we propose a method to improve existing transformation-based defenses. We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images. Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images. Our method is generic and can be integrated with existing transformation-based defenses.

Published as a conference paper at ICLR 2020
ENHANCING TRANSFORMATION-BASER DEFENSES AGAINST ADVERSARIAL ATTACKS WITH A DISTRIBU- TION CLASSIFIER
Connie Kou-.2, Hwee Kuan Lee Ee-Chien Chang', Teek Khim Ng
School of Computing, National University of Singapore BBioinformatics Institute, STAR Singapore
"Image and Pervasive Access Lab (IPAL), CNRS UMI 2955
"Singapore Eye Research Institute
(koukl, changec, rggk @comp. nus edu. sg,
.g
ABSTRACT
Adversarial attacks on convolutional neural networks (CNN) have gained signifi- cant attention and there have been active research efforts on defense mechanisms. Stochastic input transformation methods have been proposed, where the idea to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples. However, the transfor- mation improves the accuracy adversarial images at the expense of the accuracy on clean images. While it is intuitive that the accuracy on clean images would de- teriorate, the exact mechanism in which how this occurs is unclear. In this paper, we study the distribution of softmax induced by stochastic transformations. We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction. Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribu- tion of softmax are similar to the distributions from clean images. With these observations, we propose method to improve existing transfornation-bat fenses. We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images. Our em- pirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images. Our method is generic and can be integrated with existing transformation-basee defenses.
de-
INTRODUCTION
There has been widespread of convolutional neural networks (CNN) in many critical real-life applications such as facial recognition (Parkhi et al. (015)) and self-driving cars (Jung et al.) 2016). However, has been found that CNNs could misclassify the input image when the image has been corrupted by an imperceptible change (Szegedy et al. 2013). In other words, CNNs are not robust to small, carefully-crated image perturbations. Such images are called adversarial examples and there have been active research efforts in designing attacks that show the susceptibility of CNNs. Corre- spondingly, many defense methods that aim to increase robustness to attacks have been proposed.
Stochastic transformation-bee based defenses have shown considerable success in recovering from ad- versarial attacks. Under these defenses, the input image is transformed a certain way before feeding into the CNN, such that the transformed adversarial image would no longer be adversarial. As the transformation is random, by feeding in samples of the transformed image through the CNN, accumulate set of CNN softmax outputs and predictions. As such, existing transformation- based defenses take a majority vote of the CNN predictions from the randomly transformed image (Prakash et al. 2018 Guo al. 2017). Transformation-baii based defenses are desirable there is need to retrain the CNN model. However, they suffer from deterioration of performance on clean images. With increasing number of pixel deflections (Prakash al. 2018), there improvement

---

Published as a conference paper at ICLR 2020
to T(x)
distributions
density estimation
Distribttion
'bus"
classifier
(2)
M
In transformation-base defenses, the image transformed stochastically where each sample to is drawn from the distribution T(x) and then fed the CNN (blue box). In our defense method, for each input image I, we build the marginal distribution of softmax probabilities from the transformed samples 1 1/2 .... The distributions are fed separate distribution classi- fier which performs the final classification. Note that our distribution classifier is trained only on distributions obtained from clean images while tested both clean and adversarial images.
Figure 1:
the performance on adversarial images, but this comes with rapid deterioration of performance on clean images.
The exact mechanism of the deterioration in performance clean images is unclear. We believe that the softmax distribution induced by the random transformation contains rich information which not captured by majority vote that simply counts the final class predictions from the transformed samples. Now, an interesting question is whether the features in the distribution of softmax could be better utilized. In this paper, to elucidate how the deterioration in accuracy clean images occurs, we study the effects of the random image transformations on the distribution of the softmax outputs and make some key observations. After the image transform, some clean images show distributions of softmax with modes at an incorrect class, reflecting the deterioration in voting accuracy as ob- served before. While the shifting of the distribution mode the incorrect class is detrimental the voting prediction, the resulting distribution of softmax contains features that is useful for correct- ing the prediction. In addition, we observe that the adversarial counterparts show similar shifts in the distributions of softmax as the clean images. We also look into the distribution shapes for the transformed clean and adversarial images and find that they are similar.
With these observations, we propose simple method to improve existing transformation-bar de- fenses, illustrated in Figure We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed clean images and predict the class label. Without retraining the original CNN, our distribution classifier improves the per- formance of cansformation-bassee defenses both clean and adversarial images. On the MNIST dataset, the improvements in accuracy over majority voting are 1.7% and 5.9% on the clean and adversarial images respectively. On CIFAR10. 10, the improvements Note that the distributions obtained from the adversarial images are not included in the training of the distribution classifier. In real-world settings, the type of attack ing the distribution classifier on a specific attack may cause the classifier to overfit to that attack. Hence, is advantage that our defense method attack-agnostic. Our experimental findings show that the features of the distribution in the softmax are useful and can be used to improve existing transformation- based defenses. Our contributions are as follows:
6.4% and 3.6% respectively.
known beforehand. Train-
1. We analyze the effects of image transformation in existing defenses on the softmax out- puts for clean and adversarial images, with key finding that the distributions of softmax obtained from clean and adversarial images share similar features.
2. We propose a method that trains a distribution classifier on the distributions of the softmax outputs of transformed clean images only, but show improvements in both clean and adver- sarial images. This method is agnostic to the attack method. does not require retraining of the CNN and can be integrated with existing cansformation-basee methods.
2

---

Published as a conference paper at ICLR 2020
2 RELATED WORK: ATTACKS AND DEFENSES
Given an image dataset {(x1, 8/) ... (.M yns]} and a classifier Fg that has been trained with this dataset with parameters 0, the aim of the attack is to produce an adversarial image aqdv such that Fo # Vi. and ||œı zı|l is small. We focus on four gradient-based untargeted attacks. The Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2014) is a single-step attack that uses the sign of the gradient of the classification loss to perturb the image. Iterative Gradient Sign Method (IGSM) (Kurakin et al., 2016a) is an iterative version of FGSM. In DeepFool
al. (016), each iteration, the attack approximates the classifier with a linear decision boundary and generates the minimal perturbation to cross the boundary. Finally, the Carlini & Wagner (C& W) (Carlini & Wagner 2017) L2 attack jointly minimizes the perturbation Lę norm and a differentiable loss function based on the classifier's logit outputs. Besides gradient-bassee attacks. there are also black-box attacks where the CNN model is not known and only the softmax output or final prediction
is given (Brendel et al. 2017 Ilyas et al. 2018 Cheng et al. 2018).
Defense methods have been proposed to make the classifiers more robust. In adversarial training. the CNN model is trained on adversarial examples generated from itself (Madry et al., 2017 Kurakin ct al. 2016b) or from an ensemble of models (Tramèr et al., 2017). Other methods involve training auxiliary neural networks on mixture of clean and adversarial images, for instance, by denoising
the imputs with a neural network before feeding into the CNN (Liao et al. 2018 Song et al. 2017
Samangouei et al. 2018) or by training neural network on the CNN logits (Li et al. 2019) In the next section, we introduce another class of defense: transformation- based defenses.
2.1 TRANSFORMATION--R
DEFENSES
Transformation based defenses aim to recover from adversarial perturbations, that is for input trans- formation T, we want F& (T (z(Td))) = Vi- At the same time, the accuracy the clean images has to be maintained, ie. Fo (Txx)) = yi- Note that transformation-base defenses are imple- mented at test time and this is different from saining-tims data augmentation. Here introduce two transformation based defenses that we experiment on.
Pixel deflection (PD) (Prakash et al. 2018): : Pixel deflection corrupts an image by locally distributing pixels. At each step, it selects a random pixel and replaces it with another randomly selected pixel in local neighborhood. The probability of pixel being selected is inversely pro- portional to the class activation map (Zhou et al. 2016). Lastly, there is a denoising step based on wavelet transform. In our experiments, we did not use robust activation maps for our datasets we found that this omission did not cause significant difference in performance (see Appendix D.3)
Random resize and padding (RRP) (Xie al., 2017)> : Each image is first resized to a random size and then padded with zeroes to a fixed size in random manner.
In many transformation-basee methods, the transformation is stochastic. Hence there can be dif- ferent samples of the transformation of an image: t₂ T(x), where to represents a transformed sample. Existing transformation defenses benefit from improved performance by taking the major- ity vote across samples of random transformations. The advantage of transformation- -bsed methods that there is no retraining of the CNN classifier. However, weakness, as identified by PPakashh et al. 2018)), is that the transformation increases the accuracy on adversarial images at the expense of the accuracy on clean images. The exact mechanism of the deterioration in performance on clean images is unclear. In this paper, we elucidate how the deterioration in accuracy clean images oc- curs by studying the effects of the random image transformations on the distribution of the softmax outputs.
3 ANALYSIS ON DISTRIBUTIONS OF SOFTMAX WITH RANDOM IMAGE TRANSFORMATIONS
Due to the randomness of the transforms, samples of the transformed image will have different soft- max outputs. With each image, we obtain a distribution over the softmax outputs accumulated from multiple samples of the transformation. These the steps obtain the distribution of softmax:
3

---

Published as a conference paper at ICLR 2020
Class label 8
Clean di 300
5 o 7B
-
Adv. no noise
60
0:15
Class
0.15
Class
cleat-cles
Axcraye sherggeoo
Auceage 15 chergerco
40
2.30
adlver san sarial-advei sal-addersarial
Class label B
tsams class)
Claan = 300
20
3.05
clean-adrersarid
Adtv d 300
0
0
10
20
clean- clan (differer. clase)
Adv. no noise
0.00
0.00
Nummerr deflections
Number deflections
(a)
(b)
Figure 2: MNIST dataset: (a) Effect of number of pixel deflections (d) on the voting predictions of clean and adversarial images for MNIST digits 6 and 8. The numbers represent the percentage of images predicted to the various classes, eg. for class label 8, 78% of the transformed clean images predicted correctly. For class label with the image transformation, there is recovery the adversarial images (24% recovery), but on the clean images, there is a deterioration in performance some images are misclassified. However, is interesting that the misclassifications have the same voting classes as the transformed adversarial images. Similar overlap in voting classes is observed for digit (b) Effect of increasing number of pixel deflections on the distances of distributions obtained from clean and adversarial images. The standard error bars, taken over 3 random seeds for the transformation, are smaller than the plot points. Best viewed in color.
1. For each input image z, obtain N transformed samples: (t) T(z), ,N
2. The transformed samples of the image (tf.)
(N)) are fed into the CNN individ- ually to obtain their softmax probabilities. Let dn be the softmax vector derived from (l and o( (t)
(8)
=.jン for j 1, ,., C, be the j-th component of the softmax vector. denotes the number of classes for the classification task. With each input image and transforma- tion method, there exists an underlying joint distribution of the CNN softmax probabilities, from which we estimate with N samples.
3. The underlying joint distribution of the softmax has a dimension equal to the number of classes (eg. 10-D for MNIST). Performing accurate density estimation in high dimensions is challenging due to the curse of dimensionality. Here we make an approximation by com- puting the marginal distributions over each class. When we use the term 'distribution of softmax', we are referring to the marginalized distributions. We use kernel density estima- (N),
tion with Gaussian kernel. Let h be the distribution accumulated from o 2,3" aaj"
(I)
N
(s 0 22.
h (s)
Σ exp
N 2
28
where 8 is the kernel width and s E [0, is the support the softmax output. The distribution is then discretized into bins.
In this section, we study the effect of image transformation the distribution of the softmax and make several interesting observations. In the following analyses, we study a LeNet5 CNN (LeCun) al. 1998) trained with MNIST. The adversarial images are generated using FGSM and for the transformation defense, we use pixel deflection, with N=100 transformation samples per image. The image transformation magnitude controlled by the number of pixel deflections, d. In the analysis here and in the experimental results in Section [5] when reporting the accuracies, on clean images, we consider images that have been correctly predicted by the CNN, hence without any transformation defense, the test accuracy is 100%. This is following the setup of Prakash et al. (0118) where the misclassified images by CNN are excluded as is not meaningful to evaluate any attack (and subsequent defense) methods on these images. For adversarial images, consider the images that have been successfully attacked, so the test accuracy reflects the recovery rate and without any defense the accuracy is 0%.
4

---

Published as a conference paper at ICLR 2020
In Figure Za we show how the image transformation affects the voting predictions on two MNIST classes. For each MNIST class, we take all the clean and adversarial test images, perform the transformation and then feed through the CNN to obtain the final voting prediction. We observe that for class lahel 8, there is some recovery from the attack as some adversarial images are voted to the correct class after the transformation. However, some clean images get misclassified to other classes (eg. 2 and 3). Although this means there is a deterioration of the accuracy on clean images, it is interesting that the misclassifications have the same voting classes as the transformed adversarial images. A similar pattern is observed for class label 6 where the clean images are misclassified to classes 4 and 5, which overlap with the vote predictions of some adversarial images at d = 300.
With the above analysis, we characterize the relationship between the clean and adversarial images in terms of the IS divergence of the distributions of the softmax at increasing number of pixel deflections. For each MNIST digit class, we quantify the (1) distance of the distributions among the clean images (clean-clean, same class), (2) distance of the distributions among the adversarial images ladversarial-adversaan adversarial images aeleaaaad same class) and (4) the distance of the distributions between clean images of this class and all other classes clean, different class). Here we give details on the calculation of the 4 distance measures. First, the distance between the distributions of softmax where
same class), (3) the distance of the distributions between clean and
output for two input images, #1 and x2 is given by d(hs,. ha.) = _ Ej (Js(hen.j) DJs is the lensen-Shannon divergence. Distance measures of (1) and (2) computed by taking the average distance of each image distribution to the centroid distribution which is computed with "(hhz.):" i)) XT. E. he:j- (3) is computed by the distance between the centroids of the clean and adversarial distributions. Finally, (4) is computed by the distance of the centroid distribution of the clean images of the particular class with the centroid distribution of another class, averaged over the other 9 classes.
In Figure 2b we show results for two MNIST classes, but similar trends are observed across all classes (see Figure in Appendix A.. The clean-clean (same-class) distance starts off low initially as all clean samples will give high scores at the correct class. With increasing number of deflec- tions, there is increased variability in the softmax outputs and the resulting distributions. Next, the adversarial images of the same class are initially predicted as different incorrect classes without any transformation, and hence the adversarial adversarial (same-class) distance starts off high and decreases with more transformation. The alean-adversariall (same-class) distance decreases with increasing image transformation which shows that the distributions of softmax from the clean and adversarial images are becoming more similar. Finally, the clean-clean (different class) distance decreases as well, which is expected because we already know that with more transformation, the clean image voting accuracy deteriorates. However, we observe that clean-clean (different class) distance decreases less rapidly and remains higher than clean-clean (same-class) distance d=300. This means the transformation still retains information about the differences between the classes. At d=800, all 4 distance measures converge, which suggests the number of deflections is too large and the differences between the classes are no longer retained.
adversariil-zddversaiaall
Next, we visualize the morphing of the distributions with increasing number of pixel deflections for an example image in Figure 3 For the purpose of visualization. instead of the per-class marginal distributions of the softmax, we perform kernel density estimation (kde) on the softmax values for the marginals on class 5 and 6. The softmax values of the other 8 classes are not shown. We have not excluded the areas where performing kde results in sum probability exceeding onc, and our visualization still conveys our ideas and the distribution shapes well. Without any image transfor- mation, as expected, the softmax outputs of the clean and adversarial images are very different. As the number of pixel deflections increases, each point evolves to distribution due to the randomness of the transformation. The voting mechanism is straightforward; an image is classified to the class where the distribution mass is largest. In this example, the distribution shapes for the clean and adversarial image become more similar, and result in the same incorrect voting prediction at d=300. This shows the similarity of distributions obtained from clean and adversarial images after image transformation, which was illustrated in Figure 2b)
In Figure 4 we show more examples of the distributions obtained from clean images (A-H) and their adversarial counterparts (Á-fi at d=300. For clean images A-D, voting predicts correctly but on the adversarial counterparts A-D, voting predicts wrongly. For clean images E-H and the adversarial counterparts Ë-H, voting predicts wrongly. For completeness, we also show in Figure
5

---

Published as a conference paper at ICLR 2020
clean
CNN predict: 6
1.00 d:
1.00
200,
1.00
300, vote::
0.75
0.75
9.75
0.75
6
0.50
Clere
0.30
0.50
Clas
0.30
0.25
0.25
0.25
0.0
0.5
10
0.00
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Class
adversarial counterpart
1,00
CNN predict:
1.00
d: 100, vote:
1.00
200, vete:
1.00
d:
300, vote:
6.50
0.30
9.50
0.300
6.00
0.5
1.0
0.00
0.
6.5
0.0
0.5
0.00
0.5
1.0
No perturbation
Increasing number pixel deflections
Figure 3: An example image and its adversarial counterpart (ground truth: class 6) undergoing increasing number of pixel deflections (d). The clean image gets misclassified by voting as the distribution mass shifts to the incorrect class. Note that for the purpose of visualization, instead of the per-class marginal distributions of the softmax, here we show the joint distribution over the softmax values of classes and 6. The softmax values the other classes are not shown.
Clean images
Class label
A
14
1A
jas
jss
vote correct
3.5
CB0
E
F
1.0g
vote:
H
1.
$5
vote wrong
Clink
Adversarial counterparts
ohhe vote
BS
vote wrong
Oan
1.c
SEV VITE
vote wrong
050
Click
Figure 4: Distributions of the softmax obtained from 8 selected clean images of MNIST digit (A -H) and their adversarial counterparts (Á-H) at d=300. Here we show the joint distribution of the softmax class and The clean and adversarial images misclassified by voting show similar distribution shapes, indicated by the groupings and arrows, eg. between E and A, E, F. between F and B and

---

Published as a conference paper at ICLR 2020
in Appendix B examples of adversarial images where the transformation defense, coupled with vot- ing, has successfully recovered the correct class. With the random image transformation, there are similarities in the distribution shapes between the clean and adversarial images, shown by the groupings and arrows (eg. between E and A, E, F). This further supports earlier observations. After the image transformation, the voting accuracy on the clean images deteriorates, but the result- ing distributions have similar features as the distributions from the adversarial counterparts. This gives us an idea to enhance existing transformation-baal fier on the distributions obtained from clean images only, while improving the performance on both clean and adversarial images.
defenses: to train distribution classi-
4 ENHANCING TRANSFORMATION BASED DEFENSES WITH DISTRIBUTION CLASSIFIER
Instead of voting, to reduce the drop in performance on clean images, we train separate compact distribution classifier to recognize patterns the distributions of softmax probabilities of clean images, as illustrated in Figure [T] For each clean image, the marginal distributions obtained are inputs to the distribution classifier, which learns to associate this distribution with the correct class label. If the individual transformed images were initially misclassified by the CNN, our distribution classifier should learn to recover the correct class. During the test phase, for any input image, clean adversarial, build the distribution of softmax from N transformed samples and feed them into our trained distribution classifier to obtain the final prediction. Note that our defense method does not require retraining of the original CNN, is agnostic to the attack method and can be integrated methods.
with most existing stochastic cransormatioonbase
Distribution classifiers: We investigate three distribution classification methods. First, we adapt a state-of-the-art iistribution-toddiitrbbbtoo regression method, called distribution regression net- work (DRN) (Kou et al., 2018) (details are included in Appendix O We also experimented on random forest (RF), which averages the outputs from multiple decision trees. Finally, we experi- mented on multilayer perceptrons (MLP) which are fully connected neural networks, with softmax output layer. For this distribution classification task, we concatenate the distribution bins from the softmax classes into a single input vector for RF and MIP. For DRN and MIP, we use the cross entropy loss and the network architectures are chosen by cross- validation. For random forest, the Gini impurity is used as the splitting criterion and the number of trees and maximum depth are tuned by validatoi........... The hyperparameter values included in Appendix D.4
EXPERIMENTS AND DISCUSSION
In the following section, we describe our experimental setup to evaluate the performance on clean and adversarial images with our distribution classifier method.
Datasets and CNN networks: We use the MNIST (LeCun et al. 1998). CIFARIO and CIFAR100 Krizhevsky & Hinton 2009) datasets. For the CNN model for MNIST, we use LeNet5 (LeCun ct al. 1998) that has 98.7% test accuracy. For CIFARI0 10 and CIFARIO0, we use wide ResNet
ZZgoruyko & Komodakis 2016) with test accuracies of 95.7% and 78.9% respectively.
Attack methods: As introduced in Section 2] we use four adversarial attacks in the untargeted setting. In Appendix [ we have included the distortion metries, the success rates and the hyper- parameters. The attacks are implemented using the CleverHans library (Papernot et al. 2018).
Transformation-based defenses: As a baseline, we use a random pixel noise (RPN) as a defense method, where each pixel noise is sampled with uniform distribution with Loo measure. In addi- tion, we use two existing transformation--aasse methods: pixel deflection (PD) (Prakash et al. 2018) and image random resize and pad (RRP) (Xie et al.) 2017). Although these two methods have not been tested for MNIST, CIFARIO and CIFARIOO, we find that they work considerably well and present the results here. The hyperparameter tuning for each defense is conducted sets. We select hy byperparameters that give the best recovery from adversarial attack, regardless of the deterioration in accuracy on clean images. The hyperparameters are included in Appendix D.2
the validation
7

---

Published as a conference paper at ICLR 2020
FGSM
IGSM
DeepFool
C&W
Voting *DRN
MLP
gams
wm
A
Test accuracy (%6)
8
0.5
00
XXX
4.4.
64.
20
MNIST
CIFAR10
CIFAR100
RPN RPN PD PD RRP RRP RPN RPN PD PD RRP RRP RPN RPN
PD PD RRP RRP
clean adv. clean adv. clean adv. clean adv. clean adv. clean adv. clean adv. clean adv. clean adv.
Figure 5: MNIST, CIFARIO and CIFARI00 results: For each attack and transformation-basee de- fense. we compare the clean and adversarial (adv.) test accuracies with baseline majority voting and the three distribution classifier methods. The distribution classifiers generally show improvements over voting. Best viewed in color.
To test the effectiveness of the transformation-base defenses before integrating with defense method, we perform majority voting on the transformed image samples. This sets the baseline for distribution classifier defense method. When reporting the test accuracies, clean images, we consider images that have been correctly predicted by the CNN, hence without any defense method, the test accuracy 100%. For adversarial images, we consider the images that have been successfully attacked, so the test accuracy reflects the recovery rate and without any defense the accuracy is 0%.
5.1 MNIST RESULTS
For the MNIST dataset, N = 100 transformation samples were used for voting and for constructing the distribution of softmax. We found that the distribution classifiers required only 1000 training data, which is a small fraction out of the original 50,000 data. 3 shows the test accura- cies of the three transformation-biit based defenses with majority voting and with the three distribution classifiers. Table 11 in Appendix D.5 shows the numerical figures of the results. First, observe that the recovery on adversarial images with majority voting for the iterative methods IGSM, Deep- Fool and C&W is much better compared to single-step FGSM. This is in line with the observations by Xie et al. (017) where they found their defense to be more effective for iterative attacks.
The distribution classifiers have improved accuracy the voting accuracy was already high (eg. 100% voting accuracy for PD on DeepFool). The mean improvement of the accuracy the clean images is 1.7% for DRN. Hence, method is stronger than voting, Voting simply takes the mode of the softmax probabilities of the transformed image, disregarding properties such as variance across the classes. In contrast, the distribution classifier learns from the distinctive features of the distribution of softmax.
voting the clean images, except when
distribution classifier
Without training on the distributions obtained from adversarial images, our method has managed to improve the recovery rate, with a mean improvement of 5.9% for DRN. The three distribution classifier methods are comparable, except for some cases where DRN outperforms other classifiers (eg. PD adv., IGSM) and where MLP and RF have lower accuracy than voting (eg. RPN adv., DeepFool and C). In Figure 4 we show that after image transformation, the distributions of softmax between the clean and adversarial images show some similarities and distinctive features. In fact, all of the clean (A-H) and adversarial (Á-H) images (class 6) are classified correctly by the distribution classifier. Even though the distribution classifier was only trained on distributions from the clean images (A-H), the distribution classifier can recover the correct class for the adversarial images where voting has failed (Á-Hi). The distribution classifier does so by learning the distinctive shapes of the distributions associated with the digit class from the clean images, and is able to apply this to the adversarial images with similar distribution shapes. Furthermore, classifier is able to pick up subtle differences in the distribution features. Figure (alure//a/sowww shows examples of clean images with class label 5 that correctly classified by our distribution classifier. It is
distribution
8

---

Published as a conference paper at ICLR 2020
0.300
Cless label 5, clean
Voting cleen
ng adv.
vote:
vate:
DRN clean
UMV
0.50
0.225
Cure
0.80
0.155
+8.0
Class 6
0.5
8.0
0.5 Coss
No. traraformed samp per imagee
80
100
No. of tramafo samples per image
42
102
(a)
(b)
(c)
Figure 6: MNIST dataset: (a) Example of two clean images with class label 5, where the distribution shapes look similar to Č and in Figure 4 but the distribution classifier can still discriminate be- tween the distributions from class labels 5 and 6 and classify them correctly. (b) On the clean images, both voting and DRN accuracies improve with more number of transformed samples. As number of samples increases, voting saturates while DRN continues improve. (c) On the adversarial images, the accuracies stay more of less the same.
interesting that although the distribution shapes for adversarial images C and G shown in Figure look similar, our distribution classifier is able to distinguish between the shapes for class 5 and 6.
5.1.1 NUMBER OF TRANSFORMED SAMPLES REQUIRED
We used N=100 transformed samples in our experiments. Hence, the evaluation time will be 100 times longer than taking a single sample. Here we study the effect of the number of samples. Figure [b]and[6c]shoov the classification accuracies for voting and DRN as the number of transformed sam- ples increases. On the clean images, both voting and DRN accuracies improve with more number of samples, with the performance of voting saturating while DRN's performance continues to increase with widening gap. This shows that a sufficient number of samples is required capture the features of the distribution of softmax. On the adversarial images, the accuracies stay more of less the same. Although having more transformed samples beneficial for the performance on clean images, our distribution classifier improves the voting performance regardless of the number of samples.
5.2 CIFARIO AND CIFARIO0 100 RESULTS
For the CIFARIO and CIFAR100 datasets, N = 50 image transformation samples and 10,000 train- ing data were used. Figure (middle) shows the results for CIFAR10. All three distribution clas- sifiers gave comparable improvements over voting, except for MLP which performs worse than voting for adversarial images with RPN on DeepFool. For CIFAR 100 (Figure right), the distribu- tion classifiers mostly show improved performance over voting. There are exceptions where DRN (eg. PD adv., FGSM) and MLP (eg. RPN adv., DeepFool) have lower accuracy than voting. This suggests that for datasets with more classes, random forest may perform better than other classifiers.
As explained in Section 3 in the results Figure 5 we have excluded clean images which are misclassified by the CNN and the images where the attack has failed. To check that our method works on these images, we evaluated these images for CIFAR 100 with FGSM attack, random resize and padding and random forest classifier. Our results in Table 14 in the Appendix show that our distribution classifier method still outperforms majority voting.
END-TO-END ATTACK ON DISTRIBUTION CLASSIFIER METHOD
Here we evaluate end-to-end attacks on our distribution classifier method (with DRN) on MNIST and CIFARIO. We use Boundary Attack (Brendel et al. (017) which is a black-box decision based attack. We performed the attack on the base CNN classifier (CNN), CNN with pixel deflection and voting (Vote), and CNN with pixel deflection and distribution classifier trained on clean images (DRN). In addition, we trained the distribution classifier on a mixture distributions obtained from both clean and adversarial images obtained with IGSM on the base CNN, which can be seen lightweight adversarial training (DRN LAT) except that the CNN is kept fixed. Finally, we tested the attack on adversarially-traiin CNN (Ady trained CNN) by Madry al. (007)) with allowed

---

Published as a conference paper at ICLR 2020
0.075
0.075
0.050
12
0.050
0.025
0.025
0.000
0.000
original
CNN
Vote
DRN
DRN Adv trained
original
CNN
Vote
DRN
DRN Adv trained
LAT
CNN
LAT
CNN
7
7
7
4
4
4
advarsarial images
lowar advarsaiaa images
(a) MNIST
(b) CIFARIO
Figure 7: Black-box boundary attack MNIST and CIFARIO, with the average L2 of the attack perturbation shown, with the bars denoting the standard error the means. Note that the adversarial images for vote, DRN and DRN LAT are of lower quality.
perturbations of Loc S. 0.3. Since Boundary Attack uses the L2 measure, the ad adversarially-trann trained CNN which uses the Loc metric is not expected to perform well. For details of our implementation of Boundary Attack, please refer Appendix [ Figure shows the mean L2 of the perturbations over 100 test images, with maximum of 5000 iterations for the attack. CNN and the adversarially- trained CNN have very low perturbations. The stochastic models, Vote, DRN and DRN LAT, have much higher perturbations with lower quality adversarial images, and the difficulty of the attack increases that order. This shows that the distribution classifier and the lightweight adversarial training extension are more difficult to attack by the Boundary Attack method compared to voting.
Athalye al. (2018) have shown that under the white-box setting where the attacker has full knowl- edge of the CNN model and the defense, random transformation defenses are susceptible to fur- ther attack by estimating the gradients using multiple transformation samples, in method called Expectation over Transformation((OOT), To employ white-box attack on our distribution classifier method, there are a few potential challenges. First, we use 50 to 100 transformed samples per image to accumulate the distribution of softmax. Attacking method with EOT will be very cine-consuming as it requires taking multiple batches transformations, each with 50-100 sam- ples. Next, we have shown the non-differentiable random forest. While there have been attacks proposed for random forests Kantchelian al. 2016). it is unclear how feasible it is combine these attacks with EOT. We leave the evaluation of white-box attacks on our distribution classifier method for future work.
method works with different distribution classifier models, including
7 CONCLUSION
Adversarial attacks on convolutional neural networks have gained significant research attention and stochastic input transformation defenses have been proposed. However, with transformation-based defenses, the performance on clean images deteriorates and the exact mechanism in which how this happens is unclear. In this paper, we conduct in-depth analysis on the effects of stochastic transformation-basee defenses on the softmax outputs of clean and adversarial images. We observe that after image transformation, the distributions of softmax obtained from clean and adversarial images share similar distinct features. Exploiting this property, we propose a method that trains a distribution classifier on the distributions of the softmax outputs of transformed clean images only, but show improvements in both clean and adversarial images over majority voting. In our current work, have considered untargeted attacks on the CNN and it is interesting classifier method with targeted attacks.
distribution
ACKNOWLEDGGMETS
We thank Harold Soh, Wang Wei, Terence Sim, Mahsa Paknezhad and Kaicheng Liang for their constructive discussions. This work supported, part, by the Biomedical Research Council of the Agency for Science, Technology and Research and the National University of Singapore.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses adversarial examples. arXiv preprint (al2:088000044. 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision becseion-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv: 171..04248, 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEFE, 2017.
Minhao Cheng, Thong Le, Pin- Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsich. Query- efficient hard-label black-box attack: An optimization-base approach. arXiv preprint arXiv: 1807.04477. 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.657. 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. arXiv preprint 1xxv:1711.001111 2017.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. arXiv preprint arXiv:1804.08596 2018.
Scokwoo Jung, Unghui Lee, Jiwon Jung, and David Hyunchul Shim. Real-time traffic sign recog- nition system with deep convolutional neural network. In 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAT), pp. 31-34. IEEE, 2016.
Alex Kantchelian, J Doug Tygar, and Anthony Joseph. Evasion and hardening of tree ensemble classifiers. In International Conference on Machine Learning, pp. 2387-2396, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Connie Khor Li Kou, Hwee Kuan Lee, and Teck Khim Ng. A compact network learning model for distribution regression. Neural Networks, 2018
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 16X1::1607 2016a.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint wrXiv:6611.01236 2016b.
Yann LeCun, Léon Bottou, Yoshua Bengio. Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-22333 2324, 1998.
Yifeng Li, Lingxi Xie, Ya Zhang, Rui Zhang, Yanfeng Wang, and Qi Tian. Defending adversarial attacks by correcting logits. arXiv preprint
2019.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks, arXiv preprint arXiv:1706.000 06083, 2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- pufer vision and pattern recognition, pp. 2574-2582, 2016.

---

Published as a conference paper at ICLR 2020
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy. Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang. Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint arXiv:1610.00766 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. Deep face recognition. In bmve, volume 1, pp. 6, 2015.
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8571-8580, 2018.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan grrense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint 1805.06660 2018.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint 10t0. 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 2013.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 17051777507..4 2017.
Cihang Xie, Jianyu Wang. Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint arXiv: 2017.
2017.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks. arXiv preprint
arXiv:1605.07146. 2016.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016
A DISTANCE BETWEEN DISTRIBUTIONS OF SOFTMAX FOR ALL MNIST CLASSES
In Section 3 we studied the distance metrics for the distribution of softmax. Figure g shows the distance metrics for all ten MNIST classes with increasing number of pixel deflections.
B EXAMPLES OF VOTING RECOVERING FROM ADVERSARIAL ATTACK
In Figure we show examples where pixel deflection with voting recovers from the adversarial attack.
C ADAPTATION OF DRN FOR DISTRIBUTION CLASSIFICATION
For one the distribution classifier methods, we adapt a state-of-the-ar regression method, called distribution regression network (DRN) (Kou et al., 2018). DRN encodes an entire distribution each network node and this compact representation allows it to achieve higher prediction accuracies for the distribution regression task compared to conventional neural networks. Since DRN shows superior regression performance, we adapt DRN for distribution clas- sification in this work.
12

---

Published as a conference paper at ICLR 2020
0.15
0.15
Class
0:15
Averaee 15 divegennce
15 divergerce
Averae 15 divergence
diveggence
15 divergencee
0.10
0.10
0.10
0.05
0.05
5.5
0.05
0.09
50
300
800
350
500
50
303
850
803
300
850
Number deflections
Number deflectooss
Number deflections
Number deflections
Number of deflections
Class
Average divegennce
Werrgg diergence
Averuge 55 divergence
Average 5 dhergence
wwwsuae 35 divergence
0.10
0.10
0.10
0.30
0.10
0.05
0.05
30
860
303
800
300
Number deflections
Number of deflections
Number of deflections
Number deflections
Number of deflections
clean-clean (same class)
acaverssraalaceesrrriaall (same class)
ldvversrraaall (same class)
clean-clean (different class)
Figure 8: Effect of increasing number of pixel deflections on the distances of distributions obtained from clean and adversarial images of the MNIST dataset. Best viewed in color.
Class label 6, adv.
adv. vote:
adv.:
0.8
0.0
094
04
255
1.0
Figure 9: Adversarial examples where pixel deflection with voting recovers from adversarial attack, where ground truth label is 6 but CNN predicts as O.
Our adaption of the distribution classifier is shown on the right of Figure 10 The network consists of fully- oolly-connected layers, where each node encodes distribution. The number of hidden layers and nodes per hidden layer are chosen by cross validation. The number discretization bins for each distribution for the input layer and hidden layers is also tuned as hyperparameters. To adapt DRN for our distribution classification task, for the final layer, we have C nodes representing each class and we use 2 bins for each distribution to represent the logit output for the corresponding class. The cost functiin for the distribution classifier is the cross entropy loss on the logits. The distribution classifier is optimized by backpropagation using the Adam optimizer (Kingma & Baj 2014). The weight initialization method follows Kou et al. (2018), where the weights are sampled from uniform random distribution.
D EXPERIMENTAL SETUP
D.1 ADVERSARIAL ATTACK HYPERPARAMETES
Tables to 3 show the hyperparameter settings used for the adversarial attacks. The attacks are implemented using the CleverHans library (Papernot et al. 2018). For DeepFool and C&W W, the other hyperparameters used are the default values set in CleverHans. For L2 norm, use the not-mean-square distortion normalized by total number of pixels, following previous works.
D.2 IMAGE TRANSFORMATION DEFENSE PARAMETERS
Tables soow the image transformation parameters used for MNIST and CIFARIO respectively. The hyperparameter tuning for each defense method is conducted the validation set for each
13

---

Published as a conference paper at ICLR 2020
logits
softmax
0.2
0.09
Illmr
0.8
0.17
0.2
0.09
'bus'
nilla
0.4
0.11
Figure 10: Adaptation of DRN for the distribution classifier task.
Table 1: Hyperparameter settings for the four adversarial attack methods on the MNIST dataset.
MNIST
Attack success rate (%)
Distortion
Settings
FGSM
Lo = 0.39
53.2
100 steps,
IGSM
Loc = 0.24
91.2
step size=0.004
DeepFool
1.2 0.06
97.6
Max iter.=30
C&W
L2 =
94.9
Max itcr:=500
Table 2: Hyperparameter settings for the four adversarial attack methods on the CIFAR10 dataset.
CIFARIO
Attack success rate
Distortion
Settings
(%)
FGSM
Los 0.031
58.2
20 steps,
IGSM
Loo 0.031
100.0
step size=0.004
DeepFool
L 0.042
95.3
Max iter.=
C&W
L2 0.023
94.1
Max iter.=10
dataset. We select hyperparameters that give the best recovery from adversarial attack, regardless of the deterioration accuracy on clean images.
D.3 CLASS ACTIVATION MAPS FOR PIXEL DEFLECTION
The pixel deflection (Prakash al., 2018 defense uses class activation maps (CAMs) (Zhou al, 2016) to randomly select pixels to undergo the deflection step. In our experiments, we did not use class activation maps and instead randomly select pixels with equal probabilities. First, for MNIST dataset, CAMs are unsuitable because the LeNet (LeCun al. 1998) architecture does not have global average pooling layers which are required for CAMs. For the CIFAR10 dataset, the wide ResNet (Zagorukko & Komodakis 2016) architecture uses final layer of global average pooling and so we tested CAMs on it. Table 7 compares the performance on clean and adversarial images using the FGSM and IGSM attacks, with and without CAMs, which shows that using CAMs does
14

---

Published as a conference paper at ICLR 2020
Table 3: Hyperparameter settings for the four adversarial attack methods the CIFARI00 dataset.
CIFAR 100
Attack success rate
Distortion
Settings
(%)
FGSM
Lo 0.031
78.5
20 steps,
IGSM
Lo 0.031
99.9
step size=0.004
DeepFool
L2 0.051
91.8
Max iter.=5
C&W
L2 0.026
96.0
Max iter.=10
Table Details of the image transformation parameters for MNIST. The three transformation-baal methods tested are random pixel noise (RPN), pixel deflection (PD) and random resize and padding (RRP). For RPN, the noise magnitude is unnormalized (out 255). For PD, the number of deflections, w is the window size and a is the denoising parameter.
FGSM
IGSM
DeepFool
C&W
RPN PD
=130
=90
=10=10
Lx==10
d=300, w=20, o=0
d=100, w=20, o=0
d=10, w=20, a=0.08
d=100, w=25, a=0.08 resize range=[23,28)
RRP
resize range=[44228
resize range=[22,28]
resize range=(23,28)
not cause significant difference in performance. This may be because CAMs are more effective on larger images such those in ImageNet where there many more background pixels.
D.4 DISTRIBUTION CLASSIFIER HYPERPARAMETER
Our defense method uses distribution classifier train distributions of softmax probabilities obtained from transformed samples of the clean images. For each image, we build the marginal distributions of the softmax for each class using kernel density estimation with Gaussian kernel. The kernel width is optimized to be 0.05. For DRN and MLP, the network architecture of the distribution classifier and optimization hyperparameters are chosen by cross-validation. For random forest, the number of trees and maximum depth of the trees are tuned by cross-validation. The hyperparameters used are shown Tables 8 10
D.5 ACCURACY RESULTS FOR DISTRIBUTION CLASSIFIERS
Here include the detailed numerical figures for the accuracies of majority voting and the distri- bution classifier methods. Tables [1] to [13 show the clean and adversarial test accuracies and the attack methods and the defense methods.
E DETAILS OF IMPLEMENTATION OF BOUNDARY ATTACK
For Vote, DRN and DRN LAT, the model outputs are random because of the random image trans- formation. At each step of Boundary Attack, we allow the attack query the model once, and this involves taking 50- 50-100 transformed samples for the image to perform voting or to feed to the
Table 5: Details of the image transformation parameters for CIFARIO.
FGSM
IGSM
DeepFool
C&W
RPN PD
=40
T0
To
T1c=10
d=60, w=10, 7=0.06
d=80, w=10, o=0.06
d=20, w=10, a=0.04
d=80, w=10, o=0.04 resize range=[23332]
RRP
resize range=[20,25)
resize range=[19.25)
resize range=(28832)
15

---

Published as a conference paper at ICLR 2020
Table 6: Details of the image transformation parameters for CIFAR100.
FGSM
IGSM
DeepFool
C& W
RPN
=40
=
= d=20, w=10, 0=0.22
=x
PD
d=90, w=10, o=0.66
d=80, w=10, o=0.06
d=30, w=10, o=0.04 resize rangc=((5530)
RRP
resize range=[23,26)
resize range=[23326]
resize range=(27,321
Table 7: Performance for pixel deflection on the CIFARIO dataset with and without class activation maps (CAM). Numbers shown are the test accuracies (%).
FGSM
IGSM
Without CAM
With CAM
Without CAM
With CAM
clean
75.96 (0.07)
75.98 (0.02)
75.71 (0.04)
75.70 (0.06)
adv.
36.35 (0.07)
36.40 (0.02)
51.66 (0.07)
51.59 (0.08)
Table 8: DRN network architecture for the distribution classifier, where 2x10 represents 2 hidden layers of 10 nodes.
FGSM
IGSM
DeepFool
C&W
RPN
Ix10
2x10
1x10
1x10
MNIST
PD
Ix10
Ix10
Ix20
Ix20
RRP
1x10
Ix10
1x20
1x20
RPN
1x10
Ix20
1x20
Ix20
CIFAR1O
PD
1x20
Ix10
1x20
1x10
RRP
Ix20
Ix10
1x20
1x10
RPN
1x100
1x100
1x100
1x100
CIFARIO0
PD
1x100
1x100
1x100
1x100
RRP
1x100
100
1x100
1x100
Table 9: MLP network architecture for the distribution classifier, where 2x10 represents 2 hidden layers of 10 nodes.
FGSM
IGSM
DeepFool
C&W
RPN
1x20
1x20
1x50
1x20
MNIST
PD
1x20
1x50
Ix50
1x50
RRP
1x50
1x20
1x50
1x50
RPN
Ix50
Ix50
Ix50
Ix20
CIFARIO
PD
1x50
Ix50
Ix50
Ix50
RRP
1x50
1x50
Ix20
1x20
RPN
Lx2000
1x1000
Ix500
Ix500
CIFARI00
PD
Ix500
Ix1000
2x100
Ix1000
RRP
1x200
1x1000
2x100
Ix1000
16

---

Published as a conference paper at ICLR 2020
Table 10:
Random forest hyperparameters for the distribution classifier, where v represents the number trees and d represents the maximum depth of the trees.
FGSM
IGSM
DeepFool
C&W
RPN
n=100,
d=20
n=100, d=20
n=200, d=50
n=200, d=50
MNIST
PD
n=100. d=20
n=100, d=20
n=200, d=50
n=200, d=50
RRP
n=200, d=20
n=100, d=20
n=200, d=50
n=200, d=50
RPN
n=200, d=50
n=200, d=50
n=200, d=50
n=200, d=50
CIFARIO
PD
n=200, d=50
n=200, d=50
n=200, d=50
n=200, d=50
RRP
n=200, d=50
n=200, d=50
n=200, d=50
n=200, d=50
RPN
n=200, d=200
n=200, d=200
n=200, d=200
n=200, d=200
CIFAR100
PD
n=200, d=200
n=200, d=200
n=200, d=100
n=200, d=200
RRP
n=200, d=200
n=200, d=200
n=200, d=100
n=200, d=100
Table 11: MNIST results: For each attack, we compare the clean and adversarial (adv.) test accura- cies with majority voting (Vote) and the three distribution classifier methods: distribution regression network (DRN), random forest (RF) and multilayer perceptron (MLP). The three transformation- based defenses random pixel noise (RPN), pixel deflection (PD) and random resize and padding (RRP). With no defense, the clean accuracy is 100% and the adversarial accuracy is 0%.
MNIST test accuracies (%)
RPN
PD
RRP
clean
adv.
clean
adv.
clean
adv.
Vote
95.4
17.0
87.6
17.1
91.4
36.4
DRN
97.8
32.1
96.9
22.6
98.2
39.4
FGSM
RF
97.8
35.2
97.5
32.8
98.7
31.7
MLP
98.1
34.4
97.3
31.6
98.8
24.0
Vote
97.2
77.9
98.9
51.3
98.7
71.1
DRN
98.4
90.9
98.8
67.3
98.4
82.4
IGSM
RF
98.4
91.1
99.3
49,1
99.0
75.4
MLP
98.5
91.4
99.3
45.4
99.0
65.2
Vote
100
99.3
100
99.4
98.9
93.6
DRN
99.9
98.3
99.9
99.4
99.1
95.2
DeepFool
RF
99.9
93.1
99.8
97.0
99.1
96.1
MLP
99.8
77.0
99.8
92.8
99.0
95.4
Vote 99.9
99.7
99.0
93.8
98.9
95.3
DRN
99.7
99.5
99.4
98.0
99.2
97.4
C&W
RF
99.9
96.8
99.3
96.6
99.1
97.0
MLP
99.7
89.6
99.4
96.3
99.0
96.7
17

---

Published as a conference paper at ICLR 2020
Table 12: CIFARIO results: For each attack, we compare the clean and adversarial (adv.) test accura- cies with majority voting (Vote) and the three distribution classifier methods: distribution regression network (DRN), random forest (RF) and multilayer perceptron (MLP). The three transformation- based defenses are random pixel noise (RPN), pixel deflection (PD) and random resize and padding (RRP). With no defense, the clean accuracy is 100% and the adversarial accuracy is 0%.
CIFARIO test accuracies (%)
RPN
PD
RRP
clean
adv.
clean
adv.
clean
adv.
Vote
47.3
16.9
75.9
36.4
81.9
39.6
DRN
68.4
24.8
84.1
39.0
88.4
41.2
FGSM
RF
73.8
28.8
85.2
38.3
88.7
41.1
MLP
72.2
26.8
84.6
37.7
88.6
40.5
Vote
47.3
26.5
75.7
51.7
79.8
56
DRN
68.8
37.7
83.1
58.2
87.9
61.3
IGSM
RF
73.8
40.7
85.2
59.4
88.5
60.9
MLP
72.2
39.2
84.9
58.7
88.4
61.0
Vote
97.8
91.4
93.5
91.1
97.6
91.3
DRN
98.6
93.0
94.6
92.3
97.7
91.5
DeepFool
RF
98.6
93.0
94.5
92.2
97.9
91.9
MLP
98.3
87.3
94.2
91.1
97.7
91.0
Vote 95.8
85.5
92.5
88.5
95.8
87.7
DRN
97.5
87.6
94.1
90.8
97.0
89.3
C&W W
RF 97.7
87.6
94.2
90.6
96.9
89.2
MLP
97.3
86.8
93.9
89.9
96.8
88.8
Table 13: CIFAR100 results: For each attack, compare the clean and adversarial (adv.) test accuracies with majority voting (Vote) and the three distribution classifier methods: distribu- tion regression network (DRN), random forest (RF) and multilayer perceptron (MLP). The three transformation-be based defenses are random pixel noise (RPN), pixel deflection (PD) and random re- size and padding (RRP). With defense, the clean accuracy 100% and the adversarial accuracy 0%.
CIFARI00 test accuracies (%)
RPN
PD
RRP
clean
adv.
clean
adv.
clean
adv.
Vote
22.4
6.2
61.7
24.5
65.7
19.8
DRN
22.8
9.5
59.5
21.8
73.5
23.0
FGSM
RF
46.9
18.9
71.8
28.9
79.5
27.0
MIPP
22.8
8.9
69.8
27.4
78.0
25.5
Vote
46.7
16.1
61.9
40.3
65.7
34.0
DRN
48.6
17.8
61.8
40.6
73.5
39.9
IGSM
RF
65.7
25.2
71.9
47.1
79.6
44.7
MLP
59.8
21.4
69.4
44.6
78.1
43.7
Vote
97.7
92.4
96.1
92.8
89.4
82.3
DRN
97.7
92.2
95.9
92.5
91.2
84.6
DeepFool
97.5
90.8
95.8
92.1
91.5
84.5
MLP
91.4
55.6
91.3
81.7
89.2
80.8
Vote 88.3
72.3
84.6
78.8
82.4
72.0
DRN
89.1
73.0
84.1
77.8
86.2
75.5
C&W
RF
90.6
74.9
86.5
80.7
88,2
77.9
MLP
88.8
71.0
85.1
78.4
87.1
77.1
18

---

Published as a conference paper at ICLR 2020
Table 14: Distribution classifier outperforms majority voting for clean images that are misclassified by CNN and images where the attack has failed. The results are for CIFAR100 with FGSM attack, random resize and padding and random forest classifier.
Test accuracy (%)
No transformation
Voting
Distribution classifier
Clean images misclassified by CNN
12.3
18.1
Images where attack has failed
100
65.7
79.5
distribution classifier to obtain prediction. To avoid overfitting fixed transformation pattern, the transformation is random at each step. Our criteria for an image being adversarial is that out of queries, the image misclassified at least once. Because of the randomness the model, the image returned by Boundary Attack may be classified to the correct class, and we increase the perturba- tion by increasing amounts until the image misclassified. Note that to overcome the randomness, could have performed multiple queries each attack step, but because models already use 50- 100 transformed samples per query, this will be computationally infeasible.
19


Reviews:
[{"review_id": "HJeltjbHcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors analyze the use of image transformation as a defense against adversarial examples, where a challenge is to prevent the deterioration of performance on clean images. To do this, they show that the softmax distributions for clean and adversarial images share similar \"features\", and therefore one can apply a trained distribution classifier which takes the softmax distribution to return the class label. This is as opposed to original approach of making a prediction for each sample (a random transformation of the input image) followed by majority voting.\n\nI have one major concern about their analysis, which makes me lean weakly towards not accepting the paper. I'm happy to change my score after discussions and further clarifications. In particular, I'm not convinced that the phenomena of softmax distributions appearing similarly for clean and adversarial images is not simply a result of the stochastic transformation converging to a fixed (or possibly neighborhood of fixed) distributions. Does the transformation retain information about the difference between classes? That is, while the divergence of softmax distributions from clean and adversarial images decreases as one applies more transformations, does the divergence of softmax distributions from clean images of two different classes also decrease?\n\nAs a non-expert in the field, I found the paper well-written. It motivates the idea well by identifying challenges in a promising technique (random tranfsormations) and provides decent background explanation.\n\nFigure 2 is an interesting example showing the relationship of the softmax distributions for clean and adversarial examples. This raises a few questions which are unclear to me:\n\n1. I'm a bit surprised at how remarkably bad transformation-based defenses seem to be in degrading classifier performance. Is it really the case that the MNIST model only gets 78% accuracy on class label 8 and 36% on class label 6? What about without the transformation?\n2. This phenomena of softmax distributions being similar seems to largely depend on the stochastic transformation T and form of attack. This is especially seen in Figure 2b, which suggests that T may be acting like a transition kernel which takes an arbitrary initial state and may have it converge to a fixed stationary distribution. What does the divergence look like for two class distributions of clean images over the number of pixel deflections?\n\nFor the defense, I'm also curious what happens if the model used the distribution classifier with random transformations at training time, instead of a separate softmax output layer.\n\nRegarding experiments, I only assessed their sensibility. Unfortunately, I don't know enough about the field to tell how significant the results are. \n\n1. For the choice of number of transformations and training set size for the distribution classifier, how were they chosen? They seem to vary arbitrarily across the datasets.\n2. It seems like the improvements appear only in single-step FGSM, which also tends to be the worst performing, where the other methods are iterative. What's the intuition for this?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgUEOWkqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel defense method to make the classification more robust. The motivation is based on the observation: the distribution of the soft-max for the cleaned image and its transformed images for one class is similar to the distribution of the soft-max for the adversarial image and its transformed images for the same class, and the distributions of the soft-max for the cleaned image and its transformed images for different classes are different. Then, a distribution based method is proposed to classify the distribution of the soft-max for the cleaned (or adversarial) image and its transformed images. \n\nAfter I read the core part several times, finally I understood the paper. Overall I think this paper is well motivated, and the empirical results also support the claims/observations. But I think this paper could be improved by (1) checking the performance over large datasets (such as ImageNet); (2) providing possible analysis on the observation. Otherwise, the readers cannot be fully convinced. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlcBGi3cr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n===== Post Rebuttal ===== \n\nThe authors addressed almost all of my comments. I think the revised paper is of higher quality and clarity as a result. Figure 3 is the main remaining concern. However,  given the listed strengths of the paper, I am happy to change my rating to weak accept .\n\n===== Summary ===== \n\nOne strategy for defense against adversarial attacks is to perturb the input sample multiple times and then aggregate their corresponding outputs into a single output. One recent variant of this method (Prakash et al. 2018) shows using this type of defense mechanism degrades the accuracy on clean (unperturbed) inputs. This paper tries to mitigate the accuracy drop on clean inputs. It does so by learning an additional distribution classifier which takes as input the distribution of perturbed samples’ outputs. The method is trained on MNIST, CIFAR10, CIFAR100 and using 4 different adversarial attacks. It shows significant improvement in several cases over standard input-perturbation methods.\n\n\n===== Strengths and Weaknesses ===== \n\n+ The paper picks an important problem, proposes a simple technique and achieves significant improvements in certain cases.\n+ The experiments are done on 3 datasets and 4 different adversarial attack techniques.\n+ The final experiment on end-to-end attack is interesting.\n\nConcerns regarding the main paper’s description\n- From Figure 1, it seems that the distribution classifier does a binary classification for the distribution of each class separately. If this is the case, how is the final classification done (using the binomial distribution per class)? The figure suggests that the maximum probability of the binary classifications (bus for yellow class with 0.8 score) is taken. This would be strange since they are not optimized to compete with each other (e.g. as in a more proper cross entropy). In any case, the details of the distribution classifier is missing from the paper, and section 4 refers to the figure. The details should come in the main text using precise language and math formulation. \n\n- page 4 is very hard to follow since it tries to give a textual description to mathematical concepts. I think it is important to provide the mathematical definition such that 1) it is easier to follow 2) the concepts are unambiguously defined. This includes a definition of softmax distribution, joint distribution of output and transformation, marginalized distributions, clean intra-class distance, adversarial intra-class distance, and clean-adversarial inter-class distance.\n\n- page 4: it seems it is implied in the definition of “clean” images that these images are classified “correctly” by the network. Otherwise, in Figure 2.a, the clean row is uninterpretable not knowing whether the clean image was classified correctly or not.\n\n- page 4,5: what is a “transformation magnitude”? It could refer to both the number of transformations and also the number of perturbed pixels. The behavior in Figure 3 suggests that the latter should be the case, but it’s important to disambiguate.\n\n- figure 3 is counterintuitive. The support (possible distributions) should normally live on a simplex (due the sum-to-one probability constraint). In figure 3, it seems there is support in the full unit hypercube. In particular, we can see density along the horizontal/vertical line of 1-density for one class (i.e. non-zero density for others in this case)\n\n- page 6: “[...] to train a distribution classifier on the distributions obtained from clean images only [...]“. I cannot follow the argument at the end of the first paragraph that led to the proposed *training on clean images only*. I understand that from the qualitative demonstration of 8 different examples in Figure 4 the distributions of perturbed adversarial examples *might be* redundant to the perturbed wrongly-classified clean examples. However, even if it is redundant, why should it be harmful to train the distribution classifier with adversarial examples as well as clean examples?\n\nRegarding Experiments:\n- why is the evaluation done only based on accuracy of *correctly-classified* clean images and the recovery of *wrongly-classified* adversarial images? Couldn’t it be that the baseline (majority voting) does a better job than the distribution classifier on the “wrongly-classified” clean images and the “correctly-classified” adversarial images? \n- I think it's still interesting and informative to train the distribution classifier using both clean and perturbed inputs to compare the performance.\n- page 6: why is there a need for applying a kernel density estimator followed by a discretization (e.g. for instance instead of a simple histogram)?\n\n===== Final Decision ===== \nMy current “weak reject” rating is primarily based on the unclarity of the main paper and secondary regarding the concerns for experiments. \n\n===== Points of Improvement ===== \nI believe the paper will become stronger if the proposal is described and motivated more formally.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates tradeoffs between preserving accuracy on clean samples and increasing robustness on adversarial samples by using transformations and majority votes. Observations on the distribution of the induced softmax show that existing methods could be improved by leveraging information from that distribution to correct predictions, as confirmed by experiments.
The problem space is important and reviewers find the approach interesting. Authors have provided some necessary clarifications during rebuttal and additional experiments. While some reservations remain, this paper's premise and its experimental results appear sufficiently interesting to justify an acceptance recommendation.
Decision: Accept (Poster)