Enhancing Transformation-Based Defenses Against Adversarial Attacks with a Distribution Classifier

1 INTRODUCTION . There has been widespread use of convolutional neural networks ( CNN ) in many critical real-life applications such as facial recognition ( Parkhi et al. , 2015 ) and self-driving cars ( Jung et al. , 2016 ) . However , it has been found that CNNs could misclassify the input image when the image has been corrupted by an imperceptible change ( Szegedy et al. , 2013 ) . In other words , CNNs are not robust to small , carefully-crafted image perturbations . Such images are called adversarial examples and there have been active research efforts in designing attacks that show the susceptibility of CNNs . Correspondingly , many defense methods that aim to increase robustness to attacks have been proposed . Stochastic transformation-based defenses have shown considerable success in recovering from adversarial attacks . Under these defenses , the input image is transformed in a certain way before feeding into the CNN , such that the transformed adversarial image would no longer be adversarial . As the transformation is random , by feeding in samples of the transformed image through the CNN , we accumulate a set of CNN softmax outputs and predictions . As such , existing transformationbased defenses take a majority vote of the CNN predictions from the randomly transformed image ( Prakash et al. , 2018 ; Guo et al. , 2017 ) . Transformation-based defenses are desirable as there is no need to retrain the CNN model . However , they suffer from deterioration of performance on clean images . With increasing number of pixel deflections ( Prakash et al. , 2018 ) , there is improvement on the performance on adversarial images , but this comes with a rapid deterioration of performance on clean images . The exact mechanism of the deterioration in performance on clean images is unclear . We believe that the softmax distribution induced by the random transformation contains rich information which is not captured by majority vote that simply counts the final class predictions from the transformed samples . Now , an interesting question is whether the features in the distribution of softmax could be better utilized . In this paper , to elucidate how the deterioration in accuracy on clean images occurs , we study the effects of the random image transformations on the distribution of the softmax outputs and make some key observations . After the image transform , some clean images show distributions of softmax with modes at an incorrect class , reflecting the deterioration in voting accuracy as observed before . While the shifting of the distribution mode to the incorrect class is detrimental to the voting prediction , the resulting distribution of softmax contains features that is useful for correcting the prediction . In addition , we observe that the adversarial counterparts show similar shifts in the distributions of softmax as the clean images . We also look into the distribution shapes for the transformed clean and adversarial images and find that they are similar . With these observations , we propose a simple method to improve existing transformation-based defenses , as illustrated in Figure 1 . We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed clean images and predict the class label . Without retraining the original CNN , our distribution classifier improves the performance of transformation-based defenses on both clean and adversarial images . On the MNIST dataset , the improvements in accuracy over majority voting are 1.7 % and 5.9 % on the clean and adversarial images respectively . On CIFAR10 , the improvements are 6.4 % and 3.6 % respectively . Note that the distributions obtained from the adversarial images are not included in the training of the distribution classifier . In real-world settings , the type of attack is not known beforehand . Training the distribution classifier on a specific attack may cause the classifier to overfit to that attack . Hence , it is an advantage that our defense method is attack-agnostic . Our experimental findings show that the features of the distribution in the softmax are useful and can be used to improve existing transformation-based defenses . Our contributions are as follows : 1 . We analyze the effects of image transformation in existing defenses on the softmax outputs for clean and adversarial images , with a key finding that the distributions of softmax obtained from clean and adversarial images share similar features . 2 . We propose a method that trains a distribution classifier on the distributions of the softmax outputs of transformed clean images only , but show improvements in both clean and adversarial images . This method is agnostic to the attack method , does not require retraining of the CNN and can be integrated with existing transformation-based methods . 2 RELATED WORK : ATTACKS AND DEFENSES . Given an image dataset { ( x1 , y1 ) · · · ( xM , yM ) } and a classifier Fθ that has been trained with this dataset with parameters θ , the aim of the attack is to produce an adversarial image xadvi such that Fθ ( x adv i ) 6= yi , and ||xadvi − xi|| is small . We focus on four gradient-based untargeted attacks . The Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2014 ) is a single-step attack that uses the sign of the gradient of the classification loss to perturb the image . Iterative Gradient Sign Method ( IGSM ) ( Kurakin et al. , 2016a ) is an iterative version of FGSM . In DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , at each iteration , the attack approximates the classifier with a linear decision boundary and generates the minimal perturbation to cross the boundary . Finally , the Carlini & Wagner ( C & W ) ( Carlini & Wagner , 2017 ) L2 attack jointly minimizes the perturbation L2 norm and a differentiable loss function based on the classifier ’ s logit outputs . Besides gradient-based attacks , there are also black-box attacks where the CNN model is not known and only the softmax output or final prediction is given ( Brendel et al. , 2017 ; Ilyas et al. , 2018 ; Cheng et al. , 2018 ) . Defense methods have been proposed to make the classifiers more robust . In adversarial training , the CNN model is trained on adversarial examples generated from itself ( Madry et al. , 2017 ; Kurakin et al. , 2016b ) or from an ensemble of models ( Tramèr et al. , 2017 ) . Other methods involve training auxiliary neural networks on mixture of clean and adversarial images , for instance , by denoising the inputs with a neural network before feeding into the CNN ( Liao et al. , 2018 ; Song et al. , 2017 ; Samangouei et al. , 2018 ) or by training a neural network on the CNN logits ( Li et al. , 2019 ) . In the next section , we introduce another class of defense : transformation-based defenses . 2.1 TRANSFORMATION-BASED DEFENSES . Transformation-based defenses aim to recover from adversarial perturbations , that is for input transformation T , we want Fθ ( T ( xadvi ) ) = yi . At the same time , the accuracy on the clean images has to be maintained , ie . Fθ ( T ( xi ) ) = yi . Note that transformation-based defenses are implemented at test time and this is different from training-time data augmentation . Here we introduce two transformation-based defenses that we experiment on . Pixel deflection ( PD ) ( Prakash et al. , 2018 ) : Pixel deflection corrupts an image by locally redistributing pixels . At each step , it selects a random pixel and replaces it with another randomly selected pixel in a local neighborhood . The probability of a pixel being selected is inversely proportional to the class activation map ( Zhou et al. , 2016 ) . Lastly , there is a denoising step based on wavelet transform . In our experiments , we did not use robust activation maps for our datasets as we found that this omission did not cause significant difference in performance ( see Appendix D.3 ) . Random resize and padding ( RRP ) ( Xie et al. , 2017 ) : Each image is first resized to a random size and then padded with zeroes to a fixed size in a random manner . In many transformation-based methods , the transformation is stochastic . Hence there can be different samples of the transformation of an image : tx ∼ T ( x ) , where tx represents a transformed sample . Existing transformation defenses benefit from improved performance by taking the majority vote across samples of random transformations . The advantage of transformation-based methods is that there is no retraining of the CNN classifier . However , a weakness , as identified by Prakash et al . ( 2018 ) , is that the transformation increases the accuracy on adversarial images at the expense of the accuracy on clean images . The exact mechanism of the deterioration in performance on clean images is unclear . In this paper , we elucidate how the deterioration in accuracy on clean images occurs by studying the effects of the random image transformations on the distribution of the softmax outputs . 3 ANALYSIS ON DISTRIBUTIONS OF SOFTMAX WITH RANDOM IMAGE TRANSFORMATIONS . Due to the randomness of the transforms , samples of the transformed image will have different softmax outputs . With each image , we obtain a distribution over the softmax outputs accumulated from multiple samples of the transformation . These are the steps to obtain the distribution of softmax : 1 . For each input image x , obtain N transformed samples : t ( i ) x ∼ T ( x ) , i = 1 , · · · , N 2 . The transformed samples of the image ( t ( 1 ) x , t ( 2 ) x , · · · , t ( N ) x ) are fed into the CNN individ- ually to obtain their softmax probabilities . Let σ ( i ) x be the softmax vector derived from t ( i ) x , and σ ( i ) x , j , for j = 1 , · · · , C , be the j-th component of the softmax vector . C denotes the number of classes for the classification task . With each input image and a transformation method , there exists an underlying joint distribution of the CNN softmax probabilities , from which we estimate with N samples . 3 . The underlying joint distribution of the softmax has a dimension equal to the number of classes ( eg . 10-D for MNIST ) . Performing accurate density estimation in high dimensions is challenging due to the curse of dimensionality . Here we make an approximation by computing the marginal distributions over each class . When we use the term ‘ distribution of softmax ’ , we are referring to the marginalized distributions . We use kernel density estimation with a Gaussian kernel . Let hx , j be the distribution accumulated from σ ( 1 ) x , j , · · · , σ ( N ) x , j : hx , j ( s ) = 1 N √ 2πδ N∑ i exp ( − ( s− σ ( i ) x , j ) 2 2δ2 ) , ( 1 ) where δ is the kernel width and s ∈ [ 0 , 1 ] is the support of the softmax output . The distribution is then discretized into bins . In this section , we study the effect of image transformation on the distribution of the softmax and make several interesting observations . In the following analyses , we study a LeNet5 CNN ( LeCun et al. , 1998 ) trained with MNIST . The adversarial images are generated using FGSM and for the transformation defense , we use pixel deflection , with N=100 transformation samples per image . The image transformation magnitude is controlled by the number of pixel deflections , d. In the analysis here and in the experimental results in Section 5 , when reporting the accuracies , on clean images , we consider images that have been correctly predicted by the CNN , hence without any transformation defense , the test accuracy is 100 % . This is following the setup of Prakash et al . ( 2018 ) where the misclassified images by CNN are excluded as it is not meaningful to evaluate any attack ( and subsequent defense ) methods on these images . For adversarial images , we consider the images that have been successfully attacked , so the test accuracy reflects the recovery rate and without any defense the accuracy is 0 % . In Figure 2a , we show how the image transformation affects the voting predictions on two MNIST classes . For each MNIST class , we take all the clean and adversarial test images , perform the transformation and then feed through the CNN to obtain the final voting prediction . We observe that for class label 8 , there is some recovery from the attack as some adversarial images are voted to the correct class after the transformation . However , some clean images get misclassified to other classes ( eg . 2 and 3 ) . Although this means there is a deterioration of the accuracy on clean images , it is interesting that the misclassifications have the same voting classes as the transformed adversarial images . A similar pattern is observed for class label 6 where the clean images are misclassified to classes 4 and 5 , which overlap with the vote predictions of some adversarial images at d = 300 . With the above analysis , we characterize the relationship between the clean and adversarial images in terms of the JS divergence of the distributions of the softmax at increasing number of pixel deflections . For each MNIST digit class , we quantify the ( 1 ) distance of the distributions among the clean images ( clean-clean , same class ) , ( 2 ) distance of the distributions among the adversarial images ( adversarial-adversarial , same class ) , ( 3 ) the distance of the distributions between clean and adversarial images ( clean-adversarial , same class ) and ( 4 ) the distance of the distributions between clean images of this class and all other classes ( clean-clean , different class ) . Here we give details on the calculation of the 4 distance measures . First , the distance between the distributions of softmax output for two input images , x1 and x2 is given by d ( hx1 , hx2 ) = 1 C ∑C j DJS ( hx1 , j , hx2 , j ) , where DJS is the Jensen-Shannon divergence . Distance measures of ( 1 ) and ( 2 ) computed by taking the average distance of each image distribution to the centroid distribution which is computed with µ ( { hx1 , j , · · · , hxM , j } ) = 1M ∑M i hxi , j . ( 3 ) is computed by the distance between the centroids of the clean and adversarial distributions . Finally , ( 4 ) is computed by the distance of the centroid distribution of the clean images of the particular class with the centroid distribution of another class , averaged over the other 9 classes . In Figure 2b , we show results for two MNIST classes , but similar trends are observed across all classes ( see Figure 8 in Appendix A ) ) . The clean-clean ( same-class ) distance starts off low initially as all clean samples will give high scores at the correct class . With increasing number of deflections , there is increased variability in the softmax outputs and the resulting distributions . Next , the adversarial images of the same class are initially predicted as different incorrect classes without any transformation , and hence the adversarial-adversarial ( same-class ) distance starts off high and decreases with more transformation . The clean-adversarial ( same-class ) distance decreases with increasing image transformation which shows that the distributions of softmax from the clean and adversarial images are becoming more similar . Finally , the clean-clean ( different class ) distance decreases as well , which is expected because we already know that with more transformation , the clean image voting accuracy deteriorates . However , we observe that clean-clean ( different class ) distance decreases less rapidly and remains higher than clean-clean ( same-class ) distance at d=300 . This means the transformation still retains information about the differences between the classes . At d=800 , all 4 distance measures converge , which suggests the number of deflections is too large and the differences between the classes are no longer retained . Next , we visualize the morphing of the distributions with increasing number of pixel deflections for an example image in Figure 3 . For the purpose of visualization , instead of the per-class marginal distributions of the softmax , we perform kernel density estimation ( kde ) on the softmax values for the marginals on class 5 and 6 . The softmax values of the other 8 classes are not shown . We have not excluded the areas where performing kde results in sum probability exceeding one , and our visualization still conveys our ideas and the distribution shapes well . Without any image transformation , as expected , the softmax outputs of the clean and adversarial images are very different . As the number of pixel deflections increases , each point evolves to a distribution due to the randomness of the transformation . The voting mechanism is straightforward ; an image is classified to the class where the distribution mass is largest . In this example , the distribution shapes for the clean and adversarial image become more similar , and result in the same incorrect voting prediction at d=300 . This shows the similarity of distributions obtained from clean and adversarial images after image transformation , which was illustrated in Figure 2b . In Figure 4 , we show more examples of the distributions obtained from clean images ( A-H ) and their adversarial counterparts ( Ã-H̃ ) at d=300 . For clean images A-D , voting predicts correctly but on the adversarial counterparts Ã-D̃ , voting predicts wrongly . For clean images E-H and the adversarial counterparts Ẽ-H̃ , voting predicts wrongly . For completeness , we also show in Figure 9 in Appendix B examples of adversarial images where the transformation defense , coupled with voting , has successfully recovered the correct class . With the random image transformation , there are similarities in the distribution shapes between the clean and adversarial images , as shown by the groupings and arrows ( eg . between E and Ã , Ẽ , F̃ ) . This further supports our earlier observations . After the image transformation , the voting accuracy on the clean images deteriorates , but the resulting distributions have similar features as the distributions from the adversarial counterparts . This gives us an idea to enhance existing transformation-based defenses : to train a distribution classifier on the distributions obtained from clean images only , while improving the performance on both clean and adversarial images .

Summary: This paper presents a novel defense method to make the classification more robust. The motivation is based on the observation: the distribution of the soft-max for the cleaned image and its transformed images for one class is similar to the distribution of the soft-max for the adversarial image and its transformed images for the same class, and the distributions of the soft-max for the cleaned image and its transformed images for different classes are different. Then, a distribution based method is proposed to classify the distribution of the soft-max for the cleaned (or adversarial) image and its transformed images.