The Limitations of Adversarial Training and the Blind-Spot Attack

1 INTRODUCTION . Since the discovery of adversarial examples in deep neural networks ( DNNs ) ( Szegedy et al. , 2013 ) , adversarial training under the robustness optimization framework ( Madry et al. , 2018 ; Sinha et al. , 2018 ) has become one of the most effective methods to defend against adversarial examples . A recent study by Athalye et al . ( 2018 ) showed that adversarial training does not rely on obfuscated gradients and delivers promising results for defending adversarial examples on small datasets . Adversarial training approximately solves the following min-max optimization problem : min θ E ( x , y ) ∈X [ max δ∈S L ( x+ δ ; y ; θ ) ] , ( 1 ) where X is the set of training data , L is the loss function , θ is the parameter of the network , and S is usually a norm constrained ` p ball centered at 0 . Madry et al . ( 2018 ) propose to use projected gradient descent ( PGD ) to approximately solve the maximization problem within S = { δ | ‖δ‖∞ ≤ } , where = 0.3 for MNIST dataset on a 0-1 pixel scale , and = 8 for CIFAR-10 dataset on a 0-255 pixel scale . This approach achieves impressive defending results on the MNIST test set : so far the best available white-box attacks by Zheng et al . ( 2018 ) can only decrease the test accuracy from approximately 98 % to 88 % 1 . However , on CIFAR-10 dataset , a simple 20-step PGD can decrease the test accuracy from 87 % to less than 50 % 2 . 1 https : //github.com/MadryLab/mnist_challenge 2 https : //github.com/MadryLab/cifar10_challenge The effectiveness of adversarial training is measured by the robustness on the test set . However , the adversarial training process itself is done on the training set . Suppose we can optimize ( 1 ) perfectly , then certified robustness may be obtained on those training data points . However , if the empirical distribution of training dataset differs from the true data distribution , a test point drawn from the true data distribution might lie in a low probability region in the empirical distribution of training dataset and is not “ covered ” by the adversarial training procedure . For datasets that are relatively simple and have low intrinsic dimensions ( MNIST , Fashion MNIST , etc ) , we can obtain enough training examples to make sure adversarial training covers most part of the data distribution . For high dimensional datasets ( CIFAR , ImageNet ) , adversarial training have been shown difficult ( Kurakin et al. , 2016 ; Tramèr et al. , 2018 ) and only limited success was obtained . A recent attack proposed by Song et al . ( 2018 ) shows that adversarial training can be defeated when the input image is produced by a generative model ( for example , a generative adversarial network ) rather than selected directly from the test examples . The generated images are well recognized by humans and thus valid images in the ground-truth data distribution . In our interpretation , this attack effective finds the “ blind-spots ” in the input space that the training data do not well cover . For higher dimensional datasets , we hypothesize that many test images already fall into these blindspots of training data and thus adversarial training only obtains a moderate level of robustness . It is interesting to see that for those test images that adversarial training fails to defend , if their distances ( in some metrics ) to the training dataset are indeed larger . In our paper , we try to explain the success of robust optimization based adversarial training and show the limitations of this approach when the test points are slightly off the empirical distribution of training data . Our main contributions are : • We show that on the original set of test images , the effectiveness of adversarial training is highly correlated with the distance ( in some distance metrics ) from the test image to the manifold of training images . For MNIST and Fashion MNIST datasets , most test images are close to the training data and very good robustness is observed on these points . For CIFAR , there is a clear trend that the adversarially trained network gradually loses its robustness property when the test images are further away from training data . • We identify a new class of attacks , “ blind-spot attacks ” , where the input image resides in a “ blind-spot ” of the empirical distribution of training data ( far enough from any training examples in some embedding space ) but is still in the ground-truth data distribution ( well recognized by humans and correctly classified by the model ) . Adversarial training can not provide good robustness on these blind-spots and their adversarial examples have small distortions . • We show that blind-spots can be easily found on a few strong defense models including Madry et al . ( 2018 ) , Kolter & Wong ( 2018 ) and Sinha et al . ( 2018 ) . We propose a few simple transformations ( slightly changing contrast and background ) , that do not noticeably affect the accuracy of adversarially trained MNIST and Fashion MNIST models , but these models become vulnerable to adversarial attacks on these sets of transformed input images . These transformations effectively move the test images slightly out of the manifold of training images , which does not affect generalization but poses a challenge for robust learning . Our results imply that current adversarial training procedures can not scale to datasets with a large ( intrinsic ) dimension , where any practical amount of training data can not cover all the blind-spots . This explains the limited success for applying adversarial training on ImageNet dataset , where many test images can be sufficiently far away from the empirical distribution of training dataset . 2 RELATED WORKS . 2.1 DEFENDING AGAINST ADVERSARIAL EXAMPLES . Adversarial examples in DNNs have brought great threats to the deep learning-based AI applications such as autonomous driving and face recognition . Therefore , defending against adversarial examples is an urgent task before we can safely deploy deep learning models to a wider range of applications . Following the emergence of adversarial examples , various defense methods have been proposed , such as defensive distillation by Papernot et al . ( 2016 ) and feature squeezing by Xu et al . ( 2017 ) . Some of these defense methods have been proven vulnerable or ineffective under strong attack methods such as C & W in Carlini & Wagner ( 2017 ) . Another category of recent defense methods is based on gradient masking or obfuscated gradient ( Buckman et al . ( 2018 ) ; Ma et al . ( 2018 ) ; Guo et al . ( 2017 ) ; Song et al . ( 2017 ) ; Samangouei et al . ( 2018 ) ) , but these methods are also successfully evaded by the stronger BPDA attack ( Athalye et al . ( 2018 ) ) . Randomization in DNNs ( Dhillon et al. , 2018 ; Xie et al. , 2017 ; Liu et al. , 2018 ) is also used to reduce the success rate of adversarial attacks , however , it usually incurs additional computational costs and still can not fully defend against an adaptive attacker ( Athalye et al. , 2018 ; Athalye & Sutskever , 2017 ) . An effective defense method is adversarial training , which trains the model with adversarial examples freshly generated during the entire training process . First introduced by Goodfellow et al. , adversarial training demonstrates the state-of-the-art defending performance . Madry et al . ( 2018 ) formulated the adversarial training procedure into a min-max robust optimization problem and has achieved state-of-the-art defending performance on MNIST and CIFAR datasets . Several attacks have been proposed to attack the model release by Madry et al . ( 2018 ) . On the MNIST testset , so far the best attack by Zheng et al . ( 2018 ) can only reduce the test accuracy from 98 % to 88 % . Analysis by Athalye et al . ( 2018 ) shows that this adversarial training framework does not rely on obfuscated gradient and truly increases model robustness ; gradient based attacks with random starts can only achieve less than 10 % success rate with given distortion constraints and are unable to penetrate this defense . On the other hand , attacking adversarial training using generative models have also been investigated ; both Xiao et al . ( 2018 ) and Song et al . ( 2018 ) propose to use GANs to produce adversarial examples in black-box and white-box settings , respectively . Finally , a few certified defense methods ( Raghunathan et al. , 2018 ; Sinha et al. , 2018 ; Kolter & Wong , 2018 ) were proposed , which are able to provably increase model robustness . Besides adversarial training , in our paper we also consider several certified defenses which can achieve relatively good performance ( i.e. , test accuracy on natural images does not drop significantly and training is computationally feasible ) , and can be applied to medium-sized networks with multiple layers . Notably , Sinha et al . ( 2018 ) analyzes adversarial training using distributional robust optimization techniques . Kolter & Wong ( 2018 ) and Wong et al . ( 2018 ) proposed a robustness certificate based on the dual of a convex relaxation for ReLU networks , and used it for training to provably increase robustness . During training , certified defense methods can provably guarantee that the model is robust on training examples ; however , on unseen test examples a non-vacuous robustness generalization guarantee is hard to obtain . 2.2 ANALYZING ADVERSARIAL EXAMPLES . Along with the attack-defense arms race , some insightful findings have been discovered to understand the natural of adversarial examples , both theoretically and experimentally . Schmidt et al . ( 2018 ) show that even for a simple data distribution of two class-conditional Gaussians , robust generalization requires significantly larger number of samples than standard generalization . Cullina et al . ( 2018 ) extend the well-known PAC learning theory to the case with adversaries , and derive the adversarial VC-dimension which can be either larger or smaller than the standard VC-dimension . Bubeck et al . ( 2018b ) conjecture that a robust classifier can be computationally intractable to find , and give a proof for the computation hardness under statistical query ( SQ ) model . Recently , Bubeck et al . ( 2018a ) prove a computational hardness result under a standard cryptographic assumption . Additionally , finding the safe area approximately is computationally hard according to Katz et al . ( 2017 ) and Weng et al . ( 2018 ) . Mahloujifar et al . ( 2018 ) explain the prevalence of adversarial examples by making a connection to the “ concentration of measure ” phenomenon in metric measure spaces . Su et al . ( 2018 ) conduct large scale experiments on ImageNet and find a negative correlation between robustness and accuracy . Tsipras et al . ( 2019 ) discover that data examples consist of robust and non-robust features and adversarial training tends to find robust features that have strongly-correlations with the labels . Both adversarial training and certified defenses significantly improve robustness on training data , but it is still unknown if the trained model has good robust generalization property . Typically , we evaluate the robustness of a model by computing an upper bound of error on the test set ; specifically , given a norm bounded distortion , we verify if each image in test set has a robustness certificate ( Zhang et al. , 2018 ; Dvijotham et al. , 2018 ; Singh et al. , 2018 ) . There might exist test images that are still within the capability of standard generalization ( i.e. , correctly classified by DNNs with high confidence , and well recognized by humans ) , but behaves badly in robust generalization ( i.e. , adversarial examples can be easily found with small distortions ) . Our paper complements those existing findings by showing the strong correlation between the effectiveness of adversarial defenses ( both adversarial training and some certified defenses ) and the distance between training data and test points . Additionally , we show that a tiny shift in input distribution ( which may or may not be detectable in embedding space ) can easily destroy the robustness property of an robust model .

Summary: This paper provides some insights on influence of data distribution on robustness of adversarial training. The paper demonstrates through a number of analysis that the distance between the training an test data sets plays an important role on the effectiveness of adversarial training. To show the latter, the paper proposes an approach to measure the distance between the two data sets using combination of nonlinear projection (e.g. t-SNE), KDE, and K-L divergence. The paper also shows that under simple transformation to the test dataset (e.g. scaling), performance of adversarial training reduces significantly due to the large gap between training and test data set. This tends to impact high dimensional data sets more than low dimensional data sets since it is much harder to cover the whole ground truth data distribution in the training dataset.