On the Sensitivity of Adversarial Robustness to Input Data Distributions

1 INTRODUCTION . Neural networks have been demonstrated to be vulnerable to adversarial examples ( Szegedy et al. , 2013 ; Biggio et al. , 2013 ) . Since the first discovery of adversarial examples , great progress has been made in constructing stronger adversarial attacks ( Goodfellow et al. , 2014 ; Moosavi-Dezfooli et al. , 2016 ; Madry et al. , 2017 ; Carlini and Wagner , 2017 ) . In contrast , defenses fell behind in the arms race ( Carlini and Wagner , 2016 ; Athalye et al. , 2017 ; 2018 ) . Recently a line of works have been focusing on understanding the difficulty in achieving adversarial robustness from the perspective of data distribution . In particular , Tsipras et al . ( 2019 ) demonstrated the inevitable tradeoff between robustness and clean accuracy in some particular examples . Schmidt et al . ( 2018 ) showed that the sample complexity of “ learning to be robust ” learning could be significantly higher than that of “ learning to be accurate ” . In this paper , we contribute to this growing literature from a new angle , by studying the relationship between adversarial robustness and the input data distribution . We focus on the adversarial training method , arguably the most popular defense method so far due to its simplicity , effectiveness and scalability ( Goodfellow et al. , 2014 ; Huang et al. , 2015 ; Kurakin et al. , 2016 ; Madry et al. , 2017 ; Erraqabi et al. , 2018 ) . Our main contribution is the finding that adversarial robustness is highly sensitive to the input data distribution : A semantically-lossless shift on the data distribution could result in a drastically different robustness for adversarially trained models .. Note that this is different from the transferability of a fixed model that is trained on one data distribution but tested on another distribution . Even retraining the model on the new data distribution may give us a completely different adversarial robustness on the same new distribution . This is also in sharp contrast to the clean accuracy of standard training , which , as we show in later sections , is insensitive to such shifts . To our best knowledge , our paper is the first work in the literature that demonstrates such sensitivity . Our investigation is motivated by the empirical observations on the MNIST dataset and the CIFAR10 dataset . In particular , while comparable SOTA clean accuracies ( the difference is less than 3 % ) are achieved by MNIST and CIFAR10 ( Gastaldi , 2017 ) , CIFAR10 suffers from much lower achievable robustness than MNIST in practice.1 Results of this paper consist of two parts . First in theory , we start with analyzing the difference between the regular Bayes error and the robust error , and show that the regular Bayes error is invariant to invertible transformations of the data distribution , but the robust error is not . We further prove that if the input data is uniformly distributed , then the perfect decision boundary can not be robust . However , we also manage to find a robust model for the binarized MNIST dataset ( semantically almost identical to MNIST , later described in Section 3 ) . The certification method by Wong and Kolter ( 2018 ) guarantees that this model achieves at most 3 % robust error . Such a sharp contrast suggests the important role of the data distribution in adversarial robustness , and leads to our second contribution on the empirical side : we design a series of augmented MNIST and CIFAR10 datasets to demonstrate the sensitivity of adversarial robustness to the input data distribution . Our finding of such sensitivity raises the question of how to properly evaluate adversarial robustness . In particular , the sensitivity of adversarial robustness suggests that certain datasets may not be sufficiently representative when benchmarking different robust learning algorithms . It also raises serious concerns about the deployment of believed-to-be-robust training algorithm in a real product . In a standard development procedure , various models ( for example different network architectures ) would be prototyped and measured on the existing data . However , the sensitivity of adversarial robustness makes the truthfulness of the performance estimations questionable , as one would expect future data to be slightly shifted . We illustrate the practical implications in Section 4 with two practical examples : 1 ) the robust accuracy of PGD trained model is sensitive to gamma values of gamma-corrected CIFAR10 images . This indicates that image datasets collected under different light conditions may have different robustness properties ; 2 ) both as a “ harder ” version of MNIST , the fashion-MNIST ( Xiao et al. , 2017 ) and edge-fashion-MNIST ( an edge detection variant described in Section 4.2 ) exhibit completely different robustness characteristics . This demonstrates that different datasets may give completely different evaluations for the same algorithm . Finally , our finding opens up a new angle and provides novel insights to the adversarial vulnerability problem , complementing several recent works on the issue of data distributions ’ influences on robustness . Tsipras et al . ( 2019 ) hypothesize that there is an intrinsic tradeoff between clean accuracy and adversarial robustness . Our studies complement this result , showing that there are different levels of tradeoffs depending on the characteristics of input data distribution , under the same learning settings ( training algorithm , model and training set size ) . Schmidt et al . ( 2018 ) show that different data distributions could have drastically different properties of adversarially robust generalization , theoretically on Bernoulli vs mixtures of Gaussians , and empirically on standard benchmark datasets . From the sensitivity perspective , we demonstrate that being from completely different distributions ( e.g . binary vs Gaussian or MNIST vs CIFAR10 ) may not be the essential reason for having large robustness difference . Gradual semantics-preserving transformations of data distribution can also cause large changes to datasets ’ achievable robustness . We make initial attempts in Section 5 to further understand this sensitivity . We investigated perturbable volume and inter-class distance as the natural causes of the sensitivity ; model capacity and sample complexity as the natural remedies . However , the complexity of the problem has so far defied our efforts to give a definitive answer . 1.1 NOTATION AND PROBLEM SETUP . We specifically consider the image classification problem where the input data is inside a high dimensional unit cube . We denote the data distribution as a joint distribution P ( x , y ) , where x ∈ [ 0 , 1 ] d , d is the number of pixels , and y ∈ { 1 , 2 , . . . , k } is the discrete label . We assume the support of x is the whole pixel space [ 0 , 1 ] d. When x is a random noise ( or human perceptually unclassifiable image ) , one can think of P ( y |x ) being closed to uniform distribution on labels . In the standard setting , the samples ( xi , yi ) can be interpreted as xi is independently sampled from the marginal distribution P ( x ) , and then yi is sampled from P ( x |xi ) . In this paper , we discuss P ( x ) ’ s influences on adversarial robustness , given a fixed P ( y|x ) . 1After PGD adversarial training ( Madry et al. , 2017 ) , MNIST has 89.3 % accuracy under perturbation δ with ‖δ‖∞ ≤ 0.3 under the strongest PGD attack , but CIFAR10 has only 45.8 % accuracy , even under a much smaller δ with ‖δ‖∞ ≤ 8/255 . In our experiments , we only discuss the whitebox robustness , as it represents the “ intrinsic ” robustness . We use models learned by adversarially augmented training ( Madry et al. , 2017 ) ( PGD training ) , which has the SOTA whitebox robustness . We consider bounded ` ∞ attack as the attack for evaluating robustness for 2 reasons : 1 ) PGD training can defend against ` ∞ relatively well , while for other attacks , how to train a robust model is still an open question ; 2 ) in the image domain ` ∞ attack is the mostly widely researched attack . Let H denote the universal set of all the measurable functions . Given a joint distribution P ( x , y ) on the space X × Y , we define the Bayes error R∗ = infh∈H EP ( x , y ) L ( y ; h ( x ) ) = R∗ ( P ( x , y ) ) , where L is the objective function . In other words , Bayes error is the error of the best possible classifier we can have , h∗ , without restriction on the function space of classifiers . We further define ( adversarial ) robust error RR ( h ) = EP ( x , y ) max‖δ‖∞ < L ( y ; h ( x+ δ ) ) = RR ( P ( x , y ) ) . We denote RR∗ = RR ( h∗ ) to be the robust error achieved by the Bayes classifier h∗ . For simplicity , we assume our algorithm can always learn h∗ , which reduces clean accuracy to be ( 1 − Bayes error ) , and robust accuracy of the Bayes classifier to be ( 1−RR∗ ) . 2 THEORETICAL ANALYSES AND PROVABLE CASES . As mentioned in the introduction , although the SOTA clean accuracies are similar for MNIST and CIFAR10 , the robust accuracy on CIFAR10 is much more difficult to achieve , which indicates the different behaviors of the clean accuracy and robust accuracy . The first result in this section is to further confirm this indication in a simple setting , where the clean accuracy remains the same but the robust accuracy completely changes under a distribution shift . Based on results from the concentration of measure literature , we further show that under uniform distribution , no algorithm can achieve good robustness , as long as they have high clean accuracy . On the other hand , we examine the performance of a verifiable defense method on binarized MNIST ( pixels values rounded to 0 and 1 ) , and the result suggests the exact opposite : provable adversarial robustness on a MNISTlike dataset is achievable . Such contrast thus suggests the important role of the data distribution in achieving adversarial robustness . 2.1 DISENTANGLE CLEAN ACCURACY AND ROBUST ACCURACY . One immediate result is that Bayes error remains the same under any distribution shift induced by an injective map T : X → X . To see that , simply note that T−1 exists and h∗ ◦ T−1 gives the same Bayes error for the shifted distribution . However , such invariance property does not hold for the robust error of the Bayes classifier . Furthermore , the following two examples show that Bayes error can have completely different behavior from its robust error . Although both examples have 0 Bayes error , they have completely different robust errors . Example 1 . Assume x is uniformly distributed in [ 0 , 1 ] d and y = 1 , for all x with x > e1 > 1/2 and y = 0 , for x > e1 ≤ 1/2 , where e1 is the one-hot vector . We use the 0-1 loss here . Note that the Bayes error decision boundaries are given by the following hyperplane : HP1 = { x ∈ [ 0 , 1 ] d : x1 = 0 } , and thus R∗ = 0 ; RR∗ = 2 , under the budget ‖δ‖∞ < . In this case , the robust error is tolerable and relatively robust measured by the fraction of points that are successfully attacked , 2 . Moreover , consider an injective map T which maps { x : x > e1 > 1/2 } to { x : x > 1 > d2 } , and { x : x > e1 ≤ 1/2 } to { x : x > 1 ≤ d2 } 2 . The Bayes error on the new distribution remains 0 , as T is invertible . In contrast , the robust error is much worse . In fact , RR∗ ≥ 1− 1 4d 2 . Remark 2.1 . Note that here the robust error of the Bayes classifier will grow to 1 as the dimensionality increases , for a fixed budget .

Summary: A nice paper that clarifies the difference between the clean accuracy (accuracy of models on non-perturbed examples) and the robust accuracy (accuracy of models on adversarially perturbed examples) and it shows that changing the marginal distribution of the input data P(x) while preserving its semantic P(y|x) fixed affects the robustness of the model. Therefore, testing the robustness of the model should be performed in a careful manner. Comprehensive experiments were performed to show that changing the distribution of the MINST (smoothing) and CIFAR (saturation) data could lead to a significant difference in robust accuracy while the clean accuracy is almost steady. In addition, a set of experiments were performed in an attempt to search for the criteria required for choosing a proper dataset for testing adversarial attack to measure the robustness.