Are adversarial examples inevitable?

A wide range of defenses have been proposed to harden neural networks against adversarial attacks . However , a pattern has emerged in which the majority of adversarial defenses are quickly broken by new attacks . Given the lack of success at generating robust defenses , we are led to ask a fundamental question : Are adversarial attacks inevitable ? This paper analyzes adversarial examples from a theoretical perspective , and identifies fundamental bounds on the susceptibility of a classifier to adversarial attacks . We show that , for certain classes of problems , adversarial examples are inescapable . Using experiments , we explore the implications of theoretical guarantees for real-world problems and discuss how factors such as dimensionality and image complexity limit a classifier ’ s robustness against adversarial examples . 1 INTRODUCTION . A number of adversarial attacks on neural networks have been recently proposed . To counter these attacks , a number of authors have proposed a range of defenses . However , these defenses are often quickly broken by new and revised attacks . Given the lack of success at generating robust defenses , we are led to ask a fundamental question : Are adversarial attacks inevitable ? In this paper , we identify a broad class of problems for which adversarial examples can not be avoided . We also derive fundamental limits on the susceptibility of a classifier to adversarial attacks that depend on properties of the data distribution as well as the dimensionality of the dataset . Adversarial examples occur when a small perturbation to an image changes its class label . There are different ways of measuring what it means for a perturbation to be “ small ” ; as such , our analysis considers a range of different norms . While the ` ∞-norm is commonly used , adversarial examples can be crafted in any ` p-norm ( see Figure 1 ) . We will see that the choice of norm can have a dramatic effect on the strength of theoretical guarantees for the existence of adversarial examples . Our analysis also extends to the ` 0-norm , which yields “ sparse ” adversarial examples that only perturb a small subset of image pixels ( Figure 2 ) . As a simple example result , consider a classification problem with n-dimensional images with pixels scaled between 0 and 1 ( in this case images live inside the unit hypercube ) . If the image classes each occupy a fraction of the cube greater than 12 exp ( −π 2 ) , then images exist that are susceptible to adversarial perturbations of ` 2-norm at most . Note that = 10 was used in Figure 1 , and larger values are typical for larger images . Finally , in Section 8 , we explore the causes of adversarial susceptibility in real datasets , and the effect of dimensionality . We present an example image class for which there is no fundamental link between dimensionality and robustness , and argue that the data distribution , and not dimensionality , is the primary cause of adversarial susceptibility . 1.1 BACKGROUND : A BRIEF HISTORY OF ADVERSARIAL EXAMPLES . Adversarial examples , first demonstrated in Szegedy et al . ( 2013 ) and Biggio et al . ( 2013 ) , change the label of an image using small and often imperceptible perturbations to its pixels . A number of defenses have been proposed to harden networks against attacks , but historically , these defenses have been quickly broken . Adversarial training , one of the earliest defenses , successfully thwarted the fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2014 ) , one of the earliest and simplest attacks . However , adversarial training with FGSM examples was quickly shown to be vulnerable to more sophisticated multi-stage attacks ( Kurakin et al. , 2016 ; Tramèr et al. , 2017a ) . More sophisticated defenses that rely on network distillation ( Papernot et al. , 2016b ) and specialized activation functions ( Zantedeschi et al. , 2017 ) were also toppled by strong attacks ( Papernot et al. , 2016a ; Tramèr et al. , 2017b ; Carlini & Wagner , 2016 ; 2017a ) . The ongoing vulnerability of classifiers was highlighted in recent work by Athalye et al . ( 2018 ) and Athalye & Sutskever ( 2017 ) that broke an entire suite of defenses presented in ICLR 2018 including thermometer encoding ( Buckman et al. , 2018 ) , detection using local intrinsic dimensionality ( Ma et al. , 2018 ) , input transformations such as compression and image quilting ( Guo et al. , 2017 ) , stochastic activation pruning ( Dhillon et al. , 2018 ) , adding randomization at inference time ( Xie et al. , 2017 ) , enhancing the confidence of image labels ( Song et al. , 2017 ) , and using a generative model as a defense ( Samangouei et al. , 2018 ) . Rather than hardening classifiers to attacks , some authors have proposed sanitizing datasets to remove adversarial perturbations before classification . Approaches based on auto-encoders ( Meng & Chen , 2017 ) and GANs ( Shen et al. , 2017 ) were broken using optimization-based attacks ( Carlini & Wagner , 2017b ; a ) . A number of “ certifiable ” defense mechanisms have been developed for certain classifiers . Raghunathan et al . ( 2018 ) harden a two-layer classifier using semidefinite programming , and Sinha et al . ( 2018 ) propose a convex duality-based approach to adversarial training that works on sufficiently small adversarial perturbations with a quadratic adversarial loss . Kolter & Wong ( 2017 ) consider training a robust classifier using the convex outer adversarial polytope . All of these methods only consider robustness of the classifier on the training set , and robustness properties often fail to generalize reliably to test examples . One place where researchers have enjoyed success is at training classifiers on low-dimensional datasets like MNIST ( Madry et al. , 2017 ; Sinha et al. , 2018 ) . The robustness achieved on more complicated datasets such as CIFAR-10 and ImageNet are nowhere near that of MNIST , which leads some researchers to speculate that adversarial defense is fundamentally harder in higher dimensions – an issue we address in Section 8 . This paper uses well-known results from high-dimensional geometry , specifically isoperimetric inequalities , to provide bounds on the robustness of classifiers . Several other authors have investigated adversarial susceptibility through the lens of geometry . Fawzi et al . ( 2018 ) study adversarial susceptibility of datasets under the assumption that they are produced by a generative model that maps random Gaussian vectors onto images . Gilmer et al . ( 2018 ) do a detailed case study , including empirical and theoretical results , of classifiers for a synthetic dataset that lies on two concentric spheres . Simon-Gabriel et al . ( 2018 ) show that the Lipschitz constant of untrained networks with random weights gets large in high dimensions . Shortly after the original appearance of our work , Mahloujifar et al . ( 2018 ) presented a study of adversarial susceptibility that included both evasion and poisoning attacks . Our work is distinct in that it studies adversarial robustness for arbitrary data distributions , and also that it rigorously looks at the effect of dimensionality on robustness limits . 1.2 NOTATION . We use [ 0 , 1 ] n to denote the unit hypercube in n dimensions , and vol ( A ) to denote the volume ( i.e. , ndimensional Lebesgue measure ) of a subset A ⊂ [ 0 , 1 ] n. We use Sn−1 = { x ∈ Rn| ‖x‖2 = 1 } to denote the unit sphere embedded in Rn , and sn−1 to denote its surface area . The size of a subset A ∈ Sn−1 can be quantified by its ( n − 1 dimensional ) measure µ [ A ] , which is simply the surface area the set covers . Because the surface area of the unit sphere varies greatly with n , it is much easier in practice to work with the normalized measure , which we denote µ1 [ A ] = µ [ A ] /sn−1 . This normalized measure has the property that µ1 [ Sn−1 ] = 1 , and so we can interpret µ1 [ A ] as the probability of a uniform random point from the sphere lying in A . When working with points on a sphere , we often use geodesic distance , which is always somewhat larger than ( but comparable to ) the Euclidean distance . In the cube , we measure distance between points using ` p-norms , which are denoted ‖z‖p = ( ∑ i |zi|p ) 1/p if p > 0 , and ‖z‖0 = card { zi | zi 6= 0 } . Note that ‖ · ‖p is not truly a norm for p < 1 , but rather a semi-norm . Such metrics are still commonly used , particularly the “ ` 0-norm ” which counts the number of non-zero entries in a vector . 2 PROBLEM SETUP . We consider the problem of classifying data points that lie in a space Ω ( either a sphere or a hypercube ) into m different object classes . The m object classes are defined by probability density functions { ρc } mc=1 , where ρc : Ω → R. A “ random ” point from class c is a random variable with density ρc . We assume ρc to be bounded ( i.e. , we don ’ t allow delta functions or other generalized functions ) , and denote its upper bound by Uc = supx ρc ( x ) . We also consider a “ classifier ” function C : Ω → { 1 , 2 , . . . , m } that partitions Ω into disjoint measurable subsets , one for each class label . The classifier we consider is discrete valued – it provides a label for each data point but not a confidence level . With this setup , we can give a formal definition of an adversarial example . Definition 1 . Consider a point x ∈ Ω drawn from class c , a scalar > 0 , and a metric d. We say that x admits an -adversarial example in the metric d if there exists a point x̂ ∈ Ω with C ( x̂ ) 6= c , and d ( x , x̂ ) ≤ . In plain words , a point has an -adversarial example if we can sneak it into a different class by moving it at most units in the distance d. We consider adversarial examples with respect to different ` p-norm metrics . These metrics are written dp ( x , x̂ ) = ‖x − x̂‖p . A common choice is p = ∞ , which limits the absolute change that can be made to any one pixel . However , ` 2-norm and ` 1-norm adversarial examples are also used , as it is frequently easier to create adversarial examples in these less restrictive metrics . We also consider sparse adversarial examples in which only a small subset of pixels are manipulated . This corresponds to the metric d0 , in which case the constraint ‖x − x̂‖0 ≤ means that an adversarial example was crafted by changing at most pixels , and leaving the others alone . 3 THE SIMPLE CASE : ADVERSARIAL EXAMPLES ON THE UNIT SPHERE . We begin by looking at the case of classifiers for data on the sphere . While this data model may be less relevant than the other models studied below , it provides a straightforward case where results can be proven using simple , geometric lemmas . The more realistic case of images with pixels in [ 0 , 1 ] will be studied in Section 4 . The idea is to show that , provided a class of data points takes up enough space , nearly every point in the class lies close to the class boundary . To show this , we begin with a simple definition . Definition 2 . The -expansion of a subsetA ⊂ Ω with respect to distance metric d , denotedA ( , d ) , contains all points that are at most units away from A . To be precise A ( , d ) = { x ∈ Ω | d ( x , y ) ≤ for some y ∈ A } . We sometimes simply write A ( ) when the distance metric is clear from context . Our result provides bounds on the probability of adversarial examples that are independent of the shape of the class boundary . This independence is a simple consequence of an isoperimetric inequality . The classical isoperimetric inequality states that , of all closed surfaces that enclose a unit volume , the sphere has the smallest surface area . This simple fact is intuitive but famously difficult to prove . For a historical review of the isoperimetric inequality and its variants , see Osserman et al . ( 1978 ) . We will use a special variant of the isoperimetric inequality first proved by Lévy & Pellegrino ( 1951 ) and simplified by Talagrand ( 1995 ) . Lemma 1 ( Isoperimetric inequality ) . Consider a subset of the sphere A ⊂ Sn−1 ⊂ Rn with normalized measure µ1 ( A ) ≥ 1/2 . When using the geodesic metric , the -expansion A ( ) is at least as large as the -expansion of a half sphere . The classical isoperimetric inequality is a simple geometric statement , and frequently appears without absolute bounds on the size of the -expansion of a half-sphere , or with bounds that involve unspecified constants ( Vershynin , 2017 ) . A tight bound derived by Milman & Schechtman ( 1986 ) is given below . The asymptotic blow-up of the -expansion of a half sphere predicted by this bound is shown in Figure 3 . Lemma 2 ( -expansion of half sphere ) . The geodesic -expansion of a half sphere has normalized measure at least 1− ( π 8 ) 1 2 exp ( −n− 1 2 2 ) . Lemmas 1 and 2 together can be taken to mean that , if a set is not too small , then in high dimensions almost all points on the sphere are reachable within a short jump from that set . These lemmas have immediate implications for adversarial examples , which are formed by mapping one class into another using small perturbations . Despite its complex appearance , the result below is a consequence of the ( relatively simple ) isoperimetric inequality . Theorem 1 ( Existence of Adversarial Examples ) . Consider a classification problem with m object classes , each distributed over the unit sphere Sn−1 ⊂ Rn with density functions { ρc } mc=1 . Choose a classifier function C : Sn−1 → { 1 , 2 , . . . , m } that partitions the sphere into disjoint measurable subsets . Define the following scalar constants : • Let Vc denote the magnitude of the supremum of ρc relative to the uniform density . This can be written Vc : = sn−1 · supx ρc ( x ) . • Let fc = µ1 { x|C ( x ) = c } be the fraction of the sphere labeled as c by classifier C. Choose some class c with fc ≤ 12 . Sample a random data point x from ρc . Then with probability at least 1− Vc ( π 8 ) 1 2 exp ( −n− 1 2 2 ) ( 1 ) one of the following conditions holds : 1. x is misclassified by C , or 2. x admits an -adversarial example in the geodesic distance . Proof . Choose a class c with fc ≤ 12 . LetR = { x|C ( x ) = c } denote the region of the sphere labeled as class c by C , and letR be its complement . R ( ) is the -expansion ofR in the geodesic metric . BecauseR covers at least half the sphere , the isoperimetric inequality ( Lemma 1 ) tells us that the epsilon expansion is at least as great as the epsilon expansion of a half sphere . We thus have µ1 [ R ( ) ] ≥ 1− ( π 8 ) 1 2 exp ( −n− 1 2 2 ) . Now , consider the set Sc of “ safe ” points from class c that are correctly classified and do not admit adversarial perturbations . A point is correctly classified only if it lies inside R , and therefore outside of R. To be safe from adversarial perturbations , a point can not lie within distance from the class boundary , and so it can not lie within R ( ) . It is clear that the set Sc of safe points is exactly the complement of R ( ) . This set has normalized measure µ1 [ Sc ] ≤ ( π 8 ) 1 2 exp ( −n− 1 2 2 ) . The probability of a random point lying in Sc is bounded above by the normalized supremum of ρc times the normalized measure µ1 [ Sc ] . This product is given by Vc ( π 8 ) 1 2 exp ( −n− 1 2 2 ) . We then subtract this probability from 1 to obtain the probability of a point lying outside the safe region , and arrive at equation 1 . In the above result , we measure the size of adversarial perturbations using the geodesic distance . Most studies of adversarial examples measure the size of perturbation in either the ` 2 ( Euclidean ) norm or the ` ∞ ( max ) norm , and so it is natural to wonder whether Theorem 1 depends strongly on the distance metric . Fortunately ( or , rather unfortunately ) it does not . It is easily observed that , for any two points x and y on a sphere , d∞ ( x , y ) ≤ d2 ( x , y ) ≤ dg ( x , y ) , where d∞ ( x , y ) , d2 ( x , y ) , and dg ( x , y ) denote the l∞ , Euclidean , and geodesic distance , respectively . From this , we see that Theorem 1 is actually fairly conservative ; any -adversarial example in the geodesic metric would also be adversarial in the other two metrics , and the bound in Theorem 1 holds regardless of which of the three metrics we choose ( although different values of will be appropriate depending on the norm ) .

Summary: This paper explores the inevitability of adversarial examples with concentration inequalities. It is motivated by the difficulties of achieving adversarial robustness in literature. It derives isoperimetric inequalities on a cube, and then discuss the adversarial robustness of data distributed inside the cube, with the assumption that the data has bounded density. These inequalities are established on different norms. The authors then discuss limitation of the proposed bounds when analyzing practical data distribution and discussed the influence of dimensionality on adversarial robustness.