A single gradient step finds adversarial examples on random two-layers neural networks

1 Introduction . We study the random two-layers neural network model , f : Rd ! R defined by f ( x ) = 1p k kX ` =1 a ` ( w ` · x ) , ( 1 ) where : R ! R is a fixed non-linearity , the weight vectors w ` 2 Rd are i.i.d . from a Gaussian N 0 , 1 d Id ( so they are roughly unit norm vectors ) , and the coefficients a ` 2 R are independent from the weight vectors and i.i.d . uniformly distributed in { 1 , +1 } . With this parametrization , the central limit theorem says that , for x 2 p d · Sd 1 ( so that w ` · x ⇠ N ( 0 , 1 ) ) and large width k , the distribution of f ( x ) is approximately a centered Gaussian with variance EX⇠N ( 0,1 ) [ ( X ) 2 ] . Our goal is to study the concept of adversarial examples in this random model . We say that 2 Rd is an adversarial perturbation at x 2 Rd if k k ⌧ kxk and sign ( f ( x ) ) 6= sign ( f ( x + ) ) . In this case , we call x+ an adversarial example . Our main result is that , while |f ( x ) | = O ( 1 ) with high probability , a single gradient step on f ( i.e. , a perturbation of the form = ⌘rf ( x ) for some ⌘ 2 R ) suffices to find such adversarial examples , with roughly k k ' kxkp d = 1 . Note , here that gradients are taken with respect to the input to the network as opposed to the weights of the network . We prove this statement for networks of subexponential width ( e.g. , k ⌧ exp ( o ( d ) ) ) with both smooth and ReLU activation functions . We first state our result for smooth activation functions in the following theorem . Theorem 1 . Let 2 ( 0 , 1 ) and be non-constant , Lipschitz and with Lipschitz derivative . There exists constants C1 , C2 , C3 , C4 depending on such that the following holds true . Assume k ⇤Canada CIFAR AI Chair 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . C1 log . 3 ( 1/ ) and d C2 log ( k/ ) log ( 1/ ) , and let ⌘ 2 R such that |⌘| = C3 p log ( 1/ ) krf ( x ) k2 and sign ( ⌘ ) = sign ( f ( x ) ) . Then , with probability at least 1 , one has : sign ( f ( x ) ) 6= sign ( f ( x+ ⌘rf ( x ) ) ) . Moreover we have k⌘rf ( x ) k C4 p log ( 1/ ) . Note that our proof of Theorem 1 in Section 2 easily gives explicit values for C1 , C2 , C3 , C4 . Also note that by re-arranging the constraint on d in Theorem 1 and setting = 1/poly ( d ) , the subexponential width condition is of the form k ⌧ exp ( o ( d ) ) . Our second main result establishes similar behavior for the non-smooth ReLU activation unit . Theorem 2 . Let 2 ( 0 , 1 ) and ( t ) = max ( 0 , t ) . There exist constants C1 , C2 , C3 , C4 , C5 such that the following holds true . Assume C1 log 6 ( d ) log ( 1/ ) k , C2 log3 ( d ) log ( 1/ ) d , C3 log4 ( k ) log ( 1/ ) d log ( d ) and let ⌘ 2 R such that |⌘| = C4 p log ( 1/ ) krf ( x ) k2 and sign ( ⌘ ) = sign ( f ( x ) ) . Then , with probability at least 1 , one has : sign ( f ( x ) ) 6= sign ( f ( x+ ⌘rf ( x ) ) ) . Moreover , we have k⌘rf ( x ) k C5 p log ( 1/ ) . As before , note that the subexponential condition on the width in the above Theorem is of the form k ⌧ exp ( d0.24 ) . In fact by modifying a bit the proof we can get a condition of the form k ⌧ exp ( d⇢ ) for any ⇢ < 1/2 , but for the sake of clarity we only prove the weaker version stated above . The proof for the ReLU activation is broken into two separate cases focusing on the overlapping regimes : Case 1 : k & d log2 ( d ) and Case 2 : log6 ( d ) log ( 1/ ) . k . d log3 d , as the proofs for the two settings use distinct arguments . The proof for the first regime is similar to the proof for Theorem 1 while the second uses a refinement of an argument by Daniely and Schacham [ 2020 ] . These arguments are carried out in Section 3 . We would like to note that our results and corresponding proofs are extendible to the setting where a ` are drawn from a normal distribution with minor modifications . We include the simpler setting with Bernoulli activations in the last layer for the sake of conceptual clarity . Intuitively , the strong concentration ( and anti-concentration ) properties of the normal distribution coupled with a conditioning argument on the magnitudes of the weights in the final layer yield similar results for gaussian setting as well . Additionally , due to the scale invariance of the ReLU activation function , our results also hold true for any distribution over x as long as P { x = 0 } = 0 and for smooth activations when P n kxk = ⇥ ( p d ) o = 1 . Finally , our results also hold when a bias unit is introduced ; in this setting , we simply project our perturbation onto the ( ( d 1 ) -dimensional ) subspace of perturbations which leave the bias unit unchanged . 1.1 Related works . The existence of adversarial examples in neural network architectures was first evidenced in the seminal paper of Szegedy et al . [ 2014 ] , where the authors found adversarial examples by using the L-BFGS optimization procedure . Shortly after this work , it was hypothesized in Goodfellow et al . [ 2015 ] that the existence of adversarial examples stems from an excessive “ linearity ” of neural network models . This hypothesis was experimentally confirmed by showing that a single step of gradient descent suffices to find adversarial perturbations ( the so-called fast gradient sign method -FGSM ) . Our theorems can be thought of as a theoretical confirmation of the hypothesis in Goodfellow et al . [ 2015 ] . In fact , as explained in Section 1.2 , our proofs proceed exactly by showing that “ most ” two-layers neural networks behave “ mostly ” linearly over “ vast ” regions of input space . We note that not all networks are susceptible to one-step gradient attacks to find adversarial examples . Indeed , in Goodfellow et al . [ 2015 ] , it was shown that adversarial training can be used to build networks that are somewhat robust to one-step gradient attacks . Interestingly , Madry et al . [ 2018 ] showed that such models remain susceptible to multi-steps gradient attacks , and empirically demonstrated that better robustness can be achieved with adversarial training using multi-steps gradient attacks . Understanding this phenomenon theoretically remains a challenge , see for example AllenZhu and Li [ 2020 ] for a proposed approach , and Moosavi-Dezfooli et al . [ 2019 ] , Qin et al . [ 2019 ] for discussion/algorithmic consequences of the relation with the phenomenon of gradient obfuscation ( Papernot et al . [ 2017 ] , Athalye et al . [ 2018 ] ) . Our work is a direct follow-up of Daniely and Schacham [ 2020 ] ( which itself follows Shamir et al . [ 2019 ] ) . Daniely and Schacham prove that multi-steps gradient descent finds adversarial examples for ReLU random networks of the form ( 1 ) , as long as the number of neurons is much smaller than the dimension ( i.e. , k = o ( d ) ) . They explicitly conjecture that this condition is not necessary , and indeed we exponentially improve it in Theorem 2 ( see below for a discussion of k exponential in the dimension ) . We note that Daniely and Schacham went beyond two-layers neural networks , and conjectured ( and proved for shrinking layers ) that gradient descent finds adversarial examples on random multi-layers neural networks . We give some experimental confirmation of this multi-layer conjecture in Section 4 . The ultra-wide case k = exp ( ⌦ ( d ) ) remains open . This exponential size case seems of a different nature than the polynomial size we tackle here , at least for the ReLU activation function . In particular , it is likely that the behavior with exponential width would be closely tied to the actual limit case k = +1 , where the random model ( 1 ) yields a Gaussian process . Namely , for k = +1 , f is a Gaussian process indexed by the sphere ( say if we restrict to inputs x 2 p d · Sd 1 ) , with f ( x ) ⇠ N ( 0 , EX⇠N ( 0,1 ) [ ( X ) ] ) and E [ f ( x ) f ( y ) ] = EX , Y⇠N ( 0,1 ) : E [ XY ] =x·y [ ( X ) ( Y ) ] . For example if the activation function is a Hermite polynomial of degree p , then f would be a spherical p-spin glass model . This polynomial case is particularly well-understood , and in fact the landscape we describe below in Section 1.2 was already described in this case by Ben Arous et al . [ 2020 ] ( in particular Corollary 59 ) indicating that adversarial examples would likely continue to persist in the ultra-wide setting . It would be interesting to see if the p-spin glass landscape literature can be extended to non-polynomial activation functions , and to a finite ( but possibly exponential in d ) k. A step in this latter direction was recently taken in Eldan et al . [ 2021 ] , where convergence rates to the Gaussian process limit where given both for polynomial activations and for the ReLU . Finally we note that for a smooth activation it might be that there is a more direct argument to remove the subexponential width condition in Theorem 1 ( in technical terms , the proof of Proposition 2 could leverage a better argument than our naive upper bound on Lip ( ) ) . Finally , we note that , in practice , it has been found that there exists “ universal ” adversarial perturbations that generalize across both inputs and neural networks , Moosavi-Dezfooli et al . [ 2017 ] . For the case of ReLU activation ( Theorem 2 ) , we could in fact prove our result by replacing the gradient step with a step in the direction P k ` =1 a ` w ` , which is indeed a direction independent of the input x , thus proving the existence of “ universal ” perturbations ( generalizing across inputs ) for our model .

Summary: The paper studies adversarial examples in random depth-2 ReLU networks.  They consider depth-2 networks, where the first layer is drawn from a normal distribution such that the norms of the weights vectors are roughly 1, and the second layer is drawn from the uniform distribution on {-1/\sqrt{k},1/\sqrt{k}} where k is the width of the network. They assume that k is at most sub-exponential in the input dimension, and is bounded below by some polylogarithmic expression. For an input x on the sphere of radius \sqrt{d} the output of the network is w.h.p. O(1) (the input x is fixed, and the probability is over the choice of the network). The authors show that there is w.h.p. \delta with ||\delta||=O(1) such that sign(f(x)) \neq sign(f(x+\delta)). Moreover, such \delta can be found with a single gradient step. Thus, although the inputs are of size \sqrt{d}, there is w.h.p. a perturbation of size O(1) that changes the sign of the output. They prove the result for both smooth and ReLU activations. Finally, they validate the results empirically, and also demonstrate that ש similar phenomenon occurs in deeper networks.  Daniely and Schacham [2020] showed that such a property holds in ReLU networks where the sizes d_1,…,d_t  of the layers (where d_1 is the input dimension) are such that for every j we have d_{j+1} = o(d_j) and d_t = \omega(1). For the special case of depth-2 networks it required width k=o(d). Thus, the improvement here w.r.t. k is exponential.  The proofs roughly follow the following steps. First, they show an O(1) upper bound on f(x) and an \Omega(1) lower bound on ||\nabla f(x)||. Then, they bound the variation of the gradient near x. Finally, they use standard arguments to bound f(x-\eta \nabla f(x)) for an appropriate \eta. The proofs of these steps for the case of the ReLU activation is different from the case of smooth activations, and the proofs for the ReLU activation with small k is different from the case of large k.