A single gradient step finds adversarial examples on random two-layers neural networks

1 Introduction . We study the random two-layers neural network model , f : Rd ! R defined by f ( x ) = 1p k kX ` =1 a ` ( w ` · x ) , ( 1 ) where : R ! R is a fixed non-linearity , the weight vectors w ` 2 Rd are i.i.d . from a Gaussian N 0 , 1 d Id ( so they are roughly unit norm vectors ) , and the coefficients a ` 2 R are independent from the weight vectors and i.i.d . uniformly distributed in { 1 , +1 } . With this parametrization , the central limit theorem says that , for x 2 p d · Sd 1 ( so that w ` · x ⇠ N ( 0 , 1 ) ) and large width k , the distribution of f ( x ) is approximately a centered Gaussian with variance EX⇠N ( 0,1 ) [ ( X ) 2 ] . Our goal is to study the concept of adversarial examples in this random model . We say that 2 Rd is an adversarial perturbation at x 2 Rd if k k ⌧ kxk and sign ( f ( x ) ) 6= sign ( f ( x + ) ) . In this case , we call x+ an adversarial example . Our main result is that , while |f ( x ) | = O ( 1 ) with high probability , a single gradient step on f ( i.e. , a perturbation of the form = ⌘rf ( x ) for some ⌘ 2 R ) suffices to find such adversarial examples , with roughly k k ' kxkp d = 1 . Note , here that gradients are taken with respect to the input to the network as opposed to the weights of the network . We prove this statement for networks of subexponential width ( e.g. , k ⌧ exp ( o ( d ) ) ) with both smooth and ReLU activation functions . We first state our result for smooth activation functions in the following theorem . Theorem 1 . Let 2 ( 0 , 1 ) and be non-constant , Lipschitz and with Lipschitz derivative . There exists constants C1 , C2 , C3 , C4 depending on such that the following holds true . Assume k ⇤Canada CIFAR AI Chair 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . C1 log . 3 ( 1/ ) and d C2 log ( k/ ) log ( 1/ ) , and let ⌘ 2 R such that |⌘| = C3 p log ( 1/ ) krf ( x ) k2 and sign ( ⌘ ) = sign ( f ( x ) ) . Then , with probability at least 1 , one has : sign ( f ( x ) ) 6= sign ( f ( x+ ⌘rf ( x ) ) ) . Moreover we have k⌘rf ( x ) k C4 p log ( 1/ ) . Note that our proof of Theorem 1 in Section 2 easily gives explicit values for C1 , C2 , C3 , C4 . Also note that by re-arranging the constraint on d in Theorem 1 and setting = 1/poly ( d ) , the subexponential width condition is of the form k ⌧ exp ( o ( d ) ) . Our second main result establishes similar behavior for the non-smooth ReLU activation unit . Theorem 2 . Let 2 ( 0 , 1 ) and ( t ) = max ( 0 , t ) . There exist constants C1 , C2 , C3 , C4 , C5 such that the following holds true . Assume C1 log 6 ( d ) log ( 1/ ) k , C2 log3 ( d ) log ( 1/ ) d , C3 log4 ( k ) log ( 1/ ) d log ( d ) and let ⌘ 2 R such that |⌘| = C4 p log ( 1/ ) krf ( x ) k2 and sign ( ⌘ ) = sign ( f ( x ) ) . Then , with probability at least 1 , one has : sign ( f ( x ) ) 6= sign ( f ( x+ ⌘rf ( x ) ) ) . Moreover , we have k⌘rf ( x ) k C5 p log ( 1/ ) . As before , note that the subexponential condition on the width in the above Theorem is of the form k ⌧ exp ( d0.24 ) . In fact by modifying a bit the proof we can get a condition of the form k ⌧ exp ( d⇢ ) for any ⇢ < 1/2 , but for the sake of clarity we only prove the weaker version stated above . The proof for the ReLU activation is broken into two separate cases focusing on the overlapping regimes : Case 1 : k & d log2 ( d ) and Case 2 : log6 ( d ) log ( 1/ ) . k . d log3 d , as the proofs for the two settings use distinct arguments . The proof for the first regime is similar to the proof for Theorem 1 while the second uses a refinement of an argument by Daniely and Schacham [ 2020 ] . These arguments are carried out in Section 3 . We would like to note that our results and corresponding proofs are extendible to the setting where a ` are drawn from a normal distribution with minor modifications . We include the simpler setting with Bernoulli activations in the last layer for the sake of conceptual clarity . Intuitively , the strong concentration ( and anti-concentration ) properties of the normal distribution coupled with a conditioning argument on the magnitudes of the weights in the final layer yield similar results for gaussian setting as well . Additionally , due to the scale invariance of the ReLU activation function , our results also hold true for any distribution over x as long as P { x = 0 } = 0 and for smooth activations when P n kxk = ⇥ ( p d ) o = 1 . Finally , our results also hold when a bias unit is introduced ; in this setting , we simply project our perturbation onto the ( ( d 1 ) -dimensional ) subspace of perturbations which leave the bias unit unchanged . 1.1 Related works . The existence of adversarial examples in neural network architectures was first evidenced in the seminal paper of Szegedy et al . [ 2014 ] , where the authors found adversarial examples by using the L-BFGS optimization procedure . Shortly after this work , it was hypothesized in Goodfellow et al . [ 2015 ] that the existence of adversarial examples stems from an excessive “ linearity ” of neural network models . This hypothesis was experimentally confirmed by showing that a single step of gradient descent suffices to find adversarial perturbations ( the so-called fast gradient sign method -FGSM ) . Our theorems can be thought of as a theoretical confirmation of the hypothesis in Goodfellow et al . [ 2015 ] . In fact , as explained in Section 1.2 , our proofs proceed exactly by showing that “ most ” two-layers neural networks behave “ mostly ” linearly over “ vast ” regions of input space . We note that not all networks are susceptible to one-step gradient attacks to find adversarial examples . Indeed , in Goodfellow et al . [ 2015 ] , it was shown that adversarial training can be used to build networks that are somewhat robust to one-step gradient attacks . Interestingly , Madry et al . [ 2018 ] showed that such models remain susceptible to multi-steps gradient attacks , and empirically demonstrated that better robustness can be achieved with adversarial training using multi-steps gradient attacks . Understanding this phenomenon theoretically remains a challenge , see for example AllenZhu and Li [ 2020 ] for a proposed approach , and Moosavi-Dezfooli et al . [ 2019 ] , Qin et al . [ 2019 ] for discussion/algorithmic consequences of the relation with the phenomenon of gradient obfuscation ( Papernot et al . [ 2017 ] , Athalye et al . [ 2018 ] ) . Our work is a direct follow-up of Daniely and Schacham [ 2020 ] ( which itself follows Shamir et al . [ 2019 ] ) . Daniely and Schacham prove that multi-steps gradient descent finds adversarial examples for ReLU random networks of the form ( 1 ) , as long as the number of neurons is much smaller than the dimension ( i.e. , k = o ( d ) ) . They explicitly conjecture that this condition is not necessary , and indeed we exponentially improve it in Theorem 2 ( see below for a discussion of k exponential in the dimension ) . We note that Daniely and Schacham went beyond two-layers neural networks , and conjectured ( and proved for shrinking layers ) that gradient descent finds adversarial examples on random multi-layers neural networks . We give some experimental confirmation of this multi-layer conjecture in Section 4 . The ultra-wide case k = exp ( ⌦ ( d ) ) remains open . This exponential size case seems of a different nature than the polynomial size we tackle here , at least for the ReLU activation function . In particular , it is likely that the behavior with exponential width would be closely tied to the actual limit case k = +1 , where the random model ( 1 ) yields a Gaussian process . Namely , for k = +1 , f is a Gaussian process indexed by the sphere ( say if we restrict to inputs x 2 p d · Sd 1 ) , with f ( x ) ⇠ N ( 0 , EX⇠N ( 0,1 ) [ ( X ) ] ) and E [ f ( x ) f ( y ) ] = EX , Y⇠N ( 0,1 ) : E [ XY ] =x·y [ ( X ) ( Y ) ] . For example if the activation function is a Hermite polynomial of degree p , then f would be a spherical p-spin glass model . This polynomial case is particularly well-understood , and in fact the landscape we describe below in Section 1.2 was already described in this case by Ben Arous et al . [ 2020 ] ( in particular Corollary 59 ) indicating that adversarial examples would likely continue to persist in the ultra-wide setting . It would be interesting to see if the p-spin glass landscape literature can be extended to non-polynomial activation functions , and to a finite ( but possibly exponential in d ) k. A step in this latter direction was recently taken in Eldan et al . [ 2021 ] , where convergence rates to the Gaussian process limit where given both for polynomial activations and for the ReLU . Finally we note that for a smooth activation it might be that there is a more direct argument to remove the subexponential width condition in Theorem 1 ( in technical terms , the proof of Proposition 2 could leverage a better argument than our naive upper bound on Lip ( ) ) . Finally , we note that , in practice , it has been found that there exists “ universal ” adversarial perturbations that generalize across both inputs and neural networks , Moosavi-Dezfooli et al . [ 2017 ] . For the case of ReLU activation ( Theorem 2 ) , we could in fact prove our result by replacing the gradient step with a step in the direction P k ` =1 a ` w ` , which is indeed a direction independent of the input x , thus proving the existence of “ universal ” perturbations ( generalizing across inputs ) for our model .

Summary: The authors prove that two-layer neural networks can be perturbed adversarially at a fixed data point and at random initialization (with high probability), i.e.\ for a neural network $f$ with randomly initialized parameters and a data point $x$, there exists a small perturbation $\delta = O(1)$ such that $|f(x)| = \Omega(1)$, but $f(x+\delta)\cdot f(x) < 0$. The result holds for a wide range of widths and activation functions. They present numerical evidence for their theoretical claims.