Adversarial Examples in Multi-Layer Random ReLU Networks

1 Introduction and Main Result . Since the phenomenon of adversarial examples was first observed in deep networks [ SZS+14 ] , there has been considerable interest in why this extreme sensitivity to small input perturbations arises in deep networks [ GSS15 , SSRD19 , BLPR19 , DS20 , BCGdC21 ] and how it can be detected and avoided [ CW17a , CW17b , FCSG17 , MMS+18 , QMG+19 ] . Building on the work of Shamir et al [ SSRD19 ] , Daniely and Schacham [ DS20 ] prove that small perturbations ( measured in the Euclidean norm ) can be found for any fixed input and most Gaussian parameters in certain ReLU networks—those in which each layer has vanishing width relative to the previous layer—and conjectured the same result without this strong constraint on the architecture . Bubeck , Cherapanamjeri , Gidel and Tachet des Combes [ BCGdC21 ] prove that the same phenomenon occurs in general two-layer ReLU networks , and give experimental evidence of its presence in deeper ReLU networks . In this paper , we prove that adversarial examples also arise in deep ReLU networks with random weights for a wide variety of network architectures—those with constant depth and polynomiallyrelated widths . The key fact underlying this phenomenon was already observed in [ DS20 ] : a high-dimensional linear function f ( x ) = w > x with input x 6= 0 and random parameter vector w with a uniformly chosen direction will satisfy krf ( x ) k kxk |f ( x ) | with high probability . This implies 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the existence of a nearby adversarial example for this linear function : a perturbation of x of size |f ( x ) |/krf ( x ) k ⌧ kxk in the direction f ( x ) rf ( x ) will flip the sign of f ( x ) . This observation can be extended to nonlinear functions that are locally almost linear . Indeed , it is easy to show that for all x , u 2 Rd , |f ( x + u ) ( f ( x ) + hu , rf ( x ) i ) | kuk sup krf ( x ) rf ( x + v ) k : v 2 Rd , kvk kuk , and thus to demonstrate the existence of an adversarial example near x for a function f , it suffices to show the smoothness property : for all v 2 Rd with kvk . |f ( x ) |/krf ( x ) k , krf ( x ) rf ( x + v ) k ⌧ krf ( x ) k. ( 1 ) We show that for a deep ReLU network with random parameters and a high-dimensional input vector x , there is a relatively large ball around x where the function computed by the network is very likely to satisfy this smoothness property . Thus , adversarial examples arise in deep ReLU networks with random weights because the functions that they compute are very close to linear in this sense . It is important to notice that ReLU networks are not smooth in a classical sense , because the nondifferentiability of the ReLU nonlinearity implies that the gradient can change abruptly . But for the smoothness condition ( 1 ) , it suffices to have krf ( x ) rf ( x + v ) k ✏ + ( kvk ) , for some increasing function : R+ ! R+ , provided that ✏ + ( kvk ) ⌧ krf ( x ) k. We prove an inequality like this for ReLU networks , where the ✏ term decreases with width . Consider a network with input dimension d , ` +1 layers , a single real output , and complete connections between layers . Let d1 , . . . , d ` denote the dimensions of the layers . The network has independent random weight matrices Wi 2 Rdi⇥di 1 for i 2 [ ` + 1 ] , where we set d0 = d and d ` +1 = 1 . For input x 2 Rd , the network output is defined as follows : f ( x ) = W ` +1 · ( W ` · ( W ` 1 · ( · · · ( W1 · x ) · · · ) ) ) where ( x ) i = max { xi , 0 } W ` +1 ⇠ N ( 0 , I/d ` ) and 8i 2 [ ` ] , ( Wi ) j , k i.i.ds N ( 0 , 1/di 1 ) . ( NN-DEF ) Note that the scale of the parameters is chosen so that all the real-valued signals that appear throughout the network have roughly the same scale . This is only for convenience : because the ReLU is positively homogeneous ( that is , for ↵ > 0 , ( ↵x ) = ↵ ( x ) ) , the scaling is not important for our results ; the 1/di 1 in ( NN-DEF ) could be replaced by any constant without affecting the ratio between the norm of an input vector and that of a perturbation required to change the sign of the corresponding output . The following theorem is the main result of the paper . Theorem 1.1 . Fix ` 2 N. There are constants c1 , c2 , c3 that depend on ` for which the following holds . Fix 2 ( 0 , 1 ) and let f ( · ) be an ( ` + 1 ) -layer ReLU neural network defined by ( NN-DEF ) with input dimension d and intermediate layers of width { di } ` i=1 . Suppose that the widths satisfy dmin c1 ( log dmax ) c2 log 1/ where dmin = min { di } ` i=1 , d , dmax = max { di } ` i=1 , d . Then for any fixed input x 6= 0 , with probability at least 1 , |f ( x + ⌘rf ( x ) ) | |f ( x ) | and sign ( f ( x + ⌘rf ( x ) ) ) 6= f ( x ) , for an ⌘ satisfying k⌘rf ( x ) k kxk c3 r log 1/ d . It suffices to choose c1 = ( C1 ` ) c2 , c2 = C2 ` , c3 = C ` 3 , for some absolute constants C1 , C2 , C3 . This theorem concerns networks of fixed depth , and the constants in the size of the perturbation and in the requirement on the network width are larger for deeper networks . We also prove a converse result that illustrates the need for some constraint on the depth . Theorem 3.1 shows that when the depth is allowed to grow polynomially in the input dimension d , the function computed by a random ReLU network is essentially constant , which rules out the possibility of adversarial examples . The heart of the proof of Theorem 1.1 is to show a smoothness property like ( 1 ) . It exploits a decomposition of the change of gradient between two input vectors . Define Hi : Rd ! Rdi⇥di as Hi ( x ) jk = 1 { j = k , vi ( x ) j 0 } with vi ( x ) = Wi ( · · · ( W1x ) ) . For two input vectors x , y 2 Rd , we will see in Section 2.4 that we can decompose the change of gradient as rf ( x ) rf ( y ) = ` X j=1 W ` +1 j+1Y i= ` Hi ( x ) Wi ! · ( Hj ( x ) Hj ( y ) ) Wj · 0 @ 1Y i=j 1 Hi ( y ) Wi 1 A . ( Here and elsewhere , indices of products of matrices run backwards , so Q k i=j Mi = I when j < k. ) For the jth term in the decomposition , we need to control the scale of : the gradient of the mapping from the input to the output of layer j , the change in the layer j nonlinearity Hj ( x ) Hj ( y ) , and the gradient from layer j to the output . It turns out that controlling these quantities depends crucially on the width of the narrowest layer before layer j—we call this the bottleneck layer for layer j . This width determines the dimension of the image at layer j of a ball in the input space . In proving bounds on gradients and function values that hold uniformly over pairs of nearby vectors x and y , this dimension—the width of the bottleneck layer—dictates the size of a discretization ( an ✏-net ) that is a crucial ingredient in the proof of these uniform properties . Our analysis involves working separately with the segments between these bottleneck layers . We show that for an input x 2 Rd satisfying kxk = p d and any y in a ball around x , with high probability krf ( x ) rf ( y ) k = o ( 1 ) , but |f ( x ) | is no more than a constant and krf ( x ) k is at least a constant . This implies the existence of a small ( o ( kxk ) ) perturbation of x in the direction f ( x ) rf ( x ) that flips the sign of f ( x ) . These results suggest several interesting directions for future work . First , our results show that for high-dimensional inputs , adversarial examples are inevitable in random ReLU networks with constant depth , and unlikely in networks with polynomial depth . Beyond this , we do not know how the sensitivity to input perturbations decreases with depth . Similarly , both results are restricted to networks with subexponential width , and it is not clear what happens for very wide networks . Finally , we show that networks with random weights suffer from adversarial examples because their behavior is very similar to that of random linear functions . It would be worthwhile to determine whether randomly initialized trained networks retain this nearly linear behavior , and hence suffer from adversarial examples for the same reason . Related Work : Related theoretical work include the recent result of Bubeck and Sellke [ BS21 ] who , following up on Bubeck , Li and Nagaraj [ BLN21 ] , show that only mildly over-parameterized networks when trained on random data have large Lipschitz constants . While these results apply to a broader class of networks including those potentially trained on data , this weaker property on its own does not suffice to explain the prevalence of adversarial examples and computational ease of finding them . Indeed , establishing this fact requires understanding the behavior of the local landscape of the function computed by the network which these approaches do not capture . Closely related empirical works include work by Madry , Makelov , Schmidt , Tsipras and Vladu [ MMS+18 ] and by Qin , Martens , Gowal , Krishnan , Dvijotham , Fawzi , De , Stanforth and Kohli [ QMG+19 ] . We note that both these works emphasize the importance of the interplay between the local behavior of the function and the existence of adversarial perturbations . [ MMS+18 ] identify local near-linearity as a cause of adversarial examples and propose a robust training procedure attempting to eliminate this property . On the other hand , [ QMG+19 ] suggest an alternative robust training procedure that retains local near-linearity but eliminates adversarial examples by ensuring that the learnt network ( despite being locally linear ) is robust to single-step perturbations . Our work lends theoretical grounding to this phenomenon showing that local linearity arises naturally at initialization .

Summary: EDIT: I will be keeping my (positive) score.  The paper studies robustness properties of ReLU networks that have random weights. The main finding is that random ReLU nets of constant depth are susceptible to adversarial examples for every input vector. The main theorem relies on some conditions about the widths in the networks' layers. To complement this result, the authors show that in the large depth regime, such a result cannot hold.  The paper builds on prior work and  their main result generalizes results of Daniely and Schacham (2020) for networks with rapidly decreasing width and results of Bubeck et al (2021) for two-layer networks. Some ideas and intuition come actually from prior works:  the main goal is to demonstrate the existence of an adversarial example near input x,  and to do this for a function f, it turns out that it suffices to show the smoothness property: see (1) . This "smoothness" property is not in the common sense of the word "smooth" function, but basically says that for a deep ReLU network with random parameters and a high-dimensional input vector x, there is a relatively large ball around x where f satisfies this smoothness property.   The main result is Theorem 1.1 that proves how taking a small step away from a point x in the direction of the gradient can flip the sign of the evaluated function f at x thus corresponding to an adversarial example. The most interesting condition is that the min depth of the net $d_\min \ge c_1(\log d_\max)^{c_2} \log 1/\delta$.  The authors also prove a converse result to justify why their results could only be true in the constant depth regime. This complementary result is given in Theorem 3.1 which shows that when depth grows polynomially in the input dimension d, the function computed by a random ReLU network can essentially be constant, which rules out the possibility of adversarial examples.  For theorem 1, an important aspect of the architecture of the network that allows for adversarial examples or not, depends crucially on the width of the narrowest layer before layer j—that  is "the bottleneck layer" for layer j as the authors call it. This width determines the dimension of the image at layer j of a ball in the input space. In proving bounds on gradients and function values that hold uniformly over pairs of nearby vectors x and y, this dimension—the width of the bottleneck layer—dictates the size of a discretization that is a crucial ingredient in their proof.