Adversarial Robustness is at Odds with Lazy Training
Authors: Yunjuan Wang
Enayat Ullah
Poorya Mianjy
Raman Arora
Venue: NeurIPS 2022 Accept (2022)

Abstract: Recent works show that adversarial examples exist for random neural networks [Daniely and Schacham, 2020] and that these examples can be found using a single step of gradient ascent [Bubeck et al., 2021]. In this work, we extend this line of work to ``lazy training'' of neural networks -- a dominant model in deep learning theory in which neural networks are provably efficiently learnable. We show that over-parametrized neural networks that are guaranteed to generalize well and enjoy strong computational guarantees remain vulnerable to attacks generated using a single step of gradient ascent. 

Adversarial Robustness is at Odds with Lazy Training
Yunjuan Wang
Enayat Ullah
Department of Computer Science Johns Hopkins University
Department Computer Science Johns Hopkins University
Baltimore, MD, 21218 ywang509e edu
Baltimore, MD, 21218 enayat@jhu. edu
Poorya Mianjy
Raman Arora
Department of Computer Science Johns Hopkins University Baltimore, MD, 21218 mi mianjy@jhu edu
Department of Computer Science Johns Hopkins University Baltimore, MD, 21218 aroracca juu edu
Abstract
Recent works show that adversarial examples exist for random neural net- works [Daniely and Shacham, 2020] and that these examples can be found using a single step of gradient ascent [Bubeck et al., 2021]. In this work, we extend this line of work to "lazy Iraining" of neural networks a dominant model in deep learning theory in which neural networks are provably efficiently learnable. We show that weerparametrrzed neural networks that are guaranteed to generalize well and enjoy strong computational guarantees remain vulnerable attacks generated using single step of gradient ascent.
Introduction
Despite the tremendous success of deep learning, recent works have demonstrated that neural networks are extremely susceptible to attacks. One such vulnerability is due to arbitrary adversarial corruption of data at the time of prediction, commonly referred as inference-tine attacks [Biggio al., 2013, Szegedy et al., 2014]. Such attacks are catastrophically powerful, especially in settings where adversaries can directly access the model parameters. By adding crafted imperceptible perturbations on the natural input, such attacks are easily realized across various domains, including in computer vision where an autonomous driving car may misclassify traffic signs [Sitawarin al., 2018], in natural language processing where automatic speech recognition misinterprets the meaning of sentence [Schönherr al., 2019], among many others.
The potential threat of adversarial examples has led to serious concerns regarding the security and trustworthiness neural network-based models that are increasingly being deployed in real-world systems. is crucial, therefore, to understand why trained neural networks classify clean data with high accuracy yet remain extraordinarily fragile due strategically induced perturbations.
One of the earliest works to describe adversarial examples was that of Szegedy et al. [2014], motivated by the need for networks that not only generalize well but also robust to small perturbations of its input. This was followed by a body of works that finds adversarial examples using various methods [Goodfellow al., 2015, Papernot al., 2016, Madry al., 2018, Carlini and Wagner, 2017a,b) Although lots of defense algorithms have been designed, most of them are quickly defeated by stronger attacks [Carlini and Wagner, 2017a, Carlini al., 2019, Tramèr et al., 2020]. Much of the prior work has focused on this empirical "arms race" between adversarial defenses and attacks; yet, deep theoretical understanding of why adversarial examples exist has been somewhat limited.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Recently, line of work theoretically constructs adversarial attacks for random neural networks. Inspired by the work of Shamir al. [2019], Daniely and Shacham [2020] prove that small no-m adversarial perturbations can be found by multi-step gradient ascent method for random ReLU networks with small width each layer has vanishingly small width compared to the previous layer. Bubeck et al. [2021] generalize this result to two-layer randomly initialized networks with relatively large network width and show that a single step of gradient ascent suffices to find adversarial examples. Bartlett et al. [2021] further generalize this result to random multilayer ReLU networks. However, all of the works above focus on random neural networks and fail to explain why adversarial examples exist for neural networks trained using stochastic gradient descent. In other words, why do neural networks that are guaranteed to generalize well remain vulnerable to adversarial attacks?
In this paper, we make progress toward addressing that question. In particular, we show that adversarial examples can be found easily using a single step of gradient ascent for neural networks trained using first-order methods. Specifically, we focus on over- wver-parametrized two layer ReLU networks which have been studied extensively recent years due to their favorable computational aspects. Several works have shown that under mild over-parametriizatoon (size of the network is polynomial in the size of the training sample), the weights of two-layer ReLU networks stay close to initialization throughout the training of the network using gradient descent-based methods [Allen-Zhu et al., 2019, Ji and Telgarsky. 2020, Arora et al., 2019]. In such a "lazy regime" of training, the networks are locally linear which allows training loss (aka, optimization) as well as bound the test loss (aka, generalization). The purpose of work is to show the deficiency of the lazy regime even though it is the dominant framework in the theory of deep learning.
to give computational guarantees for minimizing the
Our key contributions are as follows. We investigate the robustness of over-parametrized neural networks, trained using lazy training, against inference-time attacks. Building on the ideas of Bubeck et al. [2021], we show that such trained neural networks still suffer from adversarial examples which can be found using a single step of gradient ascent.
In particular, we show that an adversarial perturbation of size O [Ixl]] in the direction of the gradient Vdl
suffices to flip the prediction of two-layer ReLU networks trained in the lazy regime these are networks with all weights
1 from their initialization, where is the width of distance of O ym
the network. To the best of our knowledge, this is the first work which shows that networks with small generalization error still vulnerable adversarial attacks.
Further, we validate our theory empirically. We confirm that perturbation of size 0(1/Vd) suffices to generate a strong adversarial example for a trained two-layer ReLU networks.
The rest of the paper is organized as follows. In Section 2, problem setup, and discuss the related work. We present our main result Section 3, and give proof sketch in Section 4. In Section 5 we provide empirical support for our theory, and conclude with discussion in Section 6.
we introduce the preliminaries, give the
2 Preliminaries
Notation Throughout the paper, we denote scalars, vectors and matrices, respectively, with low- ercase italics lowercase bold and uppercase bold Roman letters, e.g. u, u, and U. We use [m] to denote the set {1, 2, m.,. We use || || or || |l2 exchangeably for E2-norm. Given a matrix U = [u]..... um,] E Rdxmm we define ||U||2,0o = max,gjm] ||u,]l- We use (2 R) to denote the l ball centered at u of radius R. For any U E gdxm, we use 02
= {U' E ddxm | ||U' 0|12,0 < RR to denote the l2.00 ball centered at U of radius R. For any function f: Rd + R. ff denotes the gradient vector. We define the standard normal distribution as N 1), and the standard multivariate normal distribution [0, I2). We use gd-1 to denote the unit sphere in d dimensions. We use the standard O-notation (O and Ω2).
2.1 Problem Setup
Let X' C Rd and y denote the input space and the label space, respectively. In this paper, we focus on the binary classification setting where y = -11, We assume that the data (x, y) is drawn from an unknown joint distribution D on X x y. For a function fw. X - y parameterized by w in

---

some parameter space W, the generalization envor captures the probability that fw makes a mistake on a sample drawn from D:
generalization error == P(x.))>D(Ufm - 0).
For fixed X, we consider norm-bounded adversarial perturbations given by B2 R), for some fixed perturbation budget R. The robust loss on x is defined as
(R(W; v)
max 1 (xfm(x') 0). (x,
The robust error then captures the probability that there exists an adversarial perturbation on samples drawn from D such that fw makes a mistake on the perturbed point
Ln(w; D) ((s.))>D(<(1(we
In this work, focus two-layer neural networks with ReLU activation, parameterized by pair of weight matrices (a, W), computing the following function:
f(x;a, W) =
Σ xw.) a=1
Here, m corresponds the number of hidden nodes, i.e., the width of the network; a(2) max }} ... is the ReLU activation function, and W [Wi,..., Wm.] E Rdxm and (0.,--.; :0m] R denote the top and the bottom layer weight matrices, respectively.
In this work, we study the robustness of neural networks which are close to their initialization. In particular, we are interested in the lazy regime, defined as follows. Definition 2.1 (Lazy Regime). Initialize the top and bottom layer weights as a, and W. ~ (O, La), v's E [m]. Let Wo [W........ Wa,m E Ruxm The lazy regime is the set of all networks parametrized by (a, W), such that W E B2 (Wo, Co/ /m), for some constant Co.
+1})
2.2 Related Work
Adversarial Examples. The field of adversarially robust machine learning has received significant attention starting with the seminal work of Szegedy al. [2014]. Most existing works focus on proposing methods to generate adversarial examples, for example, the fast gradient sign method (FGSM) [Goodfellow et al., 2015], the projected gradient descent (PGD) method [Madry et al., 2018], the Carlini & Wagner attack [Carlini and Wagner, 2017b) (017b). to name a few.
Most related to work are those on proving the existence of adversarial examples random neural networks. This line of work initiated with the work of Shamir et al. [2019], where authors propose an algorithm generate bounded no-mr adversarial perturbation with guarantees for arbitrary deep networks. Shortly afterwards, Daniely and Shacham [2020] showed that multi-step gradient descent can find adversarial examples for random ReLU networks with small width (i.e. m o(d). More recently, work of Bubeck et al. [2021] shows that single gradient ascent update finds adversarial examples for sufficiently wide randomly initialized ReLU networks. This was later extended to multi-layer networks with the work of Bartlett et al. [2021]. A very recent work of Vardi et al. [2022] shows that gradient flow induces bias towards non-robust networks. Our focus and techniques here are different we show that independent of the choice of the training algorithm, all neural networks trained the lazy regime remain vulnerable adversarial attacks.
Over-parametrized Neural Networks. Our investigation into the existence of adversarial examples in the lazy regime is motivated by recent advances deep learning theory. In particular, a series of recent works establish generalization error bounds for first-order optimization methods under the assumption that the weights of the network do not move much from their initialization [Jacot et al., 2018, Allen-Zhu et al., 2019, Arora et al., 2019, Chen et al., 2021, Ji and Telgarsky, 2020, Cao and Gu, 2020]. In particular, Chizat al. [2019] recognize that "lazy training" phenomenon is due to a choice of scaling that makes the model behave as its linearization around the initialization. Interestingly, linearity has been hypothesized as key reason for existence of adversarial examples in neural networks [Goodfellow et al., 2015]. Furthermore, this was used to prove existence of adversarial attacks for random neural networks [Bubeck et al., 2021, Bartlett al., 2021 1) and also provide the basis of our analysis in this work.
3

---

3
Main Results
In this section, we present our main result. We assume that the data is normalized so that ||x||2 1. We show that under the lazy regime assumption, a single gradient step on f suffices to find an adversarial example flip the prediction, given the network width is sufficiently wide but not excessively wide.
Theorem 3.1 (Main Result). Let Ws,0 ~ (0, Ia), ~ unif((1, +1}) and E (0, For any given x E S6-1, with probability at least 1 7, the following holds simultaneously for all W e B2 Wo, Co/ /m)
sign(f(x;a, a, W)) # sign(f(x + a;a, W)) where d v/Vxf( W). provided that the following conditions are satisfied:
C2
1. Step Size: |n|
[V W))22
2. Width requirement: max d2 ( Cglog
())
S m S C4 24
where Co. C2. C3, C1 are constants independent of width m and dimension d.
Several remarks are in order.
We note that the setting in Theorem 3.1 is motivated by the popular gradient ascent based attack model which has been shown to be fairly effective in practice. While in practice, we perform multiple steps of projected gradient ascent generate an adversarial perturbation, here we show that a single step of gradient suffices with a small step size of O(1/d). This attests to the claim that over-parameterized neural networks trained in the lazy regime remain quite vulnerable to adversarial examples. We confirm our findings empirically in Section 5.
Our analysis suggests that an attack size of O(1/Vd) suffices, i.e., for any x a perturbation á of size ||o]| = 0(1/ can flip the sign of f(x). This is a relatively small budget for adversarial perturbations, especially for for high-dimensional data. Indeed, in practice, we find that noise budget of this size allows for adversarial attacks on neural networks trained on real datasets such as MNIST and CIFAR-10 [Madry et al., 2018]. Interestingly, we find in our experiments, that O(1/Vā) the "right" perturbation budget in the sense that there is a sharp drop in robust accuracy. More details are provided in Section 5.
Our investigation focuses on the settings wherein the weights of the trained neural network stay close to the initialization. Further, given the initialization scheme we consider, the 0(1 deviation of the incoming weight vector any neuron of the trained network from its initialization precisely captures the neural tangent kernel (NTK) regime studied in several prior works [Ji and Telgarsky, 2020, Du al., 2018]. This O(L/vm) deviation bound is also the largest radius that we can allow in our analysis; see Section 4 for further details. In fact, our results would also hold for adversarial training given the same initialization and the bound the deviation of weights is met. Curiously, our empirical results exhibit a phase transition around the perturbation budget of O 0(1//m) we see in Section 5 that robust accuracy increases as we allow the network weights deviate more.
There is an interesting interplay between the network width and the input dimensionality. Our results require that the network is sufficiently wide (i.e., polynomially large in d), but not excessively wide (is sub-exponential in d). The upper bound is large enough to allow for the wveepparameterization required in the NTK setting, as long as the input dimensionality is sufficiently large.
To get an intuition about how the result follows, we first paraphrase the arguments from Bubeck et al. [2021], Bartlett et al. [2021] for random neural networks with weights Wo. We consider linear model fww;x) Va Ed Ws,nXs. Since the initial weights are independent (sampled i.i.d. from (o 1)), using standard concentration arguments, we have that for large d, with high probability [[( < O(1). In contrast, V Vx/(wnn xWo;x) = w, / x)] |/w]], which is roughly ((Vī). Hence, for large enough step of gradient ascent suffices to change the sign of f.
The above reasoning can be extended to weights that are close to random weights. In particular, for arbitrary weights W close to the initialization, consider the linear model: fw; = Va Ed= W,X,.

---

Since l|w, - Wa,oll = (1), thus even in the worst case, the function still behaves as a constant translation of the initial prediction model f(wo; x), and thus is constant. Similarly, [ x)] ((vd).
The work of Bubeck et al. [2021] extends the above observation to ReLU activation exploiting the fact that the function, even though non-linean is still locally linear. Note that, in a different context, this local linearity property is central to the Neural Tangent Kernel regime. In particular, the key underlying idea there is based on the linear approximation to the prediction function. Furthermore, with appropriate initialization, a linear approximation is not only good enough model close to the initialization, but also enables tractable analysis. We collapse this "good" linearization property under the lazy training assumption since that suffices for our purposes.
Beyond Lazy regime. Our result can be easily extended to a broader class of neural networks. In particular, consider the cone C' = {W Br > 0,rW E c2 (Wo: ( see Fig- ure 1. Note that for binary classification, scaling the network weights with positive factor does not change the prediction. As result, Theorem 3.1 still holds for network weights which belong to the cone C but may not be the lazy regime.
W
Cone containing the lazy- regime ball around initialization Wg.
Figure 1:
Robust Test Error. As corollary to Theorem 3.1, we have that none of the networks in the lazy regime are robust, i.e., for all networks in the lazy regime, the robust test error is bounded from below by a constant with high probability.
Corollary 3.2. Let Wa00 ~ (O, a ~ unif((-1,+1)) R O(1/vd) and y E (0. In the parameter settings Theorem 3.1, for any data distribution D supported on the sphere gd-1, with probability at least 10y (over the draw of Wo and a), for all W s.t. W E (Wo, Co/ mm) the robust error Ln W D)= 2 0.9.
We note that our result above is not a contradiction to the prior result of Gao et al. [2019], which claims that adversarial training finds robust classifiers in overparameterized models. Firstly, our result shows that there is no robust classifier in the lazy regime (Definition 2.1). In contrast, Gao et al. [2019] simply assume the existence of robust classifier in an NTK setting; see [Gao et al., 2019, Assumption 5.2]. Indeed, ignoring some of the (potentially unimportant) differences between our setting and that of Gao al. [2019]. (e.g., smooth vs. non-smooth activations and tied vs. non-tied weights in the last layer), then our result may suggest that the assumption in Gao et al. [2019] is incorrect. Secondly, the main result of Gao al. [2019] is show that the excess robust training error of the output of adversarial training is small. However, this only implies a small robust training error the best-in-class robust training error is also small, which, again, as we discussed above, does not hold. Finally, despite the validity of the key assumption in Gao et al. [2019], the authors only provide bound on robust training error. In contrast, our result shows that robustness may not hold in the sense of robust test error.
4 Proof Sketch
In this section, we provide outline of the proof of Theorem 3.1. Recall that our goal is to show that for any given x E gd-1, the vector = J a, W) is such that [|ö]| << ||x||, and sign(f(x; a, W)) & sign(f(x+ sign()) = 1. From the fundamental theorem of calculus, we have f(x+5)= = f(x)+ f")x+ Using simple algebraic manipulations (see full proof in Appendix B), we have the following,
W)). Assume without loss of generality that f(x a, W) > 0, then
0(1)
a(va)
f(x+nVf(xa a);a, W)
< f(x;a, W) -Im (xx WI
o(va)
+lml
sup SER || Svi ((
VV + W) - Vf(x;;, W)
The goal now to control each of the three terms on the right hand side. Note that the suggested
bounds on each of the terms will suffice to flip the label, thereby establishing the main result.
5

---

Given any x and W 02 (Wo, Cu ym
we need to control:
1. The function value of neural network fx;; W)
2. The size of the gradient of neural network || V (x; W)ll with respect to the input x
The size of the difference gradients f W) Vf(x o; w) for ||ô|| - O(1/Vd)
4.1 Bounding the function value
Lemma 4.1. (Informal) For any with probability at least 1 1 (x W) S 2 2/lo(22/7) Co
holds for all W E B2, Wor
Ca. vm
provided that 2 (((oog(/7)))
Proof Sketch:
From triangle incquality.
[xxxa, W) (x(x;a, Wo)l + (x;a, W) f(x; Wo)|-
For the first term, we have from Bubeck et al. [2021] that [x; a, Wo)| = O(1). For the second term (xx; W) - [xx; a, Wo)l S O (1) owing the Lipschitzness of ReLU. Note that O(1//m) is the largest deviation on ||w, w.,oll that we can allow get the constant bound here.
We next move to the second and third terms. We note that while bounding the first term simply follows from the Lipschitzness of the network, as we show the following sections, the second and third terms require a finer analysis, due to the non non-smoothness of ReLU.
4.2 Bounding the gradient
Lemma 4.2. (Informal) For any x, with probability at least 1 - 7, VV a, W 2 \vā holds for log((m/>))
all W B2.00 (Wo,
Co ym
provided that m 2 ((log(1/7)) and d 2 2 m
Proof Sketch: P la xx be the projection matrix onto the orthogonal complement of x. Using triangle incquality.
a W)ll [|VV W)ll 2 [x;a Wo)l [XVV w) PV;((;; Wo)ll-
The first term IP fx: Wo)l 2 ((vd)), which follows from Bubeck et al. [2021] with minor changes arising from different scaling of the initialization.
We further decompose the second term,
/ W) - a Wo)l
<
Σ
a.P(W.o' (..'o(w. (w]])
Vm
m
a PW - a <o(wTox)))
sup
WcB
(Wo,
Cg mm
m
The first term is O(1) simply from the fact that ||,,- arguments are as follows: by definition of the term Pw,,0 is independent of o' o'(w]] x) ('(w.., therefore, in Bubeck et al. [2021], we can bound both of these terms independently. Note that a.Pw..a ~ (O, Therefore, the second term equal in distribution the following random variable:
- woll S Co.
To bound the second term, the key
Vm
2
Do o'(3
SuPweB, Co vm
where L(),
The term co((wxxx [ a' a'(w [w only when the s-th neuron has different signs for incoming weights Ws,0 and w,. So, at high-level, we want to bound the number neurons changing sign between weights and w. To do this, we draw insights from results in the lazy training regime that

---

bound the number of such neurons in terms of deviations in weights. In particular, the following lemma shows that with high probability, for all W E 00 (Wo, have sign different from their initialization.
Co vm ) at most ((ym) neurons can
Lemma 4.3. Let Sy == { W 02 (Wo:
Co. VmL
1[( a. x) > 0J + ([ > 0]}. Then,
with probability at least - we have that [S,] S Cov'm + m log(1/7)
2
The key idea is that we can absorb the union bound in the definition of the set above without compromising any increase in the size of the set. Putting all of these pieces together yields the bound claimed in Lemma 4.2.
4.3 Bounding the difference of gradients
Lemma 4.4. For any x, with probability at least - y, for all W E B2,00 (oo. vm
Ca.
we have that
sup v ax w) vf((x a;a, w|ll s o(vd), provided that ||ô|| - O(1/Vd), and d S m
S 0 (1).
Proof Sketch: It follows from the definition of norm that
[V;(x; W)
fx + a;a, w)ll
sup r LEER" V
Σ a.w. (d'(w.x) - o'(w.(x 5)))
The strategy, then, is to bound the right-hand side for fixed and 8, and then use an e-net argument to bound the supremum. To that end, define the following quantities:
Xa0 = ag a.w. To (w - o' d'(wIu(x + 8))), X, = a. r(o d''( +8)).
Then, the norm of the difference of the gradients is upper bounded by
mm Ex= X.,0
Jm Σdh (X, Xx,o).
The first term Vmn D=.1XX..0 O ((o//llg(
which follows from Bubeck al. [2021] with minor changes due the different scale of the initialization. However, the second term requires more careful analysis. In particular, the difference X, X..u includes deviations of both variables x and W. Therefore, we need to expand the difference in terms of each variable and carefully bound the resulting terms. Similar to the ideas presented in the previous section, here we need to bound the size of the set of neurons that can change sign under perturbations on both x and W. It turns out that the size of this set is in the same order as the one in Lemma 4.3, as long as the perturbation in x is O(1).
5 Experiments
The primary goal of this section is to provide empirical support for our theoretical findings. In particular, want to verify how tight our theoretical bounds in practice, in terms of the size of the perturbation needed and the interplay of robustness with training in the lazy regime. For instance, are the trained models indeed vulnerable to small perturbations of size O(1 (d) as suggested in Theorem 3.1? ? the assumption of training in the lazy regime merely for analytical convenience or do we see markedly different behavior if we allow the iterates to deviate more than O(1/ m)) from initialization?
Datasets & Model specification. We utilize the MNIST dataset for our empirical study. MNIST dataset of 28 x 28 greyscale handwritten digits [LeCun et al., 1998]. We extract examples corresponding to images of the digits "0' and '1', resulting in 12665 training examples and 2115 test examples. To study the behaviour of different quantities as a function of the input feature dimensionality, we downsample the images by different factors to simulate data with d ranging between 25 and 784 (the original dimension). We use two-l ReLUU networks with a fixed top layer to match our theory and use the initialization described in Section 2.
7

---

Addh-1dd
6.77
benchrurk
0.25
6077
with 110
1endhoork 0.//
0.00
20C
700
750
0.00
305
demendzee
dimendos
Minimal size of the perturbation vector needed to flip the predicted label (left), the norm of the gradient (x(x;a, W) for W trained in the lazy regime (middle), and the corresponding step size (right), as a function of the input dimension for fixed value of Cu = 10 and different values of network widths (m)
Figure 2:
0.14
20.0
20
0.3
17.5
Q-40
g0.2
5425
g00
0.
5.-
100
200
300
400
5.0
600
700
000
110
200
322
210
500
003
736
80
100
200
303
420
0003
Figure 3: Minimal size of the perturbation vector needed to flip the predicted label (left), the norm of the gradient f a, W) for W trained in the lazy regime (middle), and the corresponding step size (right), as a function of the input dimension d for a fixed value of network width m = 10 and different values of Co-
5.1 Lazy regime for standard SGD
Setting. For each d {5?, 72, 102, 142, 20 255,, 282), we set the width of the network m e {1e3, 1e4, 1e5]. We train the network using standard SGD on the logistic loss using a learning rate of 0.1. We denote the weight matrix at the thh iterate of SGD W, and the incoming weight vector into the 4hh hidden node at iteration & as W.,t- The training terminates if the iterates exit the lazy training regime, i.e., if supisesm ||Wa,r W.ooll Co. We experiment with Co E {10, 20, 30, 40). We note that regardless of the above stopping criterion, each of the trained models achieves test error of less than 1%. In other words, the models that our training algorithm returns are guaranteed to satisfy the lazy regime assumption and generalize well.
Vmm
Given a trained neural network with parameters (W, a)), for each test example x, we find the smallest step size 1) such that the perturbation ō = v/ (x) flips the sign of the prediction, i.e., fw (x) 5) < O. For each experimental setting (i.e, for each value of d, Co. and m), we report results averaged over 5 independent random runs.
Results. The key quantities in our analysis are the step size 7) the norm of the gradient [VV((x; a, W)ll, and the size of the perturbation vector which is given as the product of the step size and the norm of the gradient. In Figure 2, we plot these quantities as a function of input dimen- sion d, for a fixed Co. for various values of width m E (1e3, le4, 1e5) (corresponding to the curves in red, orange, and blue, respectively). The green curves represent the corresponding theoretical bound. As predicted by Theorem 3.1, the gradient norm behaves roughly as ((vd). Furthermore, the minimum step-size required to flip the label, and the minimum size of the perturbation vector (normalized by the input norm) behave as (1/d)) and ((IVd)), respectively.
In Figure 3, we explore the behaviour of the key quantities as we allow for larger deviations in the lazy regime, by allowing for larger values of Co, for fixed network width of m 10 We see that as Co increases, both the size of the gradient and the minimum step size required to flip the label increase, thereby increasing the size the perturbation vector needed to change the sign of the prediction. This suggests that allowing the trained networks to deviate more from the initialization potentially leads to models that are more robust (against the attack model based on single step of gradient ascent).
8

---

60
60
565
633
611
Figure 4: (Left):Rousst test accuracy as a function of perturbation size ||ô|| and dimension d, for a fixed network width of m 1000 and fixed value of maximal deviation of weight vectors of
V = Co with Co = 10. (Right):Robust test accuracy as a function of width m and maximal ym
deviation of weight vectors, V, for fixed dimension of d 784 and fixed perturbation size | 0.2.
Remarkably, the minimum perturbation size, closely follows the 0(1/vd) behaviour predicted by the theorem. This begets further investigation into the required perturbation size, which is the focus of the next subsection.
5.2 Lazy regime for adversarial training
We begin by noting that theoretical results not limited to any particular learning algorithm. In fact, the results hold simultancously for all networks in the lazy regime. In this section, we further verify our theoretical results for networks that are adversarially trained in the lazy regime.
Setting. We downsample images as in the previous section to simulate data with E (52 62, 72,.... ,..., ) For this set of experiments, to closely match the setting of our theorem. We use projected gradient descent (PGD) [Madry al., 2018] for adversarial training. To ensure that the updates remain in the lazy regime, after each gradient step, we project the weights onto 32 V); see Algo. 1 in the appendix for details. We experiment with values of V and the size of [|ō]| in the intervals 0.05 < V and 0.05 < ||o|| S 1, respecivelly These ranges are wide enough include the O(1/Vā) and 0(1/vm) theoretical bounds that we aim to verify here. We note that the projection step is not commonly used in adversarial training; however, in our experiments, it does not hurt the test accuracy of the trained models, perhaps due to the simplicity of the classification task. At test time, we generate the adversarial examples using the PGD attack instead of a single gradient attack as in Section 5.1. We report the robust test accuracy averaged over independent random runs of the experiment.
also normalize the data, that ||x||
Results. For the experiments reported in Figure 4, left panel, we fix the network width to m 1000 and set V
Co V1000 0.316, to check how the dimension d and the perturbation budget
v/m
affect the robust accuracy. We observe a sharp drop in robust accuracy about the 5/ threshold for the perturbation budget ||ō|| as predicted by Theorem 1. We can see that for perturbation sizes slightly larger than 5/ Vd, the robust accuracy drops to O, while the trained networks are robust against smaller perturbations.
For the experiments reported in the right panel of Figure 4, we fix the input dimensionality to d = 28 and the perturbation size to ||]] 0.2, check how the network width m and the maximal deviation in weights, given by V. affect the robust test accuracy. We observe a phase transition in the robust test accuracy at value of V around 0(1/vm) as required by Theorem 3. particular, for any fixed width even slight deviation from the 25/ /m/ radius allows for training networks that are completely robust against the PGD attacks. With a smaller radius, however, the robust test accuracy quickly drops to O. This experiment suggests that adversarial robustness may require going beyond the lazy regime.
Why MNIST? We remark that our empirical demonstration on a simple dataset makes stronger case for our theoretical result. This is because the perturbation size of ((1 prescribed by

---

theoretical result is worst-case, regardless of the dataset. So, in an easy setting like binary-MNIST, we would expect our bound be pessimistic and a smaller perturbation be sufficient. On the contrary, we see that even MNIST exhibits a perturbation-diimnniio relationship that closely follows our theory. This can be regarded as an empirical evidence of the tightness of our bounds.
Conclusion
In this paper, study the robustness of two-layer neural networks trained in the lazy regime against inference-time attacks. In particular, we show that the class of networks for which the weights are close to the initialization after training, still vulnerable to adversarial attacks based on a single step of gradient ascent with perturbation of size O [IN
va
Our works suggests several research directions for future work. First, the focus here is on two-layer networks, natural question is to ask whether the same holds for multi-layer networks. Second, while we show that single step of gradient ascent-based attack is sufficient for our purposes, we expect to get stronger results if we consider stronger attacks, for instance, with gradient ascent based attack that run convergence. Third, our experiments highlight an intriguing relationship between the width, the input dimension, and the robust accuracy. So, natural avenue is to understand these dependencies more carefully. These include, figuring out the dependence on the input dimensionality of the minimal perturbation required to generate adversarial example. Similarly, how does the robust error behave as a function of the maximal distance of the weight vectors from the initialization.
Acknowledgments and Disclosure of Funding
This research supported, in part, by the DARPA GARD award HR001 12020004, NSF CAREER award IIS- 1943251, and the Spring 2022 workshop on "Learning and Games" at the Simons Institute for the Theory of Computing.
References
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference Machine Learning, pages 322-332. PMLR, 2019.
Peter Bartlett, Sébastien Bubeck, and Yeshwanth Cherapanamjeri. Adversarial examples in multi- layer random ReLU networks. Advances in Neural Information Processing Systems, 34, 2021.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šındić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joinl European conference on machine learning and knowledge discovery in databases, pages 387-402, Springer, 2013.
Sébastien Bubeck, Yeshwanth Cherapanamjeri, Gauthier Gidel, and Rémi Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. Advances in Neural Information Processing Systems, 2021.
Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over- parameterized deep ReLU networks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 3349 3356, 2020.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop artificial intelligence and security, pages 3-14, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 20/7 IEEE symposium security and privacy (SP), pages 39-57. IEEE, 2017b.
10

---

Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 1:XXiv:9902.070003 2019.
Zixiang Chen, Yuan Cao, Difan Zou, and Quanquan Gu. How much over-parameetriizatoon is sufficient to learn deep relu networks? In 9th International Conference on Learning Representations, 2021.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 2019.
Amit Daniely and Hadas Shacham. Most ReLU networks suffer from e adversarial perturbations. Advances in Neural Information Processing Systems, 33:6629-6636, 2020.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes wwer-parameterized neural networks. In International Conference on Learning Representations, 2018.
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsich, Liwei Wang, and Jason D Lee. Convergence of adversarial training in overparametrized neural networks. Advances in Neural Information Processing Systems, 12:13029-1300 13040, 2019.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks. In 8th International Conference on Learning Representations, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227-22224 1998.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium security and privacy (EuroS&P), P), pages 372-387. IEEE, 2016.
Lea Schönherr, Katharina Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding. In 26th Annual Network and Distributed System Security Symposium, 2019.
Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. A simple explanation for the existence of adversarial examples with small hamming distance. arXiv preprint arXiv: 10861, 2019.
Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Prateek Mittal, and Mung Chiang. Rogue signs: Deceiving traffic sign recognition with malicious ads and logos. arXiv preprint 1rX1v:1801.02280, 2018.
Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, 2014.
Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems, 2020.
Gal Vardi, Gilad Yehudai, and Ohad Shamir. Gradient methods provably converge to non-robust networks. arXiv preprini arXiv:2202.04344 2022.
Martin J Wainwright. High-dimensional statistics: non-asymptotic viewpoint, volume 48. Cam- bridge University Press, 2019.

---

Checklist
A. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
B. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
C. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Figure Figure 3.
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [N/A]
D. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
E. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "jg2PWIbYgx6", "reviewer": "Reviewer_7pem", "summary": "This paper studies model robustness in the context of lazy training, i.e, the model parameters are in the neighborhood of its initialization.\nSpecifically, the authors find that the lazy trained models are quite vulnerable against adversarial attacks, although it has good generalization on clean inputs.\nIt is proved that a single step attack with a small step size can flip the prediction of the two-layer networks under lazy training. This claim is validated in the numerical stimulation.", "questions": "I have pointed out the major questions in the weakness part in the previous section. In addition, I have one additional question:\n\n1. In the right part of Figure 2, why the step size $\\eta$ is significantly bigger in the small $C_0$ and small $d$ cases? This is not consistent with what the authors say.\n\nIn addition, the presentation of this paper needs improvements, such as:\n\n1. Definition of $L_R$ (between line 95 and 96), $L_R$ is not a function of $(x, y)$ because they are averaged for expectation calculation.\n\n2. Theorem 3.1, $C_1$ is not defined.\n\n3. Corollary 3.2, line 176. it should be $a \\sim unif\\\\{-1, +1\\\\}$\n\n4. Lemma 4.4,  line 251, $c$ is not defined.", "limitations": "The authors briefly discussed the limitation of this work in the end of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. It is interesting to study the model's robustness in the context of lazy training, which makes the model efficiently learnable.\n2. The theoretical proof is sound.\n\nWeakness:\n\nI have several concerns listed below, addressing them can improve the manuscript.\n\n1. In Theorem 3.1, the width needs to be both upper bounded and lower bounded (Line 145). However, the lower bounds increases with the decrease of $\\gamma$, and the upper bound decreases with the decreases of $\\gamma$. That is to say, in some cases, the $\\gamma$ can not be arbitrarily small, i.e. $\\gamma \\to 0$ may make $m$ infeasible under the constraints of this theorem. This is a critical concern especially in the case of small input dimension $d$. The author should comprehensively discuss this case.\n\n2. The main theorem (Theorem 3.1) of this paper is applicable to any model parameters in the neighborhood of the initialization. However, in the empirical experiments only study the cases of training or adversarial training, could it be the vulnerability as shown in the experiments arise from the training methods, i.e, the parameters prior encoded by your training objective and optimizer?\n\n3. In general, I think the real interesting part of this work is to show there is no robust models in the neighborhood of a random Gaussian initialization, **even with adversarial training**. I suggest the author highlight the adversarial training part, because in vanilla training, even without constraining the model parameters in the initial neighborhood, the model is not robust at all.\n\n4. For the presentation of Theorem 3, the ``constant'' $C_2$, $C_3$, $C_4$, and $C_5$ depends on $C_0$. I agree that for notation simplicity, there is no need to provide the exact analytical form of these values. However, I do suggest the authors discuss how these values increases or decreases with $C_0$. In addition $C_1$ is not defined in this Theorem.\n\nIn general, I like the topic and the problem settings of this paper, but there are several concerns pointed in this section and the next one. It can be a good paper if the concerns above are adequately addressed. I am happy to discuss with the authors in the rebuttal and re-evaluate this paper after that.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VA4l-YaptZ9", "reviewer": "Reviewer_e718", "summary": "This paper studies the existence of adversarial examples (found via a single gradient step) for two-layer ReLU networks in settings where the trained weights remain close to their initialized values. The authors prove that adversarial examples exist in this specific case, and support their theoretical findings via experiments on MNIST.  \n", "questions": "- How does this perturbation budget relate to what is typically considered an indistinguishable attack in adversarial training studies on MNIST?\n\nMinor suggestions\n- Typo line 26: lead -> led, incomplete sentence line 47, extra “a” in line 73, missing \\citep throughout\n- All the plots in Figures 1-3 are difficult to read -- they should be enlarged. \n", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is mostly well written.\n- This paper presents a novel theoretical analysis of adversarial robustness in the context of what is referred to as the \"lazy regime\" in this work, where the trained weights do not stray far from their initialization.  \n- The authors present empirical results on MNIST that back up their theoretical findings. \n\nWeaknesses\n- It's not immediately clear what the importance/significance of this work is, and how this extends beyond two-layer ReLU networks, and this very specific setting. \n- The proof sketch reads a bit rushed -- the readability could be improved here. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T9ISxWgqi1J", "reviewer": "Reviewer_ztAA", "summary": "This paper studies the robustness of two-layer NNs against adversarial examples in the “lazy-training” regime, extending a line of work which studied this for randomly initialized two-layer NNs. The paper shows that if after training, the weights of the NN are close to the randomly initialized weights, single-step adversarial perturbations of size O(1/\\sqrt(d)) flips the prediction of this NN. The authors empirically validate their theory via experiments on MNIST.\n", "questions": "- Would similar results hold for multi-class classification problems?\n- It would be more convincing to see empirical evidence supporting the theory on datasets beyond MNIST (e.g., CIFAR-10 or ImageNet), and in general on tasks beyond image classification. Is there a reason why the authors didn’t try these?\n", "limitations": "The  authors adequately addressed the limitations of their work. They didn’t include potential negative societal impact though. While I don’t think there is a direct negative societal impact of this paper, I believe the authors might be able to talk about how readers might get a wrong perspective of non-robustness of NNs in general. The readers should be aware that the claims of this paper hold only for two layer NNs and in the lazy training regime only, i.e., that there is no proof yet that it is guaranteed to find adversarial examples for general NNs.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n#### Strengths:\n- The findings in the paper are original to the best of my knowledge. The paper extends a line of work that theoretically shows that NNs are susceptible to single-step adversarial attacks.\n- The claims of the paper are sounds and are well-supported by the experiments.\n- I think the work is decently significant as it is the first paper to attempt to bring the theory of robustness of neural networks beyond the random-initialization regime.\n- The paper is very well written and flows smoothly. I didn’t check the proof sketch in detail, but everything else was easy to grasp and follow.\n- The paper lists a number of potential research directions for building on top of their work which I think is helpful and puts the contributions of the paper into context.\n\n#### Weaknesses:\n- Overall the paper is pretty good, but it can benefit from more experiments to support the theory (more datasets, multiple tasks beyond image classification)\n- A discussion of extension to multi-class classification problems is missing.\n\n\n#### Minor changes:\n- Line 73: repeated “a”.\n- Equation after line 93: extra parentheses next to the zero\n- Line 134-158-164: references should be inside parentheses\n- Line 146: C1 doesn’t show up in theorem 3.1 nor in the main paper.\n- Line 141 and 176: a ∼ unif{1, +1} -> a ∼ unif({-1, +1})\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AXgZifOW5BA", "reviewer": "Reviewer_TJvx", "summary": "The paper takes a solid step towards explaining adversarial sensitivity of neural networks based on random networks. The paper shows that networks can be attacked by a single gradient descent step in the lazy regime where parameters are close to initialization. ", "questions": "See the weakness part.", "limitations": "The authors adequately addressed the potential negative societal impact of their work? I think fixing $a$ is a big limitation for the technical results.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper takes a solid step towards explaining adversarial sensitivity of neural networks based on random networks [1]. In the lazy regime like NTK, while networks can show good clean perfomrance, they can easily attacked by a single gradient step.\n\n2. The paper is easy to follow with clear presentation.\n\n3. I appreciate the adversarial training in the empirical experiments, which shows that the theorem holds for arbitrary $w$ near initialization not constraint by the training algorithms.\n\nWeaknesses\n\n1. My major concern is that the Theorem only optimizes the first layer $W$ while $a$ is not included. In the experiment, $a$ is also a tunable parameter. This is inconsistent. Besides, allowing tunable $a$ may bring contradictory result. In [2], the authors show that adversarial training is possible to converge in the overparameterized models. It seems to indicate that adversarial robustness is not impossible in the lazy regime.\n\n2. I recommend the authors to report clean performance in the experiment parts to show the network actually achieves good clean generlization near initialization.\n\n3. One conclusion is misaligned with empirical findings. The adversarial trained networks need larger size of adversarial example to attack the network (Figure 4(b) in [3]). And the common $L_2$ adversarial budget set for ImageNet (dimension is $224\\times224\\times 3$) is also higher than CIFAR10 (dimension is $32\\times32\\times 3$). This is contradictory to the conclusion that the label will be flipped with $O(\\frac{1}{\\sqrt{d})$ adversarial noise.\n\n[1] S´ebastien Bubeck et.al. A single gradient step ﬁnds adversarial examples on random two-layers neural networks, NeurIPS 2021\n\n[2] Ruiqi Gao et.al. Convergence of Adversarial Training in Overparametrized Neural Networks, NeurIPS 2019\n\n[3] Ali Shafahi et.al. Are adversarial examples inevitable?, ICLR 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept