Gradient Methods Provably Converge to Non-Robust Networks
Authors: Gal Vardi
Gilad Yehudai
Ohad Shamir
Venue: NeurIPS 2022 Accept (2022)
Keywords: implicit bias
deep learning theory
robustness

Abstract: Despite a great deal of research, it is still unclear why neural networks are so susceptible to adversarial examples. 	In this work, we identify natural settings where depth-$2$ ReLU networks trained with gradient flow are provably non-robust (susceptible to small adversarial $\ell_2$-perturbations), even when robust networks that classify the training dataset correctly exist.	Perhaps surprisingly, we show that the well-known implicit bias towards margin maximization induces bias towards non-robust networks, by proving that every network which satisfies the KKT conditions of the max-margin problem is non-robust.

Gradient Methods Provably Converge to Non-Robust Networks
Gal Vardi* TTI-Chicago and Hebrew University galvardi@ttisn
Gilad Yehudai"
Weizmann Institute of Science
gilad. yehudaihueeeeeeeeee aettaaiiweeimmmn. ac.
Ohad Shamir
Weizmann Institute of Science ohad, shamir r @weei zmann. ac
Abstract
Despite a great deal of research, is still unclear why neural networks are so susceptible to adversarial examples. In this work, identify natural settings where 2epth-2 ReLU networks trained with gradient flow are provably non-robust (susceptible to small adversarial &y fy-perturbations), even when robust networks that classify the training dataset correctly exist. Perhaps surprisingly, we show that the well-known implicit bias towards margin maximization induces bias towards robust networks, by proving that every network which satisfies the KKT conditions of the max-margin problem is non-robust.
Introduction
In seminal paper, Szegedy et al. [2013] observed that deep networks are extremely vulnerable to
adversarial examples. They demonstrated that in trained neural networks very small perturbations to the input change the predictions. This phenomenon has attracted considerable interest, and
various attacks (e.g., Carlini and Wagner [017]. Papernot et al. [0117. Athalye et al. [2018), Carlini
and Wagner 2018). Wu et al. [2020]) and defenses (c.g.. Papernot et al. [2016] Madry al. [007
Wong and Kolter [2018]. Croce and Hein 2020)) were developed. Despite great deal of research,
still unclear why neural networks are susceptible adversarial examples [Goodfellow et al.)
2014
Fawzi et al., 20188 Shafahi et al.] 2018 Schmidt et al., 2018 Khoury and Hadfield-Ment
2018 Bubeck et al. 2019 Allen-Zhu and Li 2020, Wang al. 2020, Shah al.) 2020 Shamir Daniely and Shacham 2020). Specifically, it is not well-understood why
et al. 2021 Ge et al. 202T
gradient methods learn non-robust networks, namely, networks that susceptible to adversarial examples, even in cases where robust classifiers exist.
In recent string of works, it was shown that small adversarial perturbations can be found for any fixed input in certain ReL networks with random weights (drawn from a Gaussian distribution). Building on Shamir al. 2019]. it was shown in Daniely and Shacham (2020) that small adversarial perturbations (measured in the Euclidean norm) can be found in random ReLU networks where each layer has vanishing width relative to the previous layer. Bubeec al. [2021] extended this result to general two-layers ReLU networks, and Bartlett et al. [2021] extended it to a large family of ReLU networks of constant depth. These works aim to explain the abundance of adversarial examples in ReLU networks, since they imply that adversarial examples are common in random networks, and in particular random initializations of gradient-based methods. However, trained networks are clearly
"Equal contribution
Work done while the author was at the Weizmann Institute of Science
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

not random, and properties that hold in random networks may not hold in trained networks. Hence, finding a theoretical explanation to the existence of adversarial examples in trained networks remains a major challenge.
In this work, we show that in 22pth-2 ReI.U networks trained with the logistic loss or the exponential loss, gradient flow is biased towards non-robust networks, even when robust networks that classify the training dataset correctly exist. We focus on the setting where we train the network using a binary classification dataset {(x;; )1)),==
have (x,,x)) = o(d). E.g., this assumption holds w.h.p. if the inputs are drawn i.idd. from the uniform distribution on the sphere of radius vd. On the one hand, we prove that the training dataset can be correctly classified by a (sufficiently wide) 2epth-2 ReLU network, where for each example x; in the dataset flipping the sign of the output requires a perturbation of size roughly vd (measured in the Euclidean norm). On the other hand, we prove that for depth-2 ReLU networks of any width, gradient flow converges to networks, such that for every example x; in the dataset flipping the sign of the output can be done with a perturbation of size much smaller than vd. Moreover, the same adversarial perturbation applies to all examples in the dataset.
(vd 1 (-1,1), such that for each i +
For example, if we have m = ((d)) examples and maxity [xx, x,)| O(1), namely, the examples are "almost orthogonal". then we show that in the trained network there are adversarial perturbations of size (1) for each example in the dataset. Also, if we have = Õ (d) examples that are drawn i..d. from the uniform distribution the sphere of radius vd, then w.h.p. there are adversarial perturbations of size (dd//)) of o(vi) for each example in the dataset. In both cases, the dataset can be correctly classified by depth-2 ReLU network such that perturbations of size o(vd) cannot flip the sign for any example in the dataset.
limitation of our negative result is that it assumes an upper bound of O maxiya xx,x,))) on the size of the dataset. Hence, it does not apply directly to large datasets. Therefore, we extend our result to the case where the dataset might be arbitrarily large, but the size of the subset of examples that attain exactly the margin is bounded. Thus, instead of assuming upper bound on the size of the training dataset, suffices to assume an upper bound on the size of the subset of examples that attain the margin in the trained network.
The tendency of gradient flow to converge to non-robust networks even when robust networks exist can be seen as an implication of its implicit bias. While existing works mainly consider the implicit bias of neural networks in the context of generalization (see Vardi 2022 for survey), we show that it is also useful technical tool in the context of robustness. In order to prove our negative result, we utilize known properties of the implicit bias in depth-2 ReLU networks trained with the logistic or the exponential loss. By Lyu and Li [2199] and Ji and Telgarsky| (2020). if gradient flow in homogeneous models (which include depth-2 2 ReLU networks) with such losses converges to zero loss, then converges in direction to a KKT point of the max-margin problem in parameter space. In our proof we show that under our assumptions on the dataset every network that satisfies the KKT conditions of the max- mar-ma problem is non-robust. This fact may seem surprising, since our geometric intuition on linear predictors suggests that maximizing the margin is equivalent to maximizing the robustness. However, once we consider more complex models, we show that robustness and margin maximization in parameter space are two properties that do not align, and can even contradict each other.
We complement our theoretical results with an empirical study. As we already mentioned. limitation of our negative result that it applies to the case where the size of the dataset is smaller than the input dimension. We show empirically that the same small perturbation from our negative result is also able to change the labels of almost all the examples in the dataset, even when it is much larger than the input dimension. In addition, our theoretical negative result holds regardless of the width of the network. We demonstrate empirically, by showing that changing the width does not change the size of the perturbation that flips the labels of the examples in the dataset.
2 Preliminaries
Notations. We use bold-face letters to denote vectors, e.g., x = (II,... x.)) For x E Rd we denote by ||x[] the Fuclidean norm. We denote by (.) the indicator function, for example 1 [ 2 5) equals 1 if 2 5 and 0 otherwise. We denote sign(2) 1( 2 0). For integer d 2 we denote
2

---

(1.... dd. For set A we denote by (AA) the uniform distribution over A. We use standard asymptotic notation (-) to hide constant factors, and (-) to hide logarithmic factors.
Neural networks. The ReLU activation function is defined by (2)) = (0, In this work we consider depth-2 ReLU neural networks. Formally, a depth-2 network No of width l is parameterized by 0 = [W1.... Wk, b, v| where w; E _R for all i E [k] and b, v E IR,, and for every input x E I2d we have (x) = Σjele [wo(w] x + b,). We sometimes view 0 as the vector obtained by concatenating the vectors Wi,..., Wk, b, Thus, ||0|| denotes the l2 norm of the vector 0.
We denote x0):x) == (x)... We say that a network is homogeneous if there exists L such that for every a > 0 and 0, x we have ((a0;x)= = a² @0: ;0xxx; x). Note that 2 ReLU networks as defined above are homogeneous (with L 2).
Robustness. Given some function R(-), We say that neural network No is R(d)- w.r.t. inputs XI...., Xn, E _d if for every E [n]. (()())) and x' E HRd with ||x xx'Il S we have (ig = sign(>N(xx)).. Thus, changing the labels of the examples cannot done with perturbations of size ((R(d)). Note that we consider here &2 perturbations.
In this work we focus on the case where the inputs xI...., Xn. are on the sphere of radius vd, denoted by vd. gd-1, which generally corresponds to components of size O(1). Then, the distance between every two inputs is at most O O(Vd). and therefore perturbation of size O(vd) clearly suffices for flipping the sign of the output (assuming that there is at least one imput with // (xx) >> O and one input with A (x;) < O). Hence, the best we can hope for is Vd-robustness. In our results we show setting where a v d-robust network exists, but gradient flow converges to network where we can flip the sign of the outputs with perturbations of size much smaller than vd (and hence it is not vd-robust).
Note that in homogeneous neural networks, for every a > 0 and every x, we have sign (f(a0; sign (a % db (0; x)) sign (#(0:x)) x)). Thus, the robustness of the network depends only the direction of e and does not depend on the scaling of 0. [p]
Gradient flow and implicit bias. training dataset. Let (0:-)) _ IR be neural network parameterized by 0. For loss function e IR + IR the empirical loss of & (0;-) on the dataset S is
Let {xx, ,) S X (-1,1- be a binary classification
(())= = Σ (0:
(1)
1=1
We focus on the exponential loss E(g) = and the logistic loss E(g) = log(1 + t
We consider gradient flow on the objective given in (I). This setting captures the behavior of gradient descent with an infinitesimally small step size. Let @(t) be the trajectory of gradient flow. Starting from an initial point (0), the dynamics of (() is given by the differential equation do(1) E 8 (()()), Here, d denotes the Clarke subdifferential, which is a generalization of the derivative for non-differentiable functions (see Appendix A for a formal definition).
We say that a trajectory O(t) converges in direction to 0 if lim, 0
O())
Throughout this
work we use the following theorem:
Theorem 2.1 (Paraphrased from Lyu and Li 2019 Ji and Telgarsky 2020 Let 400-)) be homogeneous ReLU neural network parameterized by 6. Consider minimizing either the exponential or the logistic loss over binary classification dataset (.s,.
using gradient flow. Assume that
there exists time to such that (()()))) < 1, namely, ye (0 x;) > 0 for every x,. Then, gradient flow converges in direction to first order stationary point (KKT point) of the following maximum margin problem in parameter space:
min 0
|0
s.t. Vi ei [m] (0 21
Moreover. L(O(t)) + 0 and ||((t)] as co.
3

---

Note that in ReLU networks Problem 2 is non-smooth. Hence, the KKT conditions are defined using the Clarke subdifferential. See Appendix A for more details of the KKT conditions. Theo- rem 2.1 characterized the implicit bias of gradient flow with the exponential and the logistic loss for homogeneous ReLU networks. Namely, even though there are many possible directions T that classify the dataset correctly, gradient flow converges only to directions that are KKT points of Problem (2). We note that such KKT point is not necessarily a global/local optimum (cf. Vardi et al. (202T)) Thus, under the theorem's is assumptions, gradient flow may not converge to an optimum of Problem 2 but it is guaranteed to converge to a KKT point.
3 Robust networks exist
We first show that for datasets where the correlation between every pair of examples is not too large, there exists a robust 2 ReL.U network that labels the examples correctly. Intuitively, such a network exists since we can choose the weight vectors and biases such that each neuron is active for exactly one example x; in the dataset, and hence only one neuron contributes to the gradient at X; Also, the weight vectors are not too large and therefore the gradient x; is sufficiently small. Formally, we have the following:
Theorem 3.1. Let (x),,)))==> S (Vd. x [-1,1} be a dataset. Let < c < 1 be a constant independent of d and suppose that KX,,,)), Se c. d for every i + j. Let k 2 Then, there exists a 2 ReLU network N of width i such that yxN(xx;) 2 for every i E [m], and for every x; flipping the sign of the output requires a perturbation of size larger than 1-c vd. Thus, N is v ddrrobust w.E.t. Xi...., Xm-
Proof. We prove the claim for k = m. The proof for & >> m follows immediately by adding zero- weight neurons. Consider the network N(x) Σド= 1y0 (w] b,) such that for every j E [m] ltc 2x; and bj -ic For every i E [m] we have we have fj- = Vj. Wi d(1-c)
2 1|xx 2
1+c
w. x; + b;
1-c =1,
d(1 c)
and for every i # j we have
2x x
1 <
2c
1
w x; + bj = d(1 c)
-1.
- c 1-c
1-c
Hence, (xx;) = = y for every i E [m].
We now prove that M is / Voobust Let i E [m] and let x² E Rd such that ||x; - x11l S عع Vd.
We show that sign(N((x)) = sx,,(N(xx)). We have
-2
vd(1 -c c)
w. x/+b; = w. (x/-xx w, x, 2 - |/ - x:|++1
+1 2
vd(1 - c)
4
Also, for every i + we have
2
vd(11 - c)
wJ w = w (x x;) + J x, S ||,,|111x - x+1 vd(1 c)
4
2
Therefore, sign(N(xx))) sign( (v;) sign(y;) si,n(N(xx)).
It is not hard to show that the condition on the inner products in the above theorem holds w.h.p. when m (dly(d) and xj...., Xm are drawn from the uniform distribution on the sphere of radius vd. Indeed, the following lemma implies that in this case the inner products can be bounded by J and can even be bounded by Vdl log(d) (See Appendix B for the proof).
Lemma 3.1. Let Xi...., Xm. be i.i.d. such that x, U($a-1) for all i E [m]) where d for some constant k. Then, with probability at least 1 - d (2) (d-3)/2 1(x;, Xy)1 S 3 for all i j. Moreover, with probability least 1 - (2+11mm(d)/1 1 (1(1) have (xx,, x,>| < log(d) for all i + j. Vd
oa(1) we have

---

Gradient flow converges to non-robust networks
We now show that even though robust networks exist, gradient flow is biased towards non-robust networks. For homogeneous networks, Theorem 2.1 implies that gradient flow generally converges in direction to a KKT point Problem (2). Moreover, as discussed previously, the robustness of the network depends only the direction of the parameters vector. Thus, it suffices to show that every is non-robust. Formally, we have:
network that satisfies the KKT conditions of Problem 2
Theorem 4.1. Let {(x;, k)) S (Vl.a/-))x)) (-1,)) be training dataset. We denote I = [m]]
It == fiE 1 g: = 1} and I == ii E 1:3: -1,, and assume that min {
[rtl [ 2 cfr
some > O. Furthermore,
assume that m S 3(max.p3 [(xx)))++] Let No be a depth-2 ReLU
d-1
network such that is a KKT point of Problem 2 Then, there is a vector z = Eier yix, with
7 > 0 and ||]|| = (V
d
such that for every E I+ we have No(x: - =) I 1, and for every
i I- we have No + 21
Example Assume that (from the above theorem) a constant independent of d, m. Consider the following cases:
If maxizi [xx;xx)1 O(1) and |||| = O(1. Thus, in this case the data points are "almost orthogonal", and gradient flow converges to highly non-robust solutions, since even very small perturbations can flip the signs of the outputs for all examples in the dataset.
(dd) then the adversarial pertarbation z satisfies
If the inputs are drawn i.idd. from U(yd 1 then by Lemma 3.1 we have w.h.p. that max43 ((x, x1)1 = 00 (Vdlog(d)). and hence for m -
vd
the adversarial perturba-
(ogldd) tion z satisfies ||z|| = (VVallog(a) = = (22/1) o(v7).
Note that in the above cases the size of the adversarial perturbation is much smaller than vd. Also, by Theorem 3.1 and Lemma 3.1 there exist v /d-robust networks that classify the dataset correctly.
Thus, under the assumptions of Theorem 4. gradient flow converges non-robust networks,
when robust networks exist by Theorem 3.1 We discuss the proof ideas in Section 5 We note that Theorem 4.1 assumes that the dataset can be correctly classified by network No. which is indeed true (in fact, even by a width-2 network, since by assumption we have m S d). Moreover, we note that the assumption of the inputs coming from vd -dd-1 is mostly for technical convenience, and believe that it can be relaxed have all points approximately of the same norm (which would happen, e.g., if the inputs are sampled from a standard Gaussian distribution).
The result in Theorem 4.1 has several interesting properties:
It does require any assumptions on the width of the neural network.
It does not depend on the initialization, and holds whenever gradient flow converges to zero loss.
Note that if gradient flow converges to zero loss then by Theorem 2. converges in direction to KKT point of Problem (2) (regardless of the initialization of gradient flow) and hence the result holds.
It proves the existence of adversarial perturbations for every example in the dataset.
The same vector is used adversarial perturbation (up sign) for all examples. It corresponds to the well-known empirical phenomenon of universal adversarial perturbations, where one find single perturbation that simultancously flips the label of many inputs (cf.
Moosavi-Dezfooll et al. 2017]. Zhang et al. [2021)
The perturbation z depends only on the training dataset. Thus, for a given dataset, the same perturbation applies to all depth-2 networks which gradient flow might converge to. corresponds to the well-known empirical phenomenon of transferubility in adversarial examples, where one can find perturbations that simultaneously flip the labels of many different trained networks (cf.
Liu et al. 20166. Akhtar and Mian (018)).
5

---

E.g. we discussed in Example 1 if the data points orthogonal then need m ( and if they are random then we need m Ö(vd). Moreover, the conditions of the theorem do not allow datasets that contain clusters, where the inner products between data points are large, and do not allow data points which are not of norm vd. In the following corollary we extend Theorem 4.1 to allow for such scenarios. Here, the technical assumptions are only on the subset of data points that attain the margin. In particular, if this subset satisfies the assumptions, then the dataset may be arbitrarily large and may contain clusters and points with different norms.
limitation of Theorem 4.1 that it holds only for datasets of size m O mmxx xx,,,
Corollary 4.1. Let {(x, y)) C R {-1,1} be training dataset. Let No be depth-2 ReLU network such that 0 is a KKT point of Problem ( Let := {i E [n] NoNo(se (x:) = 1}.
I+ = {i E I V/ 1} and I == {i E yi -1}. Assume that for all E I we
have ||xi|| = Vd. Let = 17, and assume that min |rtl [ 2 f some > 0, and
that m 3(maxizjer (xxx))+++1** Then, there is vector z n Eier Vi4; with nj and
d+1
|z||| = =0 we have Nx. + 1
such that for every i E I+ we have No 2) f 1, and for every i E I-
The proof of the corollary can be easily obtained by slightly modifying the proof of Theorem (see Appendix for details). Note that in Theorem 4.1 the set of size m contains all the examples in the
dataset, and hence for all points in the dataset there are adversarial perturbations of size O
c.m
while in Corollary 4.1 the set / contains only the examples that attain exactly margin 1, the adversarial perturbations provably exist for examples in 1, and their size O
(V
d
depends on the size m
c
of L.
5
Proof sketch of Theorem 4.1
We now discuss the main ideas in the proof of Theorem 4.1 For the formal proof see Appendix D
5.1 A simple example
We start with simple example to gain some intuition. Consider dataset {(xi, Mi) ) such that for all i [d] we have x; vd e; where e1,. Od the standard unit vectors in Rd and d is even. Suppose that y, for < and y: -1 for i > 2.
First, consider the robust network N of width dl from Theorem 3.1 that correctly classifies the dataset. In the proof of Theorem 3.1 we constructed the network N(x) = Σj=1 (' x + by) such that for every j E [d] we have f/ = V/- Wj 2x, and by = 1. Note that we have y,NV = for all i E [d]. In this network, each input x, is in the active region (i.e., the region of inputs where the ReLU is active) of exactly one neuron, and has distance of yd from the active regions of the other 2
neurons. Hence, adding a perturbation smaller than sd to an input x, can affect only the contribution 2
of one neuron to the output, and will not flip the output's sign.
Now, we consider a network N²(x) Σÿ En= (w", x + b5), such that for all j E [d] we have v/ = Øý. w' 2 and b/, 0. Thus, the weights w', are in the same directions the weights w, of the network M, and in the network ' the bias terms equal O. is easy to verify that for all E [d] we have yxN"(x;) = 1. Since ||w]]l < ||w;i| and bb) < |by for all j E [d], then the network A'' is better than A in the sense of margin maximization. However, A' is much less robust than N. Indeed, note that in the network A each input x; is on the boundary the active regions of all neurons j * i, that is, for all j # i we have wJ x; + b/ = O. As result, perturbation can affect the contribution of all neurons to the output. Let i < d and consider adding to x, the perturbation z = $ Σ & +1 X,. Thus, z is spanned by all the inputs X, where y; = 11, and affects the (negative) contribution of the corresponding neurons. It is not hard to show that |z|| = 2v2 and (x + z) -1. Therefore,
2

---

is much less robust than N (which required perturbations of size Y Thus, bias towards margin maximization might have a negative effect on the robustness. Of course, this is just an example, and in the following subsection we provide a more formal overview of the proof.
5.2 Proof overview
We denote No(x)= - ESE( (w x x by). Thus, No a network width k. where the weights in the first layer are Wi...., Wk. the bias terms are bi....,bu dy. and the weights in the second layer 01,-.., U/- We assume that No satisfies the KKT conditions of Problem (2). We denote J := [R:] J
J+ = JJ: UU 2 0), and J == {j E J : Vy < 0). Note that since the dataset contains both examples with label and examples with label -1 then J+ and J are non-empty. For simplicity, we assume that luyl = 1 for all j E J. That is, vj = 1 for j E J and 0j = -1 for j J- We emphasize that we focus here on the case where |uyl = in order to simplify the description of the proof idea, and in the formal proof we do not have such an assumption. We denote
(x,, x-)|- Thus, by our assumption we have m S d+1 p = max
3(++1)
Since θ satisfies the KKT conditions (see Appendix A for the formal definition) of Problem ( then there are A....., Am such that for every E J we have
W Σ AvV (o.No(x))) = Σ Xinoooo,Ex.. RET
(3)
EET
where o' is a subgradient of at w X; + bj. i.., if J x; + bj + then d'. = sign(w (w] J x; + bj).
and otherwise o', is some value in [0, 1] (we note that in this case o', can be any value in [o, 1] and in our proof we do not have any further assumptions on it). Also we have A, 0 for all i, and X 0
(x,)) * 1. Likewise, we have
by = Σ Vo, (v.No(x:)) Exvrooo
iet
In the proof we use ( and (4) in order to show that No is non-robust. We focus here on the case where E I and we show that Nx z) S -1. The result for i E I can be obtained in similar manner. We denote x x,
The proof consists of three main components:
1. We show that (x,) = 1, namely, x, attains exactly margin 1. (x)
2. For every J have w x + by wJ x; + by. Since for j J have ", it implies that when moving from x, to x the non-negative contribution of the neurons in J to the output does not increase.
3. When moving from x; x" the total contribution of the neurons in J to the output (which is non-positive) decreases by at least 2.
Note that the combination of the above properties imply that Nx(xx) < -1 as required. In Ap-
pendix C we provide some more details on the main ideas for the proof of each part.
Experiments
We complement our theoretical results by an empirical study on the robustness of depth-2 ReLU networks trained on synthetically generated datasets. Theorem 4. shows that networks trained with gradient flow converge non-robust networks, but there are still a couple of questions remaining regarding the scope and limitations of this result. First, although the theorem limits the number of samples, we show here that the result applies also in cases when there are much more training samples. Second, the theorem does not depend on the width of the trained network, and we show that even when the size of the training set is much larger than the input dimension, the width of the network does not affect the size of the minimal perturbation that changes the label of the samples.
Experimental setting. In all of our experiments we trained a depth-2 fully-connected neural network with ReI.U activations using SGD with batch size of 5, 000. For experiments with less than 5, 000 samples, this equivalent to full batch gradient descent. We used the exponential loss, although we also tested on logistic loss and obtained similar results. Each experiment done using
7

---

0
Width: 100
30
4.5
Width: 500
Minimum perturbation size
Width: 1000
8.0
Width: 2000
20
2.5
Width: 4000
vd
Width:
3.0
Manimum
2.0
1.5
1.0
dimension
Input dimension
(a)
(b)
Figure 1: The effects of the width and the number of samples on the minimal perturbation size. The X-axis corresponds the input dimension. The y-axis corresponds the minimal perturbation size to change the labels of all the samples on the margin. We defined the perturbation direction as in Theorem
Σiez y,X;, where / is the set of samples that are on the margin. (a) The minimal perturbation size plotted for different sample sizes. Note that in all of the experiments, the minimal perturbation size is well beyond the plot of dd, for which the perturbation is not adversarial. (b) The minimal perturbation size for different widths of the network. Here, the number samples m
different random seeds, and we present the results in terms of the average and (when relevant) standard deviation over these runs. We used increasing learning rate accelerate the convergence to the KKT point (which theoretically is only reached at infinity). We began training with learning rate of 10-5 and increased it by factor of 1.1 every 100 iterations. We finished training after we achieved a loss smaller than 30, We emphasize that since we use an exponentially tailed loss, the gradients are extremely small at late stages of training, hence to achieve such small loss must use an increasing learning rate. implemented our experiments using Py PyTorch (Paszke et al. (019)). )
Dataset. In all of our experiments we sampled (x, y) R (-1,1} where x uuva (VVd-ad-1) and is uniform on {-1, 1}. We also tested on x sampled from Gaussian distribution with variance and obtained similar results. Here we only report the results on the uniform distribution.
Margin. In our experimental results we defined the margin in the following way: We train a network (x)) over datuset (X1, 311 Bi...... (Xm: Vm) Rd 1). Suppose that after training all the samples are classified correctly (this happened in all of our experiments), i.e. N(x))y/ >> argmii,,Wxxx, (x;)):- Finally,
We define mmrry
say that sample (x, is the margin if N(x:); I 1.1 (x )iinng. In words, we consider Xumg to be a sample which is exactly on the margin, but we also allow 10% slack for other samples to be on the margin. We must allow some slack, because in practice we cannot converge exactly to the KKT point, where all the samples on the margin have the exact same output.
6.1 Results
Minimum perturbation size. Figure aa) shows that the perturbation defined in Theorem 4.1 can change the labels of all the samples on the margin, even when there are much more samples than stated d samples, where
in Theorem 4.1 To this end, we trained our model and d E {100, 200,... ..., 1000). Note that Theorem 4.1 only considers the case of a = 0.5 for data which is uniformly distributed. The width of the network is 1, 000. After training is completed, considered perturbations in the direction z = Σig Y;X,, where represents the set of samples that are the margin. The y-axis represents the minimal > 0 such that for all
0.5, 0.8, 1.2, 1.5
we have that
N x; y:C y: < O. In words, we plot the minimal size of the perturbation which changes the labels of all the samples on the margin. We emphasize that we used the same perturbation for all the samples.

---

1.000
m des
samples margin
Number of samples outside margin
300
0.975
0.950
250
d
= d
200
0.900
150
6,875
g0s
de0
zd12
0.025
0.800
dimension
Input dimension
(a)
(b)
Figure 2: The number of samples on the margin. The x-axis is the input dimension. Each line plot represents different number of samples, scaling with the input dimension. (a) The ratio of the number of samples on the margin out of the total number of samples. (b) The number of samples not on the margin.
We also plot vd, as perturbation above this line can trivially change the labels of all points. Recall that by Theorem 3.1 there exists v d-robust network (if the width of the network is at least the size of the datasct). From Figure [a). is clear that the minimal perturbation size much smaller than /d. We also plot the standard deviation over the 5 different random seeds, showing that our results are consistent.
It is also important to understand how many samples lie the margin, since our perturbation changes the label of these samples. Figure plots the ratio of samples on the margin out of the total number of samples, and Figure ((b) plots the number of samples not the margin. These plots correspond to the same experiments as in Figure Ta). where the number of samples depends on the input dimension. For m d.. 5 all the samples are the margin, as was proven in Lemma D.1 For = 0.8, 1, 1.2, 1.5, it can be seen that at least 92% of the samples are on the margin. Together with aa). shows that a single perturbation with small magnitude can change the label of almost all the samples. We remind that this happens when we sample from the uniform distribution, and not when there a cluster structure as discussed before Corollary 4.1
d where
Effect of the width. Figure [bb shows that the width of the network does not have significant effect on its robustness. The y-axis is the same as Figure a and in all the experiments the number of samples is equal to the input dimension (i.e. width varying from 100 to 8000. The minimal perturbation size all the experiments is almost
d). We tested neural networks with
same, regardless of the width. This finding matches the result from Theorem 4.1 but for datasets much larger than the bound from our theory.
7 Discussion
In this paper, we showed that in depth-2 ReLU networks gradient flow is biased towards non-robust solutions even when robust solutions exist. To that end, we utilized prior results on the implicit bias of homogeneous models with exponentially-taile losses. While the cxisting works on implicit bias mainly in the context of generalization, in this work make first step towards understanding its implications on robustness. We believe that the phenomenon adversarial examples an implication of the implicit bias in neural networks, and hence the relationship between these two topics should be further studied. Thus, understanding the implicit bias may be key to explaining the existence of adversarial examples.
note that we show non-robustness w.r.t. the training dataset rather than on test data. Intuitively, achieving robustness on the training set should be easier than test data, and we show a negative result already for the former. We believe that extending our approach to robustness on data an interesting direction for future research.

---

There are some additional important open questions that naturally arise from our results, First, our theoretical negative results assume that the size of the dataset (or the size of the subset of examples
that attain the margin) is upper bounded by O msxigt (xxx,))] This assumption is required in proof for technical reasons, but we conjecture that it can be significantly relaxed, and our experiments support this conjecture. Another natural question is extend our results to more architectures, such as deeper ReLU networks. Finally, it would be interesting to study whether other optimization methods have different implicit bias which directs toward robust networks.
Acknowledgments and Disclosure of Funding
This research is supported in part by European Research Council (ERC) grant 754705.
References
N. Akhtar and A. Mian. Threat of adversarial attacks deep learning in computer vision: A survey. leee Access, 6: 6:44400-14430, 2018
Z. Allen-Zhh Zhu and Y. Li. Feature purification: How adversarial training performs robust deep learning. arXiv preprint A0t::22005.019900 2020.
A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circum- venting defenses to adversarial examples. In International conference machine learning, pages 274-283. PMLR, 2018.
P. Bartlett, S. Bubeck, and Y. Cherapanamjeri. Adversarial examples in multi- random relu networks. Advances in Neural Information Processing Systems, 34, 2021.
S. Bubeck, Y. T. Lee, E. Price, and I. Razenshteyn. Adversarial examples from computational constraints. In International Conference on Machine Learning, pages 831-840. PMLR, 2019.
Bubeck, Y. Cherapanamjeri, G. Gidel, and R. Combes. single gradient step finds adversarial examples random two-layers neural networks. Advances in Neural Information Processing Systems, 34, 2021.
N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 3-14, 2017.
N. Carlini and D. Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE Security and Privacy Workshops (SPW), pages 1-7. IEEE, 2018
F. H. Clarke, Y. S. Ledyaev, R. J. Stern, and P. R. Wolenski. Nonsmooth analysis and control theory, volume 178. Springer Science & Business Media, 2008
Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free free attacks. In International conference on machine learning, pages 2020.
PMLR,
A. Daniely and H. Shacham. Most relu networks suffer from lg adversarial perturbations. Advances in Neural Information Processing Systems, 33, 2020.
J. Dutta, K. Deb, R. Tulshyan, and R. Arora. Approximate kkt points and a proximity measure for termination. Journal of Global Optimization, 56(4): 2013.
K. W. Fang. Symmetric multivariate and related distributions. Chapman and Hall/CRC, 2018.
A. Fawzi, H. Fawzi, and O. Fawzi. Adversarial vulnerability for any classifier. arXiv preprint arXiv: 1802.80088888 2018.
S. Ge, Singla, R. Basri, and D. Jacobs. Shift invariance can reduce adversarial robustness. arXiv preprint arXiv:2103.022 02699, 2021.
10

---

J. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. arXiv preprint arXiv:2006.066.57 2020.
M. Khoury and D. Madfield-Menell On the geometry of adversarial examples. arXiv preprint arXiv:1811.00525, 2018.
Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint 161 2016.
K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint 1906. X1v:00900005890, 2019.
A. Madry, A. Makelov, Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0000 06083, 2017.
S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE sympostum security and privacy (SP), pages 582-597. IEEE, 2016.
N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-box attacks against machine learning. Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high- perforrrffreeec deep learning library. Advances in neural information processing systems, 32:8026-800 2019.
L. Schmidt, Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust generalization requires more data. arXiv preprint arXiv: 2018.
A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? arXiv preprint arviv: 1809.02000 2018
H. Shah, K. Tamuly, A. Raghunathan. P. Jain, and P. Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint wwXiv:2006.07710 2020.
A. Shamir, I. Safran, E. Ronen, and O. Dunkelman. A simple explanation for the existence of adversarial examples with small hamming distance. arXiv preprint 1rXiv:1901.10861, 2019.
A. Shamir, O. Melamed, and O. BenShmuel. The dimpled manifold model of adversarial examples in machine learning, arXiv preprint arXiv:2106.100 10151, 2021.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13 12.6119, 2013.
G. Vardi. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208. 12591, 2022.
G. Vardi, O. Shamir, and N. Srebro. On margin maximization in linear and relu networks. arXiv preprint arXiv:2/ 00.02732, 2021.
H. Wang, X. Wu, Z. Huang, and E. P. Xing. High-frequency component helps explain the generaliza- tion of convolutional neural networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9684-8694, 2020.
E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. International Conference on Machine Learning, pages 5286-5295. PMLR, 2018.
11

---

Z. Wu, S.-N. Lim, L. S. Davis, and T. Goldstein. Making an invisibility cloak: Real world adversarial attacks on object detectors. European Conference on Computer Vision, pages 1-17. Springer, 2020.
C. Zhang, P. Benz, C. Lin, A. Karjauv, J. Wu, and I. S. Kweon. A survey on universal adversarial attack. arXiv preprint arXiv:1103.01498, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [N/A] We haven't 't included the code, since we only ran simple simulations, but provided the details about them for reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A] We did not use significant computing resources (such as GPUs, clusters or cloud services).
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "kTJJgbenZui", "reviewer": "Reviewer_GV2f", "summary": "This paper studied the adversarial examples from a theoretical perspective. Specifically, the authors considered a depth-2 ReLU homogeneous neural network trained with gradient flow, and then showed that it is non-robust (under $l_2$ adversarial perturbations), even when a robust neural network does exist. ", "questions": "1. The authors showed that when training data points are neither too many (compared with the dimension) nor too correlated, the gradient flow converges to non-robust solutions in spite of robust ones existing. What if there are a lot of data points and many of them are correlated? Will the result be the same or do adversarial examples only exist for some training data points?\n\n2. In the paper, the authors assumed all the input features have the same norm, however, it may not hold in the real application, for instance, different images have different norms. Thus, I was wondering how this assumption will affect the results. \n\n3. In Figure 2 (a), the purple line with the most data points, why is the ratio of sample on the margin higher than the green and red lines?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The authors used a depth-2 ReLU neural network to provide a theoretical explanation for the existence of adversarial examples in the trained network. \n\n2. The authors conducted some numerical experiments, and their results supported the theoretical results.\n\n3. The result of the paper is very important, which explains that even with a well-trained neural network, there always exist some adversarial examples to fool it.\n\nWeakness:\n\nThe assumption for the theoretical results is quite strong in the big data era, for example, in Theorem 4.1, the data size is even smaller than the dimension; in Corollary 4.1, just curious, can we always get such a network $N_\\theta$ with KKT point and the data set $I:=  {i\\in [n]: y_iN_{\\theta}(x_i) = 1\\}$ is non-empty. Maybe the authors should elaborate on this a bit more.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "fABUXfrTO3B", "reviewer": "Reviewer_EoRx", "summary": "The paper provides a theoretical construction in which a one hidden-layer ReLU network trained with gradient flow (under some assumptions) provably converges to a non-robust network, while at the same time a much more robust network exists. Robustness here is meant in the sense of the minimum L2 perturbations and is studied only on the training set (but this is sufficient to illustrate the point). The technical tools rely on the results from the previous works about convergence to the max-margin classifier in the parameter space which, as the authors show, can lead to the lack of robustness. This shows an interesting discrepancy between the max-margin classifier in linear models vs. one hidden-layer networks.", "questions": "I agree with the main point of the paper, but I wonder if the proposed theoretical constructions reflect the observed phenomena happening in practice, let’s say, for image classification. I wonder if the authors could provide some further comments on that. Some elements of the theoretical setup seem a bit artificial. For example, for a robust network in Theorem 3.1, one selects a separate weight vector for each input point which leads to only one active ReLU unit which appears to be an extreme case of memorization of the training data. Also, it seems to me that the proposed theorems (perhaps except Corollary 4.1) don’t take into account any structure of the data in the sense that they allow an arbitrary labeling of the input points on the sphere. This makes me think that the construction may be a bit artificial and not necessarily reflect some standard setting relevant to the application of neural networks in practice. It would be great to have further comments on that, in my opinion.\n", "limitations": "Perhaps the only major limitation of the analysis presented in this paper is the set of assumptions on the training data. But the main limitations are discussed quite clearly throughout the text.\n\n\n---\n\n**Update after the rebuttal.**\n*I don't have any significant concerns about this paper and the questions I had were more of a speculative nature. I'm happy about the paper and the rebuttal. I think the paper is worth to be published at NeurIPS.*", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality.** The paper presents an interesting and novel analysis for an important problem of understanding why standard gradient-based training can lead to non-robust models, although robust models provably exist. The paper nicely points out that margin maximization in the parameter space is not aligned in general with the L2 robustness. Moreover, I think the paper is an interesting contribution towards the discussion of adversarial examples being “bugs vs. features” (https://arxiv.org/abs/1905.02175) where the authors of the current paper give theoretical evidence that adversarial examples can also arise as “bugs” from the particular optimization algorithm we use for training.\n- **Quality.** High, the theory is sound, illustrated with simple-to-grasp examples, and the overall story is consistent and clear.\n- **Clarity.** The paper is well-written.\n- **Significance.** The previous works (Daniely and Shacham [2020], Bubeck et al. [2021], Bartlett et al. [2021]) mostly focused on analyzing the abundance of adversarial examples for random networks. However, as the authors point out, networks trained with GD are clearly not random and understanding the implicit bias of GD on robustness is thus key. Some assumptions on the input data may seem a bit restrictive (the input points are on a sphere, there is a low upper bound on the number of training points which are allowed, the network width has to be at least equal to the number of training points for the robust network) and perhaps are worth further commenting on. However, I think the paper advances our current theoretical understanding of adversarial vulnerability via the presented constructions, and it’s important that the proposed theoretical setup is well-documented and published. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_ADmKzkeYWG", "reviewer": "Reviewer_ysvx", "summary": "The paper theoretically shows that for a 2-layer ReLU network trained by gradient methods, it provably converges to non-robust networks (regards to l2 perturbation) even when a robust network exits. They support their theoretical findings by experiments with a 2-layer fully connected network with ReLU activation on synthetically generated datasets.\n", "questions": "1. Will it be easy to extend the proof to other kinds of robustness, for example,  $L_{\\inf}$ robustness? \n\n2. How is the magnitude of the perturbations used in the theoretical analysis compared to those used in the practical settings in Cifar or ImageNet? \n", "limitations": "The authors fairly addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper mainly focuses on the theoretical analysis of why adversarial examples exist. They prove that given a two-way classification problem, a robust 2-layer ReLU network exists, but gradient flow converges to non-robust networks. The paper is interesting to read although I did not read every line of the proof, and the claim that converging to non-robust networks is the implicit bias seems interesting and novel to me. The properties of the theorem are interesting and consistent with the existing findings. \n\nIt will be even more interesting if it can be extended to more realistic or practical settings, i.e., more complicated networks and robustness other than l2 distance. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WOCiqDw9nCi", "reviewer": "Reviewer_51Wy", "summary": "This paper firstly shows that in depth-2 ReLU networks gradient flow is biased towards non-robust solutions even when robust solutions exist theoretically, and then verifies it experimentally on synthetically generated datasets.", "questions": "There is perhaps a minor problem in the statement of Theorem 3.1. The proof uses the condition $m \\le k$ but it do not appear in the statement of the theory (Usually, we regard the architectures as fixed before learning, so I think it is better to fix $k$ at first and state that $m \\le k$).", "limitations": "The paper stated the limitations about the theory, such as the limitations of the training set size, the results are just guaranteed for the training set, and so on.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper shows a novel result that in depth-2 ReLU networks gradient flow is biased towards non-robust solutions even when robust solutions exist theoretically. And then further verifies it in the experiments.\n- This paper gives some intuitive examples that help understanding the theorys and ideas to prove the theory.\n- This paper expresses its' ideas clearly.\n- This paper connects the theoretical results to some empirical phenomenons (line 198-206), although the empirical phenomenons are usually discussed for the test set and the theory in this paper is for the training set.\n\nWeaknesses:\n- This main contribution of this paper is just Theorem 4.1 and a naive result Theorem 3.1 and simulation experiments, and the models and datas is somehow simple, I think the amount of work is not so enough in some sense, but the phenomenon it reveals is quite interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept