Title: The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial robustness

neural networks

implicit bias

generalization
Abstract: In this work, we study the implications of the implicit bias of gradient flow on generalization and adversarial robustness in ReLU networks.  We focus on a setting where the data consists of clusters and the correlations between cluster means are small, and show that in two-layer ReLU networks gradient flow is biased towards solutions that generalize well, but are vulnerable to adversarial examples. Our results hold even in cases where the network is highly overparameterized.  Despite the potential for harmful overfitting in such settings, we prove that the implicit bias of gradient flow prevents it.  However, the implicit bias also leads to non-robust solutions (susceptible to small adversarial $\ell_2$-perturbations), even though robust networks that fit the data exist.

The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks

Abstract

In this work, we study the implications of the implicit bias of gradient flow on generalization and adversarial robustness in ReLU networks. We focus on a setting where the data consists of clusters and the correlations between cluster means are small, and show that in two-layer ReLU networks gradient flow is biased towards solutions that generalize well, but are vulnerable to adversarial examples. Our results hold even in cases where the network is highly overparameterized. Despite the potential for harmful overfitting in such settings, we prove that the implicit bias of gradient flow prevents it. However, the implicit bias also leads to non-robust solutions (susceptible to small adversarial ℓ 2 -perturbations), even though robust networks that fit the data exist.

1 Introduction

A central question in the theory of deep learning is how neural networks can generalize even when trained without any explicit regularization, and when there are more learnable parameters than training examples. In such optimization problems there are many solutions that label the training data correctly, and gradient descent seems to prefer solutions that generalize well [Zha+17]. Thus, it is believed that gradient descent induces an implicit bias towards solutions which enjoy favorable properties [Ney+17]. Characterizing this bias in various settings has been a subject of extensive research in recent years, but it is still not well understood when the implicit bias provably implies generalization in non-linear neural networks.

An additional intriguing phenomenon in deep learning is the abundance of adversarial examples in trained neural networks. In a seminal paper, Szegedy et al. [Sze+14] observed that deep networks are extremely vulnerable to adversarial examples, namely, very small perturbations to the inputs can significantly change the predictions. This phenomenon has attracted considerable interest, and various attacks (e.g., [GSS15; CW17; Pap+17; ACW18; CW18; Wu+20]) and defenses (e.g., [Pap+16; KGB17; Mad+18; WK18; CH20; WRK20]) were developed. However, the fundamental principles underlying the existence of adversarial examples are still unclear, and it is believed that for most tasks where trained neural networks suffer from a vulnerability to adversarial attacks, there should exist other neural networks which can be robust to such attacks. This is suggestive of the possible role of the optimization algorithms used to train neural networks in the existence of adversarial examples.

In this work, we study the implications of the implicit bias for generalization and robustness in ReLU networks, in a setting where the data consists of clusters (i.e., Gaussian mixture model) and the

correlations between cluster means are small. We show that in two-layer ReLU networks trained with the logistic loss or the exponential loss, gradient flow is biased towards solutions that generalize well, albeit they are non-robust. Our results are independent of the network width, and hence they hold even where the network has significantly more parameters than training examples. In such an overparameterized setting, one might expect harmful overfitting to occur, but we prove that the implicit bias of gradient flow prevents it. On the flip side, in our setting the distances between clusters are large, and thus one might hope that gradient flow will converge to a robust network. However, we show that the implicit bias leads to non-robust solutions.

Our results rely on known properties of the implicit bias in two-layer ReLU networks trained with the logistic or the exponential loss, which were shown by Lyu and Li [LL20] and Ji and Telgarsky [JT20]. They proved that if gradient flow in homogeneous models (which include two-layer ReLU networks) with such losses reaches a small training loss, then it converges (in direction) to a KKT point of the maximum-margin problem in parameter space. We show that in clustered data distributions, with high probability over the training dataset, every network that satisfies the KKT conditions of the maximum-margin problem generalizes well but is non-robust. Thus, instead of analyzing the trajectory of gradient flow directly in the complex setting of training two-layer ReLU networks, we demonstrate that investigating the KKT points is a powerful tool for understanding generalization and robustness. We emphasize that our results hold in the rich (i.e., feature learning ) regime, namely, the neural network training does not lie in the kernel regime, and thus we provide guarantees which go beyond the analysis achieved using NTK-based results.

In a bit more detail, our main contributions are the following:

Suppose that the data distribution consists of k clusters, and the training dataset is of size n ≥ ˜ Ω( k ) . We show that with high probability over the sizen dataset, if gradient flow achieves training loss smaller than 1 n at some time t 0 , then it converges in direction to a network that generalizes well (i.e., has a small test error). Thus, gradient-flow-trained networks cannot harmfully overfit even if the network is highly overparameterized. The sample complexity ˜ Ω( k ) in this result is optimal (up to log factors), since we cannot expect to perform well on unseen data using a training dataset that does not include at least one example from each cluster.

In the same setting as above, we prove that gradient flow converges in direction to a nonrobust network, even though there exist robust networks that classify the data correctly. Specifically, we consider data distributions on R d such that the distance between every pair of clusters is Ω( √ d ) , and we show that there exists a two-layer ReLU network where flipping the output sign of a test example requires w.h.p. an ℓ 2 -perturbation of size Ω( √ d ) , but gradient flow converges to a network where we can flip the output sign of a test example with an ℓ 2 -perturbation of size much smaller than √ d . Moreover, the adversarial perturbation depends only on the data distribution, and not on the specific test example or trained neural network. Thus, the perturbation is both universal [Moo+17; Zha+21] and transferable [Liu+17; AM18]. We argue that clustered data distributions are a natural setting for analyzing the tendency of gradient methods to converge to non-robust solutions. Indeed, if positive and negative examples are not well-separated (i.e., the distances between points with opposite labels are small), then robust solutions do not exist. Thus, in order to understand the role of the optimization algorithm, we need a setting with sufficient separation between positive and negative examples.

The remainder of this paper is structured as follows: Below we discuss related work. In Section 2 we provide necessary notations and background, and introduce our setting and assumptions. In Sections 3 and 4 we state our main results on generalization and robustness (respectively), and provide some proof ideas, with all formal proofs deferred to the appendix. We conclude with a short discussion (Section 5).

Related work

Implicit bias in neural networks. The literature on implicit bias in neural networks has rapidly expanded in recent years, and cannot be reasonably surveyed here (see Vardi [Var22] for a survey). In what follows, we discuss results that apply to two-layer ReLU or leaky-ReLU networks trained with gradient flow in classification settings.

By Lyu and Li [LL20] and Ji and Telgarsky [JT20], homogeneous neural networks (and specifically two-layer ReLU networks, which are the focus of this paper) trained with exponentially-tailed classification losses converge in direction to a KKT point of the maximum-margin problem. Our analysis of the implicit bias relies on this result. We note that the aforementioned KKT point may not be a global optimum of the maximum-margin problem [VSS22]. Recently, Kunin et al. [Kun+22] extended this result by showing bias towards margin maximization in a broader family of networks called quasi-homogeneous .

Lyu et al. [Lyu+21], Sarussi, Brutzkus, and Globerson [SBG21], and Frei et al. [Fre+23b] studied implicit bias in two-layer leaky-ReLU networks with linearly-separable data, and proved that under some additional assumptions gradient flow converges to a linear classifier. Chizat and Bach [CB20] studied the dynamics of gradient flow on infinite-width homogeneous two-layer networks with exponentially-tailed losses, and showed bias towards margin maximization w.r.t. a certain function norm known as the variation norm. Phuong and Lampert [PL20] studied the implicit bias in two-layer ReLU networks trained on orthogonally separable data .

Safran, Vardi, and Lee [SVL22] proved implicit bias towards minimizing the number of linear regions in univariate two-layer ReLU networks, and used this result to obtain generalization bounds. Similarly to our work, they used the KKT conditions of the maximum-margin problem in parameter space to prove generalization in overparameterized networks. However, our setting is significantly different. Implications of the bias towards KKT points of the maximum-margin problem were also studied in Haim et al. [Hai+22], where they showed that this implicit bias can be used for reconstructing training data from trained ReLU networks.

Theoretical explanations for non-robustness in neural networks. Despite much research, the reasons for the abundance of adversarial examples in trained networks are still unclear [GSS15; FFF18; Sha+19; Sch+18; KH18; Bub+19; AL21; Wan+20; Sha+20; SMB21; Sin+21; Wan+22; DB22]. Below we discuss several prior theoretical works on this question.

In one line of work, it has been shown that small adversarial perturbations can be found for any fixed input in certain neural networks with random weights (drawn from the Gaussian distribution) [DS20; Bub+21; BBC21; MW22]. These works differ in the assumptions about the width and depth of the networks as well as the activation functions considered. However, since trained networks are non-random, these works are unable to capture the existence of adversarial examples in trained networks.

The result closest to ours was shown in Vardi, Yehudai, and Shamir [VYS22]. Similarly to our result, they used the KKT conditions of the maximum-margin problem in parameter space, in order to prove that gradient flow converges to non-robust two-layer ReLU networks under certain assumptions. More precisely, they considered a setting where the training dataset S consists of nearly-orthogonal points, and proved that every KKT point is non-robust w.r.t. S . Namely, for every two-layer network that satisfies the KKT conditions of the maximum-margin problem, and every point x i from S , it is possible to flip the output's sign with a small perturbation. Their result has two main limitations: (1) It considers robustness w.r.t. the training data, while the more common setting in the literature considers robustness w.r.t. test data, as it is often more crucial to avoid adversarial perturbations in test examples; (2) Since they assume near orthogonality of the training data, the size of the dataset S must be smaller than the input dimension. 2 Thus, they considered a high dimensional setting. We note that high-dimensional settings often have a different generalization behavior than low-dimensional settings (e.g., overfitting can be benign in the high-dimensional setting, but harmful in a low-dimensional setting [KYS23]). Our result does not suffer from these limitations, since we consider robustness w.r.t. test data, and the size of our training dataset might be very large. In our results, we essentially require near orthogonality of the cluster means, as opposed to near orthogonality of the training dataset in their result.

Finally, in Bubeck, Li, and Nagaraj [BLN21] and Bubeck and Sellke [BS21], the authors proved (under certain assumptions) that overparameterization is necessary if one wants to interpolate training data using a neural network with a small Lipschitz constant. Namely, neural networks with a small number of parameters are not expressive enough to interpolate the training data while having a small

Lipschitz constant. These results suggest that overparameterization might be necessary for robustness. In this work, we show that even if the network is highly overparameterized, the implicit bias of the optimization method can prevent convergence to robust solutions.

2 Preliminaries

We use bold-face letters to denote vectors, e.g., x = ( x 1 , . . . , x d ) . For x ∈ R d we denote by ∥ x ∥ the Euclidean norm. We denote by ✶ [ · ] the indicator function, for example ✶ [ t ≥ 5] equals 1 if t ≥ 5 and 0 otherwise. We denote sign( z ) = 1 if z > 0 and -1 otherwise. For an integer d ≥ 1 we denote [ d ] = { 1 , . . . , d } . For a set A we denote by U ( A ) the uniform distribution over A . We denote by N ( µ, σ 2 ) the normal distribution with mean µ ∈ R and variance σ 2 , and by N ( µ , Σ) the multivariate normal distribution with mean µ and covariance matrix Σ . The identity matrix of size d is denoted by I d . We use standard asymptotic notation O ( · ) and Ω( · ) to hide constant factors, and ˜ O ( · ) , ˜ Ω( · ) to hide logarithmic factors. We use log for the logarithm with base 2 and ln for the natural logarithm.

In this work, we consider depth2 ReLU neural networks. The ReLU activation function is defined by ϕ ( z ) = max { 0 , z } . Formally, a depth2 network N θ of width m is parameterized by θ = [ w 1 , . . . , w m , b , v ] where w i ∈ R d for all i ∈ [ m ] and b , v ∈ R m , and for every input x ∈ R d we have

We sometimes view θ as the vector obtained by concatenating the vectors w 1 , . . . , w m , b , v . Thus, ∥ θ ∥ denotes the ℓ 2 norm of the vector θ . We note that in this work we train both layers of the ReLU network.

We denote Φ( θ ; x ) := N θ ( x ) . We say that a network is homogeneous if there exists L > 0 such that for every α > 0 and θ , x we have Φ( α θ ; x ) = α L Φ( θ ; x ) . Note that depth2 ReLU networks as defined above are homogeneous (with L = 2 ).

We next define gradient flow and remind the reader of some recent results on the implicit bias of gradient flow in two-layer ReLU networks. Let S = { ( x i , y i ) } n i =1 ⊆ R d × {-1 , 1 } be a binary classification training dataset. Let Φ( θ ; · ) : R d → R be a neural network parameterized by θ . For a loss function ℓ : R → R the empirical loss of Φ( θ ; · ) on the dataset S is

We focus on the exponential loss ℓ ( q ) = e -q and the logistic loss ℓ ( q ) = log(1 + e -q ) .

We consider gradient flow on the objective given in Eq. (1). This setting captures the behavior of gradient descent with an infinitesimally small step size. Let θ ( t ) be the trajectory of gradient flow. Starting from an initial point θ (0) , the dynamics of θ ( t ) is given by the differential equation d θ ( t ) dt ∈ -∂ ◦ L ( θ ( t )) . Here, ∂ ◦ denotes the Clarke subdifferential [Cla+08], which is a generalization of the derivative for non-differentiable functions.

We now remind the reader of a recent result concerning the implicit bias of gradient flow over the exponential and logistic losses for homogeneous neural networks. Note that since homogeneous networks satisfy sign(Φ( α θ ; x )) = sign(Φ( θ ; x )) for any α > 0 , the sign of the network output of homogeneous networks depends only on the direction of the parameters θ . The following theorem provides a characterization of the implicit bias of gradient flow by showing that the trajectory of the weights θ ( t ) converge in direction to a first-order stationary point of a particular constrained optimization problem, where θ converges in direction to ˜ θ means lim t →∞ θ ( t ) ∥ θ ( t ) ∥ = ˜ θ ∥ ˜ θ ∥ . Note that since ReLU networks are non-smooth, the first-order stationarity conditions (i.e., the Karush-KuhnTucker conditions, or KKT conditions for short) are defined using the Clarke subdifferential (see Lyu and Li [LL20] and Dutta et al. [Dut+13] for more details on the KKT conditions in non-smooth optimization problems).

Theorem 2.1 (Paraphrased from Lyu and Li [LL20] and Ji and Telgarsky [JT20]) . Let Φ( θ ; · ) be a homogeneous ReLU neural network parameterized by θ . Consider minimizing either the exponential or the logistic loss over a binary classification dataset { ( x i , y i ) } n i =1 using gradient flow. Assume that there exists time t 0 such that L ( θ ( t 0 )) < 1 n (and thus y i Φ( θ ( t 0 ); x i ) > 0 for every x i ). Then, gradient flow converges in direction to a first-order stationary point (KKT point) of the following

maximum margin problem in parameter space:

Theorem 2.1 gives a characterization of the implicit bias of gradient flow with the exponential and the logistic loss for homogeneous ReLU networks. Note that the theorem makes no assumption on the initialization, training data, or number of parameters in the network; the only requirement is that the network is homogeneous and that at some time point in the gradient flow trajectory, the network is able to achieve small training loss. The theorem shows that although there are many ways to configure the network parameters to achieve small training loss (via overparameterization), gradient flow only converges (in direction) to networks which satisfy the KKT conditions of Problem (2). It is important to note that satisfaction of the KKT conditions is not sufficient for global optimality of the constrained optimization problem [VSS22]. We further note that if the training data are sampled i.i.d. from a distribution with label noise (e.g., a class-conditional Gaussian mixture model, or a distribution where labels y i are flipped to -y i with some nonzero probability), networks which have parameters that are feasible w.r.t. the constraints of Problem (2) have overfit to noise, and understanding the generalization behavior of even globally optimal solutions to Problem (2) in this setting is the subject of significant research [Mon+19; CL21; Fre+23a].

̸

Finally, we introduce the distributional setting that we consider. We consider a distribution D clusters on R d ×{-1 , 1 } that consists of k clusters with means µ (1) , . . . , µ ( k ) ∈ R d and covariance σ 2 I d (i.e., a Gaussian mixture model), such that the examples in the j -th cluster are labeled by y ( j ) ∈ {-1 , 1 } . More formally, ( x , y ) ∼ D clusters is generated as follows: we draw j ∼ U ([ k ]) and x ∼ N ( µ ( j ) , σ 2 I d ) , and set y = y ( j ) . We assume that there exist i, j ∈ [ k ] with y ( i ) = y ( j ) . Moreover, we assume the following:

Assumption 2.2. We have:

∥ ∥ µ ( j ) ∥ ∥ = √ d for all j ∈ [ k ] .

0 < σ ≤ 1 .

̸

Example 1. Below we provide simple examples of settings that satisfy the assumption:

√

Suppose that the cluster means satisfy |⟨ µ ( i ) , µ ( j ) ⟩| = ˜ O ( d ) for every i = j . This condition holds, e.g., if we choose each cluster mean i.i.d. from the uniform distribution on the sphere √ d · S d -1 (see, e.g., Vardi, Yehudai, and Shamir [VYS22, Lemma 3.1]). Let σ = 1 , namely, each cluster has a radius of roughly √ d . Then, the assumption can be satisfied by choosing k = ˜ O ( √ d ) .

̸

̸

Suppose that the cluster means are exactly orthogonal (i.e., ⟨ µ ( i ) , µ ( j ) ⟩ = 0 for all i = j ), and σ = 1 / √ d . Then, the assumption can be satisfied by choosing k = ˜ O ( d ) .

̸

If the number of clusters is k = ˜ O (1) , then the assumption may hold even where max i = j |⟨ µ ( i ) , µ ( j ) ⟩| = ˜ Θ( d ) (for any 0 < σ ≤ 1) .

√

A few remarks are in order. First, the assumption that ∥ ∥ µ ( j ) ∥ ∥ is exactly d is for convenience, and we note that it may be relaxed (to have all cluster means approximately of the same norm) without affecting our results significantly. Note that in the case where σ = 1 , the radius of each cluster is roughly of the same magnitude as the cluster mean. Second, we assume for convenience that the noise (i.e., the deviation from the cluster's mean) is drawn from a Gaussian distribution with covariance matrix σ 2 I d . However, we note that this assumption can be generalized to any distribution D noise such that for every unit vector e the noise ξ ∼ D noise satisfies w.h.p. that ⟨ ξ , e ⟩ = ˜ O (1) and ∥ ξ ∥ = ˜ O ( √ d ) . This property holds, e.g., for a d -dimensional Gaussian distribution N (0 , Σ) , where tr[Σ] = d and ∥ Σ ∥ 2 = O (1) (see Frei et al. [Fre+23b, Lemma 3.3]), and more generally for a class of sub-Gaussian distributions (see Hu et al. [Hu+20, Claim 3.1]). Third, note that the third part of Assumption 2.2 essentially requires that the number of clusters k cannot be too large

̸

and the correlations between cluster means cannot be too large. Finally, we remark that when k is small, our results may be extended to the case where σ > 1 . For example, if k = ˜ O (1) and max i = j |⟨ µ ( i ) , µ ( j ) ⟩| = ˜ O ( √ d ) , our generalization result (Theorem 3.1) can be extended to the case where σ = ˜ O ( d 1 / 8 ) . We preferred to avoid handling σ > 1 in order to simplify the proofs.

Moreover, it is worth noting that Assumption 2.2 implies that the data is w.h.p. linearly separable (see Lemma 2.1 below, and a proof in Appendix B). However, in this work we consider learning using overparameterized ReLU networks, and it is not obvious a priori that gradient methods do not harmfully overfit in this case. Indeed, it has been shown that ReLU networks trained by gradient descent can interpolate training data and fail to generalize well in some distributional settings [Kou+23].

Lemma 2.1. Let u = ∑ q ∈ [ k ] y ( q ) µ ( q ) . Then, with probability at least 1 -2 d 1 -ln( d ) / 2 = 1 -o d (1) over ( x , y ) ∼ D clusters, we have y = sign( u ⊤ x ) .

3 Generalization

In this section, we show that under our assumptions on the distribution D clusters, gradient flow does not harmfully overfit. Namely, even if the learned network is highly overparameterized, the implicit bias of gradient flow guarantees convergence to a solution that generalizes well. Moreover, we show that the sample complexity is optimal. The main result of this section is stated in the following theorem:

Theorem 3.1. Let ϵ, δ ∈ (0 , 1) . Let S = { ( x i , y i ) } n i =1 ⊆ R d ×{-1 , 1 } be a training set drawn i.i.d. from the distribution D clusters, where n ≥ k ln 2 ( d ) . Let N θ be a depth2 ReLU network such that θ = [ w 1 , . . . , w m , b , v ] is a KKT point of Problem (2). Provided d is sufficiently large such that δ -1 ≤ 1 3 d ln( d ) -1 and n ≤ min {√ δ 3 · e d/ 32 , √ δ 3 · d ln( d ) / 4 , ϵ 4 · d ln( d ) / 2 } , then with probability at least 1 -δ over S , we have

The sample complexity requirement in Theorem 3.1 is n = ˜ Ω( k ) . Essentially, it requires that the dataset S will include at least one example from each cluster. Clearly, any learning algorithm cannot perform well on unseen clusters. Hence the sample complexity requirement in the theorem is tight (up to log factors).

The assumptions in Theorem 3.1 include upper bounds on δ -1 and n . Note that the expressions in these upper bounds are super-polynomial in d , and in particular if n, δ -1 , ϵ -1 = poly( d ) , then these assumptions hold for a sufficiently large d . Admittedly, enforcing an upper bound on the training dataset's size is uncommon in generalization results. However, if n is exponential in d , it is not hard to see that there will be clusters which have both positive and negative examples within radius σ of the cluster center, essentially introducing a form of label noise to the problem. Since KKT points of Problem (2) interpolate the training data, this would imply that the network has interpolated training data with label noise-in other words, it has 'overfit' to noise. Understanding the generalization behavior of interpolating neural networks in the presence of label noise is a very technically challenging problem for which much is unknown, especially if one seeks to understand this by only relying upon the properties of KKT conditions for margin maximization. It is noteworthy that all existing non-vacuous generalization bounds for interpolating nonlinear neural networks in the presence of label noise require n < d [FCB22; Cao+22; XG23; Fre+23a; Kou+23].

Combining Theorem 3.1 with Theorem 2.1, we conclude that w.h.p. over a training dataset of size n ≥ k ln 2 ( d ) (and under some additional mild requirements), if gradient flow reaches empirical loss smaller than 1 n , then it converges in direction to a neural network that generalizes well. This result is width-independent, thus, it holds irrespective of the network width. Specifically, even if the network is highly overparameterized, the implicit bias of gradient flow prevents harmful overfitting. Moreover, the result does not depend directly on the initialization of gradient flow. That is, it holds whenever gradient flow reaches small empirical loss after some finite time. Thus, by relying on the KKT conditions of the max-margin problem instead of analyzing the full gradient flow trajectory, we can prove generalization without the need to prove convergence.

3.1 Proof idea

The proof of Theorem 3.1 is given in Appendix A. Here we discuss the high-level approach. Let θ = [ w 1 , . . . , w m , b , v ] be a KKT point of Problem (2). Thus, we have N θ ( x ) = ∑ j ∈ [ m ] v j ϕ ( w ⊤ j x + b j ) . Since θ satisfies the KKT conditions of Problem (2), then there are λ 1 , . . . , λ n such that for every j ∈ [ m ] we have

̸

where ϕ ′ i,j is a subgradient of ϕ at w ⊤ j x i + b j , i.e., if w ⊤ j x i + b j = 0 then ϕ ′ i,j = ✶ [ w ⊤ j x i + b j ≥ 0] , and otherwise ϕ ′ i,j is some value in [0 , 1] . Also we have λ i ≥ 0 for all i , and λ i = 0 if y i N θ ( x i ) = 1 . Likewise, we have

̸

In the proof, using a careful analysis of Eq. (3) and (4) we show that w.h.p. N θ classifies correctly a fresh example. More precisely, the main argument can be described as follows. We denote J := [ m ] , J + := { j ∈ J : v j > 0 } , and J -:= { j ∈ J : v j < 0 } . Moreover, we denote I := [ n ] and Q := [ k ] . For q ∈ Q we denote I ( q ) = { i ∈ I : x i is in cluster q } . Consider the network's output for an input x from cluster r ∈ Q with y ( r ) = 1 . Since N θ ( x ) = ∑ j ∈ J + v j ϕ ( w ⊤ j x + b j )+ ∑ j ∈ J -v j ϕ ( w ⊤ j x + b j ) and ϕ ( z ) ≥ z , we have

This suggests the following possibility: if we can ensure that ∑ j ∈ J + v j ( w ⊤ j x + b j ) is large and positive while ∑ j ∈ J -v j ϕ ( w ⊤ j x + b j ) is not too negative, then the network will accurately classify the example x . Using Eq. (3) and (4) and that y ( r ) = 1 (so y i = 1 for i ∈ I ( r ) ), the first term in the above decomposition is equal to

̸

Since x comes from cluster r and the clusters are nearly orthogonal, the pairwise correlations x ⊤ i x will be large and positive when i ∈ I ( r ) but will be small in magnitude when i ∈ I ( q ) for q = r . Thus, we can hope that this term will be large and positive if we can show that the quantity ∑ i ∈ I ( r ) ∑ j ∈ J + λ i v 2 j ϕ ′ i,j is not too small relative to the quantity ∑ q ∈ Q \{ r } ∑ i ∈ I ( q ) ∑ j ∈ J + λ i v 2 j ϕ ′ i,j . By similar arguments, in order to show the second term in Eq. (5) is not too negative, we need to understand how the quantity ∑ i ∈ I ( q ) ∑ j ∈ J -λ i v 2 j ϕ ′ i,j varies across different clusters q ∈ Q . Hence, in the proof we analyze how the quantities ∑ i ∈ I ( q ) ∑ j ∈ J + λ i v 2 j ϕ ′ i,j , ∑ i ∈ I ( q ) ∑ j ∈ J -λ i v 2 j ϕ ′ i,j relate to each other for different clusters q ∈ Q , and show that these quantities are all of the same order. Then, we conclude that w.h.p. x is classified correctly.

4 Robustness

We begin by introducing the definition of R ( · ) -robustness.

Definition 4.1. Given some function R ( · ) , we say that a neural network N θ is R ( d ) -robust w.r.t. a distribution D x over R d if for every r = o ( R ( d )) , with probability 1 -o d (1) over x ∼ D x , for every x ′ ∈ R d with ∥ x -x ′ ∥ ≤ r we have sign( N θ ( x ′ )) = sign( N θ ( x )) .

Thus, a neural net N θ is R ( d ) -robust if changing the label of an example cannot be done with a perturbation of size o ( R ( d )) . Note that we consider here ℓ 2 perturbations.

For the distribution D clusters under consideration, it is straightforward to show that classifiers cannot be R ( d ) -robust if R ( d ) = ω ( √ d ) : since the distance between examples in different clusters is w.h.p.

O ( √ d ) , it is clearly possible to flip the sign of an example with a perturbation of size O ( √ d ) . In particular, the best we can hope for is √ d -robustness. In the following theorem, we show that there exist two-layer ReLU networks which can both achieve small test error and the optimal level of √ d -robustness.

Theorem 4.1. For every r ≥ k , there exists a depth2 ReLU network N : R d → R of width r such that for ( x , y ) ∼ D clusters, with probability at least 1 -d -ω d (1) we have y N ( x ) ≥ 1 , and flipping the sign of the output requires a perturbation of size larger than √ d 8 (for a sufficiently large d ). Thus, N classifies the data correctly w.h.p., and it is √ d -robust w.r.t. D x .

√

Thus, we see that d -robust networks exist. In the following theorem, we show that the implicit bias of gradient flow constrains the level of robustness of trained networks whenever the number of clusters k is large.

Theorem 4.2. Let ϵ, δ ∈ (0 , 1) . Let S = { ( x i , y i ) } n i =1 ⊆ R d ×{-1 , 1 } be a training set drawn i.i.d. from the distribution D clusters, where n ≥ k ln 2 ( d ) . We denote Q + = { q ∈ [ k ] : y ( q ) = 1 } and Q -= { q ∈ [ k ] : y ( q ) = -1 } , and assume that min { | Q + | k , | Q -| k } ≥ c for some c > 0 . Let N θ be a depth2 ReLU network such that θ = [ w 1 , . . . , w m , b , v ] is a KKT point of Problem (2). Provided d is sufficiently large such that δ -1 ≤ 1 3 d ln( d ) -1 and n ≤ min {√ δ 3 · e d/ 32 , √ δ 3 · d ln( d ) / 4 , ϵ 4 · d ln( d ) / 2 } , with probability at least 1 -δ over S , there is a vector z = η · ∑ j ∈ [ k ] y ( j ) µ ( j ) with η > 0 and ∥ z ∥ ≤ O (√ d / c 2 k ) , such that

̸

Note that the expressions in the upper bounds on n and δ -1 are super-polynomial in d , and hence these requirements are mild (e.g., they hold for a sufficiently large d when n, δ -1 , ϵ -1 = poly( d ) ). As we mentioned in the discussion following Theorem 3.1, we believe removing the requirement for an upper bound on n would be highly nontrivial.

Theorem 4.2 implies that if c 2 k = ω d (1) , then w.h.p. over the training dataset, every KKT point of Problem (2) is not √ d -robust. Specifically, if c is constant, namely, at least a constant fraction of the clusters have positive labels and a constant fraction of the clusters have negative labels, then the network is not √ d -robust if k = ω d (1) . Recall that by Theorem 3.1, we also have w.h.p. that every KKT point generalizes well. Overall, combining Theorems 2.1, 3.1, 4.1, and 4.2, we conclude that for c 2 k = ω d (1) , w.h.p. over a training dataset of size n ≥ k ln 2 ( d ) , if gradient flow reaches empirical loss smaller than 1 n , then it converges in direction to a neural network that generalizes well but is not √ d -robust, even though there exist √ d -robust networks that generalize well. Thus, in our setting, there is bias towards solutions that generalize well but are non-robust.

̸

Example 2. Consider the setting from the first item of Example 1. Thus, the cluster means satisfy |⟨ µ ( i ) , µ ( j ) ⟩| = ˜ O ( √ d ) for every i = j , and we have σ = 1 and k = ˜ Θ( √ d ) . Suppose that c = Θ(1) , namely, there is at least a constant fraction of clusters with each label {-1 , 1 } . Then, the adversarial perturbation z from Theorem 4.2 satisfies ∥ z ∥ = O ( √ d / k ) = ˜ O ( d 1 / 4 ) = o ( √ d ) .

Similarly to our discussion after Theorem 3.1, we note that Theorem 4.2 is width-independent, i.e., it holds irrespective of the network width. It implies that we cannot hope to obtain a robust solution by choosing an appropriate width for the trained network. As we discussed in the related work section, Bubeck, Li, and Nagaraj [BLN21] and Bubeck and Sellke [BS21] considered the expressive power of neural networks, and showed that overparameterization might be necessary for robustness. By Theorem 4.2, even when the network is overparameterized, the implicit bias of the optimization method can prevent convergence to robust solutions. Moreover, our result does not depend directly on the initialization of gradient flow. Recall that by Theorem 2.1 if gradient flow reaches small empirical loss then it converges in direction to a KKT point of Problem (2). Hence our result holds whenever gradient flow reaches a small empirical loss.

Note that in Theorem 4.2, the adversarial perturbation does not depend on the input (up to sign). It corresponds to the well-known empirical phenomenon of universal adversarial perturbations , where one can find a single perturbation that simultaneously flips the label of many inputs (cf. [Moo+17;

Zha+21]). Moreover, the same perturbation applies to all depth2 networks to which gradient flow might converge (i.e., all KKT points). It corresponds to the well-known empirical phenomenon of transferability in adversarial examples, where one can find perturbations that simultaneously flip the labels of many different trained networks (cf. [Liu+17; AM18]).

It is worth noting that Theorems 3.1 and 4.2 demonstrate that trained neural networks exhibit different properties than the 1 -nearest-neighbour learning rule, irrespective of the number of parameters in the network. For example, consider the case where σ = 1 √ d , namely, the examples of each cluster are concentrated within a ball of radius O (1) around its mean. Then, the distance between every pair of points from the same cluster is O (1) , and the distance between points from different clusters is Ω( √ d ) . In this setting, both the 1 -nearest-neighbour classifier and the trained neural network will classify a fresh example correctly w.h.p., but in the 1 -nearest-neighbour classifier flipping the output's sign will require a perturbation of size Ω( √ d ) , while in the neural network a much smaller perturbation will suffice.

Finally, we remark that in the limit σ → 0 , we get a distribution supported on µ (1) , . . . , µ ( k ) . Then, a training dataset of size n ≥ k ln 2 ( d ) will contain w.h.p. all examples in the support, and hence robustness w.r.t. test data is equivalent to robustness w.r.t. the training data. In this case, we recover the results of Vardi, Yehudai, and Shamir [VYS22] which characterized the non-robustness of KKT points of ReLU networks trained on nearly orthogonal training data. In particular, our Theorem 4.2 is a strict generalization of their Theorem 4.1.

4.1 Proof ideas

Here we discuss the main ideas in the proofs of Theorem 4.1 and 4.2 (see Appendices C and D for the formal proofs).

The proof of Theorem 4.1 follows by the following simple construction. The robust network includes k neurons, each corresponding to a single cluster. That is, we have N ( x ) = ∑ k j =1 v j σ ( w ⊤ x + b j ) , where v j = y ( j ) , w j = 4 µ ( j ) d , and b j = -2 . Note that the j -th neuron points at the direction of the j -th cluster and has a negative bias term, such that the neuron is active on points from the j -th cluster, and inactive on points from the other clusters. Then, given a fresh example ( x , y ) ∼ D clusters, we show that the network classifies it correctly w.h.p. with margin at least 1 . Also, there is w.h.p exactly one neuron that is active on x , and hence the gradient of the network w.r.t. the input is affected only by this neuron and is of size O (1 / √ d ) . Therefore, we need a perturbation of size Ω( √ d ) in order to flip the output's sign.

The intuition for Theorem 4.2 can be described as follows. Recall that in our construction of a robust network above, an example ( x , y ) ∼ D clusters is w.h.p. in an active region of exactly one neuron, and hence in the neighborhood of x the output of the network is sensitive only to perturbations in the direction of that neuron. Now, consider the linear model x ↦→ w ⊤ x , where w = ∑ k q =1 1 d y ( q ) µ ( q ) . It is not hard to verify that for ( x , y ) ∼ D clusters we have w.h.p. that 0 < y w ⊤ x ≤ O (1) . Moreover, the gradient of this linear predictor is of size ∥ w ∥ = Ω( √ k/d ) . Hence, we can flip the output's sign with a perturbation of size O ( √ d/k ) . Thus, the linear classifier is non-robust if k = ω d (1) . Intuitively, the difference between our robust ReLU network and the non-robust linear classifier is the fact that in the neighborhood of x the robust network is sensitive only to perturbations in the direction of one cluster, while the linear classifier is sensitive to perturbations in the directions of all k clusters. In the proof, we analyze ReLU networks which are KKT points of Problem (2), and show that although these ReLU networks are non-linear, they are still sensitive to perturbations in the directions of all k clusters, similarly to the above linear classifier. The formal proof follows by a careful analysis of the KKT conditions of Problem (2), given in Eq. (3) and (4).

We remark that in the proof of Theorem 4.2 we use some technical ideas from Vardi, Yehudai, and Shamir [VYS22]. However, there are significant differences between the two settings. For example, they assume that the training data are nearly orthogonal, which only holds when the dimension is large relative to the number of samples; thus, it is unclear whether the existence of small adversarial perturbations in their setting is due to the high-dimensionality of the data or if a similar phenomenon exists in the more common n > d setting. At a more technical level, their proof relies on showing that

in a KKT point all inputs must lie exactly on the margin, while in our setting they are not guaranteed to lie exactly on the margin.

5 Discussion

In this paper, we considered clustered data, and showed that gradient flow in two-layer ReLU networks does not harmfully overfit, but also hinders robustness. Our results follow by analyzing the KKT points of the max-margin problem in parameter space. In our distributional setting, the clusters are well-separated, and hence there exist robust classifiers, which allows us to consider the effect of the implicit bias of gradient flow on both generalization and robustness. Understanding generalization and robustness in additional data distributions and neural network architectures is a challenging but important question. As a possible next step, it would be interesting to study whether the approach used in this paper can be extended to the following data distributions:

First, our assumption on the data distribution (Assumption 2.2) implies that the number of clusters cannot be too large, and as a result the data is linearly separable (Lemma 2.1). We conjecture that our results hold even for a significantly larger number of clusters, such that the data is not linearly separable.

Second, it would be interesting to understand whether our generalization result holds for linearly separable data distributions that are not clustered. That is, given a distribution that is linearly separable with some margin γ > 0 and a training dataset that is large enough to allow learning with a max-margin linear classifier, are there KKT points of the max-margin problem for two-layer ReLU networks that do not generalize well? In other words, do ReLU networks that satisfy the KKT conditions generalize at least as well as max-margin linear classifiers?

Acknowledgments and Disclosure of Funding

SF, GV, PB, and NS acknowledge the support of the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning through awards DMS-2031883 and #814639, and of the NSF through grant DMS-2023505.

References

[BS21] Sébastien Bubeck and Mark Sellke. 'A universal law of robustness via isoperimetry'. In: Advances in Neural Information Processing Systems (NeurIPS) (2021) (Cited on pages 3, 8).

[Cao+22] Yuan Cao, Zixiang Chen, Mikhail Belkin, and Quanquan Gu. 'Benign Overfitting in Two-layer Convolutional Neural Networks'. In: Preprint, arXiv:2202.06526 (2022) (Cited on page 6).

[CW17] Nicholas Carlini and David Wagner. 'Adversarial examples are not easily detected: Bypassing ten detection methods'. In: Proceedings of the 10th ACM workshop on artificial intelligence and security . 2017 (Cited on page 1).

[CW18] Nicholas Carlini and David Wagner. 'Audio adversarial examples: Targeted attacks on speech-to-text'. In: IEEE Security and Privacy Workshops (SPW) . 2018 (Cited on page 1).

[CL21] Niladri S. Chatterji and Philip M. Long. 'Finite-sample analysis of interpolating linear classifiers in the overparameterized regime'. In: Journal of Machine Learning Research 22.129 (2021), pp. 1-30 (Cited on page 5).

[CB20] Lenaic Chizat and Francis Bach. 'Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss'. In: Conference on Learning Theory (COLT) . 2020 (Cited on page 3).

[Cla+08] Francis H Clarke, Yuri S Ledyaev, Ronald J Stern, and Peter R Wolenski. Nonsmooth analysis and control theory . Vol. 178. Springer Science & Business Media, 2008 (Cited on page 4).

[CH20] Francesco Croce and Matthias Hein. 'Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks'. In: International conference on machine learning (ICML) . 2020 (Cited on page 1).

[DS20] Amit Daniely and Hadas Shacham. 'Most ReLU Networks Suffer from ℓ 2 Adversarial Perturbations'. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) (Cited on page 3).

[DB22] Elvis Dohmatob and Alberto Bietti. 'On the (non-) robustness of two-layer neural networks in different learning regimes'. In: arXiv preprint arXiv:2203.11864 (2022) (Cited on page 3).

[Dut+13] Joydeep Dutta, Kalyanmoy Deb, Rupesh Tulshyan, and Ramnik Arora. 'Approximate KKT points and a proximity measure for termination'. In: Journal of Global Optimization 56.4 (2013), pp. 1463-1499 (Cited on page 4).

[FFF18] Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. 'Adversarial vulnerability for any classifier'. In: Advances in neural information processing systems (NeurIPS) (2018) (Cited on page 3).

[FCB22] Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. 'Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data'. In: Conference on Learning Theory (COLT) . 2022 (Cited on page 6).

[Fre+23a] Spencer Frei, Gal Vardi, Peter L. Bartlett, and Nathan Srebro. 'Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization'. In: Preprint, arXiv:2303.01462 (2023) (Cited on pages 5, 6).

[Fre+23b] Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro, and Wei Hu. 'Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data'. In: International Conference on Learning Representations (ICLR) . 2023 (Cited on pages 3, 5).

[GSS15] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 'Explaining and harnessing adversarial examples'. In: International Conference on Learning Representations (ICLR) . 2015 (Cited on pages 1, 3).

[Hai+22] Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. 'Reconstructing training data from trained neural networks'. In: Advances in Neural Information Processing Systems (NeurIPS) (2022) (Cited on page 3).

[Hu+20] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. 'The surprising simplicity of the early-time learning dynamics of neural networks'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2020 (Cited on page 5).

[JT20] Ziwei Ji and Matus Telgarsky. 'Directional convergence and alignment in deep learning'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2020 (Cited on pages 2-4).

[KH18] Marc Khoury and Dylan Hadfield-Menell. 'On the geometry of adversarial examples'. In: Preprint, arXiv:1811.00525 (2018) (Cited on page 3).

[KYS23] Guy Kornowski, Gilad Yehudai, and Ohad Shamir. 'From Tempered to Benign Overfitting in ReLU Neural Networks'. In: arXiv preprint arXiv:2305.15141 (2023) (Cited on page 3).

[Kou+23] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. 'Benign Overfitting for Two-layer ReLU Networks'. In: International Conference on Machine Learning (ICML) . 2023 (Cited on page 6).

[Kun+22] Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. 'The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks'. In: arXiv preprint arXiv:2210.03820 (2022) (Cited on page 3).

[KGB17] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 'Adversarial machine learning at scale'. In: International Conference on Learning Representations (ICLR) . 2017 (Cited on page 1).

[LM00] Beatrice Laurent and Pascal Massart. 'Adaptive estimation of a quadratic functional by model selection'. In: Annals of Statistics (2000), pp. 1302-1338 (Cited on page 15).

[Liu+17] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. 'Delving into transferable adversarial examples and black-box attacks'. In: International Conference on Learning Representations (ICLR) . 2017 (Cited on pages 2, 9).

[LL20] Kaifeng Lyu and Jian Li. 'Gradient descent maximizes the margin of homogeneous neural networks'. In: International Conference on Learning Representations (ICLR) . 2020 (Cited on pages 2-4).

[Lyu+21] Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. 'Gradient descent on two-layer nets: Margin maximization and simplicity bias'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2021 (Cited on page 3).

[Mad+18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 'Towards Deep Learning Models Resistant to Adversarial Attacks'. In: International Conference on Learning Representations (ICLR) . 2018 (Cited on page 1).

[Mon+19] Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. 'The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime'. In: Preprint, arXiv:1911.01544 (2019) (Cited on page 5).

[MW22] Andrea Montanari and Yuchen Wu. 'Adversarial Examples in Random Neural Networks with General Activations'. In: arXiv preprint arXiv:2203.17209 (2022) (Cited on page 3).

[Moo+17] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 'Universal adversarial perturbations'. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) . 2017 (Cited on pages 2, 8).

[Ney+17] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. 'Exploring generalization in deep learning'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2017 (Cited on page 1).

[Pap+17] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 'Practical black-box attacks against machine learning'. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security . 2017 (Cited on page 1).

[Pap+16] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 'Distillation as a defense to adversarial perturbations against deep neural networks'. In: IEEE symposium on security and privacy (SP) . 2016 (Cited on page 1).

[PL20] Mary Phuong and Christoph H Lampert. 'The inductive bias of ReLU networks on orthogonally separable data'. In: International Conference on Learning Representations (ICLR) . 2020 (Cited on page 3).

[SVL22] Itay Safran, Gal Vardi, and Jason D Lee. 'On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias'. In: Advances in Neural Information Processing Systems (NeurIPS) (2022) (Cited on page 3).

[SBG21] Roei Sarussi, Alon Brutzkus, and Amir Globerson. 'Towards understanding learning in neural networks with linear teachers'. In: International Conference on Machine Learning (ICML) . 2021 (Cited on page 3).

[Sch+18] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. 'Adversarially robust generalization requires more data'. In: Advances in neural information processing systems (NeurIPS) (2018) (Cited on page 3).

[Sha+19] Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. 'Are adversarial examples inevitable?' In: International Conference on Learning Representations (ICLR) . 2019 (Cited on page 3).

[Sha+20] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. 'The pitfalls of simplicity bias in neural networks'. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) (Cited on page 3).

[SMB21] Adi Shamir, Odelia Melamed, and Oriel BenShmuel. 'The dimpled manifold model of adversarial examples in machine learning'. In: Preprint, arXiv:2106.10151 (2021) (Cited on page 3).

[Sin+21] Vasu Singla, Songwei Ge, Basri Ronen, and David Jacobs. 'Shift invariance can reduce adversarial robustness'. In: Advances in Neural Information Processing Systems (NeurIPS) (2021) (Cited on page 3).

[Sze+14] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 'Intriguing properties of neural networks'. In: International Conference on Learning Representations (ICLR) . 2014 (Cited on page 1).

[Var22] Gal Vardi. 'On the Implicit Bias in Deep-Learning Algorithms'. In: Preprint, arXiv:2208.12591 (2022) (Cited on page 2).

[VSS22] Gal Vardi, Ohad Shamir, and Nathan Srebro. 'On Margin Maximization in Linear and ReLU Networks'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2022 (Cited on pages 3, 5).

[VYS22] Gal Vardi, Gilad Yehudai, and Ohad Shamir. 'Gradient Methods Provably Converge to Non-Robust Networks'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2022 (Cited on pages 3, 5, 9).

[Wan+20] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. 'High-frequency component helps explain the generalization of convolutional neural networks'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2020 (Cited on page 3).

[Wan+22] Yunjuan Wang, Enayat Ullah, Poorya Mianjy, and Raman Arora. 'Adversarial Robustness is at Odds with Lazy Training'. In: Advances in Neural Information Processing Systems (NeurIPS) (2022) (Cited on page 3).

[WK18] Eric Wong and Zico Kolter. 'Provable defenses against adversarial examples via the convex outer adversarial polytope'. In: International Conference on Machine Learning (ICML) . 2018 (Cited on page 1).

[WRK20] Eric Wong, Leslie Rice, and J Zico Kolter. 'Fast is better than free: Revisiting adversarial training'. In: International Conference on Learning Representations (ICLR) . 2020 (Cited on page 1).

[Wu+20] Zuxuan Wu, Ser-Nam Lim, Larry S Davis, and Tom Goldstein. 'Making an invisibility cloak: Real world adversarial attacks on object detectors'. In: European Conference on Computer Vision (ECCV) . 2020 (Cited on page 1).

[XG23] Xingyu Xu and Yuantao Gu. 'Benign overfitting of non-smooth neural networks beyond lazy training'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . Ed. by Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent. 2023 (Cited on page 6).

[Zha+21] Chaoning Zhang, Philipp Benz, Chenguo Lin, Adil Karjauv, Jing Wu, and In So Kweon. 'A survey on universal adversarial attack'. In: International Joint Conference on Artificial Intelligence (IJCAI) (2021) (Cited on pages 2, 9).

[Zha+17] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 'Delving into transferable adversarial examples and black-box attacks'. In: International Conference on Learning Representations (ICLR) . 2017 (Cited on page 1).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the implicit bias of gradient descent for two-layer ReLU networks for cluster data distribution, and shows the implicit bias towards solutions that generalize well but are vulnerable to adversarial examples.\n\nSTRENGTHS:\nThe paper builds upon previous work by Vardi et al, removing the orthogonality assumption of the training data, and considering the robustness of test data instead of training data. Although the analysis is via understanding the KKT points, leveraging the previous work result that the implicit bias of gradient low for homogeneous NN given over exp or logistic loss, such regime belongs to the rich regime instead of the lazy regime, which should be more interesting to understand. Therefore I believe the paper provides good contributions to understanding the generalization and robustness of two-layer NN via gradient methods.\n\nMoreover, the paper provides a complementary result of Bubeck and Sellke, showing that even if the network is overparameterized, the implicit bias of gradient flow prevents convergence to robust solutions. This paper gives concrete examples in which the network generalize well but is non-robust, and the perturbation is independent of the network width. Note that Bubeck and Sellke paper consider regression setting whereas this paper consider classification setting.\n\nWEAKNESSES:\nInstead of the orthogonality of the training samples as assumed by Vardi et al, this paper turns out to have a clustered assumption on data distribution. I understand such an assumption comes from the proof technique, but it still restricts the setting too much that cannot capture the real scenario.\n\nQUESTIONS:\n1. I'm wondering whether assumption 2.2 (3) can be further simplified by a relationship between k and d, currently the LHS has both d and k.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors study the implicit bias of two layers neural networks with ReLU activation, in the setting where the data is composed of independant clusters that \"dont' overlap\" (i.e the probability of having the point of a cluster falling outo the support of another is small, e.g Gaussian of subgaussian). They study both the generalization properties and robustness of the solution obtained by training such network on those data.\n\nThey show that if the network suceed to fit the data (i.e the train loss falls below some threshold) and if we follow the gradient flow:\n* then the weights converge *in direction* \n* the network generalizes well \n* however it converges to a non robust solution (if the dimension is high, the number of clusters is small, and the number of points is small too)\n* whereas in the same setting robust solutions exist! But they are not \"found\" by following the gradient flow\n* the adversarial attack associated is universal and transferable\n\nSTRENGTHS:\nThe paper does a good job at explaining the technical hypothesis, their significance, and the practical consequences.  \n  \nThe \"proof idea\" sections contain the right level of details to grasp the ideas.  \n  \nThe paper is easy to read despite its technical content.  \n  \nWhile I am not familiar with the topic, the literature review seems thorough, in particular the contributions of the paper compared to existing works are clear.  \n\nThe main conclusion is significant in my opinion: \"However, the implicit bias also leads to non-robust solutions (susceptible to small adversarial 2-perturbations), even though robust networks that fit the data exist.\", with the additionnal properties that \"the adversarial attack associated is universal and transferable\".\n\nDefinition 4.1 seems to be new, and it is an interesting concept to define robustness.\n\nWEAKNESSES:\nThe main weakness of the contribution lies in the hypothesis, that are extremely restrictive, and sometimes even more than it seems at first glance.\n\nIt is not clear if those hypothesis have a chance to hold on high dimensional image space, and if the main steps of the reasoning can be adapted to this setting, or if this is just specific to this particular setting and problem (see the section \"Limitations\" below about the setting).\n\n## Thm 4.2\n\nIn Thm 4.2 the quantity is $\\min{(Q_{+}/k,Q_{-}/k)}\\geq c$ can be discussed a bit. In essence, the condition $c^2k=\\omega(1)$ (l332) means that the number of clusters should be **big**, especially if a lot of clusters have the same labels (in which case $c\\approx 0$). But on the other hand assumption 2.2 (third item) tells that the number of clusters should be small.\n\nMoreover the dimension in which the theorems hold should be high (from what I understand of theorem 3.1), but there shouldn't be too much points per clusters (otherwise $n$ would be too big).\n\nIn overall, beyond artificial example 2., is there practical settings that fulfill those hypothesis? I'm scared that the results only apply to a high dimensional setting with very few points per cluster, that are all orthogonal, and that do not overlap.\n\nQUESTIONS:\n## Suggestion\n\nBased on the quality of the literature review, I think a \"summary\" table that contains a high level description of some significant prior works, with their associated hypothesis and main conclusions, would help to situate the paper better, in the spirit of:\n\n| Paper | Hypothesis on model | Hypothesis on data |  Conclusion\n| ----------- | ----------- |  -----------   | ----------- \n| Paper 1|  two-layers ReLu | none | converges\n| Paper 2| homogeneous | almost orthogonal | converges and generalizes\n| Our | etc... | clusters  | generalizes but not robust\n\n## Question\n\nHow much do the results depend on the hypothesis that the activation is a ReLU ? This is part of the requirements of prior work, and clearly the  case of $\\phi(z)\\geq z$ (l236). Do you think that extension to similar shaped activations is straighforward, or very difficult?\n\nLIMITATIONS:\nThe limitations of the hypothesis are discussed, but this part can be detailed in more depth. I have a few remarks and questions.\n\nl207 : I think the formula can be simplified as\n\n$$k(\\max_{i\\neq j} |\\langle \\mu_i,\\mu_j\\rangle|+C+1)\\leq \\frac{1}{10}(d - C + 1)$$\n\nwhere $C$ depends on $d$ and $\\sigma$. An interpretation can be given: \"The centroids must be almost orthogonal to each other, and there shouldn't be too much of them.\". Note sure if $\\frac{1}{10}$ plays a special role, or if any other constant small than $1$ could have done the trick in the proofs. It is worth mentioning it, because this condition is hard to interpret.\n\nWe see that \"it is worth noting that Assumption 2.2 implies that the data is w.h.p. linearly separable\" shows that the hypothesis that the clusters are almost orthogonal is quite strong.\n\nWhat I understand from \"Finally, we remark that when k is small, our results may be extended to the case where σ > 1.\" (l227) hints that what you need is a \"almost orthogonal clusters that don't overlap, so they are separable\" kind-of hypothesis. I am not sure how much additional insights it brings compared to the work of  Vardi, Yehudai, and Shamir [VYS22].\n\nIn particular, the upper bound on $n$ in Thm3.1 is quite surprising, fortunately the discussion l265 \"It is noteworthy\nthat all existing non-vacuous generalization bounds for interpolating nonlinear neural networks in the\npresence of label noise require n < d [FCB22; Cao+22; XG23; Fre+23a; Kou+23].\" helps to understand why this is required.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper studies the implicit regularization brought by the neural network itself. The authors theoretically prove that in two-layer ReLU networks trained with the logistic loss or the exponential loss, the implicit bias would lead to the solutions that generalize well but non-robust, regardless of the size of the network.

STRENGTHS:
- The topic is valuable and may shed some light on the field.
- The theoretical proof seems to be sound.

WEAKNESSES:
- Lacking of even toy experiments.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show under a special data cluster model, the implicit bias of gradient flow converge to KKT points that generalize well but are not robust under l2 perturbations, when robust solutions of the problem exists. Their results are built upon earlier works on KKT points by LL20 and JT20, and the more recent work by VSS22, and extend these results further.\n\nSTRENGTHS:\n- The authors study the gradient flow solution of 2-layer ReLU network on a mixture of Gaussian distribution, and prove that while the solution generalizes well (classifies almost every test point correctly with high probability), it is not robust (can change its classification with perturbation much less than \\sqrt{d}, the optimal achievable robustness). This is a very nice result following up on the previous line of work on this topic. \n\n- The paper is well-written and clear. While the result is technical the presentation is easy to follow and main proof ideas succinctly presented in the limited space.\n\nWEAKNESSES:\n- Given the assumption on large cluster separation (\\sqrt{d} between cluster centers) and small variance of the clusters (\\sigma<1), it feels like the clusters behave very much like individual isolated points/samples. In view of this, the differentiation with the results of VSS22 in the introduction section appear to be weaker than the authors claim, as the cluster means take the role of training samples in VSS22, which still requires near-orthogonality. Also, results wrt test data is also easier under the data assumptions used in this paper because test data essentially takes the same label from their respective cluster means.\n\nQUESTIONS:\n- Although not strictly necessary for a theroy paper, it would be nice to have some simulations on synthetic data since it is easy to set up with the data model the authors assume.\n\nLIMITATIONS:\n- The authors state clearly the assumptions used for their results. Negative societal impacts not applicable here.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization and robustness of solutions obtained by gradient flow on two-layer ReLU networks. Under a distributional setting where the data is sampled from a Gaussian mixture distribution, this paper shows that the gradient flow is biased towards solutions that generalize well, but are vulnerable to adversarial examples. The theorems are built on LL20 and JT20 (Theorem 2.1). The authors also show the existence of a robust solution in this setting and prove that any solution obtained via gradient flow is non-robust.  Although the assumption is a bit restrictive, the result is novel and interesting. The authors also provide a few examples which help to understand the assumptions.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow. The problem is well-motivated. The authors provide a thorough discussion of the related work.\n2. The theoretical results are solid and interesting, the proof ideas help to understand the results.\n3. This is an active area of research. The contribution is relevant.\n\nWEAKNESSES:\n1. The authors assume that when the training loss is small, the gradient flow will start to converge to a KKT point of some maximum margin problem. What if we have a random initialization, how is this small training loss guaranteed?\n2. This might be relatively minor, but numerical experiments would help to demonstrate the vulnerability (non-robustness) of the training of two-layer ReLU networks.\n\nQUESTIONS:\n1. The authors emphasize that the result hold in the rich regime. Could the authors be more specific, as there is no discussion about the rich regime in the results?\n2. Does such a result hold for regression, in other words, what about other loss functions, such as squared error?\n\nLIMITATIONS:\nThe authors assume that the data is clustered and the number of clusters is not too large, therefore the data is linearly separable. This is a relatively strong assumption. The relaxation of this assumption and extensions to other settings would be interesting.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}