Title: From Tempered to Benign Overfitting in ReLU Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: benign overfitting

implicit bias

interpolating predictors

neural networks

theory
Abstract: Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the overfitting profile in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.

From Tempered to Benign Overfitting in ReLU Neural Networks

Abstract

Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on 'benign overfitting', where interpolating predictors achieve nearoptimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as 'tempered overfitting', where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the overfitting profile in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.

1 Introduction

Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. Although quite standard in deep learning, the so-called interpolation learning regime challenges classical statistical wisdom regarding overfitting, and has attracted much attention in recent years.

In particular, the phenomenon commonly referred to as 'benign overfitting' [Bartlett et al., 2020] describes a situation in which a learning method overfits, in the sense that it achieves zero training error over inherently noisy samples, yet it is able to achieve near-optimal accuracy with respect to the underlying distribution. So far, much of the theoretical work studying this phenomenon focused on linear (or kernel) regression problems using the squared loss, with some works extending this to classification problems. However, there is naturally much interest in gradually pushing this research towards neural networks. For example, Frei et al. [2023] recently showed that the implicit bias of gradient-based training algorithms towards margin maximization, as established in a series of works throughout the past few years, leads to benign overfitting of leaky ReLU networks in high dimensions (see discussion of related work below).

Recently, Mallinar et al. [2022] suggested a more nuanced view, coining the notion of 'tempered' overfitting. An interpolating learning method is said to overfit in a tempered manner if its error (with

respect to the underlying distribution, and as the training sample size increases) is bounded away from the optimal possible error yet is also not 'catastrophic', e.g., better than a random guess. To be more concrete, consider a binary classification setting (with labels in {± 1 } ), and a data distribution D , where the output labels correspond to some ground truth function f ∗ belonging to the predictor class of interest, corrupted with independent label noise at level p ∈ (0 , 1 2 ) (i.e., given a sampled point x , its associated label y equals sign ( f ∗ ( x )) with probability 1 -p , and -sign ( f ∗ ( x )) with probability p ). Suppose furthermore that the training method is such that the learned predictor achieves zero error on the (inherently noisy) training data. Finally, let L cl denote the 'clean' test error, namely L cl ( N ) = Pr x ∼D ( N ( x ) · f ∗ ( x ) ≤ 0) for some predictor N (omitting the contribution of the label noise). In this setting, benign, catastrophic or tempered overfitting corresponds to a situation where the clean test error L cl of the learned predictor approaches 0 , 1 2 , or some value in (0 , 1 2 ) respectively. In the latter case, the clean test error typically scales with the amount of noise p . For example, for the one-nearest-neighbor algorithm, it is well-known that the test error asymptotically converges to 2 p (1 -p ) in our setting [Cover and Hart, 1967], which translates to a clean test error of p . Mallinar et al. [2022] show how a similar behavior (with the error scaling linearly with the noise level) occurs for kernel regression, and provided empirical evidence that the same holds for neural networks.

The starting point of our paper is an intriguing experiment from that paper (Figure 6b), where they considered the following extremely simple data distributioin: The inputs are drawn uniformly at random from the unit sphere, and f ∗ is the constant +1 function. This is perhaps the simplest possible setting where benign or tempered overfitting in binary classification may be studied. In this setting, the authors show empirically that a three-layer vanilla neural network exhibits tempered overfitting, with the clean test error almost linear in the label noise level p . Notably, the experiment used an input dimension of 10 , which is rather small. This naturally leads to the question of whether we can rigorously understand the overfitting behavior of neural networks in this simple setup, and how problem parameters such as the input dimension, number of samples and the architecture affect the type of overfitting.

Our contributions: In this work we take a step towards rigorously understating the regimes under which simple, 2-layer ReLU neural networks exhibit different types of overfitting (i.e. benign, catastrophic or tempered), depending on the problem parameters, for the simple and natural data distribution described in the previous paragraph. We focus on interpolating neural networks trained with exponentially-tailed losses, which are well-known to converge to KKT points of a max-margin problem (see Section 2 for more details). At a high level, our main conclusion is that for such networks, the input dimension plays a crucial role, with the type of overfitting gradually transforming from tempered to benign as the dimension increases. In contrast, most of our results are not sensitive to the network's width, as long as it is sufficient to interpolate the training data. In a bit more detail, our contributions can be summarized as follows:

Tempered overfitting in one dimension (Theorem 3.1 and Theorem 3.2). We prove that in one dimension (when the inputs are uniform over the unit interval), the resulting overfitting is provably tempered, with a clean test error scaling as Θ(poly( p )) . Moreover, under the stronger assumption that the algorithm converges to a local minimum of the max-margin problem, we show that the clean test error provably scales linearly with p . As far as we know, this is the first result provably establishing tempered overfitting for non-linear neural networks.

Benign overfitting in high dimensions (Theorem 4.1 and Theorem 4.3). We prove that when the inputs are sampled uniformly from the unit sphere (and with the dimension scaling polynomially with the sample size), the resulting overfitting will generally be benign. In particular, we show that convergence to a max-margin predictor (up to some constant factor) implies that the clean test error decays exponentially fast to 0 with respect to the dimension. Furthermore, under an assumption on the scaling of the noise level and the network width, we obtain the same result for any KKT point of the max-margin problem.

The role of bias terms and catastrophic overfitting (Section 4.2) . The proof of the results mentioned earlier crucially relied on the existence of bias parameters in the network architecture. Thus, we further study the case of a bias-less network, and prove that without such biases, neural networks may exhibit catastrophic overfitting, and that in general they do not overfit benignly without further assumptions. We consider this an interesting illustration of how catastrophic overfitting is possible in neural networks, even in our simple setup.

Empirical study for intermediate dimensions (Section 5). Following our theoretical results, we attempt at empirically bridging the gap between the one-dimensional and high dimensional settings. In particular, it appears that the tempered and overfitting behavior extends to a wider regime than what our theoretical results formally cover, and that the overfitting profile gradually shifts from tempered to benign as the dimension increases. This substantially extends the prior empirical observation due to Mallinar et al. [2022, Figure 6b], which exhibited tempered overfitting in input dimension 10 .

We note that in an independent and concurrent work, Joshi et al. [2023] studied ReLU neural networks with one-dimensional inputs, similar to the setting we study in Section 3, and proved that tempered and catastrophic overfitting can occur for possibly non-constant target functions, depending on the loss function. However, they focus on a regression setting, and on min-norm predictors, whereas our results cover classification and any KKT point of the max-magin problem. Moreover, the architecture considered differs somewhat from our work, with Joshi et al. including an untrained skip connection and an untrained bias term.

1.1 Related work

Following the empirical observation that modern deep learning methods can perfectly fit the training data while still performing well on test data [Zhang et al., 2017], many works have tried to provide theoretical explanations of this phenomenon. By now the literature on this topic is quite large, and we will only address here the works most relevant to this paper (for a broader overview, see for example the surveys Belkin 2021, Bartlett et al. 2021, Vardi 2022).

There are many works studying regression settings in which interpolating methods, especially linear and kernel methods, succeed at obtaining optimal or near-optimal performance [Belkin et al., 2018a,b, 2019, 2020, Mei and Montanari, 2022, Hastie et al., 2022]. In particular, it is interesting to compare our results to those of Rakhlin and Zhai [2019], Beaglehole et al. [2023], who proved that minimal norm interpolating (i.e. 'ridgeless') kernel methods are not consistent in any fixed dimension, while they can be whenever the dimension scales with the sample size (under suitable assumptions) [Liang and Rakhlin, 2020]. Our results highlight that the same phenomenon occurs for ReLU networks.

There is by now also a large body of work on settings under which benign overfitting (and analogous definitions thereof) occurs [Nagarajan and Kolter, 2019, Bartlett et al., 2020, Negrea et al., 2020, Nakkiran and Bansal, 2020, Yang et al., 2021, Koehler et al., 2021, Bartlett and Long, 2021, Muthukumar et al., 2021, Bachmann et al., 2021, Zhou et al., 2023, Shamir, 2023]. In classification settings, as argued by Muthukumar et al. [2021], many existing works on benign overfitting analyze settings in which max-margin predictors can be computed (or approximated), and suffice to ensure benign behavior [Poggio and Liao, 2019, Montanari et al., 2019, Thrampoulidis et al., 2020, Wang and Thrampoulidis, 2021, Wang et al., 2021, Cao et al., 2021, Hu et al., 2022, McRae et al., 2022, Liang and Recht, 2023]. It is insightful to compare this to our main proof strategy, in which we analyze the max-margin problem in the case of NNs (which no longer admits a closed form). Frei et al. [2022] showed that NNs with smoothed leaky ReLU activations overfit benignly when the data comes from a well-seperated mixture distribution, a result which was recently generalized to ReLU activations [Xu and Gu, 2023]. Similarly, Cao et al. [2022] proved that convolutional NNs with smoothed activations overfit benignly when the data is distributed according to a high dimensional Guassian with noisy labels, which was subsequently generalized to ReLU CNNs [Kou et al., 2023]. We note that these distributions are reminiscent of the setting we study in Section 4 for (non convolutional) ReLU NNs. Chatterji and Long [2023] studied benign overfitting for deep linear networks.

As mentioned in the introduction, Mallinar et al. [2022] formally introduced the notion of tempered overfitting and suggested the study of it in the context of NNs. Subsequently, Manoj and Srebro [2023] proved that the minimum description length learning rule exhibits tempered overfitting, though noticeably this learning rule does not explicitly relate to NNs. As previously mentioned, we are not aware of existing works that prove tempered overfitting in the context of NNs.

In a parallel line of work, the implicit bias of gradient-based training algorithms has received much attention [Soudry et al., 2018, Lyu and Li, 2020, Ji and Telgarsky, 2020]. Roughly speaking, these results drew the connection between NN training to margin maximization - see Section 2 for a formal reminder. For our first result (Theorem 3.1) we build upon the analysis of Safran et al. [2022] who studied this implicit bias for univariate inputs, though for the sake of bounding the number of linear

regions. In our context, Frei et al. [2023] utilized this bias to prove benign overfitting for leaky ReLU NNs in a high dimensional regime.

2 Preliminaries

Notation. We use bold-faced font to denote vectors, e.g. x ∈ R d , and denote by ∥ x ∥ the Euclidean norm. We use c, c ′ , ˜ c, C > 0 etc. to denote absolute constants whose exact value can change throughout the proofs. We denote by [ n ] := { 1 , . . . , n } , by ✶ {·} the indicator function, and by S d -1 ⊂ R d the unit sphere. Given sets A ⊂ B , we denote by Unif( A ) the uniform measure over a set A , by A c the complementary set, and given a function f : B → R we denote its restriction f | A : A → R . We let I be the identity matrix whenever the dimension is clear from context, and for a matrix A we denote by ∥ A ∥ its spectral norm. We use stadard big-O notation, with O ( · ) , Ω( · ) and Θ( · ) hiding absolute constants, write f ≲ g if f = O ( g ) , and denote by poly( · ) polynomial factors.

Setting. Weconsider a classification task based on noisy training data S = ( x i , y i ) i m =1 ⊂ R d ×{± 1 } drawn i.i.d. from an underlying distribution D := D x ×D y . Throughout, we assume that the output values y are constant +1 , corrupted with independent label noise at level p (namely, each y i is independent of x i and satisfies Pr[ y i = 1] = 1 -p, Pr[ y i = -1] = p ). Note that the Bayes-optimal predictor in this setting is f ∗ ≡ 1 . Given a dataset S , we denote by I + := { i ∈ [ m ] : y i = 1 } and I -= { i ∈ [ m ] : y i = -1 } . We study 2-layer ReLU networks:

where n is the number of neurons or network width, σ ( z ) := max { 0 , z } is the ReLU function, v j , b j ∈ R , w j ∈ R d and θ = ( v j , w j , b j ) n j =1 ∈ R ( d +2) n is a vectorized form of all the parameters. Throughout this work we assume that the trained network classifies the entire dataset correctly, namely y i N θ ( x i ) > 0 for all i ∈ [ m ] . Following Mallinar et al. [2022, Section 2.3] we consider the clean test error

corresponding to the misclassification error with respect to the clean underlying distribution. It is said that N θ exhibits benign overfitting if L cl → 0 with respect to large enough problem parameters (e.g. m,d → ∞ ), while the overfitting is said to be catastrophic if L cl → 1 2 . Formally, Mallinar et al. [2022] have coined tempered overfitting to describe any case in which L cl converges to a value in (0 , 1 2 ) , though a special attention has been given in the literature to cases where L cl scales monotonically or even linearly with the noise level p [Belkin et al., 2018b, Chatterji and Long, 2021, Manoj and Srebro, 2023].

Implicit bias. We will now briefly describe the results of Lyu and Li [2020], Ji and Telgarsky [2020] which shows that under our setting, training the network with respect to the logistic or exponential loss converges towards a KKT point of the margin-maximization problem . To that end, suppose ℓ ( z ) = log(1 + e -z ) or ℓ ( z ) = e -z , and consider the empirical loss ˆ L ( θ ) = ∑ n i =1 ℓ ( y i N θ ( x i )) . Suppose that θ ( t ) evolves according to the gradient flow of the empirical loss, namely d θ ( t ) dt = -∇ ˆ L ( θ ( t )) . 1 Note that this corresponds to performing gradient descent over the data with an infinitesimally small step size.

Theorem 2.1 (Rephrased from Lyu and Li, 2020, Ji and Telgarsky, 2020) . Under the setting above, if there exists some t 0 such that θ ( t 0 ) satisfies min i ∈ [ m ] y i N θ ( t 0 ) ( x i ) > 0 , then θ ( t ) ∥ θ ( t ) ∥ t →∞ -→ θ ∥ θ ∥ for θ which is a KKT point of the margin maximization problem

The result above shows that although there are many possible parameters θ that result in a network correctly classifying the training data, the training method has an implicit bias in the sense that it yields a network whose parameters are a KKT point of Problem (1). Recall that θ is called a KKT point of Problem (1) if there exist λ 1 , . . . , λ m ∈ R such that the following conditions hold:

It is well-known that global or local optima of Problem (1) are KKT points, but in general, the reverse direction may not be true, even in our context of 2-layer ReLU networks (see Vardi et al., 2022). In this work, our results rely either on convergence to a KKT point (Theorem 3.1 and Theorem 4.3), or on stronger assumptions such as convergence to a local optimum (Theorem 3.2) or near-global optimum (Theorem 4.1) of Problem (1) in order to obtain stronger results.

3 Tempered overfitting in one dimension

Throughout this section we study the one dimensional case d = 1 under the uniform distribution D x = Unif([0 , 1]) , so that L cl ( N θ ) = Pr x ∼ Unif([0 , 1]) [ N θ ( x ) ≤ 0] . We note that although we focus here on the uniform distribution for simplicity, our proof techniques are applicable in principle to other distributions with bounded density. Further note that both results to follow are independent of the number of neurons, so that in our setting, tempered overfitting occurs as soon as the network is wide enough to interpolate the data. We first show that any KKT point of the margin maximization problem (and thus any network we may converge to) gives rise to tempered overfitting.

Theorem 3.1. Let d = 1 , p ∈ [0 , 1 2 ) . Then with probability at least 1 -δ over the sample S ∼ D m , for any KKT point θ = ( v j , w j , b j ) n j =1 of Problem (1), it holds that

where c, C > 0 are absolute constants.

The theorem above shows that for a large enough sample size L cl ( N θ ) = Θ(poly( p )) , proving that the clean test error must scale roughly monotonically with the noise level p , leading to tempered overfitting. 2

We will now provide intuition for the proof of Theorem 3.1, which appears in Appendix A.1. The proofs of both the lower and the upper bounds rely on the analysis of Safran et al. [2022] for univariate networks that satisfy the KKT conditions. For the lower bound, we note that with high probability over the sample ( x i ) i m =1 ∼ D m x , approximately p 5 m of the sampled points will be part of a sequence of 5 consecutive points x i < · · · < x i +4 all labeled -1 . Based on a lemma due to Safran et al. [2022], we show any such sequence must contain a segment [ x j , x j +1 ] , i ≤ j ≤ i +3 for which N θ | [ x j ,x j +1 ] < 0 , contributing to the clean test error proportionally to the length of [ x j , x j +1 ] . Under the probable event that most samples are not too close to one another, namely of distance Ω(1 /m ) , we get that the total length of all such segments (and hence the error) is at least of order Ω( mp 5 m ) = Ω( p 5 ) .

̸

As to the upper bound, we observe that the number of neighboring samples with opposing labels is likely to be on order of pm , which implies that the data can be interpolated using a network of width of order n ∗ = O ( pm ) . We then invoke a result of Safran et al. [2022] that states that the class of interpolating networks that satisfy the KKT conditions has VC dimension d VC = O ( n ∗ ) = O ( pm ) . Thus, a standard uniform convergence result for VC classes asserts that the test error would be of order 1 m ∑ i ∈ [ m ] { sign( N θ )( x i ) = y i } + O ( √ d VC /m ) = 0 + O ( √ pm/m ) = O ( √ p ) .

While the result above establishes the overfitting being tempered with an error scaling as poly( p ) , we note that the range [ cp 5 , C √ p ] is rather large. Moreover, previous works suggest that in many cases we should expect the clean test error to scale linearly with p [Belkin et al., 2018b, Chatterji and Long, 2021, Manoj and Srebro, 2023]. Therefore it is natural to conjecture that this is also true here, at least for trained NNs that we are likely to converge to. We now turn to show such a result under a stronger assumption, where instead of any KKT point of the margin maximization problem, we consider specifically local minima. We note that several prior works have studied networks corresponding to global minima of the max-margin problem, which is of course an even stronger assumption than local minimality (e.g. Savarese et al., 2019, Boursier and Flammarion, 2023). We say that θ is a local minimum of Problem (1) whenever there exists r > 0 such that if ∥ θ ′ -θ ∥ < r and ∀ i ∈ [ m ] : y i N θ ′ ( x i ) ≥ 1 then ∥ θ ∥ 2 ≤ ∥ θ ′ ∥ 2 . 3

Theorem 3.2. Let d = 1 , p ∈ [0 , 1 2 ) . Then with probability at least 1 -δ over the sample S ∼ D m , for any local minimum θ = ( v j , w j , b j ) n j =1 of Problem (1), it holds that

where c, C > 0 are absolute constants.

The theorem above indeed shows that whenever the sample size is large enough, the clean test error indeed scales linearly with p . 4 The proof of Theorem 3.2, which appears in Appendix A.2, is based on a different analysis than that of Theorem 3.1. Broadly speaking, the proof relies on analyzing the structure of the learned prediction function, assuming it is a local minimum of the max-margin problem. More specifically, in order to obtain the lower bound we consider segments in between samples x i -1 < x i < x i +1 that are labeled y i -1 = 1 , y i = -1 , y i +1 = 1 . Note that with high probability over the sample, approximately p (1 -p ) 2 = Ω( p ) of the probability mass lies in such segments. Our key proposition is that for a KKT point which is a local minimum, N θ ( · ) must be linear on the segment [ x i , x i +1 ] , and furthermore that N θ ( x i ) = -1 , N θ ( x i +1 ) = 1 . Thus, assuming the points are not too unevenly spaced, it follows that a constant fraction of any such segment contributes to the clean test error, resulting in an overall error of Ω( p ) . In order to prove this proposition, we provide an exhaustive list of cases in which the network does not satisfy the assertion, and show that in each case the norm can be locally reduced - see Figure 3 in the appendix for an illustration.

For the upper bound, a similar analysis is provided for segments [ x i , x i +1 ] for which y i = y i +1 = 1 . Noting that with high probability over the sample an order of (1 -p ) 2 = 1 -O ( p ) of the probability mass lies in such segments, it suffices to show that along any such segment the network is positive implying an upper bound of O ( p ) on the possible clean test error. Indeed, we show that along such segments, by locally minimizing parameter norm the network is incentivized to stay positive.

4 Benign overfitting in high dimensions

In this section we focus on the high dimensional setting. Throughout this section we assume that the dataset is sampled according to D x = Unif( S d -1 ) and d ≫ m , where m is the number of samples, so that L cl ( N θ ) = Pr x ∼ Unif( S d -1 ) [ N θ ( x ) ≤ 0] . We note that our results hold for other commonly studied distributions (see Remark 4.2). We begin by showing that if the network converges to the maximum margin solution up to some multiplicative factor, then benign overfitting occurs:

Theorem 4.1. Let ϵ, δ > 0 . Assume that p ≤ c 1 , m ≥ c 2 log(1 /δ ) p and d ≥ c 3 m 2 log ( m ϵ ) log ( m δ ) for some universal constants c 1 , c 2 , c 3 > 0 . Given a sample ( x i , y i ) i m =1 ∼ D m , suppose θ = ( w j , v j , b j ) n j =1 is a KKT point of Problem (1) such that ∥ θ ∥ 2 ≤ c 4 √ p ∥ θ ∗ ∥ 2 , where θ ∗ is a max-margin solution of Eq. (1) and c 4 > 0 is some universal constant. Then, with probability at least 1 -δ over S ∼ D m we have that L cl ( N θ ) ≤ ϵ .

The theorem above shows that under the specified assumptions, benign overfitting occurs for any noise level p which is smaller than some universal constant. Note that this result is independent of

the number of neurons n , and requires d = ˜ Ω( m 2 ) . The mild lower bound on m is merely to ensure that the negative samples concentrate around a Θ( p ) fraction of the entire dataset. Also note that the dependence on ϵ is only logarithmic, which means that in the setting we study, the clean test error decays exponentially fast with d .

We now turn to provide a short proof intuition, while the full proof can be found in Appendix B.1. The main crux of the proof is showing that ∑ n j =1 v j σ ( b j ) = Ω(1) , i.e. the bias terms are the dominant factor in the output of the network, and they tend towards being positive. In order to see why this suffices to show benign overfitting, first note that N θ ( x ) = ∑ n j =1 v j σ ( w ⊤ j x + b j ) equals

All the samples (including the test sample x ) are drawn independently from Unif( S d -1 ) , thus with high probability | x ⊤ x i | = o d (1) for all i ∈ [ m ] . Using our assumption regarding convergence to the max-margin solution (up to some multiplicative constant), we show that the norms ∥ w j ∥ are bounded by some term independent of d . Thus, we can bound | x ⊤ w j | = o d (1) for every j ∈ [ n ] . Additionally, by the same assumption we show that both ∑ n j =1 ∥ w j ∥ and ∑ n j =1 | v j | are bounded by the value of the max-margin solution (up to a multiplicative factor), which we bound by O ( √ m ) . Plugging this into the displayed equation above, and using the assumption that d is sufficiently larger than m , we get that the ∑ n j =1 | v j σ ( w ⊤ j x ) | term is negligible, and hence the bias terms ∑ n j =1 v j σ ( b j ) are the predominant factor in determining the output of N θ .

Showing that ∑ n j =1 v j σ ( b j ) = Ω(1) is done in two phases. Recall the definitions of I + := { i ∈ [ m ] : y i = 1 } , I -= { i ∈ [ m ] : y i = -1 } , and that by our label distribution and assumption on p we have | I + | ≈ (1 -p ) m ≫ pm ≈ | I -| . We first show that if the bias terms are too small, and the predictor correctly classifies the training data, then its parameter vector θ satisfies ∥ θ ∥ 2 = Ω( | I + | ) . Intuitively, this is because the data is nearly mutually orthogonal, so if the biases are small, the vectors w j must have a large enough correlation with each different point in I + to classify it correctly. On the other hand, we explicitly construct a solution that has large bias terms, which satisfies ∥ θ ∥ 2 = O ( | I -| ) . By setting p small enough, we conclude that the biases cannot be too small.

̸

Remark 4.2 (Data distribution) . Theorem 4.1 is phrased for data sampled uniformly from a unit sphere, but can be easily extended to other isotropic distributions such as standard Gaussian and uniform over the unit ball. In fact, the only properties of the distribution we use are that | x ⊤ i x j | = o d (1) for every i = j ∈ [ m ] , and that ∥ XX ⊤ ∥ = O (1) (independent of m ) where X is an m × d matrix whose rows are x i .

4.1 Benign overfitting under KKT assumptions

In Theorem 4.1 we showed that benign overfitting occurs in high dimensional data, but under the strong assumption of convergence to the max-margin solution up to a multiplicative factor (whose value is allowed to be larger for smaller values of p ). On the other hand, Theorem 2.1 only guarantees convergence to a KKT point of the max-margin problem (1). We note that Vardi et al. [2022] provided several examples of ReLU neural networks where there are in fact KKT points which are not a global or even local optimum of the max-margin problem.

Consequently, in this section we aim at showing a benign overfitting result by only assuming convergence to a KKT point of the max-margin problem. For such a result we use two additional assumptions, namely: (1) The output weights v j are all fixed to be ± 1 , while only ( w j , b j ) j ∈ [ n ] are trained; 5 and (2) Both the noise level p and the input dimension d depend on n , the number of neurons. Our main result is the following:

Theorem 4.3. Let ϵ, δ > 0 . Assume that p ≤ c 1 n 2 , m ≥ c 2 n log ( 1 δ ) and d ≥ c 3 m 4 n 4 log ( m ϵ ) log ( m 2 δ ) for some universal constants c 1 , c 2 , c 3 > 0 . Assume that the output weights are fixed so that |{ j : v j = 1 }| = |{ j : v j = -1 }| while ( w j , b j ) j ∈ [ n ] are trained, and that N θ converges to a KKT point of Problem (1). Then, with probability at least 1 -δ over S ∼ D m we have L cl ( N θ ) ≤ ϵ .

Note that contrary to Theorem 4.1 and to the results from Section 3, here there is a dependence on n , the number of neurons. For n = O (1) we get benign overfitting for any p smaller than a universal constant. This dependence is due to a technical limitation of the proof (which we will soon explain), and it would be interesting to remove it in future work. The full proof can be found in Appendix B.2, and here we provide a short proof intuition.

The proof strategy is similar to that of Theorem 4.1, by showing that ∑ n j =1 v j σ ( b j ) = Ω(1) , i.e. the bias terms are dominant and tend towards being positive. The reason that this suffices is proven using similar arguments to that of Theorem 4.1. The main difficulty of the proof is showing that ∑ n j =1 v j σ ( b j ) = Ω(1) . We can write each bias term as b j = v j ∑ i ∈ [ m ] λ i y i σ ′ i,j , where σ ′ i,j = ✶ ( w ⊤ j x i + b j > 1) is the derivative of the j -th ReLU neuron at the point x i . We first show that all the bias terms b j are positive (Lemma B.5) and that for each i ∈ I + , there is j ∈ { 1 , . . . , n/ 2 } with σ ′ i,j = 1 . This means that it suffices to show that λ i is not too small for all i ∈ I + , while for every i ∈ I -, λ i is bounded from above.

We assume towards contradiction that the sum of the biases is small, and show it implies that λ i = Ω ( 1 n ) for i ∈ I + and λ i = O (1) for i ∈ I -. Using the stationarity condition we can write for j ∈ { 1 , . . . , n/ 2 } that w j = ∑ m i =1 λ i y i σ ′ i,j x i . Taking some r ∈ I + , we have that w ⊤ j x r ≈ λ r σ ′ r,j since x r is almost orthogonal to all other samples in the dataset. By the primal feasibility condition (Eq. (3)) N θ ( x r ) ≥ 1 , hence there must be some j ∈ { 1 , . . . , n/ 2 } with σ ′ r,j = 1 and with λ r that is larger than some constant. The other option is that the bias terms are large, which we assumed does not happen. Note that we don't know how many neurons w j there are with σ ′ j,r = 1 , which means that we can only lower bound λ r by a term that depends on n .

̸

To show that λ s are not too big for s ∈ I -, we use the complementary slackness condition (Eq. (5)) which states that if λ s = 0 then N θ ( x s ) = -1 . Since the sum of the biases is not large, then if there exists some s ∈ I -with λ s that is too large we would get that N θ ( x s ) < -1 which contradicts the complementary slackness. Combining those bounds and picking p to be small enough shows that ∑ n j =1 v j σ ( b j ) cannot be too small.

4.2 The role of the bias and catastrophic overfitting in neural networks

In both our benign overfitting results (Theorem 4.1 and Theorem 4.3), our main proof strategy was showing that if p is small enough, then the bias terms tend to be positive and dominate over the other components of the network. In this section we further study the importance of the bias terms for obtaining benign overfitting, by examining bias-less ReLU networks of the form N θ ( x ) = ∑ n j =1 v j σ ( w ⊤ j x ) where θ = ( v j , w j ) j ∈ [ n ] . We note that if the input dimension is sufficiently large and n ≥ 2 , such bias-less networks can still fit any training data. The proofs for this section can be found in Appendix C.

We begin by showing that without a bias term and with no further assumption on the network width, any solution can exhibit catastrophic behaviour:

Proposition 4.4 (Catastrophic overfitting without bias) . Consider a bias-less network with n = 2 which classifies a dataset correctly. If the dataset contains at least one sample with a negative label, then L cl ( N θ ) ≥ 1 2 .

While the result above is restricted to any network of width two (in particular, it holds under any assumption such as convergence to a KKT point), this already precludes the possibility of achieving either benign or tempered overfitting without further assumptions on the width of the network - in contrast to our previous results. Furthermore, we next show that in the bias-less case, the clean test error is lower bounded by some term which depends only on n (which again, holds for any network so in particular under further assumptions such as convergence to a KKT point).

Proposition 4.5 (Not benign without bias) . For any bias-less network of width n , L cl ( N θ ) ≥ 1 2 n .

Note that the result above does not depend on m nor on d , thus we cannot hope to prove benign overfitting for the bias-less case unless n is large, or depends somehow on both m and d . We emphasize that Proposition 4.5 does no subsume Proposition 4.4 although they appear to be similar. Indeed, for the case of n = 2 , Proposition 4.5 provides a bound of 1 4 which does not fall under the definition of catastrophic overfitting, as opposed to Proposition 4.4. Next, we show that even for

arbitrarily large n (possibly scaling with other problem parameters such as m,d ), there are KKT points which do not exhibit benign overfitting:

̸

Proposition 4.6 (Benign overfitting does not follow from KKT without bias) . Consider a bias-less network, and suppose that x ⊤ j x i = 0 for every i = j . Denote I -:= |{ i ∈ [ m ] : y i = -1 }| . Then there exists a KKT point θ of Problem (1) with L cl ( N θ ) ≥ 1 2 -1 2 | I -| .

̸

The result above shows that even if we're willing to consider arbitrarily large networks, there still exist KKT points which do not exhibit benign overfitting (at least without further assumptions on the data). This implies that a positive benign overfitting result in the bias-less case, if at all possible, cannot possibly apply to all KKT points - so one must make further assumptions on which KKT points to consider or resort to an alternative analysis. The assumption that x ⊤ j x i = 0 for every i = j is made to simplify the construction of the KKT point, while we hypothesize that the proposition can be extended to the case where x i ∼ Unif( S d -1 ) (or other isotropic distributions) for large enough d .

5 Between tempered and benign overfitting for intermediate dimensions

In Section 3, we showed that tempered overfitting occurs for one-dimensional data, while in Section 4 we showed that benign overfitting occurs for high-dimensional data . In this section we complement these results by empirically studying intermediate dimensions and their relation to the number of samples. Our experiments extend an experiment from [Mallinar et al., 2022, Figure 6b] which considered the same distribution as we do, for d = 10 and m varying from 300 to 3 . 6 × 10 5 . Here we study larger values of d , and how the ratio between d and m affects the overfitting profile. We also focus on 2 -layer networks (although we do show our results may extend to depth 3 ).

In our experiments, we trained a fully connected neural network with 2 or 3 layers, width n (which will be specified later) and ReLU activations. We sampled a dataset ( x i , y i ) i m =1 ∼ Unif( S d -1 ) ×D y for noise level p ∈ { 0 . 05 , 0 . 1 , . . . , 0 . 5 } . We trained the network using SGD with a constant learning rate of 0 . 1 , and with the logistic (cross-entropy) loss. Each experiment ran for a total of 20 k epochs, and was repeated 10 times with different random seeds, the plots are averaged over the runs. In all of our experiments, the trained network overfitted in the sense that it correctly classified all of the (inherently noisy) training data. Furthermore, we remark that longer training times ( 100 k epochs) were also examined, and the results remained unaffected, thus we omit such plots.

0.0

0.1

0.2

0.3

0.4

0.5

Noise level ("p")

0.0

0.1

0.2

0.3

0.4

0.5

Clean test error

500 samples

Input dimension

10000

5000

2000

1000

x=y

0.0

0.1

0.2

0.3

0.4

0.5

Noise level ("p")

0.0

0.1

0.2

0.3

0.4

0.5

Clean test error

2000 samples

Input dimension

10000

5000

2000

1000

x=y

In Figure 1 we trained a 2 -layer network with 1000 neurons over different input dimensions, with m = 500 samples (left) and m = 2000 samples (right). There appears to be a gradual transition from tempered overfitting (with clean test error linear in p ) to benign overfitting (clean test error close to 0 for any p ), as d increases compared to m . Our results indicate that the overfitting behavior is close to tempered/benign in wider parameter regimes than what our theoretical results formally cover ( d = 1 vs. d > m 2 ). Additionally, benign overfitting seems to occur for lower values of p , whereas for p closer to 0 . 5 the clean test error is significantly above 0 even for the highest dimensions we examined.

0.0

0.1

0.2

0.3

0.4

0.5

Noise level ("p")

0.0

0.1

0.2

0.3

0.4

0.5

Clean test error

d=10000, n=50

d=10000, n=3000

d=2000, n=50

d=2000, n=3000

d=250, n=50

d=250, n=3000

x=y

0.0

0.1

0.2

0.3

0.4

0.5

Noise level ("p")

0.0

0.1

0.2

0.3

0.4

0.5

Clean test error

Depth 3

Input dimension

10000

5000

2000

1000

x=y

We further experimented on the effects of the width and depth on the overfitting profile. In Figure 2 (left) we trained a 2 -layer network on m = 500 samples with input dimension d ∈ { 250 , 2000 , 10000 } and n neurons for n ∈ { 50 , 3000 } . It can be seen that even when the number of neurons vary significantly, the plots remain almost the same. It may indicate that the network width n has little effect on the overfitting profile (similar to most of our theoretical results), and that the dependence on n in Theorem 4.3 is an artifact of the proof technique (in accordance with our other results). In Figure 2 (right) we trained a 3 -layer neural networks on m = 500 samples, as opposed to a 2 -layer networks in all our other experiments. Noticeably, this plot looks very similar to Figure 1 (left) which may indicate that our observations generalize to deeper networks.

6 Discussion

In this paper, we focused on a classification setting with 2-layer ReLU neural networks that interpolate noisy training data. We showed that the implicit bias of gradient based training towards margin maximization yields tempered overfitting in a low dimensional regime, while it causes benign overfitting in high dimensions. Furthermore, we provide empirical evidence that there is a gradual transition from tempered to benign overfitting as the input dimension increases.

Our work leaves open several questions. A natural follow-up is to study data distributions for which our results do not apply. While our analysis in one dimension is readily applicable for other distributions, it is currently not clear how to extend the high dimensional analysis to other data distributions (in particular, to settings in which the Bayes-optimal predictor is not constant). Moreover, establishing formal results in any constant dimension larger than 1 , or whenever the dimension scales sub-quadratically (e.g., linearly) with the number of training samples, is an interesting open problem which would likely require different proof techniques.

At a more technical level, there is the question of whether our assumptions can be relaxed. In particular, Theorem 3.2 provides a tight bound under the assumption of convergence to a local minimum of the margin maximization problem, whereas Theorem 3.1 only requires convergence to a KKT point, yet leaves a polynomial gap. Similarly, Theorem 4.1 and Theorem 4.3 require assumptions on the predictor norm or on the noise level and width, respectively.

Our work also leaves open a more thorough study of different architectures and their possible effect on the type of overfitting. In particular, we note that our results suggest that as long as the network is large enough to interpolate, the width plays a minor role in our studied setting - which we also empirically observe. On the other hand, whether this also holds for other architectures, as well as the role of depth altogether, remains unclear. Finally, it would be interesting to further study the bias-less case and assess its corresponding overfitting behavior under suitable assumptions.

Acknowledgments and Disclosure of Funding

We thank Gal Vardi for insightful discussions regarding the implicit bias of neural networks. This research is supported in part by European Research Council (ERC) grant 754705.

References

Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli, and Thomas Hofmann. Uniform convergence, adversarial spheres and a simple remedy. In International Conference on Machine Learning , pages 490-499. PMLR, 2021.

Keith Ball. An elementary introduction to modern convex geometry. Flavors of geometry , 31(1-58): 26, 1997.

Peter L Bartlett and Philip M Long. Failures of model-dependent generalization bounds for least-norm interpolation. The Journal of Machine Learning Research , 22(1):9297-9311, 2021.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica , 30:87-201, 2021.

Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit. On the inconsistency of kernel ridgeless regression in fixed dimensions. SIAM Journal on Mathematics of Data Science , 5(4):854-872, 2023.

Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248, 2021.

Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems , 31, 2018a.

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning , pages 541-549. PMLR, 2018b.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity. arXiv preprint arXiv:2303.01353 , 2023.

Yuan Cao, Quanquan Gu, and Mikhail Belkin. Risk bounds for over-parameterized maximum margin classification on sub-gaussian mixtures. Advances in Neural Information Processing Systems , 34: 8407-8418, 2021.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250, 2022.

Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. The Journal of Machine Learning Research , 22(1):5721-5750, 2021.

Niladri S Chatterji and Philip M Long. Deep linear networks can benignly overfit when shallow ones do. Journal of Machine Learning Research , 24(117):1-39, 2023.

H. Clarke. Optimization and Nonsmooth Analysis . SIAM, 1990.

Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory , 13(1):21-27, 1967.

Joydeep Dutta, Kalyanmoy Deb, Rupesh Tulshyan, and Ramnik Arora. Approximate kkt points and a proximity measure for termination. Journal of Global Optimization , 56(4):1463-1499, 2013.

Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory , pages 2668-2703. PMLR, 2022.

Spencer Frei, Gal Vardi, Peter L Bartlett, and Nathan Srebro. Benign overfitting in linear classifiers and leaky relu networks from kkt conditions for margin maximization. arXiv preprint arXiv:2303.01462 , 2023.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Lars Holst. On the lengths of the pieces of a stick broken at random. Journal of Applied Probability , 17(3):623-634, 1980.

Tianyang Hu, Jun Wang, Wenjia Wang, and Zhenguo Li. Understanding square loss in training overparametrized neural network classifiers. Advances in Neural Information Processing Systems , 35:16495-16508, 2022.

Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems , 33:17176-17186, 2020.

Nirmit Joshi, Gal Vardi, and Nathan Srebro. Noisy interpolation learning with shallow univariate relu networks. arXiv preprint arXiv:2307.15396 , 2023.

Frederic Koehler, Lijia Zhou, Danica J Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. Advances in Neural Information Processing Systems , 34:20657-20668, 2021.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting for two-layer relu networks. arXiv preprint arXiv:2303.04145 , 2023.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. The Annals of Statistics , 48(3):1329-1347, 2020.

Tengyuan Liang and Benjamin Recht. Interpolating classifiers make few mistakes. Journal of Machine Learning Research , 24(20):1-27, 2023.

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In 8th International Conference on Learning Representations (ICLR) , 2020.

Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems , 35:1182-1195, 2022.

Naren Sarayu Manoj and Nathan Srebro. Interpolation learning with minimum description length. arXiv preprint arXiv:2302.07263 , 2023.

Andrew D McRae, Santhosh Karnik, Mark Davenport, and Vidya K Muthukumar. Harmless interpolation in regression and classification with structured features. In International Conference on Artificial Intelligence and Statistics , pages 5853-5875. PMLR, 2022.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75 (4):667-766, 2022.

Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.01544 , 2019.

Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? The Journal of Machine Learning Research , 22(1):10104-10172, 2021.

Vaishnavh Nagarajan and J Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems , 32, 2019.

Preetum Nakkiran and Yamini Bansal. Distributional generalization: A new kind of generalization. arXiv preprint arXiv:2009.08092 , 2020.

Jeffrey Negrea, Gintare Karolina Dziugaite, and Daniel Roy. In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors. In International Conference on Machine Learning , pages 7263-7272. PMLR, 2020.

Tomaso Poggio and Qianli Liao. Generalization in deep network classifiers trained with the square loss. Center for Brains, Minds and Machines (CBMM) Memo , 112, 2019.

Ronald Pyke. Spacings. Journal of the Royal Statistical Society: Series B (Methodological) , 27(3): 395-436, 1965.

Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with laplace kernels is a highdimensional phenomenon. In Conference on Learning Theory , pages 2595-2623. PMLR, 2019.

Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems , 35, 2022.

Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory , pages 2667-2690. PMLR, 2019.

Ohad Shamir. The implicit bias of benign overfitting. Journal of Machine Learning Research , 24 (113):1-40, 2023.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1): 2822-2878, 2018.

Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into multiclass classification: A high-dimensional asymptotic view. Advances in Neural Information Processing Systems , 33:8907-8920, 2020.

Gal Vardi. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591 , 2022.

Gal Vardi, Ohad Shamir, and Nati Srebro. On margin maximization in linear and relu networks. Advances in Neural Information Processing Systems , 35:37024-37036, 2022.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4030-4034. IEEE, 2021.

Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classification: All roads lead to interpolation. Advances in Neural Information Processing Systems , 34: 24164-24179, 2021.

Xingyu Xu and Yuantao Gu. Benign overfitting of non-smooth neural networks beyond lazy training. In International Conference on Artificial Intelligence and Statistics , pages 11094-11117. PMLR, 2023.

Zitong Yang, Yu Bai, and Song Mei. Exact gap between generalization error and uniform convergence in random feature models. In International Conference on Machine Learning , pages 11704-11715. PMLR, 2021.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 5th International Conference on Learning Representations, ICLR , 2017.

Lijia Zhou, Frederic Koehler, Danica J Sutherland, and Nathan Srebro. Optimistic rates: A unifying theory for interpolation learning and regularization in linear regression. ACM/IMS Journal of Data Science , 1, 2023.

A Proofs of tempered overfitting

A.1 Proof of Theorem 3.1

Throughout the proof, given a sample S = ( x i , y i ) i m =1 ∼ D m we denote S x = ( x i ) i m =1 , S y = ( y i ) i m =1 , and assume without loss of generality that x 1 ≤ x 2 ≤ · · · ≤ x m .

Lemma A.1. Denote by E x the event in which the samples ( x i ) i m =1 ∼ D m x satisfy

(maximal gap isn't too large) d max := max i ∈ [ m -1] ( x i +1 -x i ) ≤ log(8( m +1) /δ ) m +1 .

(most gaps aren't too small) |{ i ∈ [ m -1] : x i +1 -x i < 1 10( m +1) }| < m +1 8 .

̸

(no collisions) ∀ i = j ∈ [ m ] : x i = x j .

̸

Then there exists absolute m 0 ∈ N such that ∀ m ≥ m 0 : Pr S x ∼D x m [ E x ] ≥ 1 -δ 4 .

Following the lemma above, we continue by conditioning on the probable event E x , after which we will conclude the proof by the union bound. We now state a lemma due to Safran et al. [2022] which is crucial for our analysis.

Lemma A.2 (Lemma E.6. Safran et al., 2022) . Suppose that i < j are such that y i , y i +1 , . . . , y j = -1 . Then in the interval [ x i , x j ] there are at most two points at which N ′ θ increases.

We derive the following corollary:

Corollary A.3. Suppose that y i , y i +1 , . . . , y i +4 = -1 . Then there exists i ≤ ℓ ≤ i +3 for which N θ | [ x ℓ ,x ℓ +1 ] < 0 .

Proof. Assume towards contradiction that y i , . . . , y i +4 = -1 , yet for any i ≤ ℓ ≤ i +3 there exists z ℓ ∈ ( x ℓ , x ℓ +1 ) such that N θ ( z ℓ ) ≥ 0 . Recall that N θ ( x i ) , . . . , N θ ( x i +3 ) ≤ -1 by Eq. (3). Thus for each i ≤ ℓ ≤ i +2 , looking at the segment ( z ℓ , z ℓ +1 ) ∋ x ℓ +1 we see that N θ ( z ℓ ) > 0 , N θ ( x ℓ +1 ) ≤ -1 , N θ ( z ℓ ) > 0 . In particular, by the mean value theorem, any such segment must contain a point at which N ′ θ increases. Obtaining three such points which are distinct contradicts Lemma A.2.

We assume without loss of generality that m is divisible by 5 and split the index set [ m ] into groups consecutive five indices: we let I 1 = { 1 , . . . , 5 } , I 2 = { 6 , . . . , 10 } and so on up to I m/ 5 . Denoting by µ the (one dimensional) Lebesgue measure we get that under the event E x it holds that

where the last inequality follows from our conditioning on E x . To see why, note that the sum ∑ i ∈ [ m/ 5] ∑ l ∈ [ I i ] min ℓ ∈ I i ( x ℓ +1 -x ℓ ) is lower bounded by the sum of the m/ 5 smallest gaps, yet under E x this sum contains at least Ω( m ) summands larger than Ω(1 /m ) -hence it is at least some constant. We conclude that as long as E x occurs we have E S y ∼D y m [Pr x ∼D x [ N θ ( x ) < 0]] ≥ cp 5 . Moreover, we see by the analysis above that if a single label y l for some l ∈ I i ⊂ [ m ] is changed, this can affect N θ ( x ) only in the segment [ x min I i ℓ , x max I i ℓ ] which is of length at most 7 d max = O ( log( m/δ ) m ) . Thus we can apply McDiarmid's inequality to obtain that under E x , with probability at least 1 -δ/ 4 :

Overall, by union bounding over E x the inequality above holds with probability at least 1 -δ/ 2 , which proves the desired lower bound.

We now turn to prove the upper bound. Let N ∗ ( · ) be a 2-layer ReLU network of minimal width n ∗ ∈ N that classifies the data correctly, namely y i N ∗ ( x i ) > 0 for all i ∈ [ m ] . Note that n ∗ is uniquely defined by the sample while N ∗ is not. Furthermore, n ∗ is upper bounded by the number of neighboring samples with different labels. 6 Hence,

̸

̸

̸

We conclude that the expected width of N ∗ (as a function of the sample) is at most n ∗ = O ( pm ) . By Safran et al. [2022, Theorem 4.2], this implies N θ belongs to a class of VC dimension O ( n ∗ ) = O ( pm ) . Thus by denoting the 0-1 loss L 0 -1 ( N θ ) = Pr ( x,y ) ∼D [sign( N θ ( x )) = y ] and invoking a standard VC generalization bound we get

̸

By Jensen's inequality E [ √ n ∗ ] ≤ √ E [ n ∗ ] ≲ √ pm = ⇒ E [ √ n ∗ /m ] ≲ √ p , so by the law of total expectation

In order to relate the bound above to the clean test error, note that

̸

hence

̸

As in our argument for the lower bound, we now note by Eq. (6) that flipping a single label y l for some l ∈ [ m ] changes n ∗ by at most 1 , hence changing the test error by at most O (1 /m ) . Therefore we can apply McDiarmid's inequality and see that under the event E x , with probability at least 1 -δ/ 4 :

Overall, by union bounding over E x the inequality above holds with probability at least 1 -δ/ 2 , which proves the upper bound and finishes the proof.

A.2 Proof of Theorem 3.2

!

!"#

!

!

!

!$#

-1

+1

!

!"#

!

!

!

!$#

-1

+1

!

!"#

!

!

!

!$#

-1

+1

!

!"#

!

!

!

!$#

-1

+1

!

!"#

!

!

!

!$#

-1

+1

Throughout the proof, given a sample S = ( x i , y i ) i m =1 ∼ D m we denote S x = ( x i ) i m =1 , S y = ( y i ) i m =1 , and assume without loss of generality that x 1 ≤ x 2 ≤ · · · ≤ x m . Denote by E x the event in which the samples ( x i ) i m =1 ∼ D m x satisfy

and recall that by Lemma A.1 we have Pr S x ∼D x m [ E x ] ≥ 1 -δ 4 for m larger than an absolute constant. Therefore from here on throughout the proof we condition on E x , after which we can conclude using the union bound.

For any point x ∈ (0 , 1) , denote by i x the maximal index i ∈ [ m ] such that x i ≤ x . Let A x denote the event in which y i x -1 = 1 , y i x = -1 and y i x +1 = 1 . Note that for any x we have Pr S y ∼D y m [ A x ] = p (1 -p ) 2 ≥ 1 4 p , so we get

We aim to show that Pr x ∼D x [ N θ ( x ) < 0 | A x ] = Ω(1) . In order to do so, note that conditioning the uniform measure D x = Unif([0 , 1]) on the event A x results in a uniform measure over the (union of) segments between -1 labeled samples to +1 labeled samples that also satisfy the additional property that the neighboring sample on their left is labeled +1 . We will show that if θ is a local minimum of Problem (1), then along any such segment N θ ( x ) is linear from -1 to +1 :

Proposition A.4. Let i ∈ [ m -1] be such that x i -1 < x i < x i +1 with y i -1 = 1 , y i = -1 and y i +1 = 1 . If θ is a local minimum of Problem (1), it holds that N θ ( x i ) = -1 , N θ ( x i +1 ) = 1 and N θ ( · ) is linear over ( x i , x i +1 ) .

In particular, the proposition above shows that

which plugged into Eq. (8) gives

Moreover, noting that any local minimum is in particular a KKT point, we saw in the proof of Theorem 3.1 that under the event E x , changing a single label y l , l ∈ [ m ] can change the test error by at most O ( d max ) = O (log( m/δ ) /m ) . By McDiarmid's inequality this finishes the proof of the desired lower bound.

As for the upper bound, for x ∈ (0 , 1) we denote by B x the event in which y i x = 1 = y i x +1 = 1 -namely, x is between two positively labeled samples. Note that for any x we have Pr S y ∼D y m [ B x ] = (1 -p ) 2 ≥ 1 -2 p = ⇒ Pr S y ∼D y m [ B c x ] ≤ 2 p . We will show that if θ is a local minimum of the margin maximization problem, then B x implies that N θ ( x ) ≥ 0 .

Proposition A.5. Let i ∈ [ m -1] be such that y i = y i +1 = 1 , and let x ∈ [ x i , x i +1 ] . If θ is a local minimum of Problem (1), it holds that N θ ( x ) ≥ 0 .

In particular, the proposition above shows that so we get

As in the lower bound proof, recalling that any local minimum is in particular a KKT point, we saw in the proof of Theorem 3.1 that under the event E x , changing a single label y l , l ∈ [ m ] can change the test error by at most O ( d max ) = O (log( m/δ ) /m ) . Hence applying McDiarmid's inequality proves the upper bound thus finishing the proof.

Proof of Proposition A.4. Throughout the proof we fix i ∈ [ m ] for which the conditions described in the proposition hold, and we assume without loss of generality that the neurons are ordered

with respect to their activation point: -b 1 w 1 ≤ -b 2 w 2 ≤ · · · ≤ -b n w n . Moreover, we may assume without loss of generality that w 1 , . . . , w n ≥ 0 . Indeed, note that Eq. (1) is invariant under the transformation v j ←-v j , w j ←-w j , b j ←-b j which does not affect neither the parameter norm nor the parameterized network in function space. Hence, any local minimum of Eq. (1) corresponds to a local minimum with w 1 , . . . , w n ≥ 0 . Lastly, we will make frequent use of the following simple observation. For differentiable x we have

so if N ′ θ ( z 1 ) < N ′ θ ( z 2 ) for some z 1 < z 2 , then there must exist j ∈ [ n ] with v j > 0 and z 1 < -b j w j < z 2 . Similarly, N ′ θ ( z 1 ) > N ′ θ ( z 2 ) implies that there exists j ∈ [ n ] with v j < 0 , z 1 < -b j w j < z 2 .

We split the proof of Proposition A.4 into two lemmas.

Lemma A.6. N θ ( · ) is linear over ( x i , x i +1 ) .

Proof. Recall that N θ ( x i ) ≤ -1 and N θ ( x i +1 ) ≥ 1 , so N θ increases along the segment ( x i , x i +1 ) . Thus, N θ ( · ) is not linear over ( x i , x i +1 ) only if (at least) one of the following occur: (1) There exist z 1 , z 2 ∈ ( x i , x i +1 ) such that z 1 < z 2 and 0 < N ′ θ ( z 1 ) < N ′ θ ( z 2 ) ; (2) there exist z 1 , z 2 ∈ ( x i , x i +1 ) such that z 1 < z 2 , N ′ θ ( z 1 ) > 0 and N ′ θ ( z 1 ) > N ′ θ ( z 2 ) ; or (3) there exists a single z ∈ ( x i , x i +1 ) at which N θ is non-differentiable, such that N ′ θ | ( x i ,z ) ≤ 0 and N ′ θ | ( z,x i +1 ) > 0 . We will show either of these contradict the assumption that θ is a local optimum of the margin maximization problem.

Case (1). The assumption on z 1 , z 2 implies that there exists j 1 ∈ [ n ] such that -b j 1 w j 1 ∈ ( z 1 , z 2 ) and v j 1 > 0 . Let j 2 > j 1 be the minimal index j ∈ { j 1 +1 , . . . , n } for which v j < 0 . 7 For some small δ > 0 , consider the perturbed network

It is clear that ∥ θ -θ δ ∥ δ → 0 -→ 0 , and we will show that for small enough δ the network above still satisfies the margin condition, yet has smaller parameter norm. To see why the margin condition is not violated for small enough δ , notice that N θ δ ( x ) = N θ ( x ) for all x / ∈ ( x i , -b j 2 w j 2 ) so in particular N θ δ ( x l ) = N θ ( x l ) for all l ∈ [ i ] , as well as for all x l ≥ -b j 2 w j 2 . Furthermore, by minimality of j 2 we note that there cannot exist y k = -1 for k such that x k ∈ ( -b j 1 w j 1 , -b j 2 w j 2 ) . A direct computation gives that N θ δ ≥ N θ along this segment, so overall the margin condition is indeed satisfied for the

entire sample. As to the parameter norm, we have

Thus,

By construction, v j 1 > 0 and v j 2 < 0 hence -v j 1 w j 1 v j 2 w j 2 > 0 . Moreover, -b j 1 w j 1 > 0 and w j 1 > 0 so -b j 1 1 -δ > 0 for δ < 1 . Lastly, recall that j 2 > j 1 = ⇒ -b j 2 w j 2 ≥ -b j 1 w j 1 = ⇒ b j 1 -w j 1 b j 2 w j 2 ≥ 0 . Overall, plugging these into Eq. (11) shows that ∥ θ ∥ 2 > ∥ θ δ ∥ 2 for small enough δ , contradicting the assumption the θ is a local minimum.

Case (2). Since N θ ( x i -1 ) ≥ 1 and N θ ( x i ) ≤ -1 , N ′ θ must be negative somewhere along the segment ( x i -1 , x i ) . Yet, N ′ θ ( z 1 ) > 0 so there must exist j 1 ∈ [ n ] such that -b j 1 w j 1 ∈ ( x i -1 , z 1 ) and v j 1 > 0 . Moreover, the assumption on z 1 , z 2 implies that there exists j 2 ∈ [ n ] such that -b j 2 w j 2 ∈ ( z 1 , z 2 ) and v j 2 < 0 . For some small δ > 0 , consider the perturbed network

̸

It is clear that ∥ θ -θ δ ∥ δ → 0 -→ 0 , and we will show that for small enough δ the network above still satisfies the margin condition, yet has smaller parameter norm. To see why the margin condition is not violated for small enough δ , notice that N θ δ ( x ) = N θ ( x ) for all x / ∈ ( x i -1 , x i +1 ) so in particular N θ δ ( x l ) = N θ ( x l ) for all l = i . Furthermore, a direct computation yields N θ δ ( x i ) < N θ ( x i ) ≤ 1 . As to the parameter norm, by a similar computation to that leading up to Eq. (11) we get that

By construction, v j 1 w j 1 > 0 and v j 2 w j 2 < 0 hence -v j 1 w j 1 w j 2 v j 2 > 0 . Moreover, -b j 1 w j 1 > 0 and -b j 2 w j 2 > 0 so we also have -v j 1 b j 1 b j 2 v j 2 > 0 . Hence, Eq. (13) shows that ∥ θ ∥ 2 > ∥ θ δ ∥ 2 for small enough δ , contradicting the assumption the θ is a local minimum.

Case (3). The assumption implies that there exists j 1 ∈ [ n ] such that -b j 1 w j 1 = z ∈ ( x i , x i +1 ) , v j 1 > 0 and -b j 1 +1 w j 1 +1 ≥ x i +1 . Let j 2 > j 1 be the minimal index j ∈ { j 1 +1 , . . . , n } such that v j < 0 (see Footnote 7). Consider the perturbed netowrk as in Eq. (10) and continue the proof as in Case (1) verbatim.

Lemma A.7. N θ ( x i ) = -1 and N θ ( x i +1 ) = 1 .

Proof of Lemma A.7. Recall that N θ ( x i ) ≤ -1 , so assume towards contradiction that N θ ( x i ) < -1 . Note that since N θ ( x i -1 ) ≥ 1 , N θ ( · ) must decrease along ( x i -1 , x i ) , so there must exist j ∈ [ n ] with v j < 0 and -b j w j < x i . Denote by j 1 ∈ [ n ] the maximal such index. Similarly, since N θ ( x i +1 ) ≥ 1 there must exist j 2 ∈ [ n ] with v j 2 > 0 and x i -1 < -b j 2 w j 2 < x i +1 . Consider the perturbed network

for some small δ > 0 . It is clear that ∥ θ -θ δ ∥ δ → 0 -→ 0 , and we will show that for small enough δ the network above still satisfies the margin condition, yet has smaller parameter norm. To see why the margin condition is not violated for small enough δ , first notice that N θ δ ( x ) = N θ ( x ) for all x / ∈ ( x i -1 , x i +1 ) . Furthermore, N θ δ ( x i ) is continuous with respect to δ so N θ ( x i ) < -1 implies that N θ δ ( x i ) < -1 for small enough δ . As to the parameter norm, by a similar computation to that leading up to Eq. (11) we get that

By construction, v j 1 < 0 and v j 2 > 0 hence -v j 1 w j 1 w j 2 v j 2 > 0 . Moreover, -b j 1 w j 1 and -b j 2 w j 2 > 0 so we also have -v j 1 b j 1 b j 2 v j 2 > 0 . Hence, Eq. (15) shows that ∥ θ ∥ 2 > ∥ θ δ ∥ 2 for small enough δ , contradicting the assumption the θ is a local minimum.

Having proved that N θ ( x i ) = -1 , we turn to prove that N θ ( x i +1 ) = 1 . Knowing that N θ ( x i +1 ) ≥ 1 , we assume towards contradiction that N θ ( x i +1 ) > 1 . Recalling that N θ ( x i -1 ) ≥ 1 , N θ ( x i ) ≤ 1 and that N θ ( · ) is linear along ( x i , x i +1 ) due to Lemma A.6, we conclude that there must exist j 1 ∈ [ n ] such that v j 1 > 0 and x i -1 < -b j 1 w j 1 ≤ x i . Denote by j 2 > j 1 the minimal index such that v j 2 < 0 . Consider once again the perturbed network in Eq. (14) (only now j 1 , j 2 are different, as we just described). The same argument as in the previous part of the proof shows that for small enough δ the network above still satisfies the margin condition, while Eq. (15) once again implies that ∥ θ ∥ 2 > ∥ θ δ ∥ 2 for small enough δ - contradicting the assumption the θ is a local minimum.

Overall, combining Lemma A.6 and Lemma A.7 finishes the proof of Proposition A.4.

Proof of Proposition A.5. The proof is essentially the same as Case (1) in the proof of Proposition A.4.

Throughout the proof we fix i ∈ [ m -1] for which the conditions described in the proposition hold, and we assume without loss of generality that the neurons are ordered with respect to their activation point: -b 1 w 1 ≤ -b 2 w 2 ≤ · · · ≤ -b n w n . As in the proof of Proposition A.4, we may assume without loss of generality that w 1 , . . . , w n ≥ 0 . Moreover, as explained there as a consequence of Eq. (9), we observe that N ′ θ ( z 1 ) < N ′ θ ( z 2 ) for some z 1 < z 2 implies the existence of j 1 ∈ [ n ] with v j 1 > 0 and z 1 < -b j 1 w j 1 < z 2 . Similarly, N ′ θ ( z 1 ) > N ′ θ ( z 2 ) implies that there exists j 2 ∈ [ n ] with v j 2 < 0 , z 1 < -b j 2 w j 2 < z 2 . With this choice of j 1 , j 2 , the proof continues as in Case (1) in the proof of Proposition A.4 verbatim.

B Proofs of benign overfitting

B.1 Proof of Theorem 4.1

We sample a dataset x 1 , . . . , x m ∼ Unif( S d -1 ) , and labels y 1 , . . . , y m ∼ D y , for p ≤ c 1 for some universal constant c 1 . We first prove that the following properties holds with probability > 1 -δ :

∥ XX ⊤ ∥ ≤ C where C is some universal constant, and X is a matrix whose rows are equal to x i .

| I -| ≤ 3 pm 2 .

Lemma B.1. Let δ > 0 , assume that we sample x 1 , . . . x m ∼ Unif( S d -1 ) , and y 1 , . . . , y m ∼ D y for p ≤ c 1 , and that m>c 2 log ( 1 δ ) p , for some universal constant c 1 , c 2 > 0 . Then, with probability > 1 -δ properties 1, 2, 3 holds.

̸

Proof. By Lemma D.1 we have that Pr [ | x ⊤ i x j | ≥ √ 2 log ( 1 δ ′ ) d ] ≤ δ ′ . Take δ ′ = δ 3 m 2 , and use union bound over all i, j ∈ [ m ] with i = j . This shows that for every i = j we have:

̸

This proves Property 1. Next, set X to be the matrix whose rows are equal to x i . By Lemma D.2 there is a constant c ′ > 0 such that:

We can also bound ∥ XX ⊤ ∥ ≤ ∥ I ∥ + ∥ XX ⊤ -I ∥ ≤ 1 + ∥ XX ⊤ -I ∥ . Combining both bounds and using the assumption that m ≥ log ( 3 δ ) we get that there is a universal constant C > 0 such that Pr [ ∥ XX ⊤ ∥ > C ] ≤ δ 3 . This proves Property 2.

Finally, using Bernstein's inequality over the choice of the labels y i have that:

where we used that p ≤ 1 . Hence, if m ≥ c 2 log ( 1 δ ) p for some universal constant c 2 ≥ 0 , then | I -| ≤ 3 pm 2 . Applying union over those three arguments proves the lemma.

From now on we condition on the event that properties 1, 2, 3 hold, and our bounds will depend on the probability of this event.

In the following lemma we show that if θ converges to a solution of the max margin problem and the bias terms are relatively small, then the norm of θ is relatively large:

Lemma B.2. Assume that m ≥ c 2 log ( 3 δ ) and that θ = ( w j , v j , b j ) n j =1 is a solution to the max margin problem of Eq. (1) with ∑ n j =1 v j σ ( b j ) ≤ 1 2 . Then, ∑ n j =1 ∥ w j ∥ 2 + v 2 j + b 2 j ≥ C √ | I + | where C is some universal constant.

Proof. Take i ∈ [ m ] , then we have that N θ ( x i ) ≥ 1 . By our assumption, this implies that:

where in the last inequality we used Cauchy-Schwartz. Denote by S := ∑ n j =1 ∥ w j ∥ 2 + v 2 j + b 2 j . Combining the above and that √ ∑ n j =1 v 2 j ≤ √ S we get:

We sum the above inequality for every i ∈ I + to get:

where in the second to last to last inequality we used the Property 2 for some constant C > 0 , and in the last inequality we used that ∑ n j =1 ∥ w j ∥ 2 ≤ S . Rearranging the above terms yields: S ≥ √ | I + | 12 C .

We now prove a lemma which constructs a specific solution that achieves a norm bound that depends on | I + | .

Lemma B.3. Assume d ≥ 50 m 2 log ( 3 m 2 δ ) and p ≤ 1 4 . There exists weights θ = ( w j , v j , b j ) n j =1 that attain a margin of at least 1 on every sample and have ∑ n j =1 ∥ w j ∥ 2 + v 2 j + b 2 j ≤ 9 √ | I -| .

Proof. Assume without loss of generality that n is even (otherwise fix the last neuron to be 0 ). We consider the following weight assignment: For every j ≤ n 2 , consider w j = -√ 4 n √ | I -| ∑ i ∈ I -x i ,

We first show that this solution attains a margin of at least 1 . For every i ∈ I + we have:

where we Property 1, that p ≤ 1 4 hence by Property 3 | I -| ≤ m 2 and our assumption on m and d . For i ∈ I -we have:

where again we used properties 1 and 3, that p ≤ 1 4 hence | I -| ≤ m 2 and our assumption on m and d . This shows that this is indeed a feasible solution. We turn to calculate the norm of this solution. First we bound the following:

̸

where we used Property 1 and the assumption on m and d . We now use the above calculation to bound the norm of our solution:

We are now ready to prove the main theorem of this subsection:

Proof of Theorem 4.1. Denote by K := ∑ n j =1 ∥ w j ∥ 2 + v 2 j + b 2 j , and assume that K ≤ a √ p ∥ θ ∗ ∥ 2 , where a will be chosen later and θ ∗ is a solution to the max margin solution from Eq. (1). Assume on the way of contradiction that ∑ n j =1 v j σ ( b j ) ≤ 1 2 , then by Lemma B.2 we know that: K ≥ C √ | I + | ≥ C √ ( 1 -3 p 2 ) m . On the other hand, by Lemma B.3 we know that ∥ θ ∗ ∥ ≤ 9 √ | I -| ≤ 9 √ 3 p 2 m . Combining this with the assumption we have on K we get that:

Picking a to be some constant with a < C √ 2 18 √ 3 contradicts the above inequality. Thus, there exists a constant c 4 := a 2 such that if K ≤ c 4 √ p S then ∑ n j =1 v j σ ( b j ) > 1 2 .

Suppose we sample x ∼ N ( 0 , 1 d I ) , then we have:

We will now bound the terms of the above equation. Note that ∑ n j =1 v 2 j ≤ K ≤ 9 √ m . For the second term, we denote by ¯ w j := w j ∥ w j ∥ , and write:

Again, we have that ∑ n j =1 ∥ w j ∥ 2 ≤ K ≤ 9 √ m . By using the stationarity KKT condition from Eq. (2) , we get that ¯ w j ∈ span { x 1 , . . . , x m } , thus we can write ¯ w j = ∑ m i =1 α i,j x i = X α j where

X is a matrix with rows equal to x i , and ( α j ) i = α i,j . We will bound a := arg max i | α i,j | :

where the last equality is true by using the SVD decomposition of X . Namely, write X = USV ⊤ , then ∥ X ⊤ X -I ∥ = ∥ V S 2 V ⊤ -I ∥ = ∥ S 2 -I ∥ = ∥ U ⊤ S 2 U -I ∥ = ∥ XX ⊤ -I ∥ . Note that in Lemma B.1, Eq. (16) we have shown that ∥ XX ⊤ -I ∥ ≤ c ′ for some constant c ′ . Note that we can choose m large enough such that c ′ ≤ 1 2 . In total, this shows that ∥ α j ∥ 2 ≤ c ′′ for some constant c ′′ .

We now use Lemma D.1 and the union bound to get that with probability > 1 -ϵ we have for every i ∈ I that | x ⊤ x i | ≤ √ 2 log ( m ϵ ) d . We condition on this event from now on. Applying both bounds we get:

Plugging in the bounds above to Eq. (17) we get:

By choosing a constant c 3 > 0 large enough such that if d ≥ c 3 m 2 √ log ( m ϵ ) then , then 81 mc ′′ √ 2 log ( m ϵ ) √ d ≤ 1 4 we get that N θ ( x ) > 0 .

B.2 Proof of Theorem 4.3

We sample a dataset x 1 , . . . , x m ∼ Unif( S d -1 ) , and labels y 1 , . . . , y m ∼ D y , for p ≤ c 1 for some universal constant c 1 . We first prove that the following properties holds with probability > 1 -δ :

n

Lemma B.4. Let δ > 0 , assume that we sample x 1 , . . . x m ∼ Unif( S d -1 ) , and y 1 , . . . , y m ∼ D y for p ≤ c n 2 , and that m>c ′ log ( 2 δ ) , for some universal constant c, c ′ > 0 . Then, with probability > 1 -δ properties 1 and 2 holds.

The proof is the same as in Lemma B.1 so we will not repeat it for conciseness. The only different is that here we only need | I -| to be smaller than c 1 m n 2 which is independent of p . Hence, for p ≤ c n 2 we get this concentration where m depends only on the probability. From now on we condition on the event that properties 1 and 2 hold, and our bounds will depend on the probability that this event happens.

In this section we consider a networks of the form:

That is, the output weights are all fixed to be ± 1 , equally divided between the neurons. We will use the stationarity condition (Eq. (2)) freely throughout the proof. For our network it means that we can write for every j ∈ [ n ] :

where σ ′ i,j = ✶ ( w ⊤ j x i + b j > ) , v j = 1 for j ∈ { 1 , . . . , n/ 2 } and v j = -1 for j ∈ { n/ 2+1 , . . . , n } . The next lemma shows that all the biases are positive. Note that this lemma relies only on that the dimension is large enough, and that Property 1 of the data holds:

Lemma B.5. Assume that d ≥ 8 m 4 log ( 2 m 2 δ ) . Then for every j ∈ [ n ] we have b j ≥ 0 .

Proof. Assume on the way of contradiction that b j < 0 for some j ∈ [ n ] . We assume without loss of generality that j ∈ { 1 , . . . , n/ 2 } , the proof for j ∈ { n/ 2 + 1 , . . . , n } is done similarly. Denote by I ′ + = { i ∈ I + := σ i,j = 1 } and I ′ -= { i ∈ I -:= σ i,j = 1 } . We can write:

Note that λ i ≥ 0 for every i , hence I ′ -is non empty, otherwise b j ≥ 0 . Take λ r = arg max i ∈ I ′ -λ i , we have:

where we used that b j < 0 and ∥ x r ∥ = 1 . Rearranging the terms and using Property 1:

Denote λ s := arg max i ∈ I ′ + ∪ I ′ -λ i , then from the above we have shown that λ r ≤ λ s m √ 2 log ( 2 m 2 δ ) d , since | I ′ + ∪ I ′ -| ≤ m . In particular, s ∈ I ′ + , otherwise, λ r = λ s which means that λ r < 0 since

By our assumption on d , we have that m 2 √ 2 log ( 2 m 2 δ ) d ≤ 1 2 , hence b j ≥ λ s 2 . But, by our assumption, b j < 0 which is a contradiction since λ s ≥ 0 .

The following lemma is a general property of the KKT conditions:

Lemma B.6. Let i ∈ I + (resp. i ∈ I -) with λ i > 0 . Then, there is a neuron k with positive output weight (resp. negative output weight) s.t σ ′ i,k = 1 .

Proof. We prove it for i ∈ I + , the other case is similar. Assume otherwise, that is for every neurons with positive output j we have σ ′ i,j = 0 , that is w ⊤ j x i + b j ≤ 0 by the definition of σ ′ i,j . This means that N θ ( x i ) ≤ 0 , since all the positive neurons are inactive on x i , which is a contradiction to N θ ( x i ) ≥ 1 .

The next lemma shows that if the bias terms are smaller than 1 4 , then the λ i 's for i ∈ I + cannot be too small.

Lemma B.7. Assume that d ≥ 32 m 4 n 2 log ( 2 m 2 δ ) , and that ∑ n/ 2 j =1 b j -∑ n j = n/ 2+1 b j ≤ 1 4 . Then, for every i ∈ I + we have λ i ≥ 1 4 n .

Proof. Take r ∈ I + , we have:

We will bound the two terms above. For the first term, note that all the terms inside the ReLU are positive, since by Lemma B.5 we have b j ≥ 0 , and λ i ≥ 0 by Eq. (4) hence we can remove the ReLU

function. Using Property 1 we can bound:

where we used that σ ′ i,j ≤ 1 . For the second term in Eq. (18) we use the fact that the ReLU function is 1 -Lipschitz to get that:

where we again used Lemma B.5 to get that b j ≥ 0 , and Property 1. Combining Eq. (19) and Eq. (20) with Eq. (18) we get that:

Rearranging the terms above, and using our assumption on the biases we get that:

If λ r ≥ 1 4 n then we are done. Assume otherwise, then n √ 2 log ( 2 m 2 δ ) d ∑ i ∈ I \{ r } λ i ≥ 1 2 and λ r < 1 4 n . Denote λ s := arg max i ∈ [ m ] λ i , then we have that ∑ i ∈ [ m ] \{ r } λ i ≤ mλ s , which by the above inequality means that λ s ≥ √ d 2 mn √ 2 log ( 2 m 2 δ ) . We split into cases:

Case I. Assume s ∈ I + . By Lemma B.6 there is k ∈ { 1 , . . . , n/ 2 } with σ ′ s,k = 1 . For the k -th neuron we have:

and note that since λ s ≥ 0 , m √ 2 log ( 2 m 2 δ ) d < 1 and b j ≥ 0 this neuron is active. For every other neuron j with a positive output weight we have:

where we used that σ is 1 -Lipschitz and that λ s is the largest among the λ i 's. For a neuron with a negative output weight, we can write:

In total, combining the above bound with Eq. (21) and Eq. (22) we have that:

By our assumption on d we have that mn √ 2 log ( 2 m 2 δ ) d ≤ 1 2 . Hence, rearranging the terms above, we get that:

But then, looking at the output of the network on λ r (recall that by our assumption λ r ≤ 1 4 n ) we get:

By our assumption mn √ 2 log ( 2 m 2 δ ) d ≤ 1 4 , hence rearranging the terms above we get that λ s ≤ 1 2 which is a contradiction to λ s ≥ √ d 2 mn √ 2 log ( 2 m 2 δ ) > 1 .

̸

Case II. Assume s ∈ I -. By Lemma B.6 there is k ∈ { n/ 2 + 1 , . . . , n } with σ ′ s,k = 1 . Note that since λ s = 0 we have that N θ ( x s ) = -1 (i.e this sample is on the margin). We have that:

where we used that mn √ 2 log ( 2 m 2 δ ) d ≤ 1 2 and b j ≥ 0 by Lemma B.5, hence in the second to last inequality the term inside the ReLU is positive. In addition, we used that the ReLU function is 1 -Lipschitz, and that λ s is the largest amont the λ i 's. Since mn √ 2 log ( 2 m 2 δ ) d ≤ 1 2 , rearranging the terms above give us that λ s ≤ 10 4 , which is a contradiction to that λ s ≥ √ d 2 mn √ 2 log ( 2 m 2 δ ) ≥ 3 .

To conclude, both cases are not possible, hence λ r ≥ 1 4 n , which is true for every r ∈ I + .

Lemma B.8. Assume that d ≥ 16 m 4 n 4 log ( 2 m 2 δ ) , and that ∑ n/ 2 j =1 b j -∑ n j = n/ 2+1 b j ≤ 1 4 . Then, for every i ∈ I -we have λ i ≤ 10 4 .

Proof. Take λ s := arg max i ∈ [ m ] λ i . We split into two cases:

Case I. Assume s ∈ I -. Note that λ s > 0 , otherwise λ i = 0 for every i , which means that N θ ( x ) is the zero function. Hence, x s lies on the margin, and also by Lemma B.6 there is a neuron j with a negative output weight such that σ ′ s,j = 1 . By the same calculation as in Eq. (24) we have::

Using our assumption that mn √ 2 log ( 2 m 2 δ ) d ≤ 1 2 and rearranging the above terms we get: λ s ≤ 10 4 which finishes the proof.

Case II. Assume s ∈ I + . Denote by λ r := S arg max i ∈ I -λ i . By Lemma B.6 there is at least one neuron k with a positive output weight such that σ ′ s,k = 1 , for this neuron we can bound:

Every neuron j with a negative output weight can be bounded similarly by:

Combining the above bounds, and using that b j ≥ 0 by Lemma B.5 we can bound:

Rearranging the terms we get that: λ s ≤ mnλ r + 1 4 . If λ r ≤ 1 we are finished, otherwise λ s ≤ 2 mnλ r and also x r lies on the margin since λ r > 0 , hence. By doing a similar calculation to Case I we can write:

By the assumption on d we have 2 m 2 n 2 √ 2 log ( 2 m 2 δ ) d ≤ 1 2 . Thus, rearranging the terms above and using these bounds we get that λ r ≤ 10 4 .

LemmaB.9. Assume that p ≤ c 1 n 2 , m ≥ c 2 n log ( 1 δ ) and d ≥ 32 m 4 n 4 log ( 2 m 2 δ ) . Then, ∑ n/ 2 j =1 b j -∑ n j = n/ 2+1 b j > 1 4 .

Proof. Assume on the way of contradiction that ∑ n/ 2 j =1 b j -∑ n j = n/ 2+1 b j ≤ 1 4 . By Lemma B.7 we have that λ i ≥ 1 4 n for every i ∈ I + , and by Lemma B.8 we have that λ i ≤ 10 4 for every i ∈ I -. By

Lemma B.6, for every i ∈ I + there is some j ∈ { 1 , . . . , n/ 2 } with σ ′ i,j = 1 . This means that:

In a similar manner, we can bound:

Combining the two bounds above we get that:

But, by our assumptions we have that m ≥ 2 n (for a large enough constant), and by Lemma B.4 we have | I -| ≤ cm n 2 and | I + | ≥ ( 1 -c n 2 ) m for some universal constant c > 0 which we will later choose. Plugging these bounds to the displayed equation above, we get that:

where in the last inequality we used that n ≥ 1 . Taking c to be a small enough constant (i.e. c < 1 22 ), we get that ∑ n/ 2 j =1 b j -∑ n j = n/ 2+1 b j > 1 4 , which is a contradiction.

We are now ready to prove the main theorem of this section:

Proof of Theorem 4.3. By Lemma B.9 we have that ∑ n/ 2 j =1 b j -∑ n j = n/ 2+1 b j > 1 4 . Denote by λ s := arg max i ∈ [ m ] λ i , we would first like to show that λ s ≤ 3 mn . We split into two cases:

Case I. Assume s ∈ I + . Note that x s lies on the margin, otherwise λ i = 0 for every i ∈ I , which means that N θ is the zero predictor, this contradicts the assumption that N θ classifies the data correctly. Using a similar analysis to Case I in Lemma B.7 we have:

4

where we used Lemma B.6 to show that there is at least one k ∈ { 1 , . . . , n/ 2 } with σ s,k = 1 . Using that mn √ 2 log ( 2 m 2 δ ) d ≤ 1 2 for a large enough constant c 3 and rearranging the terms above we have that λ s ≤ 6 ≤ 3 mn since m,n ≥ 1 .

Case II. Assume s ∈ I -. Take λ r := arg max i ∈ I + λ i . By Lemma B.5 there is at least one neuron k ∈ { n/ 2 + 1 , . . . , n } with σ ′ s,k = 1 . We have that:

Rearranging the terms we get λ s ≤ mnλ r -1 4 ≤ mnλ r . Note that x r lies on the margin, otherwise λ s = 0 , which similarly to Case I is a contradiction. Again, using a similar analysis to Case I we get:

By our assumption, m 2 n 2 log( d )2 √ 2 √ d ≤ 1 2 for a large enough constant c 3 . Hence, rearranging the terms we get λ r ≤ 6 4 . This means that λ s ≤ 2 mnλ r ≤ 3 mn .

We now turn to calculate the output of N θ on the sample x . Suppose we sample x ∼ Unif( S d -1 ) , by Lemma D.1 we have with probability > 1 -ϵ that | x ⊤ i x | ≤ √ 2 log ( m ϵ ) d for every i ∈ [ m ] . We condition on this event for the rest of the proof. Note that for every positive neuron j we have that:

Similarly, for every negative neuron j we have:

′

In both bounds above we used that λ s is the largest among the λ i 's. Combining both bounds, we have that:

We use that 3 m 2 n 2 log( d ) √ 2 √ d ≤ 1 8 for a large enough constant c 3 , hence N θ ( x ) ≥ 1 4 -1 8 > 0 , this finishes the proof.

C Proofs from Section 4.2

C.1 Proof of Proposition 4.4.

We first note that there exists j ∈ { 1 , 2 } with v j < 0 , since otherwise the network wouldn't be able to classify samples with a negative label (which exist by assumption). Assuming without loss of generality that v 2 < 0 , for any w 1 we also have that Pr x ∼ Unif( S d -1 ) [ w ⊤ 1 x ≤ 0] = 1 2 . If this even occurs, then N θ ( x ) ≤ v 2 σ ( w 2 ⊤ x ) ≤ 0 .

C.2 Proof of Proposition 4.5

Suppose [ n ] = J + ∪ J -are such that

where v j ≥ 0 for j ∈ J + and v j < 0 for j ∈ J -. Then, for any choice of ( w j ) j ∈ J + :

C.3 Proof of Proposition 4.6

Assume without loss of generality that I -= [ k ] . Consider the weights w i = x i , v i = -1 for every i ∈ I -, and w k +1 = ∑ i ∈ I + w i , v k +1 = 1 . For this network we have y i N θ ( x i ) = 1 , thus all points lie on the margin. In addition, it is not difficult to see that this network satisfies the other KKT conditions with λ i = 1 for every i ∈ [ m ] . Note that Pr x ∼ Unif( S d -1 ) [ w ⊤ k +1 x < 0] = 1 2 and Pr x ∼ Unif( S d -1 ) [ ∀ i ∈ I -, w ⊤ i x < 0] = 1 2 k , since all the x i are orthogonal. Thus, we get that

D Additional probabilistic lemmas

D.1 Proof of Lemma A.1

Clearly, the third (no collisions) condition holds almost surely, so it suffices to analyze the gaps between samples. The distribution of distances between uniformly random points on a segment is well studied [Pyke, 1965, Holst, 1980] and the lemma can be derived by known results. Nonetheless we provide a proof for completeness.

Denote by ∆ 1 ≤ ∆ 2 ≤ · · · ≤ ∆ m +1 the ordered spacings x 1 , ( x 2 -x 1 ) , . . . , ( x m -x m -1 ) , (1 -x m ) . With this notation, note that E x occurs if ∆ m +1 ≤ log(8( m +1) /δ ) m +1 and ∆ m/ 8 ≥ 1 10 m . We will show that each of these conditions holds with probability at least 1 -δ 4 , under which we would conclude by the union bound. Let Z 1 , . . . , Z m +1 iid ∼ Exp(1) be unit mean exponential random variables, and denote their ordering by Z (1) ≤ · · · ≤ Z ( m +1) . The main well known observation which we use is that for any j ∈ [ m +1] : ∆ j d = Z ( j ) ∑ m +1 (see Holst, 1980). Hence for any t > 0 :

Pr[(

m

+1)∆

j

-

i =1 Z i log(

m

+1)

t

]

Note that ∑ m +1 i =1 Z i m +1 -1 p -→ 0 as E [ ∑ m +1 i =1 Z i m +1 ] = 1 and Var [ ∑ m +1 i =1 Z i m +1 ] = m +1 ( m +1) 2 m →∞ -→ 0 . Furthermore,

By introducing the change of variables r = t +log( m +1) m +1 we conclude that

It remains to plug in our parameters of interest. For j = m +1 we get

Noting that for r = 8 log(8( m +1) /δ ) m +1 it holds that e -( m +1) r +log( m +1) → 0 so we can use the Taylor approximation exp( z ) ≈ 1 + z and conclude that

where the last equality holds for r = log(8( m +1) /δ ) m +1 . Overall for m larger than some numerical constant, the left hand side is larger than 1 -δ/ 4 as required.

The second condition follows a similar computation for j = m +1 8 , r = log( m/ 8 log(8 /δ )) 8( m +1) , while using the fact Pr [ ∆ m/ 8 > 1 10( m +1) ] = 1 -Pr [ ∆ m/ 8 ≤ 1 10( m +1) ] .

D.2 Concentration bounds for vectors on the unit sphere

Here we present some standard concentration bounds for uniformly sampled points on the unit sphere in high dimension.

Lemma D.2 (Exercise 4.7.3 from Vershynin, 2018) . Let x 1 , . . . , x m ∼ Unif( S d -1 ) , and denote by X the matrix whose i 'th row is x i . Then, there is a universal constant c > 0 such that:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work examines asymptotic overfitting behavior on a simple class of toy classification problem for two-hidden-layer ReLU nets, finding that increasing input dimension tends to push the model from tempered towards benign overfitting.\n\nSTRENGTHS:\nOriginality: this is, to my knowledge, the first analysis of tempered overfitting for nets outside the kernel regime. The broader problem's admittedly decently well-studied, and the conclusions here are more or less in line with intuitions from other works, but the authors are doing something new here.\n\nQuality: while I haven't checked the proofs, the theoretical results seem conceptually solid. While they rely on certain assumptions (e.g., assuming you've reached a KKT point or local minimum), the authors are upfront about these. I have some Qs about the experiments below.\n\nClarity: the paper's very well organized and written. Most things were clear on a single linear readthrough.\n\nSignificance: to my mind, the nice takeaways here are (a) the identification of a very simple toy problem for the study of these regimes and (b) theoretical results for classification that complement those of Mallinar et al. (which only study (kernel) *regression*). It's nice that the authors recover the $\\Theta(p)$ scaling in their Theorem 3.2. I hadn't seen the cited results about margin maximization used in a setting like this; it's nice that they let one avoid consideration of the network dynamics entirely.\n\nWEAKNESSES:\nOne omission here seems to be comparison to the experiment of Figure 9 of Mallinar et al, which shows that Laplace kernels basically exhibit the core phenomenon found by the paper: i.e., larger input dim pushing fitting from tempered to benign (with no need to scale dim w.r.t. dataset size).\n\nI suppose the biggest weakness here is that this paper's results seem like exactly what you'd intuitively expect from (the much simpler model of) kernel regression, and so it's sort of unclear what new thing we've gained from this long set of calculations. (For the purposes of review, I wouldn't count this *against* the work -- and I do appreciate that establishing facts for neural nets is hard, and improving techniques there is useful.) It's also sort of unclear to me that the proof techniques used here will scale to other problems -- they seem clever but ad-hoc, and the tempered results rely crucially on the input space being 1D -- and so it's not obvious that, e.g., the specialized + expected results here will enable extension to other more general + powerful conclusions later.\n\nThe empirics seem generally good, but a bit odd in certain ways. For example, the input dimensions are very large! The aforementioned Figure 9 of Mallinar et al. suggests that much smaller input dims should be needed to see the correct behavior here. (Could be due to the difference between classifcation + regression; see note A below). A bigger oddity's that the difference in training set size between the two plots of Figure 1 is only 4x, which is quite small. It seems like the authors are trying to say that maybe \"when [input dim] >> [dataset size], fitting is benign, and vice versa,\" but if that's the message, there are much more compelling ways of showing that, including having much greater variation in both parameters or making a 2D heatmap at some fixed and intermediate $p$.\n\nAnother concern with the empirics is that converging can presumably take a very long time when you're relying on the exponential tail of the loss fn to push you. Seems worth confirming that these experiments won't change after a much longer training time.\n\nQUESTIONS:\nCould the authors explain the definition of \"local minimum\" before Theorem 3.2? I'm confused as to how points satisfying this condition are local minima.\n\nNote A: as an aside, just to flag it, a potentially interesting Q here seems like studying the difference between classification and regression. I suspect that Figure 1 would look different in these two cases. (In particular, for regression, the results of Mallinar et al. for KR suggest that you'd see a linear fit whose *slope* depends on input dim, whereas for classification you see this quite complicated nonlinear shape that intersections with the line $y=x$ at the endpoints. I wonder what the shape of that curve is.) This Q is AFAIK totally unstudied, even for KR, for which is seems probably tractable.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the various settings of overfitting, including benign overfitting where an interpolating estimator is optimal in spite of perfectly fitting noisy labels, tempered overfitting where an interpolating estimator is inconsistent but error is a function of the label noise, and catastrophic overfitting where an interpolating estimator has no ability to generalize (random guessing in classification or infinite risk in regression). The authors follow-up prior works in benign overfitting by showing that a two layer neural network can benignly overfit so long as the input dimension of the data grows faster than the number of training samples, irrespective of the width of the network itself (assuming convergence to a max-margin solution, and additionally show another benign overfitting result that only requires convergence to a KKT point of a max-margin problem). \n\nThey also provide a theoretical justification for tempered overfitting in two-layer ReLU networks, a phenomena that was empirically studied in prior work. The authors precisely characterize the excess risk of an interpolating network trained to fit uniform data with labels samples as +1 and flipped to -1 with probability p. The authors show that for one-dimensional data, the excess risk of such a network interpolating this problem is tightly bounded by functions of the noise level (e.g. theta(poly(p)) and theta(p) in a more restrictive setting).\n\nThe authors then observe that catastrophic overfitting can occur when forcing networks, of width at least 2, without a bias to interpolate the training data (drawn as specified before) when there is at least one sample with -1 label. They also show that you cannot obtain benign overfitting from convergence to a KKT point of the max-margin problem unless the network has a bias term.\n\nFinally, the authors provide an extended version of an experiment from prior work on tempered overfitting in which fully-connected ReLU networks are trained to interpolate data on the unit sphere with +1/-1 labels sampled with probability p, as defined before. They observe that there are regions between benign and tempered that are a function of the ratio of input data dimension to number of training samples, namely: as the input data dimension grows higher than the number of training samples, the excess risk of the model approaches 0. They additionally show experimental results on 3-layer networks to empirically assert that, while their theory holds for 2 layer networks practically the same behavior is seen for 3. Finally, they experimentally show in their setting that the width of the network is not as important as the input dimension when evaluating consistency of the estimator.\n\nEDIT: Thank you for the rebuttal and engaging response, this is indeed a very thoughtful work. I opt to keep my positive score.\n\nSTRENGTHS:\nThis paper provides a clear theoretical analysis for tempered overfitting for two-layer ReLU networks that has not appeared in prior literature. There is some earlier theoretical work on tempered overfitting in kernel regression and for minimum description length settings, but as far as I know nothing for a standard finite-width neural network. The prior cited work (Mallinar et al. 2022) shows experimental evidence with neural networks that suggests tempered overfitting, and following their formalisms the authors of this work go a step further to give proofs of excess risk of one of the experiments in that paper (more on this in later section). \n\nI really appreciate the authors providing proof outlines and intuitions in the main text, as well as citing references where they are using techniques and following proof styles introduced in prior works. The style of explanation is accessible and gives me, at a high level, the right amount of information to contextualize the theorem statement while deferring details to the appendix should I want to review them.\n\nThere are many intriguing results from this paper. For one, I am initially surprised by the one-dimensional tempered results and Theorem 4.1 on benign overfitting feature bounds that don’t depend on the width of the network. I’ve written more about this in the questions section, but I would have expected a more subtle interplay between width of the network and input dimension of the data. Notably, though, the width of the network has to be sufficiently large to interpolate the data (e.g. overparameterized) and since the authors study 2-layer networks this might imply a large width, though their experimental results show roughly similar behavior for width 3000 and width 50 networks) across varying input dimensionality.\n\nI find the role of bias section illuminating as well. To the best of my knowledge, I haven’t seen a generic negative result like this, that any bias-less network will have a risk lower bounded by a function of the network width or the support of the negative training samples.\n\nWEAKNESSES:\nThe authors do acknowledge it and give a tighter bound subsequently under stronger assumptions, but Theorem 3.1 has an upper and lower bound gap that is growing with p. It’s not really a weakness of the paper, but it would be nice if the authors wrote a bit about when this bound can be very tight (as a function of p, m?).\n\nTheorem 4.3 seems to indicate a dependence on p being smaller than some constant divided by the width of the network squared, which seems to indicate for very wide networks that this result holds for p shrinking -> 0. While interesting, I’m not sure if this is the most general result and the authors don’t discuss in too much detail the implications of this, and what settings of network width lead to various results apart from n=O(1). For instance, it could be useful to assert if p is meaningful for n >> m or n ~ m, etc. There are common settings of network width with relation to number of samples or input dimensionality and at first glance I don’t know whether this theorem will give me a useful result in many of those settings. \n\nAdditionally, this theorem trades off one assumption (convergence to a max-margin solution vs. KKT point of max-margin problem) for another (output weights are fixed as +- 1 and only the input weights are trained). They mention this theorem stems from attempting to drop the strong assumptions of the prior theorem (Theorem 4.1), but to me it seems perhaps to still include a new, strong assumption. Granted, I’m unfamiliar with the works cited in the footnote that study this setting in particular. Maybe the authors can provide a little exposition about why this setting is of particular interest and what motivates it in the grander study of neural networks?\n\nMissing concluding thoughts, would like to see some final discussion on everything and where the authors see next steps going.\n\nOther notes / fixes:\n\n- Page 5, 3rd paragraph: double “we we” in the line “we we get that the total length of all…”\n- Figures could use grid-lines as well as separate line-styles to differentiate the settings for accommodation to colorblind people and for printing in black and white\n\nQUESTIONS:\nIn the setting where all layers are trained the width does not seem to be so important. This definitely speaks to the intuition I have that the ambient dimension of the data is what matters. When studying a random feature model the ambient dimension is indicated by the width of the first layer, but when you train that layer too the ambient dimension would be the input dimension. These results make me want to understand more the interplay between intrinsic and ambient dimension, as well as bottleneck vs. non-bottleneck networks in overfitting. Do you have any further intuitions about these comparisons? Is this width-independence something you would expect to hold for deeper networks, or other architectures?\n\nDoes Proposition 4.5 subsume Proposition 4.4? Could you explain a bit more about the difference here.  n=2 in Prop 4.5 would lead to a lower bound of ¼ but in Prop 4.4 the lower bound is ½. I haven’t read deeply the proofs on this section but it might help to give a little more exposition on what assumptions are made about width that differ between 4.4 and 4.5.\n\nYou prove tempered overfitting for one-dimensional data only, but practically it seems that it is more about certain settings where the ratio of input dimension to number of samples satisfies some criteria (at least, experimentally and based on prior works). Can you speak to the difference between the theorem you propose and practice? It appears that tempered overfitting could occur at any input dimension, depending on how much training data is used. Can your proof techniques be extended to explain this?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the phenomenon of overparameterized neural networks (NNs) generalizing well even when trained on noisy data. Previous research focused on \"benign overfitting,\" where interpolating predictors achieve near-optimal performance. However, recent empirical observations suggest that NN behavior is better described as \"tempered overfitting,\" with non-optimal yet non-trivial performance that degrades with increasing noise levels. In this study, the authors provide theoretical justification and empirical validation, showing that the type of overfitting transitions from tempered to benign as the dimensionality increases in a simple classification setting with 2-layer ReLU NNs. Their findings highlight the intricate connections between input dimension, sample size, architecture, training algorithm, and resulting overfitting.\n\nSTRENGTHS:\nThis paper establishes the connection between data dimensionality and the occurrence of benign or tempered overfitting. It presents three theorems that address benign, tempered, and catastrophic overfitting, respectively, corresponding to one-dimensional data, high-dimensional data, and intermediate-dimensional data.\n\nThe study focuses on a two-layer ReLU neural network that includes a bias term, which is a more comprehensive approach compared to recent works that examined the two-layer ReLU neural network without a bias term. Additionally, a specific data distribution is considered in this paper. Furthermore, the paper explores the relationship between the bias term and the occurrence of benign/tempered overfitting.\n\nWEAKNESSES:\nThe findings presented in this paper are built upon the outcomes (specifically, the convergence to the KKT point of the max-margin problem) obtained in previous studies by Lyu and Li (2020) and Ji and Telgarsky (2020). To be more precise, the results in this paper depend on either achieving convergence to a KKT point or making stronger assumptions, such as converging to a local optimum. Nonetheless, these assumptions are reasonable considering that numerous other papers also derive their results based on the convergence to the KKT point, such as the work by Frei et al. (2023).\n\nIt is important to note that this paper exclusively focuses on a simplistic data model, where the samples originate from a unit sphere and the labels are fixed constants, specifically +1. The discussion does not extend to other more prevalent data distributions.\n\nQUESTIONS:\nIt would be particularly intriguing to investigate whether these findings remain consistent when applied to other commonly studied data distributions, such as mixture distributions. Additionally, it would be valuable to explore the possibility of extending the analysis techniques utilized in the proof to encompass a broader range of distributions.\n\nLIMITATIONS:\nSame as weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the spectrum of benign to catastrophic overfitting in a specific instance of a 2-layered neural network with ReLU activation when the training data has noisy labels with proportion $p$. The authors when the input dimension is , the test error is lower bounded proportionally to $poly(p)$ under different assumptions. When the input dimension is high, i.e. $d=poly(n)$, a NN overfitting (i.e. achieving perfect training accuracy) is benign and good generalization performance is exhibited. The authors also consider the intermediate  regime for the input dimension and show a gradual behavior for the error rate as a function of the noise factor.\n\nSTRENGTHS:\n1. The paper is very well written with a great exposition and tradeoff between results and intuition.\n2. Results presented seem comprehensive and I like the fast the authors do not neglect the intermediate regime and provide empirical evidence where theoretical analysis is not attainable.\n3. The authors make their theoretical results accessible and explain the importance of the bias for the analysis.\n\nWEAKNESSES:\n1. The paper is missing a discussion/conclusion section to tie the results to real scenarios. Consider shortening one of the sections/proof sketches in favor or the above.\n2. The assumptions of the theoretical results are not very realistic, specifically in theorem 4.3, the input dimension is big with respect to both m and n, this does not follow common practice of deep learning where the number of parameters exceeds the number of examples but usually the input dimension is not very big.\n3. The considered architecture is rather limited, it would be interesting to explore empirically with much greater depth and not only 2 and 3.\n\nQUESTIONS:\n1. Is the assumption that the ground truth labels are 1 and all the noise is labeled -1 necessary for the analysis or just a simplifying assumption for presentation?\n2. The authors mention that the width of the network probably doesn't play an important role in the benign overfitting phenomena - is this also the case for more realistic empirical setups? it seems to contradict common practice of deep learning where the number of parameters scales and benign overfitting is exhibited.\n\nLIMITATIONS:\nThe assumptions made in the theoretical setup are not very realistic, that being said, the reviewer appreciates the difficulty in obtaining such theoretical results. I would recommend that the authors discuss the assumptions and limitations in a dedicated section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-15"}