SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While the result above establishes the overfitting being tempered with an error scaling as  $\mathrm{poly}(p)$ , we note that the range  $[cp^5, C\sqrt{p}]$  is rather large. Moreover, previous works suggest that in many cases we should expect the clean test error to scale linearly with  $p$  [Belkin et al., 2018b, Chatterji and Long, 2021, Manoj and Srebro, 2023]. Therefore it is natural to conjecture that this is also true here, at least for trained NNs that we are likely to converge to. We now turn to show such a result under a stronger assumption, where instead of any KKT point of the margin maximization problem, we consider specifically local minima. We note that several prior works have studied networks corresponding to global minima of the max-margin problem, which is of course an even stronger assumption than local minimality (e.g. Savarese et al., 2019, Boursier and Flammarion, 2023). We say that  $\theta$  is a local minimum of Problem (1) whenever there exists  $r > 0$  such that if  $\| \theta' - \theta \| < r$  and  $\forall i \in [m] : y_i N_{\theta'}(x_i) \geq 1$  then  $\| \theta \|^{2} \leq \| \theta' \|^{2}$ .
Theorem 3.2. Let  $d = 1$ ,  $p \in [0, \frac{1}{2})$ . Then with probability at least  $1 - \delta$  over the sample  $S \sim \mathcal{D}^m$  for any local minimum  $\pmb{\theta} = (v_j, w_j, b_j)_{j=1}^n$  of Problem (1), it holds that
$$
c \left(p - \sqrt {\frac {\log (m / \delta)}{m}}\right) \leq L _ {\mathrm {c l}} \left(N _ {\boldsymbol {\theta}}\right) \leq C \left(p + \sqrt {\frac {\log (m / \delta)}{m}}\right),
$$
The theorem above indeed shows that whenever the sample size is large enough, the clean test error indeed scales linearly with  $p$ . The proof of Theorem 3.2, which appears in Appendix A.2, is based on a different analysis than that of Theorem 3.1. Broadly speaking, the proof relies on analyzing the structure of the learned prediction function, assuming it is a local minimum of the max-margin problem. More specifically, in order to obtain the lower bound we consider segments in between samples  $x_{i-1} < x_i < x_{i+1}$  that are labeled  $y_{i-1} = 1, y_i = -1, y_{i+1} = 1$ . Note that with high probability over the sample, approximately  $p(1-p)^2 = \Omega(p)$  of the probability mass lies in such segments. Our key proposition is that for a KKT point which is a local minimum,  $N_\theta(\cdot)$  must be linear on the segment  $[x_i, x_{i+1}]$ , and furthermore that  $N_\theta(x_i) = -1$ ,  $N_\theta(x_{i+1}) = 1$ . Thus, assuming the points are not too unevenly spaced, it follows that a constant fraction of any such segment contributes to the clean test error, resulting in an overall error of  $\Omega(p)$ . In order to prove this proposition, we provide an exhaustive list of cases in which the network does not satisfy the assertion, and show that in each case the norm can be locally reduced - see Figure 3 in the appendix for an illustration.
For the upper bound, a similar analysis is provided for segments  $[x_i, x_{i+1}]$  for which  $y_i = y_{i+1} = 1$ . Noting that with high probability over the sample an order of  $(1 - p)^2 = 1 - O(p)$  of the probability mass lies in such segments, it suffices to show that along any such segment the network is positive - implying an upper bound of  $O(p)$  on the possible clean test error. Indeed, we show that along such segments, by locally minimizing parameter norm the network is incentivized to stay positive.
4 Benign overfitting in high dimensions
In this section we focus on the high dimensional setting. Throughout this section we assume that the dataset is sampled according to  $\mathcal{D}_{\mathbf{x}} = \mathrm{Unif}(\mathbb{S}^{d - 1})$  and  $d\gg m$ , where  $m$  is the number of samples, so that  $L_{\mathrm{cl}}(N_{\pmb{\theta}}) = \operatorname*{Pr}_{\mathbf{x}\sim \mathrm{Unif}(\mathbb{S}^{d - 1})}[N_{\pmb{\theta}}(\mathbf{x})\leq 0]$ . We note that our results hold for other commonly studied distributions (see Remark 4.2). We begin by showing that if the network converges to the maximum margin solution up to some multiplicative factor, then benign overfitting occurs:
Theorem 4.1. Let  $\epsilon, \delta > 0$ . Assume that  $p \leq c_1$ ,  $m \geq c_2 \frac{\log(1 / \delta)}{p}$  and  $d \geq c_3 m^2 \log \left(\frac{m}{\epsilon}\right) \log \left(\frac{m}{\delta}\right)$  for some universal constants  $c_1, c_2, c_3 > 0$ . Given a sample  $(\mathbf{x}_i, y_i)_{i=1}^m \sim \mathcal{D}^m$ , suppose  $\boldsymbol{\theta} = (\mathbf{w}_j, v_j, b_j)_{j=1}^n$  is a KKT point of Problem (1) such that  $\|\boldsymbol{\theta}\|^2 \leq \frac{c_4}{\sqrt{p}} \|\boldsymbol{\theta}^*\|^2$ , where  $\boldsymbol{\theta}^*$  is a max-margin solution of Eq. (1) and  $c_4 > 0$  is some universal constant. Then, with probability at least  $1 - \delta$  over  $S \sim \mathcal{D}^m$  we have that  $L_{\mathrm{cl}}(N_\boldsymbol{\theta}) \leq \epsilon$ .
The theorem above shows that under the specified assumptions, benign overfitting occurs for any noise level  $p$  which is smaller than some universal constant. Note that this result is independent of
Note that this is merely the standard definition of local minimality of the Euclidean norm function, under the constraints imposed by interpolating the dataset.
4. Similarly to footnote 2, our proof also characterizes the expected clean test error for any sample size as  $\mathbb{E}_{S\sim \mathcal{D}^m}\left[L_{\mathrm{cl}}(N_\theta)\right] = \Theta (p)$ .
the number of neurons  $n$ , and requires  $d = \tilde{\Omega}(m^2)$ . The mild lower bound on  $m$  is merely to ensure that the negative samples concentrate around a  $\Theta(p)$  fraction of the entire dataset. Also note that the dependence on  $\epsilon$  is only logarithmic, which means that in the setting we study, the clean test error decays exponentially fast with  $d$ .
We now turn to provide a short proof intuition, while the full proof can be found in Appendix B.1. The main crux of the proof is showing that  $\sum_{j=1}^{n} v_{j} \sigma(b_{j}) = \Omega(1)$ , i.e., the bias terms are the dominant factor in the output of the network, and they tend towards being positive. In order to see why this suffices to show benign overfitting, first note that  $N_{\theta}(\mathbf{x}) = \sum_{j=1}^{n} v_{j} \sigma(\mathbf{w}_{j}^{\top} \mathbf{x} + b_{j})$  equals
$$
\sum_ {j = 1} ^ {n} v _ {j} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) + \sum_ {j = 1} ^ {n} v _ {j} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n} v _ {j} \sigma (b _ {j}) \geq \sum_ {j = 1} ^ {n} v _ {j} \sigma (b _ {j}) - \sum_ {j = 1} ^ {n} \left| v _ {j} \sigma \left(\mathbf {w} _ {j} ^ {\top} \mathbf {x}\right) \right|.
$$
All the samples (including the test sample  $\mathbf{x}$ ) are drawn independently from  $\mathrm{Unif}(\mathbb{S}^{d-1})$ , thus with high probability  $|\mathbf{x}^\top \mathbf{x}_i| = o_d(1)$  for all  $i \in [m]$ . Using our assumption regarding convergence to the max-margin solution (up to some multiplicative constant), we show that the norms  $\|\mathbf{w}_j\|$  are bounded by some term independent of  $d$ . Thus, we can bound  $|\mathbf{x}^\top \mathbf{w}_j| = o_d(1)$  for every  $j \in [n]$ . Additionally, by the same assumption we show that both  $\sum_{j=1}^n \|\mathbf{w}_j\|$  and  $\sum_{j=1}^n |v_j|$  are bounded by the value of the max-margin solution (up to a multiplicative factor), which we bound by  $O(\sqrt{m})$ . Plugging this into the displayed equation above, and using the assumption that  $d$  is sufficiently larger than  $m$ , we get that the  $\sum_{j=1}^n |v_j \sigma(\mathbf{w}_j^\top \mathbf{x})|$  term is negligible, and hence the bias terms  $\sum_{j=1}^n v_j \sigma(b_j)$  are the predominant factor in determining the output of  $N_\theta$ .
Showing that  $\sum_{j=1}^{n} v_j \sigma(b_j) = \Omega(1)$  is done in two phases. Recall the definitions of  $I_+ := \{i \in [m] : y_i = 1\}$ ,  $I_- = \{i \in [m] : y_i = -1\}$ , and that by our label distribution and assumption on  $p$  we have  $|I_+| \approx (1 - p)m \gg pm \approx |I_-|$ . We first show that if the bias terms are too small, and the predictor correctly classifies the training data, then its parameter vector  $\theta$  satisfies  $\|\theta\|^2 = \Omega(|I_+|)$ . Intuitively, this is because the data is nearly mutually orthogonal, so if the biases are small, the vectors  $\mathbf{w}_j$  must have a large enough correlation with each different point in  $I_+$  to classify it correctly. On the other hand, we explicitly construct a solution that has large bias terms, which satisfies  $\|\theta\|^2 = O(|I_-|)$ . By setting  $p$  small enough, we conclude that the biases cannot be too small.
Remark 4.2 (Data distribution). Theorem 4.1 is phrased for data sampled uniformly from a unit sphere, but can be easily extended to other isotropic distributions such as standard Gaussian and uniform over the unit ball. In fact, the only properties of the distribution we use are that  $|\mathbf{x}_i^\top \mathbf{x}_j| = o_d(1)$  for every  $i \neq j \in [m]$ , and that  $\| XX^\top \| = O(1)$  (independent of  $m$ ) where  $X$  is an  $m \times d$  matrix whose rows are  $\mathbf{x}_i$ .
4.1 Benign overfitting under KKT assumptions
In Theorem 4.1 we showed that benign overfitting occurs in high dimensional data, but under the strong assumption of convergence to the max-margin solution up to a multiplicative factor (whose value is allowed to be larger for smaller values of  $p$ ). On the other hand, Theorem 2.1 only guarantees convergence to a KKT point of the max-margin problem (1). We note that Vardi et al. [2022] provided several examples of ReLU neural networks where there are in fact KKT points which are not a global or even local optimum of the max-margin problem.
Consequently, in this section we aim at showing a benign overfitting result by only assuming convergence to a KKT point of the max-margin problem. For such a result we use two additional assumptions, namely: (1) The output weights  $v_{j}$  are all fixed to be  $\pm 1$ , while only  $(\mathbf{w}_j, b_j)_{j \in [n]}$  are trained; and (2) Both the noise level  $p$  and the input dimension  $d$  depend on  $n$ , the number of neurons. Our main result is the following:
Theorem 4.3. Let  $\epsilon, \delta > 0$ . Assume that  $p \leq \frac{c_1}{n^2}$ ,  $m \geq c_2 n \log \left(\frac{1}{\delta}\right)$  and  $d \geq c_3 m^4 n^4 \log \left(\frac{m}{\epsilon}\right) \log \left(\frac{m^2}{\delta}\right)$  for some universal constants  $c_1, c_2, c_3 > 0$ . Assume that the output weights are fixed so that  $|\{j : v_j = 1\}| = |\{j : v_j = -1\}|$  while  $(\mathbf{w}_j, b_j)_{j \in [n]}$  are trained, and that  $N_\theta$  converges to a KKT point of Problem (1). Then, with probability at least  $1 - \delta$  over  $S \sim \mathcal{D}^m$  we have  $L_{\mathrm{cl}}(N_\theta) \leq \epsilon$ .
5This assumption appears in prior works such as Cao et al. [2022], Frei et al. [2023], Xu and Gu [2023], and is primarily used to simplify the analysis.
Note that contrary to Theorem 4.1 and to the results from Section 3, here there is a dependence on  $n$ , the number of neurons. For  $n = O(1)$  we get benign overfitting for any  $p$  smaller than a universal constant. This dependence is due to a technical limitation of the proof (which we will soon explain), and it would be interesting to remove it in future work. The full proof can be found in Appendix B.2, and here we provide a short proof intuition.
The proof strategy is similar to that of Theorem 4.1, by showing that  $\sum_{j=1}^{n} v_j \sigma(b_j) = \Omega(1)$ , i.e., the bias terms are dominant and tend towards being positive. The reason that this suffices is proven using similar arguments to that of Theorem 4.1. The main difficulty of the proof is showing that  $\sum_{j=1}^{n} v_j \sigma(b_j) = \Omega(1)$ . We can write each bias term as  $b_j = v_j \sum_{i \in [m]} \lambda_i y_i \sigma_{i,j}'$ , where  $\sigma_{i,j}' = \mathbb{1}(\mathbf{w}_j^\top \mathbf{x}_i + b_j > 1)$  is the derivative of the  $j$ -th ReLU neuron at the point  $\mathbf{x}_i$ . We first show that all the bias terms  $b_j$  are positive (Lemma B.5) and that for each  $i \in I_+$ , there is  $j \in \{1, \dots, n/2\}$  with  $\sigma_{i,j}' = 1$ . This means that it suffices to show that  $\lambda_i$  is not too small for all  $i \in I_+$ , while for every  $i \in I_-$ ,  $\lambda_i$  is bounded from above.
We assume towards contradiction that the sum of the biases is small, and show it implies that  $\lambda_{i} = \Omega\left(\frac{1}{n}\right)$  for  $i \in I_{+}$  and  $\lambda_{i} = O(1)$  for  $i \in I_{-}$ . Using the stationarity condition we can write for  $j \in \{1, \dots, n/2\}$  that  $\mathbf{w}_{j} = \sum_{i=1}^{m} \lambda_{i} y_{i} \sigma_{i,j}^{\prime} \mathbf{x}_{i}$ . Taking some  $r \in I_{+}$ , we have that  $\mathbf{w}_{j}^{\top} \mathbf{x}_{r} \approx \lambda_{r} \sigma_{r,j}^{\prime}$  since  $\mathbf{x}_{r}$  is almost orthogonal to all other samples in the dataset. By the primal feasibility condition (Eq. (3))  $N_{\boldsymbol{\theta}}(\mathbf{x}_{r}) \geq 1$ , hence there must be some  $j \in \{1, \dots, n/2\}$  with  $\sigma_{r,j}^{\prime} = 1$  and with  $\lambda_{r}$  that is larger than some constant. The other option is that the bias terms are large, which we assumed does not happen. Note that we don't know how many neurons  $\mathbf{w}_{j}$  there are with  $\sigma_{j,r}^{\prime} = 1$ , which means that we can only lower bound  $\lambda_{r}$  by a term that depends on  $n$ .
To show that  $\lambda_{s}$  are not too big for  $s\in I_{-}$ , we use the complementary slackness condition (Eq. (5)) which states that if  $\lambda_s\neq 0$  then  $N_{\theta}(\mathbf{x}_s) = -1$ . Since the sum of the biases is not large, then if there exists some  $s\in I_{-}$  with  $\lambda_{s}$  that is too large we would get that  $N_{\theta}(\mathbf{x}_s) < -1$  which contradicts the complementary slackness. Combining those bounds and picking  $p$  to be small enough shows that  $\sum_{j = 1}^{n}v_{j}\sigma (b_{j})$  cannot be too small.
4.2 The role of the bias and catastrophic overfitting in neural networks
In both our benign overfitting results (Theorem 4.1 and Theorem 4.3), our main proof strategy was showing that if  $p$  is small enough, then the bias terms tend to be positive and dominate over the other components of the network. In this section we further study the importance of the bias terms for obtaining benign overfitting, by examining bias-less ReLU networks of the form  $N_{\theta}(\mathbf{x}) = \sum_{j=1}^{n} v_{j} \sigma(\mathbf{w}_{j}^{\top} \mathbf{x})$  where  $\boldsymbol{\theta} = (v_{j}, \mathbf{w}_{j})_{j \in [n]}$ . We note that if the input dimension is sufficiently large and  $n \geq 2$ , such bias-less networks can still fit any training data. The proofs for this section can be found in Appendix C.
We begin by showing that without a bias term and with no further assumption on the network width, any solution can exhibit catastrophic behaviour:
Proposition 4.4 (Catastrophic overfitting without bias). Consider a bias-less network with  $n = 2$  which classifies a dataset correctly. If the dataset contains at least one sample with a negative label, then  $L_{\mathrm{cl}}(N_{\theta}) \geq \frac{1}{2}$ .
While the result above is restricted to any network of width two (in particular, it holds under any assumption such as convergence to a KKT point), this already precludes the possibility of achieving either benign or tempered overfitting without further assumptions on the width of the network – in contrast to our previous results. Furthermore, we next show that in the bias-less case, the clean test error is lower bounded by some term which depends only on  $n$  (which again, holds for any network so in particular under further assumptions such as convergence to a KKT point).
Proposition 4.5 (Not benign without bias). For any bias-less network of width  $n$ ,  $L_{\mathrm{cl}}(N_{\theta}) \geq \frac{1}{2^n}$ .
Note that the result above does not depend on  $m$  nor on  $d$ , thus we cannot hope to prove benign overfitting for the bias-less case unless  $n$  is large, or depends somehow on both  $m$  and  $d$ . We emphasize that Proposition 4.5 does no subsume Proposition 4.4 although they appear to be similar. Indeed, for the case of  $n = 2$ , Proposition 4.5 provides a bound of  $\frac{1}{4}$  which does not fall under the definition of catastrophic overfitting, as opposed to Proposition 4.4. Next, we show that even for
arbitrarily large  $n$  (possibly scaling with other problem parameters such as  $m, d$ ), there are KKT points which do not exhibit benign overfitting:
Proposition 4.6 (Benign overfitting does not follow from KKT without bias). Consider a bias-less network, and suppose that  $\mathbf{x}_j^\top \mathbf{x}_i = 0$  for every  $i \neq j$ . Denote  $I_{-} := |\{i \in [m] : y_i = -1\}|$ . Then there exists a KKT point  $\pmb{\theta}$  of Problem (1) with  $L_{\mathrm{cl}}(N_{\pmb{\theta}}) \geq \frac{1}{2} - \frac{1}{2^{|T_{-}|}}$ .
The result above shows that even if we're willing to consider arbitrarily large networks, there still exist KKT points which do not exhibit benign overfitting (at least without further assumptions on the data). This implies that a positive benign overfitting result in the bias-less case, if at all possible, cannot possibly apply to all KKT points - so one must make further assumptions on which KKT points to consider or resort to an alternative analysis. The assumption that  $\mathbf{x}_j^\top \mathbf{x}_i = 0$  for every  $i \neq j$  is made to simplify the construction of the KKT point, while we hypothesize that the proposition can be extended to the case where  $\mathbf{x}_i \sim \mathrm{Unif}(\mathbb{S}^{d-1})$  (or other isotropic distributions) for large enough  $d$ .
5 Between tempered and benign overfitting for intermediate dimensions
In Section 3, we showed that tempered overfitting occurs for one-dimensional data, while in Section 4 we showed that benign overfitting occurs for high-dimensional data. In this section we complement these results by empirically studying intermediate dimensions and their relation to the number of samples. Our experiments extend an experiment from [Mallinar et al., 2022, Figure 6b] which considered the same distribution as we do, for  $d = 10$  and  $m$  varying from 300 to  $3.6 \times 10^5$ . Here we study larger values of  $d$ , and how the ratio between  $d$  and  $m$  affects the overfitting profile. We also focus on 2-layer networks (although we do show our results may extend to depth 3).
In our experiments, we trained a fully connected neural network with 2 or 3 layers, width  $n$  (which will be specified later) and ReLU activations. We sampled a dataset  $(\mathbf{x}_i, y_i)_{i=1}^m \sim \mathrm{Unif}(\mathbb{S}^{d-1}) \times \mathcal{D}_y$  for noise level  $p \in \{0.05, 0.1, \dots, 0.5\}$ . We trained the network using SGD with a constant learning rate of 0.1, and with the logistic (cross-entropy) loss. Each experiment ran for a total of  $20k$  epochs, and was repeated 10 times with different random seeds, the plots are averaged over the runs. In all of our experiments, the trained network overfitted in the sense that it correctly classified all of the (inherently noisy) training data. Furthermore, we remark that longer training times (100k epochs) were also examined, and the results remained unaffected, thus we omit such plots.
Figure 1: Training a 2-layer network with 1000 neurons on  $m$  samples drawn uniformly from  $\mathbb{S}^{d - 1}$  for varying input dimensions  $d$ . Each label is equal to  $-1$  with probability  $p$  and  $+1$  with probability  $(1 - p)$ . Left:  $m = 500$ , Right:  $m = 2000$ . The line corresponding to the identity function  $y = x$  was added for reference. Best viewed in color.
In Figure 1 we trained a 2-layer network with 1000 neurons over different input dimensions, with  $m = 500$  samples (left) and  $m = 2000$  samples (right). There appears to be a gradual transition from tempered overfitting (with clean test error linear in  $p$ ) to benign overfitting (clean test error close to 0 for any  $p$ ), as  $d$  increases compared to  $m$ . Our results indicate that the overfitting behavior is close to tempered/benign in wider parameter regimes than what our theoretical results formally cover ( $d = 1$  vs.  $d > m^2$ ). Additionally, benign overfitting seems to occur for lower values of  $p$ , whereas for  $p$  closer to 0.5 the clean test error is significantly above 0 even for the highest dimensions we examined.
Figure 2: Left: 2-layer network with  $n = 50$  and  $n = 3000$  neurons and varying input dimension. Right: 3-layer network with  $n = 1000$  neurons and varying input dimension. Both plots correspond to  $m = 500$  samples.
We further experimented on the effects of the width and depth on the overfitting profile. In Figure 2 (left) we trained a 2-layer network on  $m = 500$  samples with input dimension  $d \in \{250, 2000, 10000\}$  and  $n$  neurons for  $n \in \{50, 3000\}$ . It can be seen that even when the number of neurons vary significantly, the plots remain almost the same. It may indicate that the network width  $n$  has little effect on the overfitting profile (similar to most of our theoretical results), and that the dependence on  $n$  in Theorem 4.3 is an artifact of the proof technique (in accordance with our other results). In Figure 2 (right) we trained a 3-layer neural networks on  $m = 500$  samples, as opposed to a 2-layer networks in all our other experiments. Noticeably, this plot looks very similar to Figure 1 (left) which may indicate that our observations generalize to deeper networks.
In this paper, we focused on a classification setting with 2-layer ReLU neural networks that interpolate noisy training data. We showed that the implicit bias of gradient based training towards margin maximization yields tempered overfitting in a low dimensional regime, while it causes benign overfitting in high dimensions. Furthermore, we provide empirical evidence that there is a gradual transition from tempered to benign overfitting as the input dimension increases.
Our work leaves open several questions. A natural follow-up is to study data distributions for which our results do not apply. While our analysis in one dimension is readily applicable for other distributions, it is currently not clear how to extend the high dimensional analysis to other data distributions (in particular, to settings in which the Bayes-optimal predictor is not constant). Moreover, establishing formal results in any constant dimension larger than 1, or whenever the dimension scales sub-quadratically (e.g., linearly) with the number of training samples, is an interesting open problem which would likely require different proof techniques.
At a more technical level, there is the question of whether our assumptions can be relaxed. In particular, Theorem 3.2 provides a tight bound under the assumption of convergence to a local minimum of the margin maximization problem, whereas Theorem 3.1 only requires convergence to a KKT point, yet leaves a polynomial gap. Similarly, Theorem 4.1 and Theorem 4.3 require assumptions on the predictor norm or on the noise level and width, respectively.
Our work also leaves open a more thorough study of different architectures and their possible effect on the type of overfitting. In particular, we note that our results suggest that as long as the network is large enough to interpolate, the width plays a minor role in our studied setting — which we also empirically observe. On the other hand, whether this also holds for other architectures, as well as the role of depth altogether, remains unclear. Finally, it would be interesting to further study the bias-less case and assess its corresponding overfitting behavior under suitable assumptions.
We thank Gal Vardi for insightful discussions regarding the implicit bias of neural networks. This research is supported in part by European Research Council (ERC) grant 754705.
A Proofs of tempered overfitting
Throughout the proof, given a sample  $S = (x_i, y_i)_{i=1}^m \sim \mathcal{D}^m$  we denote  $S_x = (x_i)_{i=1}^m$ ,  $S_y = (y_i)_{i=1}^m$ , and assume without loss of generality that  $x_1 \leq x_2 \leq \dots \leq x_m$ .