SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guy Kornowski
Weizmann Institute of Science guy.kornowski@weizmann.ac.il
Weizmann Institute of Science gilad.yehudai@weizmann.ac.il
Weizmann Institute of Science ohad.shamir@weizmann.ac.il
Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the overfitting profile in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.
Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. Although quite standard in deep learning, the so-called interpolation learning regime challenges classical statistical wisdom regarding overfitting, and has attracted much attention in recent years.
In particular, the phenomenon commonly referred to as "benign overfitting" [Bartlett et al., 2020] describes a situation in which a learning method overfits, in the sense that it achieves zero training error over inherently noisy samples, yet it is able to achieve near-optimal accuracy with respect to the underlying distribution. So far, much of the theoretical work studying this phenomenon focused on linear (or kernel) regression problems using the squared loss, with some works extending this to classification problems. However, there is naturally much interest in gradually pushing this research towards neural networks. For example, Frei et al. [2023] recently showed that the implicit bias of gradient-based training algorithms towards margin maximization, as established in a series of works throughout the past few years, leads to benign overfitting of leaky ReLU networks in high dimensions (see discussion of related work below).
Recently, Mallinar et al. [2022] suggested a more nuanced view, coining the notion of "tempered" overfitting. An interpolating learning method is said to overfit in a tempered manner if its error (with
respect to the underlying distribution, and as the training sample size increases) is bounded away from the optimal possible error yet is also not "catastrophic", e.g., better than a random guess. To be more concrete, consider a binary classification setting (with labels in  $\{\pm 1\}$ ), and a data distribution  $\mathcal{D}$ , where the output labels correspond to some ground truth function  $f^{*}$  belonging to the predictor class of interest, corrupted with independent label noise at level  $p \in (0, \frac{1}{2})$  (i.e., given a sampled point  $\mathbf{x}$ , its associated label  $y$  equals  $\mathrm{sign}(f^{*}(\mathbf{x}))$  with probability  $1 - p$ , and  $-\mathrm{sign}(f^{*}(\mathbf{x}))$  with probability  $p$ ). Suppose furthermore that the training method is such that the learned predictor achieves zero error on the (inherently noisy) training data. Finally, let  $L_{\mathrm{cl}}$  denote the "clean" test error, namely  $L_{\mathrm{cl}}(N) = \operatorname{Pr}_{\mathbf{x} \sim \mathcal{D}}(N(\mathbf{x}) \cdot f^{*}(\mathbf{x}) \leq 0)$  for some predictor  $N$  (omitting the contribution of the label noise). In this setting, benign, catastrophic or tempered overfitting corresponds to a situation where the clean test error  $L_{\mathrm{cl}}$  of the learned predictor approaches  $0, \frac{1}{2}$ , or some value in  $(0, \frac{1}{2})$  respectively. In the latter case, the clean test error typically scales with the amount of noise  $p$ . For example, for the one-nearest-neighbor algorithm, it is well-known that the test error asymptotically converges to  $2p(1 - p)$  in our setting [Cover and Hart, 1967], which translates to a clean test error of  $p$ . Mallinar et al. [2022] show how a similar behavior (with the error scaling linearly with the noise level) occurs for kernel regression, and provided empirical evidence that the same holds for neural networks.
The starting point of our paper is an intriguing experiment from that paper (Figure 6b), where they considered the following extremely simple data distribution: The inputs are drawn uniformly at random from the unit sphere, and  $f^{*}$  is the constant  $+1$  function. This is perhaps the simplest possible setting where benign or tempered overfitting in binary classification may be studied. In this setting, the authors show empirically that a three-layer vanilla neural network exhibits tempered overfitting, with the clean test error almost linear in the label noise level  $p$ . Notably, the experiment used an input dimension of 10, which is rather small. This naturally leads to the question of whether we can rigorously understand the overfitting behavior of neural networks in this simple setup, and how problem parameters such as the input dimension, number of samples and the architecture affect the type of overfitting.
Our contributions: In this work we take a step towards rigorously understating the regimes under which simple, 2-layer ReLU neural networks exhibit different types of overfitting (i.e. benign, catastrophic or tempered), depending on the problem parameters, for the simple and natural data distribution described in the previous paragraph. We focus on interpolating neural networks trained with exponentially-tailed losses, which are well-known to converge to KKT points of a max-margin problem (see Section 2 for more details). At a high level, our main conclusion is that for such networks, the input dimension plays a crucial role, with the type of overfitting gradually transforming from tempered to benign as the dimension increases. In contrast, most of our results are not sensitive to the network's width, as long as it is sufficient to interpolate the training data. In a bit more detail, our contributions can be summarized as follows:
- Tempered overfitting in one dimension (Theorem 3.1 and Theorem 3.2). We prove that in one dimension (when the inputs are uniform over the unit interval), the resulting overfitting is provably tempered, with a clean test error scaling as  $\Theta(\mathrm{poly}(p))$ . Moreover, under the stronger assumption that the algorithm converges to a local minimum of the max-margin problem, we show that the clean test error provably scales linearly with  $p$ . As far as we know, this is the first result provably establishing tempered overfitting for non-linear neural networks.
- Benign overfitting in high dimensions (Theorem 4.1 and Theorem 4.3). We prove that when the inputs are sampled uniformly from the unit sphere (and with the dimension scaling polynomially with the sample size), the resulting overfitting will generally be benign. In particular, we show that convergence to a max-margin predictor (up to some constant factor) implies that the clean test error decays exponentially fast to 0 with respect to the dimension. Furthermore, under an assumption on the scaling of the noise level and the network width, we obtain the same result for any KKT point of the max-margin problem.
- The role of bias terms and catastrophic overfitting (Section 4.2). The proof of the results mentioned earlier crucially relied on the existence of bias parameters in the network architecture. Thus, we further study the case of a bias-less network, and prove that without such biases, neural networks may exhibit catastrophic overfitting, and that in general they do not overfit benignly without further assumptions. We consider this an interesting illustration of how catastrophic overfitting is possible in neural networks, even in our simple setup.
- Empirical study for intermediate dimensions (Section 5). Following our theoretical results, we attempt at empirically bridging the gap between the one-dimensional and high dimensional settings. In particular, it appears that the tempered and overfitting behavior extends to a wider regime than what our theoretical results formally cover, and that the overfitting profile gradually shifts from tempered to benign as the dimension increases. This substantially extends the prior empirical observation due to Mallinar et al. [2022, Figure 6b], which exhibited tempered overfitting in input dimension 10.
We note that in an independent and concurrent work, Joshi et al. [2023] studied ReLU neural networks with one-dimensional inputs, similar to the setting we study in Section 3, and proved that tempered and catastrophic overfitting can occur for possibly non-constant target functions, depending on the loss function. However, they focus on a regression setting, and on min-norm predictors, whereas our results cover classification and any KKT point of the max-magin problem. Moreover, the architecture considered differs somewhat from our work, with Joshi et al. including an untrained skip connection and an untrained bias term.
Following the empirical observation that modern deep learning methods can perfectly fit the training data while still performing well on test data [Zhang et al., 2017], many works have tried to provide theoretical explanations of this phenomenon. By now the literature on this topic is quite large, and we will only address here the works most relevant to this paper (for a broader overview, see for example the surveys Belkin 2021, Bartlett et al. 2021, Vardi 2022).
There are many works studying regression settings in which interpolating methods, especially linear and kernel methods, succeed at obtaining optimal or near-optimal performance [Belkin et al., 2018a,b, 2019, 2020, Mei and Montanari, 2022, Hastie et al., 2022]. In particular, it is interesting to compare our results to those of Rakhlin and Zhai [2019], Beaglehole et al. [2023], who proved that minimal norm interpolating (i.e. "ridgeless") kernel methods are not consistent in any fixed dimension, while they can be whenever the dimension scales with the sample size (under suitable assumptions) [Liang and Rakhlin, 2020]. Our results highlight that the same phenomenon occurs for ReLU networks.
There is by now also a large body of work on settings under which benign overfitting (and analogous definitions thereof) occurs [Nagarajan and Kolter, 2019, Bartlett et al., 2020, Negrea et al., 2020, Nakkiran and Bansal, 2020, Yang et al., 2021, Koehler et al., 2021, Bartlett and Long, 2021, Muthukumar et al., 2021, Bachmann et al., 2021, Zhou et al., 2023, Shamir, 2023]. In classification settings, as argued by Muthukumar et al. [2021], many existing works on benign overfitting analyze settings in which max-margin predictors can be computed (or approximated), and suffice to ensure benign behavior [Poggio and Liao, 2019, Montanari et al., 2019, Thrampoulidis et al., 2020, Wang and Thrampoulidis, 2021, Wang et al., 2021, Cao et al., 2021, Hu et al., 2022, McRae et al., 2022, Liang and Recht, 2023]. It is insightful to compare this to our main proof strategy, in which we analyze the max-margin problem in the case of NNs (which no longer admits a closed form). Frei et al. [2022] showed that NNs with smoothed leaky ReLU activations overfit benignly when the data comes from a well-separated mixture distribution, a result which was recently generalized to ReLU activations [Xu and Gu, 2023]. Similarly, Cao et al. [2022] proved that convolutional NNs with smoothed activations overfit benignly when the data is distributed according to a high dimensional Gaussian with noisy labels, which was subsequently generalized to ReLU CNNs [Kou et al., 2023]. We note that these distributions are reminiscent of the setting we study in Section 4 for (non convolutional) ReLU NNs. Chatterji and Long [2023] studied benign overfitting for deep linear networks.
As mentioned in the introduction, Mallinar et al. [2022] formally introduced the notion of tempered overfitting and suggested the study of it in the context of NNs. Subsequently, Manoj and Srebro [2023] proved that the minimum description length learning rule exhibits tempered overfitting, though noticeably this learning rule does not explicitly relate to NNs. As previously mentioned, we are not aware of existing works that prove tempered overfitting in the context of NNs.
In a parallel line of work, the implicit bias of gradient-based training algorithms has received much attention [Soudry et al., 2018, Lyu and Li, 2020, Ji and Telgarsky, 2020]. Roughly speaking, these results drew the connection between NN training to margin maximization - see Section 2 for a formal reminder. For our first result (Theorem 3.1) we build upon the analysis of Safran et al. [2022] who studied this implicit bias for univariate inputs, though for the sake of bounding the number of linear
regions. In our context, Frei et al. [2023] utilized this bias to prove benign overfitting for leaky ReLU NNs in a high dimensional regime.
Notation. We use bold-faced font to denote vectors, e.g.  $\mathbf{x} \in \mathbb{R}^d$ , and denote by  $\|\mathbf{x}\|$  the Euclidean norm. We use  $c, c', \widetilde{c}, C > 0$  etc. to denote absolute constants whose exact value can change throughout the proofs. We denote by  $[n] := \{1, \dots, n\}$ , by  $\mathbb{1}\{\cdot\}$  the indicator function, and by  $\mathbb{S}^{d-1} \subset \mathbb{R}^d$  the unit sphere. Given sets  $A \subset B$ , we denote by  $\mathrm{Unif}(A)$  the uniform measure over a set  $A$ , by  $A^c$  the complementary set, and given a function  $f: B \to \mathbb{R}$  we denote its restriction  $f|_A: A \to \mathbb{R}$ . We let  $I$  be the identity matrix whenever the dimension is clear from context, and for a matrix  $A$  we denote by  $\|A\|$  its spectral norm. We use standard big-O notation, with  $O(\cdot), \Omega(\cdot)$  and  $\Theta(\cdot)$  hiding absolute constants, write  $f \lesssim g$  if  $f = O(g)$ , and denote by  $\mathrm{poly}(\cdot)$  polynomial factors.
Setting. We consider a classification task based on noisy training data  $S = (\mathbf{x}_i, y_i)_{i=1}^m \subset \mathbb{R}^d \times \{\pm 1\}$  drawn i.i.d. from an underlying distribution  $\mathcal{D} := \mathcal{D}_{\mathbf{x}} \times \mathcal{D}_y$ . Throughout, we assume that the output values  $y$  are constant  $+1$ , corrupted with independent label noise at level  $p$  (namely, each  $y_i$  is independent of  $\mathbf{x}_i$  and satisfies  $\operatorname*{Pr}[y_i = 1] = 1 - p$ ,  $\operatorname*{Pr}[y_i = -1] = p$ ). Note that the Bayes-optimal predictor in this setting is  $f^* \equiv 1$ . Given a dataset  $S$ , we denote by  $I_+ := \{i \in [m] : y_i = 1\}$  and  $I_- := \{i \in [m] : y_i = -1\}$ . We study 2-layer ReLU networks:
$$
N _ {\boldsymbol {\theta}} (\mathbf {x}) = \sum_ {j = 1} ^ {n} v _ {j} \sigma \left(\mathbf {w} _ {j} \cdot \mathbf {x} + b _ {j}\right),
$$
where  $n$  is the number of neurons or network width,  $\sigma (z)\coloneqq \max \{0,z\}$  is the ReLU function,  $v_{j},b_{j}\in \mathbb{R}$ ,  $\mathbf{w}_j\in \mathbb{R}^d$  and  $\pmb {\theta} = (v_j,\mathbf{w}_j,b_j)_j^n\in \mathbb{R}^{(d + 2)n}$  is a vectorized form of all the parameters. Throughout this work we assume that the trained network classifies the entire dataset correctly, namely  $y_{i}N_{\pmb{\theta}}(\mathbf{x}_{i}) > 0$  for all  $i\in [m]$ . Following Mallinar et al. [2022, Section 2.3] we consider the clean test error
$$
L _ {\mathrm {c l}} (N _ {\boldsymbol {\theta}}) := \operatorname * {P r} _ {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} \left[ N _ {\boldsymbol {\theta}} (\mathbf {x}) \leq 0 \right],
$$
corresponding to the misclassification error with respect to the clean underlying distribution. It is said that  $N_{\theta}$  exhibits benign overfitting if  $L_{\mathrm{cl}} \rightarrow 0$  with respect to large enough problem parameters (e.g.  $m, d \to \infty$ ), while the overfitting is said to be catastrophic if  $L_{\mathrm{cl}} \rightarrow \frac{1}{2}$ . Formally, Mallinar et al. [2022] have coined tempered overfitting to describe any case in which  $L_{\mathrm{cl}}$  converges to a value in  $(0, \frac{1}{2})$ , though a special attention has been given in the literature to cases where  $L_{\mathrm{cl}}$  scales monotonically or even linearly with the noise level  $p$  [Belkin et al., 2018b, Chatterji and Long, 2021, Manoj and Srebro, 2023].
Implicit bias. We will now briefly describe the results of Lyu and Li [2020], Ji and Telgarsky [2020] which shows that under our setting, training the network with respect to the logistic or exponential loss converges towards a KKT point of the margin-maximization problem. To that end, suppose  $\ell(z) = \log(1 + e^{-z})$  or  $\ell(z) = e^{-z}$ , and consider the empirical loss  $\hat{\mathcal{L}}(\boldsymbol{\theta}) = \sum_{i=1}^{n} \ell(y_i N_\boldsymbol{\theta}(\mathbf{x}_i))$ . Suppose that  $\boldsymbol{\theta}(t)$  evolves according to the gradient flow of the empirical loss, namely  $\frac{d\boldsymbol{\theta}(t)}{dt} = -\nabla \hat{\mathcal{L}}(\boldsymbol{\theta}(t))$ . Note that this corresponds to performing gradient descent over the data with an infinitesimally small step size.
Theorem 2.1 (Rephrased from Lyu and Li, 2020, Ji and Telgarsky, 2020). Under the setting above, if there exists some  $t_0$  such that  $\theta(t_0)$  satisfies  $\min_{i \in [m]} y_i N_{\theta(t_0)}(\mathbf{x}_i) > 0$ , then  $\frac{\theta(t)}{||\theta(t)||} \stackrel{t \to \infty}{\longrightarrow} \frac{\theta}{||\theta||}$  for  $\theta$  which is a KKT point of the margin maximization problem
$$
\min  \quad \| \boldsymbol {\theta} \| ^ {2} \quad \text {s . t .} \quad y _ {i} N _ {\boldsymbol {\theta}} (\mathbf {x} _ {i}) \geq 1 \quad \forall i \in [ m ]. \tag {1}
$$
<sup>1</sup>Note that the ReLU function is not differentiable at 0. This can be addressed either by setting  $\sigma'(0) \in [0,1]$  as done in practical implementations, or by considering the Clarke subdifferential [Clarke, 1990] (see Lyu and Li, 2020, Dutta et al., 2013 for a discussion on non-smooth optimization). We note that this issue has no effect on our results.
The result above shows that although there are many possible parameters  $\theta$  that result in a network correctly classifying the training data, the training method has an implicit bias in the sense that it yields a network whose parameters are a KKT point of Problem (1). Recall that  $\theta$  is called a KKT point of Problem (1) if there exist  $\lambda_1,\ldots ,\lambda_m\in \mathbb{R}$  such that the following conditions hold:
$$
\boldsymbol {\theta} = \sum_ {i = 1} ^ {m} \lambda_ {i} y _ {i} \nabla_ {\boldsymbol {\theta}} N _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {i}\right) \tag {2}
$$
$$
\forall i \in [ m ]: y _ {i} N _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {i}\right) \geq 1 \quad (\text {p r i m a l f e a s i b i l i t y}) \tag {3}
$$
$$
\lambda_ {1}, \dots , \lambda_ {m} \geq 0 \quad (\text {d u a l f e a s i b i l i t y}) \tag {4}
$$
$$
\forall i \in [ m ]: \lambda_ {i} \left(y _ {i} N _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {i}\right) - 1\right) = 0 \quad (\text {c o m p l e m e n t a r y s l a c k n e s s}) \tag {5}
$$
It is well-known that global or local optima of Problem (1) are KKT points, but in general, the reverse direction may not be true, even in our context of 2-layer ReLU networks (see Vardi et al., 2022). In this work, our results rely either on convergence to a KKT point (Theorem 3.1 and Theorem 4.3), or on stronger assumptions such as convergence to a local optimum (Theorem 3.2) or near-global optimum (Theorem 4.1) of Problem (1) in order to obtain stronger results.
3 Tempered overfitting in one dimension
Throughout this section we study the one dimensional case  $d = 1$  under the uniform distribution  $\mathcal{D}_x = \mathrm{Unif}([0,1])$ , so that  $L_{\mathrm{cl}}(N_{\pmb{\theta}}) = \operatorname*{Pr}_{x\sim \mathrm{Unif}([0,1])}[N_{\pmb{\theta}}(x)\leq 0]$ . We note that although we focus here on the uniform distribution for simplicity, our proof techniques are applicable in principle to other distributions with bounded density. Further note that both results to follow are independent of the number of neurons, so that in our setting, tempered overfitting occurs as soon as the network is wide enough to interpolate the data. We first show that any KKT point of the margin maximization problem (and thus any network we may converge to) gives rise to tempered overfitting.
Theorem 3.1. Let  $d = 1$ ,  $p \in [0, \frac{1}{2})$ . Then with probability at least  $1 - \delta$  over the sample  $S \sim \mathcal{D}^m$  for any KKT point  $\pmb{\theta} = (v_j, w_j, b_j)_{j=1}^n$  of Problem (1), it holds that
$$
c \left(p ^ {5} - \sqrt {\frac {\log (m / \delta)}{m}}\right) \leq L _ {\mathrm {c l}} \left(N _ {\boldsymbol {\theta}}\right) \leq C \left(\sqrt {p} + \sqrt {\frac {\log (m / \delta)}{m}}\right),
$$
where  $c, C > 0$  are absolute constants.
The theorem above shows that for a large enough sample size  $L_{\mathrm{cl}}(N_{\pmb{\theta}}) = \Theta (\mathrm{poly}(p))$ , proving that the clean test error must scale roughly monotonically with the noise level  $p$ , leading to tempered overfitting.
We will now provide intuition for the proof of Theorem 3.1, which appears in Appendix A.1. The proofs of both the lower and the upper bounds rely on the analysis of Safran et al. [2022] for univariate networks that satisfy the KKT conditions. For the lower bound, we note that with high probability over the sample  $(x_{i})_{i = 1}^{m}\sim \mathcal{D}_{x}^{m}$ , approximately  $p^5 m$  of the sampled points will be part of a sequence of 5 consecutive points  $x_{i} < \dots < x_{i + 4}$  all labeled  $-1$ . Based on a lemma due to Safran et al. [2022], we show any such sequence must contain a segment  $[x_j,x_{j + 1}]$ ,  $i\leq j\leq i + 3$  for which  $N_{\theta}|_{[x_j,x_{j + 1}]} < 0$ , contributing to the clean test error proportionally to the length of  $[x_j,x_{j + 1}]$ . Under the probable event that most samples are not too close to one another, namely of distance  $\Omega (1 / m)$ , we get that the total length of all such segments (and hence the error) is at least of order  $\Omega (\frac{mp^5}{m}) = \Omega (p^5)$ .
As to the upper bound, we observe that the number of neighboring samples with opposing labels is likely to be on order of  $pm$ , which implies that the data can be interpolated using a network of width of order  $n^* = O(pm)$ . We then invoke a result of Safran et al. [2022] that states that the class of interpolating networks that satisfy the KKT conditions has VC dimension  $d_{\mathrm{VC}} = O(n^*) = O(pm)$ . Thus, a standard uniform convergence result for VC classes asserts that the test error would be of order  $\frac{1}{m}\sum_{i\in [m]}\mathbb{1}\left\{\mathrm{sign}(N_\theta)(x_i)\neq y_i\right\} +O(\sqrt{d_{\mathrm{VC}} / m}) = 0 + O(\sqrt{pm / m}) = O(\sqrt{p})$ .
2We remark that the additional summands are merely a matter of concentration in order to get high probability bounds. As for the expected clean test error, our proof readily shows that  $cp^5\leq \mathbb{E}_{S\sim \mathcal{D}^m}\left[L_{\mathrm{cl}}(N_\theta)\right]\leq C\sqrt{p}$