SlowGuess/NeurIPS_2023_OCR huggingface dataset:
*Erin George, *Michael Murray, William Swartworth, Deanna Needell  
Department of Mathematics, UCLA, CA, USA  
[egeo, mmurray, wswartworth, deanna]@math.ucla.edu  
*Equal contribution
We study benign overfitting in two-layer ReLU networks trained using gradient descent and hinge loss on noisy data for binary classification. In particular, we consider linearly separable data for which a relatively small proportion of labels are corrupted or flipped. We identify conditions on the margin of the clean data that give rise to three distinct training outcomes: benign overfitting, in which zero loss is achieved and with high probability test data is classified correctly; non-benign overfitting, in which zero loss is achieved but test data is misclassified with probability lower bounded by a constant; and non-overfitting, in which clean points, but not corrupt points, achieve zero loss and again with high probability test data is classified correctly. Our analysis provides a fine-grained description of the dynamics of neurons throughout training and reveals two distinct phases: in the first phase clean points achieve close to zero loss, in the second phase clean points oscillate on the boundary of zero loss while corrupt points either converge towards zero loss or are eventually zeroed by the network. We prove these results using a combinatorial approach that involves bounding the number of clean versus corrupt updates across these phases of training.
Conventional machine learning wisdom suggests that the generalization error of a complex model will typically be worse versus a simpler model when both are trained to interpolate data. Indeed, the bias-variance trade-off implies that although choosing a complex model is advantageous in terms of approximation error, it comes at the price of an increased risk of overfitting. The traditional solution to managing this trade-off is to use some form of regularization, allowing the optimizer to select a predictor from a rich class of functions while at the same time encouraging it to choose one that is in some sense simple. However, in recent years this perspective has been challenged by the observation that deep learning models, trained with minimal if any form of regularization, can almost perfectly interpolate noisy data with nominal cost to their generalization performance (Zhang et al., 2017; Belkin et al., 2018b, 2019). This phenomenon is referred to as benign overfitting.
Following these empirical observations, a line of research has emerged aiming to theoretically characterize the conditions under which various machine learning models, trained to zero loss on noisy data, obtain, at least asymptotically, optimal generalization error. To date, the majority of analyses in this regard have focused primarily on linear models, including linear regression (Bartlett et al., 2020; Muthukumar et al., 2020; Wu & Xu, 2020; Chatterji & Long, 2021; Zou et al., 2021; Hastie et al., 2022; Koehler et al., 2021; Wang et al., 2021a; Chatterji & Long, 2022; Cao et al., 2021; Shamir, 2022), logistic regression (Chatterji & Long, 2021; Muthukumar et al., 2021; Wang et al., 2021b) and kernel regression (Belkin et al., 2018a; Mei & Montanari, 2019; Liang & Rakhlin, 2020; Liang et al., 2019). With regards to understanding benign overfitting in neural networks, in the
Neural Tangent Kernel (NTK) regime (Jacot et al., 2018) the prediction of a neural network is well approximated via kernel regression (Adlam & Pennington, 2020). However, this regime typically requires unrealistically large network width and fails to capture feature learning. Indeed, and despite being the initial source of inspiration, an understanding of when and how neural networks benignly overfit in the rich, feature learning regime is not well understood.
1.1 Contributions and related work
In this work we study benign overfitting in the context of binary classification for two-layer ReLU networks, trained using gradient descent and hinge loss, on label corrupted, linearly separable data. There are a number of recent and or concurrent works which prove benign overfitting results in a similar setting Frei et al. (2022, 2023); Xu & Gu (2023); Cao et al. (2022); Kou et al. (2023); Kornowski et al. (2023), however, we emphasize that these exclusively study exponentially tailed losses, notably the popular logistic loss. Benign overfitting is intimately related to the notion of implicit bias, the preference of an algorithm for selecting minimizers with certain properties over others. The implicit bias of homogeneous networks trained with gradient descent on an exponentially tailed loss from a low initial loss is known to converge in direction to a Karush-Kuhn-Tucker (KKT) point of the associated max-margin problem Lyu & Li (2020); Ji & Telgarsky (2020). This implies at least intuitively a certain bias towards margin maximization. In a recent work Frei et al. (2023) it is shown that if the input data is sufficiently orthogonal then a shallow, leaky ReLU network evaluated on such a KKT point is equivalent to a particular linear classifier. Moreover, and under additional data assumptions, the authors show such networks benignly overfit. Another recent paper Kornowski et al. (2023) uses a similar approach to derive benign overfitting results for ReLU networks and also provides a description of the transition between benign and tempered overfitting in the univariate input case. To the best of our knowledge, equivalent results on the implicit bias of homogeneous networks trained with non-exponentially tailed losses are not characterized. Furthermore, training a linear classifier with an exponential versus non-exponential tailed loss is known to result in a different implicit bias, with the non-exponential tailed loss potentially inducing convergence in direction to a classifier with a poor margin Ji et al. (2020). As a result, a priori it is not clear if and how the choice of hinge loss impacts the propensity for a shallow ReLU network to overfit.
There are two main existing lines of work which study benign overfitting in neural networks outside of the kernel regime. Concerning perhaps the most relevant line of prior work to our own, Frei et al. (2022) consider a smooth, leaky ReLU activation function, train the network using the logistic instead of the hinge loss and assume the data is drawn from a mixture of well-separated sub-Gaussian distributions. The key result of this work is that given a sufficient number of iterations of GD, then the network will interpolate the noisy training data while also achieving minimax optimal generalization error up to constants in the exponents. A concurrent work Xu & Gu (2023) extends this result to more general activation functions including ReLU, relaxes the assumptions on the noise distribution to being centered with bounded logarithmic Sobolev constant, and also improves the convergence rate. As highlighted in Xu & Gu (2023), the fact that ReLU is non-smooth and non-leaky significantly complicates the analysis of both the convergence and generalization. A second line of work (Cao et al., 2022; Kou et al., 2023) studies benign overfitting in two-layer convolutional as opposed to feedforward neural networks. Whereas here and in Frei et al. (2022); Xu & Gu (2023) each data point is modeled as the sum of a signal and noise component, in Cao et al. (2022); Kou et al. (2023) the signal and noise components lie in disjoint patches. The weight vector of each neuron is applied to both patches separately and a non-linearity, such as ReLU, is applied to the resulting pre-activation. In this setting, the authors prove interpolation of the noisy training data and derive conditions on the clean margin under which the network benignly vs non-benignly overfits. We emphasize that the data model studied in this work is very different to the setting we study here, and as a result we primarily restrict our comparison to that with Frei et al. (2022) and the concurrent work Xu & Gu (2023). Finally, in regard to optimizing shallow ReLU networks using hinge loss, a line of work (Brutzkus et al., 2018; Wang et al., 2019; Yang et al., 2021) studies the convergence of gradient descent on generic, linearly separable data without label corruptions. These works also require additional assumptions, notably leaky ReLU instead of ReLU, insertion of noise into the optimization algorithm or changes to the loss function.
Before we discuss our contributions we remark that a previous work Mallinar et al. (2022) describes and experimentally explores a taxonomy of overfitting: benign overfitting, where the generalization error is optimal; catastrophic overfitting, where the generalization error is close to random chance;
and tempered overfitting, which lies in between. In this work, we do not consider the full breadth of this taxonomy, and use the terms "non-benign overfitting" or equivalently "harmful overfitting" to refer to overfitting that may be either tempered or catastrophic. We now summarize our contributions: in particular, under certain assumptions on the model hyperparameters, we prove conditions on the clean margin resulting in the three distinct training outcomes highlighted below. We remark also that the prior works discussed primarily focus on deriving positive benign overfitting results.
1. Benign overfitting: Theorem 3.1 provides conditions under which the training loss converges to zero and bounds the generalization error, showing that it is asymptotically optimal. This result is analogous to those of Frei et al. (2022) and Xu & Gu (2023) but for the hinge instead of logistic loss.
2. Non-benign overfitting: Theorem 3.6 provides conditions under which the network achieves zero training loss while generalization error is bounded below by a constant. Unlike Frei et al. (2022) and Xu & Gu (2023), this is not due to the non-separability of the data model but is instead a result of the neural network failing to learn the optimal classifier.
3. No overfitting: Theorem 3.8 provides conditions under which the network achieves zero training loss on points with uncorrupted label signs but nonzero loss on points with corrupted signs. Again the generalization error is bounded and shown to be asymptotically optimal.
To conclude this section we further remark that our proof techniques are quite different from those used in Frei et al. (2022); Xu & Gu (2023) and indeed the other works highlighted in this section. Again we emphasize this is due to the fact we study the hinge loss instead of the logistic loss and discuss the differences arising from this in detail in Section 3. In particular, we set up the problem in such a way that the convergence analysis reduces to counting the number of activations of clean versus corrupt points during various stages of training. Our analysis further provides a detailed description of the dynamics of the network's neurons, thereby allowing us to understand how the network fits both the clean and corrupted data.
We consider a training sample of  $2n$  pairs of points and their labels  $(\mathbf{x}_i, y_i)_{i=1}^{2n}$  where  $(\mathbf{x}_i, y_i) \in \mathbb{R}^d \times \{-1, +1\}$  for all  $i \in [2n]$ . Furthermore, we identify two disjoint subsets  $\mathcal{S}_T \subset [2n] = \{1, \dots, 2n\}$  and  $\mathcal{S}_F \subset [2n]$ ,  $\mathcal{S}_T \cup \mathcal{S}_F = [2n]$ , which correspond to the clean and corrupt points in the sample respectively. The categorization of a point as clean or corrupted is determined by its label: for all  $i \in [2n]$  we assume  $y_i = \beta(i)(-1)^i$  where  $\beta(i) = -1$  iff  $i \in \mathcal{S}_F$  and  $\beta(i) = 1$  otherwise. In addition, we assume  $|\mathcal{S}_F \cap [2n]_e| = |\mathcal{S}_F \cap [2n]_o| = k$  and  $|\mathcal{S}_T \cap [2n]_e| = |\mathcal{S}_T \cap [2n]_o| = n - k$ , where  $[2n]_e \subset [2n]$  and  $[2n]_o \subset [2n]$  are the even and odd indices, respectively. We remark that this assumption simplifies the exposition of our results but is not integral to our analysis. Each data point is assumed to have the form
$$
\mathbf {x} _ {i} = (- 1) ^ {i} \left(\sqrt {\gamma} \mathbf {v} + \sqrt {1 - \gamma} \beta (i) \mathbf {n} _ {i}\right). \tag {1}
$$
Here  $\mathbf{v} \in \mathbb{R}^d$  satisfies  $\|\mathbf{v}\| = 1$  and furthermore we refer to  $\mathbf{v}$  as the signal vector as the alignment of a clean point with  $\mathbf{v}$  determines its sign. Indeed,  $\mathrm{sign}(\langle \mathbf{x}_i, \mathbf{v} \rangle) = (-1)^i = y_i$  for  $i \in S_T$  whereas  $\mathrm{sign}(\langle \mathbf{x}_i, \mathbf{v} \rangle) = -y_i$  for  $i \in S_F$ . Thus we may view the labels of a corrupt point as flipped from their clean state. The vectors  $(\mathbf{n}_i)_{i=1}^{2n}$  are mutually independent and identically distributed (i.i.d.) random vectors drawn from the uniform distribution over  $\mathbb{S}^{d-1} \cap \operatorname{span}\{\mathbf{v}\}^\perp$ , which we denote  $U(\mathbb{S}^{d-1} \cap \operatorname{span}\{\mathbf{v}\}^\perp)$ . Clearly this distribution is symmetric, mean zero and for any  $\mathbf{n} \sim U(\mathbb{S}^{d-1} \cap \operatorname{span}\{\mathbf{v}\}^\perp)$  it holds that  $\mathbf{n} \perp \mathbf{v}$  and  $\|\mathbf{n}\| = 1$ . We refer to these vectors as noise components due to the fact that they are independent of the labels of their respective points. The real, scalar quantity  $\gamma \in [0,1]$  controls the strength of the signal versus the noise and also defines the clean margin. Finally, at test time a clean label  $y \sim U(\{-1,1\})$  is sampled and the corresponding test data point is constructed,
$$
\mathbf {x} = y \left(\sqrt {\gamma} \mathbf {v} + \sqrt {1 - \gamma} \mathbf {n}\right), \tag {2}
$$
where again  $\mathbf{n}\sim U(\mathbb{S}^{d - 1}\cap \operatorname {span}\{\mathbf{v}\}^{\perp})$
The key idea we use to characterize the training dynamics is to reduce the analysis of the trajectory of each neuron to that of counting the number of clean versus corrupt updates to it. This combinatorial approach relies on each point having similar sized signal and noise components. In order to make our analysis as clear as possible, we select a data model which ensures the signal and noise components are consistent in size across all points. We emphasize that these assumptions are not strictly necessary and we believe analogous analyses could be conducted when the signal and noise components are instead appropriately bounded. In addition, and as discussed in more detail in Section 3.2, the orthogonality of the signal and noise components allow us to demonstrate non-benign overfitting even when a perfect classifier exists.
2.2 Network architecture, optimization and initialization
We consider a densely connected, single layer feed-forward neural network  $f: \mathbb{R}^{2m \times d} \times \mathbb{R}^d \to \mathbb{R}$  with the following forward pass map,
$$
f (\mathbf {W}, \mathbf {x}) = \sum_ {j = 1} ^ {2 m} (- 1) ^ {j} \phi (\langle \mathbf {w} _ {j}, \mathbf {x} \rangle).
$$
Here  $\phi := \max \{0, z\}$  denotes the ReLU activation function and  $\mathbf{w}_j$  the  $j$ -th row of the weight matrix  $\mathbf{W} \in \mathbb{R}^{2m \times d}$ . The network weights are optimized using full batch gradient descent (GD) with step size  $\eta > 0$  in order to minimize the hinge loss over a training sample  $((\mathbf{x}_i, y_i))_{i=1}^{2n} \subset (\mathbb{R}^d \times \{-1,1\})^{2n}$  sampled as described in Section 2.1. After  $t'$  iterations this optimization process generates a sequence of weight matrices  $(\mathbf{W}^{(t)})_{t=0}^{t'}$ . For convenience, we overload our notation for the forward pass map of the network and let  $f(t, \mathbf{x}) := f(\mathbf{W}^{(t)}, \mathbf{x})$ . Furthermore, we denote the hinge loss on the  $i$ -th point at iteration  $t$  as  $\ell(t, i) := \max \{0, 1 - y_if(t, \mathbf{x}_i)\}$ . The hinge loss over the entire training sample at iteration  $t$  is therefore  $L(t) := \sum_{i=1}^{2n} \ell(t, i)$ . Let  $\mathcal{F}^{(t)} := \{i \in [2n] : \ell(t, \mathbf{x}_i) > 0\}$  and  $\mathcal{A}_j^{(t)} := \{i \in [2n] : \langle \mathbf{w}_j^{(t)}, \mathbf{x}_i \rangle > 0\}$  denote the sets of point indices that have nonzero loss and which activate the  $j$ th neuron at iteration  $t$  respectively. With
$$
\frac {\partial \ell (t , i)}{\partial w _ {j r}} = \left\{ \begin{array}{l l} 0, & \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle \leq 0, \\ - (- 1) ^ {j} y _ {i} x _ {i r}, & \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle > 0 \end{array} \right.
$$
then the GD update rule for the neuron weights at iteration  $t \geq 0$  may be written as
$$
\mathbf {w} _ {j} ^ {(t + 1)} = \mathbf {w} _ {j} ^ {(t)} + (- 1) ^ {j} \eta \sum_ {l = 1} ^ {2 n} \mathbb {1} \left(l \in \mathcal {A} _ {j} ^ {(t)} \cap \mathcal {F} ^ {(t)}\right) y _ {l} \mathbf {x} _ {l}. \tag {3}
$$
In regard to the initialization of the network parameters, for convenience we assume each neuron's weight vector is drawn mutually i.i.d. uniform from the centered sphere with radius  $\lambda_w > 0$ . We remark that results analogous to the ones presented hold if the weights are instead initialized mutually i.i.d. as  $w_{jc}^{(0)} \sim \mathcal{N}(0, \sigma_w^2)$  for sufficiently small  $\sigma_w^2$ .
For indices  $i,j\in \mathbb{Z}_{\geq 1}$  we say  $i\sim j$  iff  $(-1)^{i} = (-1)^{j}$ . We often refer to a data point or neuron by its index alone, e.g. "point  $i$  refers to the  $i$ -th training point  $(\mathbf{x}_i,y_i)$ . For two iterations  $t_0,t_1$  with  $t_1 > t_0$  we define the following.
1.  $G_{j}(t_{0},t_{1})\coloneqq \sum_{i\in S_{T}}\sum_{\tau = t_{0}}^{t_{1} - 1}\mathbb{1}(i\in \mathcal{A}_{j}^{(\tau)}\cap \mathcal{F}^{(\tau)})$  is the number of clean updates applied to the  $j$ -th neuron between iterations  $t_0$  and  $t_1$ .
2.  $B_{j}(t_{0},t_{1})\coloneqq \sum_{i\in S_{F}}\sum_{\tau = t_{0}}^{t_{1} - 1}\mathbb{1}\big(i\in \mathcal{A}_{j}^{(\tau)}\cap \mathcal{F}^{(\tau)}\big)$  is the number of corrupt updates applied to the  $j$  -th neuron between iterations  $t_0$  and  $t_1$
3.  $G(t_0, t_1) \coloneqq \sum_{j \in [2m]} G_j(t_0, t_1)$  and  $B(t_0, t_1) \coloneqq \sum_{j \in [2m]} B_j(t_0, t_1)$  are the total number of clean and corrupt updates applied to the entire network between iterations  $t_0$  and  $t_1$ .
<sup>1</sup>Although the derivative of ReLU clearly does not exist at zero, we follow the routine procedure of defining an update rule that extends the gradient update to cover this event.
4.  $T(t_0, t_1) \coloneqq G(t_0, t_1) + B(t_0, t_1)$  is the total number of updates from all points applied to the entire network between iterations  $t_0$  and  $t_1$ .
We extend all these definitions to the case  $t_0 = t_1$  by letting the empty sum be 0. Finally, we use  $C \geq 1$  and  $c \leq 1$  to denote generic, positive constants.
The main contributions of this work are Theorem 3.1, Theorem 3.6 and Theorem 3.8, which characterize how the margin of the clean data drives three different training regimes: namely benign overfitting, non-benign (or harmful) overfitting and no-overfitting respectively. We primarily distinguish between the three aforementioned training outcomes based on conditions on the signal strength  $\gamma \in [0,1]$  which controls the clean margin. Assuming the corrupt points are the minority in the training sample, then heuristically we might expect the following behavior as  $\gamma$  varies: if  $n\gamma \gg 1$ , then the signal dominates the noise during training, corrupted points are never fitted and the network generalizes well. If  $n\gamma \ll 1$ , then all points are eventually fitted based on their noise component and the network generalizes poorly. As such, we expect to observe benign overfitting when  $\gamma$  is small but not too small: in this regime the network learns the signal, thus ensuring it generalizes well, but corrupted points can still be fitted based on their noise component, thereby allowing training to zero loss.
With each theorem we provide here we give a sketch of its proof: full proofs are contained in the Supplementary Materials, which also contain supporting numerical simulations in Appendix F. Throughout this section, and in order to establish a common setting in which to observe a variety of different behaviors, we make the following assumptions on the network and data hyperparameters.
Assumption 1. For a sufficiently large constant  $C \geq 1$ , failure probability  $\delta \in (0,1/2)$  and noise inner product bound  $\rho \in (0,1)$ , let  $d \geq C\rho^{-2}\log(n/\delta)$ ,  $k \leq cn$ ,  $\lambda_w \leq c\eta$  and  $\eta \leq \xi$ , where  $\xi$  depends on  $n$ ,  $m$ ,  $k$ ,  $\gamma$ , and  $d$ .
We remark that the condition  $d \geq C\rho^{-2}\log (n / \delta)$  ensures the noise components are nearly orthogonal: in particular,  $\max_{i\neq \ell}|\langle \mathbf{n}_i,\mathbf{n}_\ell \rangle |\leq c\rho$  with high probability for some positive constant  $c$ . This near orthogonality condition on the noise terms is restrictive, but is a common assumption in the related works Frei et al. (2022); Xu & Gu (2023). We note that the value of  $\rho$  required for each of our results to hold varies. Likewise, the optimal constants  $c$  and  $C$  required in each case also vary and we will not concern ourselves with finding the tightest possible constants.
While there are differences the proofs of Theorem 3.1, 3.6 and 3.8 generally fit the following outline.
1. Use concentration to show with high probability the training data is nearly orthogonal and a certain initialization pattern is satisfied.
2. Characterize the activation pattern early in training before any point achieves zero loss.
3. Bound the activations at an iteration just before any training point achieves zero loss.
4. Based on bounds on the activations at a given iteration, derive an iteration-independent upper bound on the number of subsequent updates that can occur before convergence. At convergence all points either have zero loss or activate no neurons.
We emphasize that our proof techniques are significantly different from those used in Frei et al. (2022); Xu & Gu (2023) due to the differences between the hinge and logistic loss. In particular, letting  $\sigma(z)$  denote the logistic loss, a key step in the proof of these prior works is showing at any iteration  $t \geq 0$  that the ratio  $\sigma'(y_i f(t, \mathbf{x}_i)) / \sigma'(y_l f(t, \mathbf{x}_l))$  is upper bounded by a constant for all pairs of points  $i, l$  in the training sample. For the hinge loss this approach is not feasible: indeed, if at an iteration  $t$  some points achieve zero loss while others have not then this ratio is unbounded.
3.1 Benign overfitting
The following theorem states conditions in particular on  $\gamma$  under which the network simultaneously achieves asymptotically optimal test error and achieves zero loss on both the clean and corrupted data after a finite number of iterations. A detailed proof of this Theorem along with the associated lemmas is provided in Appendix C.
Theorem 3.1. Let Assumption 1 hold and further assume  $n \geq C \log(1/\delta)$ ,  $m \geq C \log(n/\delta)$ ,  $\rho \leq c\gamma$  and  $C \sqrt{\log(n/\delta)/d} \leq \gamma \leq cn^{-1}$ . Then there exists a sufficiently small step-size  $\eta$  such that with probability at least  $1 - \delta$  over the randomness of the dataset and network initialization the following hold.
1. The training process terminates at an iteration  $\mathcal{T}_{end} \leq \frac{Cn}{\eta}$ .
2. For all  $i\in [2n]$  then  $\ell (\mathcal{T}_{end},\mathbf{x}_i) = 0$
3. The generalization error satisfies
$$
\mathbb {P} (\operatorname {s g n} (f (\mathcal {T} _ {e n d}, \mathbf {x})) \neq y) \leq \exp \left(- c d \gamma^ {2}\right).
$$
Proof sketch. Recall the parameter  $\rho$  bounds the inner products of the noise components of the training data. Specifically, the conditions on  $d$  given in Assumption 1 ensure  $\max_{i\neq l}|\langle \mathbf{n}_i,\mathbf{n}_l\rangle |\leq \frac{\rho}{1 - \gamma}$  with high probability. We also identify the following sets of neurons for  $p\in \{-1,1\}$ ,