Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks
Authors: Etienne Boursier
Nicolas Flammarion
Venue: ICML 2025 poster (2025)
Keywords: Neural Networks
Simplicty Bias
Implicit Bias
One hidden ReLU Network
Early Alignment
Primary area: deep_learning->theory

Abstract: Understanding generalization of overparametrized models remains a fundamental challenge in machine learning. The literature mostly studies generalization from an interpolation point of view, taking convergence towards a global minimum of the training loss for granted.  This interpolation paradigm does not seem valid for complex tasks such as in-context learning or diffusion. It has instead been empirically observed that the trained models go from global minima to spurious local minima of the training loss as the number of training samples becomes larger than some level we call optimization threshold. This paper explores theoretically this phenomenon in the context of two-layer ReLU networks. We demonstrate that, despite overparametrization, networks might converge towards simpler solutions rather than interpolating training data, which leads to a drastic improvement on the test loss. Our analysis relies on the so called early alignment phase, during which neurons align toward specific directions. This directional alignment leads to a simplicity bias, wherein the network approximates the ground truth model without converging to the global minimum of the training loss. Our results suggest this bias, resulting in an optimization threshold from which interpolation is not reached anymore, is beneficial and enhances the generalization of trained models.

Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks
Etienne Boursier'
Nicolas Flammarion'
Abstract
still generalizing well to unseen data in supervised learning
tasks (Chizat and Bach, 2018; Mei et al., 2018; Rotskoff and Vanden- Vanden-Fijnden, 2022; Chizat and Bach, 2020; Boursier et al., 2022: Boursier and Flammarion, 2023). This phe- nomenon often attributed overparametriiationn enabling models to find solutions that interpolate the training data yet avoid overfitting (Belkin et al., 2019; Bartlett et al., 2021). However, the advent of generative Al paradigms-such as large language models (Vaswani et al., 2017) and diffusion models (Dhariwal and Nichol, 2021) has introduced a paradigm shift in our understanding of generalization. In these settings, models can generate new data and perform novel tasks without necessarily interpolating the training data, raising fresh questions about how and why they gen- eralize. This shift can be illustrated by two seemingly un- related applications: in-context learning with transformers and generative modeling using diffusion methods.
Understanding generalization of over- parametrized models remains a fundamental chal- lenge in machine learning. The literature mostly studies generalization from interpolation point of view, taking convergence towards global minimum of the training loss for granted. This interpolation paradigm does not seem valid for complex tasks such as in-context learning or dif- fusion. It has instead been empirically observed that the trained models go from global minima spurious local minima of the training loss the number of training samples becomes larger than some level we call optimization threshold. This paper explores theoretically this phenomenon in the context of two-layer ReLU networks. We demonstrate that, despite overparametrization. networks might converge towards simpler solu- tions rather than interpolating training data, which leads to drastic improvement the test loss. Our analysis relies on the called early align- ment phase, during which neurons align toward specific directions. This directional alignment leads to a simplicity bias, wherein the network approximates the ground truth model without converging to the global minimum of the training loss. Our results suggest this bias, resulting in optimization threshold from which interpolation is not reached anymore, is beneficial and enhances the generalization of trained models.
Firstly, in-context learning (ICL) refers to the ability of large pretrained transformer models to learn new tasks from just a few examples, without any parameter updates (Brown et al., 2020; Min al., 2022). A central question is whether ICL enables models to learn tasks significantly different from those encountered during pretraining. While prior work suggests that ICL leverages mechanisms akin to Bayesian inference (Xie et al., 2022; Garg et al., 2022; Bai et al., 2023), the limited diversity of tasks in pretraining datasets may constrain the model's ability generalize. Raventós et al. (2024) investigated this effect by focusing regres- sion problems to quantify how increasing the variety of tasks during pretraining affects ICL's capacity to generalize to new, unseen tasks, in context.
1. Introduction
Secondly, diffusion models have made remarkable strides in generating high-quality images from high nigh-dimensional datasets (Sohl-Dickstein al., 2015; Ho et al., 2020: Song et al., 2021). These models learn to generate by training denoisers to estimate the score function-the gradient of the log probability density- of the noisy data distribution (Song and Ermon, 2019). significant chal- lenge in this context is approximating a continuous density from a relatively small training set without succumbing to the curse of dimensionality. Although deep neural networks may tend to memorize training data when the dataset is small relative to the network's capacity (Somepalli et al., 2023; Carlini et al., 2023), Yoon et al. (2023); Kadkhodaie
Understanding the generalization capabilities of neural net- works remains a fundamental open question in machine learning (Zhang al., 2021; Neyshabur al., 2017). Tra- ditionally, research has focused on explaining why neu- ral networks models can achieve zero training loss while
samples
INRIA, LMO, Université Paris-Saclay, Orsay, France 'TML Lab, EPFL, Switzerland. Correspondence to: Etienne Boursier www.bouuussiirrii..........
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Simplicity bias and optimization threshold in ReLU networks
al. (2024) observed they generalize well when trained sufficiently large datasets, rendering the model's behavior nearly independent of the specific training set.
ing. Moreover, this directional concentration is believed contribute to the non-convergence to the global minimizer of the training loss. However, this characterization only pertains to the initial stage of training. Therefore we extend our analysis to provide, under a restricted data model, a comprehensive characterization of the training dynamics, demonstrating that the simplicity bias persists until the end of training when the number of training samples exceeds some optimization threshold. We here provide an informal version of our main theorem, corresponding to Theorem 4.1.
The common thread connecting these examples is fundamental change in how gradient descent behaves in overparametrized models when the number of data points exceeds certain threshold. Rather than converging to the global minimum of the training loss, gradient descent con- verges to simpler solution closely related to the true loss minimizer. In learning scenarios involving noisy data, the most effective solutions are often those that do not interpo- late the data. Despite their capacity to overfit, these models exhibit a simplicity bias, generalizing well to the underlying ground truth instead of merely fitting the noise in the training data. While simplicity bias generally refers to the tendency of models to learn features of increasing complexity, until reaching data interpolation (Arpit et al., 2017; Rahaman et al., 2019; Kalimeris et al., 2019; Huh et al., 2023); this phenomenon seems to stop before full interpolation with modern architectures (even when training for a very long time, see e.g. Raventós et al., 2024, Figure 4). This obser- vation underscores significant shift in our understanding and approach to generalization in machine learning.
Theorem 1.1 (Informal). Consider a specific regression setting with n data samples of dimension d. Using two- layer ReLU neural network trained with gradient flow and small random initialization, we show that if n Z d8 log d, then regardless of the network's 's width, the learned function closely approximates the orlinary least squares solution.
In particular, Theorem 1.1 shows that in the over- parametrized regime, the final estimator does not minimize the training loss globally, yet it achieves near near-optimal per- formance on the test data. We then confirm empirically our predictions.
In this paper, we theoretically investigate this phenomenon in the toy setting of shallow ReLU networks applied to a regression problem. While multilayer perceptrons are foundational elements shared by the aforementioned models, focusing on shallow networks remains a significant simplification with respect the architectures and algo- rithms used for training transformers and diffusion models. Despite this simplification, we aim to gain theoretical insights that shed light on similar behaviors observed in more complex models. Some recent theoretical works argued that overparametrized networks do not necessarily converge to global minima. In partieular, Qiao et al. (2024) showed this effect for unidimensional data by illustrating the instability of global minima. Boursier and Flammarion (2024) advanced different reason for this effect, given by the early alignmen! phenomenon: when initialized with sufficiently small weights, neurons primarily adjust their directions rather than their magnitudes in the early phase of training, aligning along specific directions determined by the stationary points of a certain known function.
2. Preliminaries
This section introduces the setting and the early alignment phenomenon, following the notations and definitions of Boursier and Flammarion (2024).
2.1. Notations
We denote by Sa-1 the unit sphere of Rd and B(O, 1) the unit ball. We note f(1) Op (9(1)), if there exists a constant Cp. that only depends on p such that for any t, If(t) S Cpg(t). We drop the p index, if the constant Cp is universal and does not depend on any parameter. Similarly note ft) (g(t)) if there exists constant Cp O. that only depends
such that f(t) > Opg(t) and we write ft) = (p((()) if both f(t) = Op (a())) and (() = 2g (o()). For any bounded set A, ((A) denotes the uniform probability distribution on the set A.
2.2. Setting
We consider n data points (Æk, yk [e(kk])) E IR++1 drawn i.idd. from a distribution μ E P(Rd+1), 1. We also denote by X = [=T,...,,,_] E Rdxn and y = (Vi..... yn) E R" respectively the matrix whose columns are given by the input vectors 2& and the vector with coordinates given by the labels yk- A two layer ReLU network is parametrized by 0 = (w,, ag (m E (d+11) corresponding to the prediction function
Contributions. Our first contribution is to show that this function driving the carly alignment phase concentrates around its expectation, which corresponds to the true loss function, as the number of training samples grows large. For simple teacher architectures, this expected function pos- sesses only few critical points. As a result, after the early alignment phase, the neurons become concentrated in a few key directions associated with the ground truth model. This behavior reveals a simplicity bias at the initial stages of train-
hgg : x ++ E/t a,o(w] x),
2

---

Simplicity bias and optimization threshold in ReLU networks
where a is the ReLU activation given by (=) max (0, =). While training, we aim at minimizing the empirical square loss over the training data, defined as
Early alignment. In the small initialization regime de- scribed by Equation (2), numerous works highlight an early alignment phase in the initial stage of training (Maennel et al., 2018; Atanasov et al., 2022; Boursier and Flammar- ion, 2024; Kumar and Haupt, 2024; Tsoy and Konstanti- nov, 2024). During this phase, the neurons exhibit minimal changes norm, while undergoing significant changes in direction. This phenomenon is due a discrepancy in the derivatives of the neurons' norms (which scale with A) and of their directions (which scale in (()). Specifically, for sufficiently small initialization scale X, the neurons align towards the critical directions of the following function Gn defined as
L(0; x, = 2n
- -
As the limiting dynamics of (stochastic) gradient descent with vanishing learning rates, we study a subgradient flow of the training loss, which satisfies for almost any E R.+,
(tt E - -0g (((()) x.),
(1)
where 0g I. stands for the Clarke subdifferential of I. w.r.t. 0.
2.3. Early alignment dynamics
G w w + w D, 0).
(5)
Initialization. In accordance to the feature learning regime (Chizat et al., 2019), the m neurons of the neural network are initialized as
G2 is continuous, piecewise linear and can be interpreted as the correlation between the gradient information around the origin (given by D (w, 0)) and the neuron w. The network neurons thus align with the critical directions on the sphere of G, during the early training dynamies. These critical directions are called extremal vectors, defined as follows.
1/2 (a)(O), (0)) = Am 10m2mm11/p/// (āz. 1Zy):
(2)
where À 0 is the scale of initialization and (āj. wy)) are vectors drawn i.i.d. from some distribution, satisfying the following domination property for any m E N:
Definition 2.1. A vector D E R is extremal with respect to G, if there exists w E - such that both hold
|āy| 2 |
|| for any j E [m]] and Σ cf 11 (3)
1. D E D,, 0);
This property is common and allows for a simpler analysis, as ensures that the signs of the output neurons (t) remain unchanged while training (Boursier et al., 2022).
2. D or (D) E (An(w), (,n (w)).
This definition directly follows from the KKT conditions of the maximization (or minimization) problem, constrained on the sphere, of the function G...
Neuron dynamics. In the case of two layer neural net- works with square loss and ReLU activation, Equation (1)
can be written for each neuron i [m]] as
Implications of early alignment. By the end of the early alignment, most if not all neurons are nearly aligned with some extremal vector D. Maennel al. (2018): Boursier and Flammarion (2024) argue that only few extremal vec- tors exist in typical learning models. We further explore this claim in Section 3. As a consequence, only a few directions are represented by the network's 's weights at the end of the early dynamics, even though the neurons cover all possible directions at initialization. Boursier and Flammarion (2024) even show that this quantization of directions can prevent the network from interpolating the training set at convergence despite the overparametrizatioon of the network.
ii; (t) E às (t) (t) (w) ((t)))
(4)
where the vector D, (t), f()) and set D (w)((( (()) are defined as follows, with do the subdiffcrential of the ReLU activation 5:
Dn(w, 0) Σ 1 =I wow (yl - hh(xxx))))) 1=1
Dn,ww, (w, 0)
mh (yn - he(o) (rr)))) I'n @ w)
Although this failure of interpolation has been considered a drawback by Boursier and Flammarion (2024), we show in Section 4 that it can also lead to a beneficial phenomenon of simplicity bias. Specifically, Section 4 illustrates on a simple linear example that for large number of training samples. the model does not converge to interpolation. Instead, it converges towards the ordinary least square (OLS) estimator of the data. As consequence, the model fits the true signal of the data, while effectively ignoring label noise. Before studying this example, we must first understand how extremal vectors behave as the number of training samples increases.
These derivations directly follow from the subdifferential of
the training loss. In particular, [w, 0) corresponds to a specific vector (subgradient) in the subdifferential (w. 8).
Also observe that the set Dn,ww, 0) depends on w only through its activations (w). defined as
Rd + 111,0,13* An : w + sign(wT<x))
Furthermore, D, (w. 0) only depends on 0 via the predic- tion function hg. This observation is crucial to the early alignment phenomenon.
3

---

Simplicity bias and optimization threshold in ReLU networks
3. Geometry of alignment with many samples
differentiable in this limit, and vector D R4 is extremal
with respect G if there exists w S 1 such that both
We here aim to describe the geometry of the function G.. with specific focus on the extremal vectors, as the number of training samples n becomes large. These vectors are key in driving the early alignment phase of the training, making them essential to understanding the initial dynamics of the parameters.
D
1. D ((w)
2. D 0 or |
E (w:. (6)
When n becomes large, the extremal vectors of the data then concentrate toward the vectors satisfying Equation (6).
This is precisely quantified by Proposition 3.1 below.
This section provides a general result on the concentration of gradient information D,, of the train loss and support that the early alignment behavior the infinite-data setting does not differ significantly from that in the large but finite n case. While the tail bound version of Theorem I is central to our analysis in Section 4, the results of Section 3 are not only stated in a general form for broader applicability, but also constitute standalone contributions that may be useful in future work.
x Proposition 3.1. Assume the marginal law of is continu- ous w.r.. the Lebesgue measure and that 1[ [*] < co. Then for any e >> 0, there is n (e = Oz,g (d d) such that for any n 2 (=),), with probability at least
On (+): for any extremal vector Dn of the finite data (X,y) E [Rxx(d+1), there exists a vector D E R satisfy- ing Equation (6), such that
|D D2 SE.
Despite smoothnessh of the loss (due to ReLU activa- tions), can leverage the piecewise constant structure of the vector function Dn (w), along with typical Rademacher complexity arguments, to derive uniform concentration bounds on the random function w + (w)).
Proposition 3.1 states that for large n, the extremal vectors concentrate towards the vectors satisfying Equation (6). Note that Proposition 3.1 is not a direct corollary of Theorem 3.1, but its proof relies on the tail bound version of Theorem 3.1 and continuity arguments. A complete proof is given in Appendix E.
Theorem 3.1. If the marginal law of 1 is continuous with respect to the Lebesgue measure, then for any n E N,
Early alignment towards few directions. Besides laying the ground for Theorem 4.1, Proposition 3.1 aims at describing the geometry of the early alignment when the number of training samples grows large. In particular, Proposition 3.1 shows that all extremal vectors concentrate towards the directions satisfying Equation (6). Although such a description remains abstract, we believe it is satisfied by only a few directions for many data distributions. As an example, for symmetric data distributions, it is respectively satisfied by single or two directions, when considering one neuron or linear teacher. More generally, we conjecture it should be satisfied by a small number of directions as soon as the labels are given by a small teacher network. Proving such a result is yet left for future work.
Ex sup w D. (w.0)
sup
|D,, - D(w)ll)]=
0(1 d log n
B (1l
where for any w E S.-1. D(w)= = B[1 21 00000
Theorem 3.1 indicates that grows large, the sets (1,, 0) converge the corresponding vectors for the true loss, given by D(w), at a rate log Moreover this
12
rate holds uniformly across all possible directions of Idd in expectation. A probability tail bound version of Theo- rem 3.1, which bounds this deviation with high probability, can also be derived (see Theorem D.1 in Appendix D). A complete proof of Theorem 3.1 is provided in Appendix C.
The early aligument phenomenon has been described in many works, to show that after the early training dynam- ies, only a few directions (given by the extremal vectors) are represented by the neurons (Bui Thi Mai and Lampert, 2021; Lyu et al., 2021; Boursier et al., 2022; Chistikov et al., 2023; Min et al., 2024; Boursier and Flammarion, 2024; Tsoy and Konstantinov, 2024). However, these works all rely on specific data examples, where extremal vectors can be easily expressed for finite number of samples. Propo- sition 3.1 aims at providing more general result, showing that for large n, it is sufficient to consider the directions satisfying Equation (6), which is easier to characterize from a statistical perspective. We thus believe that Proposition 3. advances our understanding of how sparse is the network representation (in directions) at the end of early alignment.
When n + 0o, the alignment dynamics are thus driven by vectors D, which are close to their expected value D(w). Furthermore, when n + co, the activations of weight (w) exactly determine the direction of this weight, every possible direction is then covered by the training inputs Zs. Specifically, for an infinite dataset indexed by N. whose support covers all directions of Rd, and defining the infinite activation function A as
A S - (-1,0, 13 + sign(w" Iл)keN
then A is injective. In this infinite data limit, the functions Gn converge to the function G w + w: D(w), which is

---

Simplicity bias and optimization threshold in ReLU networks
Proposition 3.1 implies that for large values of 22 d), the early alignment phase results in the formation of a small number of neuron clusters, effectively making the neural network equivalent to a small-width network. Empirically, these clusters appear to be mostly preserved throughout training. The neural network then remains equivalent to a small-width network along its entire training trajectory.
5. The random vector 2F 1 sub-Gaussian and the noise satisfies B[nr"] < 00.
Conditions 1, 2 and 5 in Assumption 4.1 are relatively
mild. However, item 3 is quite restrictive: it is needed to ensure that the volume of the activation cone containing 3 does not vanish when n oC. A similar assumption is considered by Chistikov et al. (2023); Tsoy and Konstanti- nov (2024), for similar reasons. Additionally, Condition 4 ensures that Ex [2xx]] 1 and 3 are in the same activation cone. This assumption allows the training dynamics to remain within a single cone after the early alignment phase, significantly simplifying our analysis.
In contrast, when the number of data is limited (n & d), this guarantee no longer holds and large number of extremal vectors may exist. For example in the case of orthogonal data (which only holds for I d). there are ((2)) extremal vectors (Boursier et al., 2022). In such cases, there would still be large number of neuron clusters at the end of the early alignment phase, maintaining a large effective width of the network. Studying how this effective width is maintained until the end of training in the orthogonal case remains an open problem. We conjecture that for a mild overparametrization (n & m < 2 we would still have a relatively large effective width (increasing with n) at the end of training.
As example, if the samples z4 are distributed i.i.d. as 3r
+ v Iv& with ve 1 (8+3+)
IK 5k |
and 5 u (1 (1-1 - -,1
for a small enough e > 0 and My a standard Gaussian distribution, then Assumption 4. 1 is satisfied. In this section, we also consider the following specific initialization scheme for any E [m]:
4. Optimization threshold and simplicity bias
The goal of this section is to illustrate the transition from interpolating the training data to nearly optimal estimator (with respect to the true loss) that can arise when increasing the size of training data. Toward this end, this section proves on a toy data example, that for a large enough number of training samples, an overparametrized network will not converge to a global minimum of the training loss, but will instead be close to the minimizer of the true loss. This is done by analyzing the complete training dynamics, whose first phase- the so-called early alignment phase-is is controlled using the tail bound version of Theorem 3.1. To this end, we consider the specific case of a linear data model:
wy (0)0000. and (0)0- ((((0)--))
1))
(8)
In addition to the regime considered in Equations (2) and (3), this initialization introduces a stronger domination condition, as [a:(0)] 2 (0)]l This condition reinforces the early alignment phase, ensuring that all neurons are nearly aligned with extremal vectors by the end of this phase. Assumption 4.1 and Equation (8) are primarily introduced to enable a tractable analysis and are discussed further in Section 4.2.
This set of assumptions allows to study the training dynamics separately on the following partition of the data:
yk = 2 3r 7) for any l [n].
S+ (k E [n] 2 8 2 0} and S_ [m]
[m] \S+.
where 77s is some noise, drawn i.idd from a centered distribution. We also introduce a specific set of assumptions regarding the data distribution.
Hereafter, we denote by X + Rdxx|++1 (resp. X-), the matrix with columns given by the vectors IA for k E S+ (resp. k E S.). Similarly, we denote by Y, E RIS (resp. Y_) the vector with coordinates given by the labels yı for k E S+ (resp. l E S.).
Assumption 4.1. The samples Ia and the noise Tik are drawn i.i.d from distributions μx, and pay satisfying, for some e 0:
1. px is symmetric, i.e., He and - follow the same distribution;
Studying separately positive (a, > O) and negative (a, < 0) neurons, we prove Theorem 4.1 below, which states that at convergence for large enough number of training samples, the sum of the positive (resp. negative) neurons correspond to the OLS estimator on the subset S+ (resp. S-).
2. (s is continuous with respect to the Lebesgue measure; c(lell)))
3. P Prepxx (LTT8M 13 S
||=||a (/ = 0:
Theorem 4.1. If Assumption 4.1 holds and the initialization scheme follows Equation (8), then there exists X +(2) and n @(d log d) such that for any X S X*, any m
4. ||Ennapxx [xx]] 1 Iallop < min 2
lla"
|Boursier al. (2022) proved effective width of 2 at the end of training when m 2 2".
N and 2 n²,, with probability 1 - o d 2 the
5

---

Simplicity bias and optimization threshold in ReLU networks
parameters (() converge some 20 such that (x) = - (-8p, for any x E Supp(ux), where Supp(ux) is the support of the distribution μxx, 3r,t = (X+ (X,X))=Xx+ Y+ + and 3n.- = =X__ -__X)_XXXX are the OLS estimator respec- rively on the data in S+ and S_.
nical and is actullly decomposed into slow growth and fast growth phases, following similar approach Lyu et al. (2021); Tsoy and Konstantinov (2024).
-
At the end of the second phase, the estimation function is al- ready close to the one described in Theorem 4.1. From then, we control the neurons using a local Polyak-Eooasiewicz inequality (see Equation (45)) to show that they remain close to their value at the end of the second phase, and ac- tually converge local minimum corresponding to the estimation function he_ described in Theorem 4.1.
Precisely, the estimator learnt at convergence for a large enough n behaves (ex-everywhere as the difference of two ReLU neurons, with nearly opposite directions (thanks to the distribution symmetry), resulting in a nearly linear estimator. These directions correspond to the OLS estimator of the data in S+ and in S_, respectively. The complete proof of Theorem 4.1 is deferred to Appendix F. We provide a detailed sketch in Section 4.1 below and discuss further Theorem 4.1 in Section 4.2.
4.2. Discussion
Theorem 4.1 shows that, under a specific linear data model, when the number of training samples exceeds certain optimization threshold, the learned function converges to the OLS estimator- even in highly overparametrized settings where m > n. This result highlights two key insights:
4.1. Sketch of Proof of Theorem 4.1
Despite overparametrization, the network can converge a suboptimal solution of the training loss when initialized at a small scale.
The proof of Theorem 4.1 examines the complete training dynamics of positive neurons (a+(0)= > 0) and negative
ones (0: < 0) separately. This decoupling is possible at the end of the early phase, due to Assumption 4, and is handled thanks to Lemma F.3 in the Appendix.
This training failure can in fact be beneficial: although suboptimal for the training loss, the resulting estimator is optimal for the test loss.
First note that for the given model, there are only two vec- tors satisfying Equation (6). corresponding to E and -4E,8* respectively, for E= En-nx [2:x: J. From then and thanks to the third point of Assumption 4.1, the results from Section 3 imply that, for large value of n and with high probability, there are only two extremal vectors, both of which are close to the expected ones mentioned above. By analyzing the early alignment phase similarly to Boursier and Flammarion (2024), we show that by the end of this early phase, (i) all neurons have small norms; (ii) positive (resp. negative) neurons are aligned with Σ (resp. -E More specifically, at time r, defined as the end of the early alignment phase, we show that
We now discuss further on Theorem 4.1 and its limitations.
A bsence of interpolation. For many years, the literature has argued in favor of the fact that, if overparametrized enough, neural networks do converge towards interpolation of the training set, i.e., to global minimum of the loss (Jacot al., 2018; Du al., 2019; Chizat and Bach, 2018; Wojtowytsch, 2020).
Yet, some recent works argued in the opposite direction that convergence towards global minima might not be achieved for regression tasks, even with infinitely overparametrrieed networks (Qiao et al., 2024; Boursier and Flammarion, 2024). Indeed, Theorem 4.1 still holds as m + oc: al- though interpolation of the data is possible from a statistical aspect?, interpolation does not occur for optimization rea- sons. In this direction, Qiao et al. (2024) claim that for large values of n and univariate data, interpolation cannot happen because of the large (i.e., finite) stepsizes used for gradient descent. Following Boursier and Flammarion (2024), we here provide a complementary reason, which is due the early alignment phenomenon and loss of omnidirectionality of the weights (i.e., the fact that the weights represent all directions in Rdd.. Note that this loss of omnidirectionality is specific to the (leaky) ReLU activation and does not hold for smooth activations (see e.g. Chizat and Bach, 2018, Lemma
(T) Σ -291110- Vi E [m], (T(
de log n n
- O(A
+
From that point onward, all positive neurons are nearly aligned and behave as a single neuron until the end of train- ing. Moreover, they remain in the same activation cone until the end of training, Namely for any E [m] and 2 T.
a:(l) x w)(() > 0 for any A: E S+, a,(t) x wt(() < 0 for any ls E S_.
We then show that during second phase, all positive neu- rons grow until they reach the OLS estimator the data in S4. Mathematically, for some time T2.+ > T,
En,, (0)>0 a, (T2, )w;())++)))) Rd Bo,+.
Although the absence of bias term in the parametrization limits the expressivity of the neural network, interpolation is still possible as long as the data r, are pairwise non-proportional (Carvalho et al., 2025, Theorem 2).
Similarly, negative neurons end up close to Br,- after a different time T2.-. Proving this second phase is quite tech-
6

---

Simplicity bias and optimization threshold in ReLU networks
optimization threshold
convergence spurious statio- nary point that nicely generalizes
underparametrized regime: ERM generalizes well
excess risk
tempered overfitting
< d
>> d
π2 >
772
Figure 1: Different regimes of generalization: in green (n < d), the trained estimator interpolates the data and leads to tempered overfitting; after the optimization threshold in blue (n > d), we converge to a spurious stationary point of the training loss which generalizes well despite overparametrization, this regime is our main focus; in the underparametrized regime in red (m > n), the global minima do not interpolate anymore and generalize well.
C.10). We experimentally confirm in Appendix A.4 that both visions are complementary, as interpolation still does not happen for arbitrarily small learning rates.
interpolation (Soudry et al., 2018; Lyu and Li, 2019; Chizat and Bach, 2020; Ji and Telgarsky, 2019), while simplicity bias still exists in absence of interpolation.
Simplicity bias. Simplicity bias has been extensively studied in the literature (Arpit et al.. 2017: Rahaman et al., 2019; Kalimeris et al., 2019; Huh et al., 2023). It is often described as the fact that networks learn features of increasing complexity while learning. In other words, simpler features are first learnt (c.g.. a linear estimator), and more complex features might be learnt later. This has been observed in many empirical studies, leading to improved performance in generalization, except from a few nuanced cases (Shah et al., 2020). Yet in all these studies, the network interpolates the training set after being trained for a long enough time. In consequence, simplicity bias has been characterized by a first feature learning phase; and is then followed by interpolating phase, where the remaining noise is fitted (Kalimeris et al., 2019).
Improved test loss, due to overparametrization thresh- old. Theorem 4.1 states that for a large enough number of training samples, the interpolating phase does not happen during training, and the estimator then resembles the OLS estimator of the training set. In that regime, the excess risk scales as O(d/n) (Hsu et al., 2011) and thus quickly decreases to 0 the number of training samples grows. In contrast when interpolation happens, we either observe tempered overfitting, where the excess risk does not go down to 0 as the number of samples grows (Mallinar et al., 2022); or even a catastrophic overfitring, where the excess risk instead diverges to infinity as the size of the training set increases (Joshi et al., 2024).
The fact that the excess risk goes down to 0 as grows in our example of Section 4 could not be due to a benign overfitting (Belkin et al., 2018; Bartlett et al., 2020), as benign overfitting occurs when the dimension d also grows to infinity. We here consider a fixed dimension instead, and this reduced risk is then solely due to the optimization threshold, i.e., the fact that for large enough n, the interpolating phase does not happen anymore. While some works rely on early stopping before this interpolating phase to guarantee such an improved excess risk (Ji et al., 2021; Mallinar et al., 2022; Frei et al., 2023), it can be guaranteed without any early stopping after this optimization threshold. A similar threshold has been empirically observed in diffusion and in-context learning (Yoon et al., 2023; Kadkhodaie et al., 2024; Raventós et al., 2024), where the trained model goes from interpolation to generalization as the number of training samples increases.
We here go further by showing that this last interpolating phase does not even happen in some cases. Theorem 4.1 indeed claims that after the first feature learning phase, where the network learns a linear estimator, nothing happens in training. The interpolating phase never starts, matter how long we wait for. While interpolation is often observed for classification problems in practice, it is generally much harder to reach for regression problems (Stewart et al., 2023; Yoon et al., 2023; Kadkhodaie et al., 2024; Raventós et al., 2024). Theorem 4.1 confirms this tendency by illustrating regression example where interpolation does not happen at convergence. Notably, we here focus on the blue regime in Figure 1 and show that while the global minima poorly generalize in this regime, the optimization scheme only converges to a spurious local minimum of the training loss, which has much better generalization properties. This in stark contrast to the underparametrized regime- n > m, in red in Figure 1 where the global minimum has good guarantees, thanks to classical generalization bounds arguments (Bartlett and Mendelson, 2002).
Limitations and generality. While Theorem 4.1 con- siders a very specific setting, it describes a more general behavior. Although condition 3 of Assumption 4.1 and the initialization scheme of Equation (8) are quite artificial, they are merely required to allow tractable analysis. The experiments of Section are indeed run without these conditions and yield results similar the predictions of Theorem 4.1 for large enough
Although implicit bias and simplicity bias often refer to the same behavior in the literature, we here distinguish the two terms: implicit bias is generally considered in the regime of
7

---

Simplicity bias and optimization threshold in ReLU networks
0.10
0.50
hg
0.45
0.08
OLS
0.40
Train Loss
Test Loss
0.35
0.30
0.01
0.25
0.20
0.02
hg OLS
0.15-
0.10
0.00
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
(a) Evolution of train loss.
(b) Evolution of test loss.
Figure 2: Evolution of both train and test losses at convergence with respect to the number of training samples. o² corresponds to the noise variance B[(r)).
More partieularly, condition 3 of Assumption 4.1 is required to ensure that only two extremal vectors exist. Without this condition, there could be additional extremal vectors, but all concentrated around these two main extremal ones. On the other hand, Equation (8) is required to enforce the early alignment phase, so that all neurons are aligned towards extremal vectors at its end. With more general initial- ization, some neurons could move arbitrarily slowly in the early alignment dynamics, ending unaligned at the end of early phase. Yet, such neurons would be very rare. Relaxing these two assumptions would make the final convergence point slightly more complex than the one in Theorem 4.1. Besides the two main ReLU components described in Theo- rem 4. few small components could also be added to the final estimator, without significantly changing the reached excess risk, as observed in Section 5. This is observed in Figure 2, where the training loss is only slightly smaller than the training loss of OLS, with a comparable test loss.
e.g., Raventós et al., 2024, Figure 4).
Lastly, Theorem 4.1 requires a very large number of samples with respect to the dimension, i.e., n 2 d log d. Our experiments confirm that the optimization threshold only appears for a very large number of training samples with respect the dimension. However, similar behaviors seem to occur for smaller orders of magnitude for n in more complex learning problems, such as the training of diffusion models (Yoon et al., 2023; Kadkhodaie et al., 2024). This dependency in might indeed be different for more complex architectures (e.g., with attention) and is worth investigating for future work.
5. Experiments
This section illustrates our results on experiments on toy model close to the setting of Section 4. More pre- cisely, we train overparametrized two-layer neural networks (m = 10 000) until convergence, on data from the linear model of Equation (7). The network is trained via stochastic gradient descent and the dimension fixed
From a higher level, Theorem 4.1 is restricted to a linear teacher and a simple network architecture. remains hard assess how well the considered setting reflects the behav- ior of more complex architectures encountered in practice. We believe that the different conclusions of our work remain valid in more complex setups. In particular, additional ex- periments in Appendix A run with a more complex teacher, GeLU activations or with Adam optimizer yield similar be- haviors: the obtained estimator does not interpolate for a large number training samples, but instead accurately ap- proximates the minimizer of the test loss. Similar behaviors have also been observed on more complex tasks as gener- ative modeling or in-context learning (Yoon et al., 2023; Kadkhodaie et al., 2024; Raventós et al., 2024). Despite overparametrizztion, the trained model goes from perfect interpolation to generalization, as fails interpolating for large number of training samples. In these works as well, this absence of interpolation does not seem due an early stopping, but rather to convergence to local minimum (see
5 allow reasonable running times. The setup here is more general than Section 4. since i) the data input 23 are drawn from standard Gaussian distribution (which does not satisfy Assumption 4.1); ii) the neurons are initialized as centered Gaussian of variance 10-5 /m (which does not satisfy Equa- tions (3) and (8)). We refer to Appendix A for details on the considered experiments and additional experiments.
Figure 2 illustrates the behavior of both train loss and test loss at convergence, when the size of the training set varies. As predicted by Theorem 4.1, when exceeds some optimization threshold, the estimator at convergence does not interpolate the training set. Instead, it resembles the optimal OLS estimator, which yields test loss close to the noise level E[7?]. In contrast for smaller training sets, the final estimator interpolates the data convergence, which
8

---

Simplicity bias and optimization threshold in ReLU networks
500
5000
1600
4000
Number of neurons
Number of neurons
1000
1200
3000
1000
2000
1000
200
-1.00
-0.75
-0.50
-02
0.00
0.25
0.50
0.75
1.00
1.00
-0.73
-0.50
-0.55 0.00
0.25
0.50
0.78
1.00
Cosine Similarity with OLS estimator
Cosine Similarity with OLS estimator
(a) 500.
(b) n = 5 000.
Figure 3: Histogram of the cosine similarities of the neurons with the true OLS estimator 3, at the end of training.
yields a much larger test loss than OLS, corresponding to the tempered overfitting regime (Mallinar et al., 2022).
While majority of the neurons is already nicely aligned with the true OLS estimator the n = 500 case, an im- portant fraction of them are not aligned with this estimator (69% of them have cosine similarity smaller than 0.9 in absolute value). These unaligned neurons contribute prediction function that significantly differs from the OLS one. On the other hand, nearly all neurons are aligned with this true estimator as n grows larger (91% % of them have a cosine similarity larger than 0.9 in absolute value), confirm- ing the predictions of Theorem 4.1. As explained above, there are still a few vectors that are disaligned with the OLS estimator here, but they are only a small fraction and thus have almost impact on the estimated function.
This optimization threshold here located around n* 3000, which suggests that the large dependency of this threshold the dimension (which is here 5) in Theo- rem 4.1 seems necessary-see Appendix A.5 for experiments with larger dimensions. We still observe few differences here with the predictions of Theorem 4.1, which are due to the two differences in the setups mentioned above. Indeed, even after this optimization threshold, the test loss of the ob- tained network is slightly larger than the one of OLS, while Theorem 4.1 predicts they should coincide. This is because in the experimental setup, few neurons remain disaligned with the extremal ones at the end of the early alignment phase. These neurons will then later in training grow in norm, trying to fit a few data points. However there are only a few such neurons, whose impact thus becomes limited- see Figure 3. As a consequence, they only manage to slightly improve the train loss and have little impact on the test loss.
6. Conclusion
This work illustrates on a simple linear example the phe- nomenon of non-convergence of the parameters towards global minimum of the training loss, despite over- parametrization. This non-convergence actually yields simplicity bias the final estimator, which can lead to optimal fit of the true data distribution. A similar phe- nomenon has been observed on more complex and realistic settings (Yoon et al., 2023; Kadkhodaie et al., 2024; Raven- tós et al., 2024). However, a theoretical analysis remains out reach in these cases. still unclear whether the observed non-convergence arises from the early alignment mechanism proposed in our work, from stability issues as suggested by Qiao et al. (2024), from other factors, or from combination of these effects.
5.1. Cosine similarity with OLS estimator
To illustrate Theorem 4.1 and the fact that neurons end up aligned with the OLS estimator beyond the optimization threshold n²*, Figure 3 shows histograms of the cosine simi- larities between all the neurons w; of the network at the end of training and the true OLS estimator 3 = (XX - for different sample complexities. This experiment follows In particular, Figure 3(a) shows this histogram for n 500, where interpolation of the training data happens (see Figure 2(a)); and Figure 3(b) shows this histogram for n = 5 000, where interpolation of the training data does not happen anymore, but the network generalizes well to unseen data.
the same setup as the one of Figure
Our result is proven via the deseription of the early align- ment phase. Besides the specific data example considered in Section 4, we also provide concentration bounds on the ex- tremal vectors driving this early alignment. We believe these bounds (Theorem 3.1) can be used in subsequent works to better understand this early phase of the training dynamics, and how it yields biases towards simple estimators.
'The cosine similarity between two vectors u, E IRd is de-
fined as cos(u, r) Tmll [

---

Simplicity bias and optimization threshold in ReLU networks
Acknowledgements
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Man- dal. Reconciling modern machine-leaming practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 1 116(32)15849--1584 2019.
This work was supported by the Swiss National Science Foundation (grant number 212111) and by an unrestricted gift from Google.
Impact Statement
Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity, Advances in Neural Information Processing Systems, 36:57795- 57824, 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Erienne Boursier and Nicolas Flammarion. Early align- ment in two-layer networks training is a two-edged sword. arXiv preprint arXiv:2401.1079) 2024.
References
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ru- osong Wang. Fine-graing lineggrinedd analysis of optimization and generalization for overparameterized two-layer neural net- works. In International Conference on Machine Learning. pages 322-332. PMLR, 2019.
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flam- marion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems, 35:20105-200166 18, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural information processing systems, 33: 13:187771901, 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
Phuong Bui Thi Mai and Christoph Lampert. The inductive bias of relu networks on orthogonally separable data. In 9th International Conference on Learning Representa- tions, 2021.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. International Conference on Learning Repre- sentations, 2022.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagiel- ski. Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
Yu Bai, Fan Chen, Huan Wang. Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with cn-context algorithm selection. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: /ttsss//ppeeeevveeeeeee / forum? 1a
Luis Carvalho, João L Costa, José Mourão, and Gonçalo Oliveira. The positivity of the neural tangent kernel. SIAM Journal on Mathematics of Data Science, 7(2):995-515, 2025.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-499 2002.
Dmitry Chistikov, Matthias Englert, and Ranko Lazic. Learning a neuron by a shallow relu network: Dynamies and implicit bias for correlated inputs. Advances in Neu- rul Information Processing Systems, 36:23748-23761 2023.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexan- der Tsigler. Benign overfitting in linear regression. Pro- ceedings of the National Academy of Sciences, 1 117(48): 30063-30070, 2020.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-paramett weeaparameterized models us- ing optimal transport. Advances in neural information processing systems, 31, 2018.
Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica, 30:87-201, 2021. doi: 10. 00.10177/099249222000000
Mikhail Belkin, Daniel Hsu, and Partha Mitra. Over- fitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems, 31, 2018
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages 1305-1338. PMLR, 2020.
10

---

Simplicity bias and optimization threshold in ReLU networks
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference Learning Theory, pages 1772-1798. PMLR, 2019.
Thomas M Cover.
Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (3):326-334) 334, 1965.
Ziwei Ji, Justin Li, and Matus Telgarsky. Early-stopped neural networks are consistent. Advances in Neural In- formation Processing Systems, 34:1805-1817, 2021.
Nirmit Joshi, Gal Vardi, and Nathan Srebro. Noisy inter-
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems, 2021.
polation learning with shallow univariate reLU networks. The Twelfth International Conference on Learning Representations, 2024.
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. The Twelfth International Conference on Learning Representations, 2024.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neu- ral networks. In International conference machine learning, pages 1675-1685. PMLR, 2019.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Ben- jamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of in- creasing complexity. Advances in neural information processing systems, 32, 2019.
Spencer Frei, Niladri S Chatterji, and Peter L Bartlett. Ran- dom feature amplification: Feature learning and general- ization in neural networks. Journal of Machine Learning Research, 24(303):1 -9, 2023.
Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint 1412. 2014.
Shivam Garg, Dimitris Tsipras, Percy $ Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-3055 2022.
Akshay Kumar and Jarvis Haupt. Directional conver- gence near small initializations and saddles in two- homogeneous neural networks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: /0tv:/666.008 15, 2016.
Qianxiao Li, Cheng Tai, and Weinan. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learn- ing Research, 10(40):1-47) 2019.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 6:24, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2019.
pages 6840-6851
Daniel Hsu, Sham M Kakade, and Tong Zhang. An anal- ysis of random design linear regression. arXiv preprint arXiv: l0m.:30006.33333 2011.
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems, 34: 14978-11229,, 2021.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low- rank simplicity bias in deep networks. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprint arXiv: arxiv:1803.003367 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of

---

Simplicity bias and optimization threshold in ReLU networks
overfitting. Advances in Neural Information Processing Systems, 35:1182-1195, 2022.
Mark Rudelson and Roman Vershynin. Hanson- in-
equality and sub-gaussian concentration. 2013.
Song Mei, Andrea Montanari, and Phan-M Than-Minh Nguyen. A mean field view of the landscape of two-layer neural net- works. Proceedings of the National Academy of Sciences, 115(33), 2018.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9588, 2020.
Hancheng Min, Enrique Mallada, and Rene Vidal. Early neuron alignment in two-layer reLU networks with small initialization. In The Tweifth International Conference on Learning Representations, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of 32nd International Conference on Machine Learning, volume 37 of Proceed- ings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zor- nitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1 1104--11000 11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6048- 6058, 2023.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double de- scent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12):124003, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Rep- resentations, 2021.
Behnam Neyshabur, Srinadh Bhojanapalli. David McAllester, and Nathan Srebro. Exploring generalization in deep learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 5949-509 5958, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gra- dient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-2878, 2018.
Dan Qiao, Kaiqi Zhang, Esha Singh, Daniel Soudry, and Yu- Xiang Wang. Stable minima cannot overfit in univariate relu networks: Generalization by large step sizes. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean- Philippe Vert. Regression as classification: Influence of task formulation on neural network features. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 11563-11582. PMLR, 2023.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pages 5301-5310. PMLR, 2019.
Nikita Tsoy and Nikola Konstantinov. Simplicity bias of two-layer networks beyond linearly separable data. In Proceedings of the 41st International Conference on Ma- chine Learning, pages 48728 01728447767, 2024.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Ad- vances in Neural Information Processing Systems, 36, 2024.
Aad W Van Der Vaart and Jon A Wellner. Weak convergence and empirical processes: with applications statistics. Springer Cham, 2023.
Grant Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathemotics, 15(9):1889-1933 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
12

---

Simplicity bias and optimization threshold in ReLU networks
Roman Vershynin. High-dimensional probability: An in- troduction with applications in data science, volume 47. Cambridge university press, 2018.
Stephan Wojtowytsch. On the convergence of gradient de- Scent training for two-layer relu-networks the mean field regime. arXiv preprint arXiv:2005. 13530, 2020.
Sang Michael Xie, Aditi Raghumathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. International Conference on Learning Reprexentations, 2022.
TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabillsstic Inference { & Generative Modeling, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communica- tions of the ACM, 64(3):107-115, 2021.
13

---

Simplicity bias and optimization threshold in ReLU networks
Appendix
Table of Contents
A Additional experiments
15
A.1 Experimental details
15
A.2 Cosine similarity with OLS estimator
15
A.3 GeLU activation
A.4 Momentum based optimizers
A.5 Stability of minima
A.6 Influence of dimensionality
18
A.7 S ReLU teacher network
18
B Proof Theorem 3.1
B.I Proof of Lemma C.I
21
C. Probability tail bound version of Theorem 3.1
22
D Proof of Proposition 3.1
23
E
Proof of Theorem 4.1
24
E.I
Notations and first classical results
24
Phase I: early alignment...
25
E.3
Decoupled autonomous systems
28
E,4
Phase 2: neurons slow growth
29
E.5 Phase 3: neurons fast growth
31
Phase 4: final convergence..
A. Addiionna experiments
A.1. Experimental details
In the experiments of Figure 2, we initialized two-layer ReLU networks (without bias term) with m 10 000 neurons, initialized i.i.d. for each component Gaussian of variance
10-3
We then generated training samples as
ym
V/ 5 Ik Tek
where 77 are drawn i.i.d. as centered Gaussian of variance 0² 0.09, 24 are drawn i..dd as centered Gaussian variables and 3* is fixed, without loss of generality, 8* (1, (1,0,..., 0). The dimension is fixed to d = We then train these networks on training datasets of different sizes (each dataset is resampled from scratch).
The neural networks are trained via stochastic gradient descent (SGD), with batch size 32 and learning rate 0.01. To ensure that we reached convergence of the parameters, we train the networks for x 10t iterations of SGD, where the training seems stabilized.
All the experiments were run on personal MacBook Pro, for total compute time of approximately 100 hours. The code can be found at github. com/ wtwprssirr/iimmiiiiii..ooooeeeee
A.2. GeLU activation
Our theoretical results can be directly extended to any homogeneous activation function, i.c., leaky ReLU activation. Yet. the theory draws different conclusions for differentiable activations functions and claims that for infinitely wide neural networks, the parameters should interpolate the data at convergence (Chizat and Bach, 2018). This result yet only holds for infinitely wide networks, and remains unknown how wide a network should be to actually reach such an interpolation in practice. Figure 4 below presents experiments similar to Section 5, replacing the ReLU activation by the differentiable GeLU activation (Hendrycks and Gimpel, 2016). This activation is standard in modern large language models. Notably, is used the GPT2 architecture, which was used in the experiments of Raventós al. (2024).
14

---

Simplicity bias and optimization threshold in ReLU networks
0.10
1.0
hg
0.08
0.8
OLS
Train Loss
Test Loss
0.01
0.02
0.2
OLS
0.00
500
1000
1500
2000
2500
3000
3500
4000
1000
1500
2000
2500
3000
3500
4000
n
71
(a) Evolution of train loss.
(b) Evolution of test loss.
Figure 4: Evolution of both train and test losses at convergence with respect to the number of training samples, with GeLU activation.
While infinitely wide GeLU networks should overfit, even very wide networks (m = 10 000) are far from this behavior in practice. In particular, we observe phenomenon similar to Section in Figure 4. Surprisingly, even seems that interpolation is harder reach with GeLU activation, as the network is already unable interpolate for 500 training samples. We believe this is due to the fact that GeLU is close to linear function around the origin (corresponding to small initialization regime), making it harder to overfit noisy labels.
A.3. Momentum based optimizers
Our theoretical results hold for Gradient Flow, which is first order approximation of typical gradient methods such as Gradient Descent (GD) or Stochastic Gradient Descent (SGD) (Li et al., 2019). Yet, recent large models implementations typically use different, momentum based algorithms, such as Adam (Kingma, 2014) or Adam (Loshchilov and Hutter, 2019). illustrate the generality of the optimization threshold we proved in specific theorctical setting, we consider in Figure below the same experiments as Section 5, with the exception that i) we used GeLU activation functions (as in Appendix A.2) and ii) minimized the training loss through the Adam optimizer, with pytorch default hyperparameters.
We focus on Adam rather than AdamW here to follow the experimental setup of Raventós et al. (2024) and because focus is on implicit regularization, thus avoiding explicit regularization techniques.
0.10
0.6
0.08
OLS
Train Loss
Test Loss
0.06
0.04
0.3
0.2
0.02
hg OLS
0.1
0.00
2000
3500
4000
1000
2000
3500
7
72
(a) Evolution of train loss.
(b) Evolution of test loss.
Figure 5: Evolution of both train and test losses at convergence with respect to the number of training samples, with GeLU activation and Adam optimizer.
The observed results are very similar to the ones of Figure 4, leading to similar conclusions than Appendix A.2 and the fact that considering Adam rather than SGD does not significantly change the final results.
A.4. Stability of minima
Qiao et al. (2024) argue that the non-convergence of the estimator towards interpolation due to the instability of global
15

---

Simplicity bias and optimization threshold in ReLU networks
0.0855
0.0850
0.0845
Train Loss
0.0840
0.0835
0.0830
0.0825
0.0820
0.0815
0.0
0.5
1.0
1.5
2.0 Iterations
2.5
3.0
3.5
4.0
x10
Figure 6: Evolution of training loss from warm restart with a decaying learning rate schedule (n 8000, d 5).
minima. More precisely they claim that for large stepsizes, gradient descent (GD) cannot stabilize around global minima of the loss for large values of n. We present additional experiment in this section, illustrating that this non-convergence is not due to an instability of the convergence point of (S)GD, but to it being a stationary point of the loss as predicted by our theory.
For that, we consider a neural network initialized from the final point (warm restart) of training for 000 samples in the experiment of Figure 2.4 We then continue training this network on the same training dataset, with decaying learning rate schedule. Precisely, start with learning of 0.01 as in the main experiment, and multiply the learning rate by 0.85 every 50 000 iterations of SGD, so that after 4 x 10th iterations, the final learning rate is of order 10
We observe on Figure 6 that the training loss does not change much from the point reached at the end of training with the large learning rate 0.01. Indeed, the training loss was around 0.082 at the end of this initial training, which is slightly less than the noise level (0.09). While there seems to be some stabilization happening at the beginning of this decaying schedule, the training loss seems to converge to slightly more than 0.0815, confirming that the absence of interpolation not due to an instability reason, but rather convergence towards a spurious stationary point of the loss.
A.5. Influence of dimensionality
Theorem 4.1 predicted an optimization threshold scaling in Old log d). However, the experiments Section 5 consider fixed dimension (d = 5), making it unelear how tight is this theoretical optimization threshold and whether similar dependency in the dimension is observed in practice. To investigate further this dependency in the dimension, present in this section experiments in the same setup described in Appendix A.1, with the sole exception that the dimension is larger, fixed to d = 10.
Figure illustrates the evolution of both the train and test losses as the number training samples increases this larger dimension setting. In that case, the optimization threshold seems much larger: interpolation stops happening around 10 000 samples, and an estimation close the OLS estimator really starts happening much larger values of n, around 80 000.
Comparing with the d = 5 case, it thus seems that the point at which interpolation stops indeed seems to roughly scale in ds. However, this scaling seems even larger for the point where the estimator corresponds to the OLS one. We believe that this discrepancy is due to the differences between our theoretical and experimental setups, and in particular to the fact that multiple intermediate neurons can grow in our experimental setup (see Limitations and generality paragraph in Section 4.2).
A.6. ReLU teacher network
This section presents an additional experiment with a more complex data model. More precisely, we consider the exact same setup than Section 5 (described in Appendix A.I), with the difference that the labels y are given by
yk = f* (Ix) + Tek
Annother relevant experiment is to train from scratch (no warm restart) with smaller learning rate. When running the experiment of Appendix with a smaller learning rate 0.001, we observe again that parameters convergence correspond to OLS estimator,
16

---

Simplicity bias and optimization threshold in ReLU networks
0.10
0.22
he O1.S
0.08
0.20
Train Loss
Test Loss
0.18
0.16
0.04
0.14
0.12
0.02
hg OLS
0.10
10000
20000 30000
40000
50000 60000 70000
80000
10000 20000 30000 40000 50000 60000 70000 S0000
7
(a) Evolution of train loss.
(b) Evolution of test loss.
Figure 7: Evolution both train and test losses at convergence with respect to the number training samples, with dimension 10.
where f* is a ReLU network:
(*(xx)
Σ (ar B+)+.
The parameters B are drawn i..dd random following standard Gaussian distribution. We use the exact same 3 across all the runs for different values of Also, Ty and f/ generated in the same way described in in Appendix A.I.
Figure 8 also presents the evolution of the train and test losses as the number of training samples varies. We observe behavior similar to Figure 2, where interpolation is reached for small values of n, and is not reached anymore after some threshold n*. While the test loss is far from the optimal noise variance before this threshold, it then becomes close to afterwards.
Yet, this transition from interpolation to generalization slower in the ReLU teacher case than in the linear one. Indeed, while interpolation does not happen anymore around 2000 in both cases, much more samples (around n* 17700) needed to have simultaneously training and testing loss close to the noise variance. These experiments suggest that the behavior predicted by Theorem 4.1 for a linear model also applies in more complex models such as the 5 ReLU teacher, but that the transition from interpolation to generalization can happen more slowly or with more training samples depending on the setting.
0.10
0.250
0.08
0.225
Train Loss
Test Loss
0.200
0.175
0.04
0.150
0.02
0.125
hea
0.100
h66
0.00
2500
5000
7500
10000
12500 15000 17500
20000
2500
5000
7500
10000
12500
15000
17500 20000
n
72
(a) Evolution of train loss.
(b) Evolution of test loss.
Figure 8: Evolution of both train and test losses at convergence with respect to the number of training samples with a 5 ReLU teacher. x² corresponds to the noise variance E[rr)).
The slight difference with Figure is that this optimization threshold here seems to appear for larger values of n.
17

---

Simplicity bias and optimization threshold in ReLU networks
B. Additional Discussions
Double descent. Double descent originally refers to the fact that the test loss obtained at convergence does not behave monotonically with the number of model parameters, Recently, different types of double descent have been proposed (Nakkiran al., 2021). Notably, Henighan et al. (2023) study data double descent, where the test loss follows a "double descent" shape when plotted against the number of training examples. The phenomenon highlight here is different, as our test loss monotonically decreases with respect to the number of training points, illustrated in both our experiments and Figure 1.
However, the toy experiments of Henighan et al. (2023) illustrate similar phenomenon: for a sufficiently large number of training points, the training loss remains high while the model learns optimal features. It remains unclear though whether this high training loss stems from an underparametrized regime (i.e., the model lacks sufficient capacity memorize the data) or if optimization fails reach the empirical risk minimizer in their setup.
Feature learning and NTK regimes. We distinguish in this work between feature learning and the NTK/lazy regime, as they involve fundamentally different training dynamies (see Chizat et al., 2019, for an in-depth discussion). Our work specifically focuses the feature learning regime with small initialization, as indicated by our initialization choice (Equation (2)), where both the inner and outer layers scale as mm
In contrast, in the NTK/azy regime (corresponding to large initialization scales), theory predicts that interpolation should occur at convergence, which is contrary to our main result. However, empirically demonstrating this interpolation in our toy model (with large n)) is computationally challenging, as it would require an extremely large number of parameters.
C. Proof of Theorem 3.1
We recall Theorem 3.1 below.
Theorem 3.1. If the marginal law of xı is continuous with respect to the Lebesgue measure, then for any n E N,
gx,y sup WES, D. (w,
sup
|| D(w)l2)]
( d d n
11 x
where for any E Sa-1, D(w) B[1 >on
Proof. We first show similar result on the following expectation
[Dn(()) D(w)ll2 = =0(
d log
Exyy
sup w
n
where we recall (n(w)= 1. Ei=1 LuT We bound this expectation using typical uniform bound techniques for empirical processes.
symmetrization argument allows to show, for i.i.d. Rademacher random variables & (-1,1) (see Van Der Vaart and Wellner, 2023, Lemma 2.3.1.):
|
Ex.y
sup [Dz(ww) - D(w)ll2 2 Ex. #EB1-1
Eg
sup WEB
Σ n
T CORVA.TR
X,y
(10)
From there, it remains bound for any value of X. y the conditioned expectation Eg[- X,y]. We consider in the following fixed value of y. Note that the vector Σn k== I wT an 2...T,, actually only depends w in the value of the vector ooookkkk)) Define
A(X, = {(1 ((1 w E Rd).
(11)
We thus have the equality:
sup -
1 w
T T2
= sup uEA(X.y)
7 k=l
7 A=
18

---

Simplicity bias and optimization threshold in ReLU networks
Moreover, classical geometric arguments (see e.g. Cover, 1965, Theorem 1) allow to bound card (A(X, y)) for any X, y:
d-1 n
1
card (A(X, 2
k
k=0 =O(n")
(12)
From there, we will bound individually for each u E y) the norm of 1 Ex= ULERyaag and use a union bound argument.
Let E y() Define Z Rdx= the matrix whose column k is given by z(k uuy.r. Then note that +E UgE Ze. Hanson- Wright inequality then allows to bound the following probability (see Rudelson and Vershynin, 2013, Theorem 2.1) for some universal constant > O and any t 2 0:
Pe (|I ZE|l2 - || t x. y) S 2e
1212
where | and 2| respectively denote the Frobenius and operator norm of Z. In particular, noting that || I || this last equation implies that for any >
Pe (I|Zell2 (1 1) X.y) s 2e -t
(13)
Moreover, note that
S n
C(Z),
n
where C(Z) = Ei= 11 does not depend on u.
Rewriting Equation (13) with this last incquality, and with ô 2e -e- we finally have for each u E y)
[C(Z)
Pe |ZE||2 > (1 + v In(2/5))
X,y
5.
n
Considering a union bound over all the u E A(X, y), we have for some universal constant > 0, thanks to Equation (12):
Fe Ju ((x))(X)), > (1 (1+ V (o((log(2//)) +
+ dooggn) +1) / C(Z)
X,
< 6.
(14)
Moreover, conditioned on X,y, y. ||Zl|| is almost surely bounded by /n|l and so by VC(Z). direct bound on the expectation can then be derived using Equation (14) with
1-
d log n - VC(Z).
Eg
sup n.
TE
Wrapping up with Equation
(11) and Equation (10) then allows to derive Equation (9),
d n
Ex.y
sup W
[D,(w) D2w)ll = O
Ex.y [C((z)]
d d n
Ex,y (C(Z))
dl do m
[|]
Lemma C.1 below then allows to conclude.
19

---

Simplicity bias and optimization threshold in ReLU networks
Lemma C.1. the marginal law of continuous with respect the Lebesgue measure, then almost surely:
sup W W (w)
sup
D - D(w)lle & sup [ D(w)ll2,
w 18
where Dn(w)= = a _ In SoW.
C.1. Proof of Lemma C.I
First observe that if the marginal law of x is continuous, then D continuous with respect to Consider any Sa-1. We recall that the set Dn (w) is defined
[0, 1] if (wf,,
Dn(w) = {
TRUKTR Vk
[n], 7k
if (1of, .xa) > 0
otherwise
If all the values Ny are non-zero, then Dn (w) is the singleton given by (,((w) and thus
sup (wEDa(w)
| D(w)ll2 = (w) D(w)
Otherwise, if wT 24 0 for least one k observe that"
(n) lim inf Conv (onv((D,(w')) | w' S and || - w'lla S {()),
620
where
S = fw' E Sa-1 w au * 0 for all k}.
In other words, for any Dn E (w), w can be approached arbitrarily closed by vectors wy such that for some convex combination 77,
Dn Σ ?h (w:)).
From then, it comes that
|D - D(w)ll < Σ Mill Dn.s))) D(w)ll
Σ mi(llDa - D(w:)l (D(,,,) D(w)l)
< sup [Dx(w') - D(w'lll + Σ mill ((w,) - D(w)ll w'eS
Since D is continuous and the w, be chosen arbitrarily close w, the right sum be chosen arbitrarily close 0. In particular, we have shown that for any D, E (w),
||D, D(w)ll S sup [DDx(w)) D(w')ll
This concludes the proof of Lemma C.I.
D. Probability tail bound version of Theorem 3.1
While Theorem 3.1 bounds the maximal deviation of D, D(w) in expectation, high probability tail bound is also possible, as given by Theorem D.I below.
"This observation directly follows from definition the Clarke subdifferential.
20

---

Simplicity bias and optimization threshold in ReLU networks
Theorem D.I. If the marginal law of is continuous with respect the Lebesgue measure, then for any N and M 2 E ||,]]*
Px.y
sup WE8 Dr (w)
sup
DD, - D(wille > 4 +1+ + dlog(n) (log((2))))
+ +1)
V n
M
S + Px,y
1| M
Proof. The proof follows the same lines the proof of Theorem 3.1 in Appendix C. In particular, we first want bound in probability the term supwes, [n, ) 2l To this end, probabilistic symmetrization argument (Van Der Vaart and Wellner, 2023, Lemma 2.3.7.) yields for any t >>0 0
1
Px.y
sup [|n(ww) ((w)l22 1
< xPxx
Pe
sup #
1 T2
Σ
ek I
X,y
(15)
where (t) = 1 in sup -S.-1
( x). In particular here, 3n(() 2 1 - #
||
Moreover, we
already showed Equation (14) in the proof of Theorem 3.1, which states
Ve'ooo(lg((///) + dllo((n) + 11) / /CXX.>)
Pe
sup w
Σ Ex 2
X,y 5o.
n
where y(X.y) Σ 11
Equation (15) then rewrites for any M > 0:
[1Dx(w) 2 > 4 (1 1 (C(log(2/0) + dlog(n) + 1}] V m
M
Px.y
sup WE
Mlner 11 2 1Ipp M M)
1
8 Px.y
n2
with
/ /M
1 1= (1 (e((lo((//)) +d dlog(n)) + +1)
V
n
Note that for any M E [1xx|12]
Bn) 2 3. which implies
Px.y sup W
- (
(1+ (1+veellose (c(log(2/0) + dlog(n) + 1)
M V
+ 3
Px.y
| 2 M
A=1
Theorem D.1 then follows, thanks to Lemma C.1.
Corollary D.1 below provides a simpler tail bound, directly applying Lemma C. with Chebyshev's inequality to bound Px.y [ Ei= || M). Stronger tail bounds can be provided with specific conditions on the random variables my and yı. but the one of Corollary D.1 is enough for our use in Section 4.
Corollary D.I. Assume the marginal law of is continuous with respect to the Lebesgue measure. Moreover, assume ||zy|| admits fourth moment. Then
DD D(w)l12 4 (1 (e((lo(2//)) + dlog(n) (llog/) +1))
2E yyx| [
Px,y
sup WEB
sup (w)
21

---

Simplicity bias and optimization threshold in ReLU networks
4
4 B
8 +
3
3
BBBll y y [|]2
Proof. This a direct consequence of Theorem D.I, using Chebyshev's inequality to bound Px,y
[1 Σk=
11 M
E. Proof of Proposition 3.1
In the following proof, we define the following subsets of the unit sphere in dimension for any 6 0:
H {w E 1 (ww) satisfies Equation (6)),
H(66 D- B(D n Sd-1:
WE
min (1 [DD[ww]]
D(w)
D(w)
A min(1,
inf wEBa-1 (1())
w|l.1l
|D(w)]| + w|l. [ID(w))))
Here, B(D(w), 6) denotes the open ball of radius 6, centered in D(w).
Proof. Since the marginal distribution of X is continuous, the function D w (ww) is continuous. In particular for any
5 > 0, the infimum defining (()) is reached, so that A 0 by definition of H. In the following, we let å = 2. Thanks to Corollary D.I, with probability at least - O₂ (
DD - D(w)ll== = On (V dllgg =0
sup in ER
sup (w)
n
dlog min(A J
In particular, we choose n (e) = =0,
(AG))²,))
large enough so that for any n (e), with
probability least 1 - O₂ (2)::
Dr, D S min (A()).E).
1
sup W D,, (w)
sup
(16)
We assume in the following of the proof that Equation (16) holds.
Consider an extremal vector D, of the finite data yX By definition, there is some E 8 such that D E (,,(w) and either
1. Dn 0.
2. or An(D,) (Dn) = (n)),
3. or (Dn) = An)
In the first case, Equation (16) yields that [ID(w) < A()). Necessarily, by definition of A(S), w E H()). This means by definition of H(5) that there exists D* E Rd satisfying Equation (6), such that
[ID(w) D*112 2
In particular, using Equation (16) again yields D|D, D- E.
In the second case (D,) = (w)), we can assume D, + 0. In that case, as Da have the same activations,
(ww = Dn,( (x( i.e., we can assume without loss of generality that w = Da E H()), then there exists D* IRd satisfying Equation (6), such that D., D* lll S
[D.ll here. Similarly to the first case,
22

---

Simplicity bias and optimization threshold in ReLU networks
Let show by contradiction that indeed w H(g). Assume w H())) In particular, IID(w))l2 2 A We can now
bound the norm of D(w)
[
(ww)
(ww)
Dn
D(w)l
[D(w)]] |
2
12
D(w) Dn
Dn.
[D [D(w)))
+
[
|
[D(w)]
2
[D(w)= - Dall
||| [(w)]]
[D(w)
[[((w]]
2 [D(w)) - -D.ll
||
A 2 24 (()))
By definition of ())), this actually implies that E H()), which contradicts the initial assumption. We thus indeed have E H()), leading to the existence of D* with the wanted properties such that D,, 12 S e. In the third case (D,) = (w)). symmetric arguments lead to the same conclusion, which concludes the proof of Proposition 3.1. 0
F. Proof of Theorem 4.1
F.. Notations and first classical results
In the whole Appendix F, we define Σ = BBxxr | and
En, Σ
T&TK
En,-
Σ
KES_
KES-
and the following set of neurons:
I+ {i [m] a;(0) 0} and I- (i E [m] (0) <0.
We first start by stating the following, known balancedness lemma (see, e.g., Arora et al., 2019; Boursier et al., 2022).
Lemma F.1. For any E [m] and E R4, (t)² (t)l = a: a:) (0)]2
Lemma F.1 can be simply proved by direct computation of the derivative of (t) [| (t)ll, Thanks to Equation (8), this yields that the sign (t(t) is constant over time, and thus partitioned by the sets of neurons I, and I_..
Also, note that with probability 1 -u--r, the sets I. and I__ are both non empty, which assumed to hold the following of the section.
In this section, all the O, O and Ω notations hide constants depending on the fourth moment of 17. the norm of 3 and the constant of Assumption 4.1. Note that due to the sub-Gaussian property of x, its k-th moment can be bounded as ||| dt
for any k.
F.2. Phase early alignment
Lemma F22 If Assumption 4.1 holds, there exists = 0()) and n* = (dd log d) such that for any X S X and
E (0, }), 2 n' and for =la(1/2) with probability 1 - O(:):
||
I. output weights do not change until
Vt < J. la) < la,(t)) S (0)(0))> 25;
2. all neurons align with +E3*
(T(T) (x(T))
d log n
Vi E [m].
Σ33 = || O X
23

---

Simplicity bias and optimization threshold in ReLU networks
Proof. We start the proof by computing Dww) for any 1
( = [
E1 T T=>0 3*]
2 Taw 1 E[ T a ") l
Σ
2
The second inequality comes from the independence between x and 7, the third one comes from the symmetry of the distribution and the last one by continuity this distribution.
Corollary D.I additionally implies that for some n (cd log d) and any n 2 n², with probability least 1 (
Dn D(w)lle S 20(v dd log n
sup
sup (w)
(17)
Vd
W
with a = min(c- -
v 11 [I) Note that a > thanks to the fourth point of Assumption 4.1. Moreover, using typical concentration inequality for sub-Gaus Gaussian vectors, we also have with probability 1 - O
Ek=n + S 2x 10 = O(d)
(18)
n1
We assume in the following of the proof that both Equations (17) and (18) hold.
Σg" Since D(w) - for any w. have for any w S D, E (w) and k: E S4:
z Dn, = = (D - D(w)) + " (Σ - (r) + z² gr
2 |sxl| (1I - D(w)ll -1 - -381 + c
2vd
a | > 0.
vd
Similarly for any k E S_, Dn < O. This directly implies here that there are only two extremal vectors here:
Dr(3*) = Lm, 9
Σ
Tk
RES+
1
Dn((3t) -
(19)
n
We can now show, similarly Boursier and Flammarion (2024), the early alignment phenomenon in the first phase. 7
1. First note that Equation (17) and the definition of imply that for any w:
|D (w))l s || ||.
(20)
We define L = minft 2 0 Σイ a,())² > 22-4
For any E [m] and E [0, 4. Equation (4) rewrites:
(t)
= [w:(t) T (t) dt
[w)][ En || 12 22-12 w
< [ax(t)]
max
n
"The additional d dependence comes from the expectation of ||y=|| in the square root. Additionally, the probability bound comes from the fact that F,, E,]llri] We could directly scale that does not depend on n.
O(1) here. E, { Theorem from Boursier and Flammarion (2024) here, but would not allow choose initialization
24

---

Simplicity bias and optimization threshold in ReLU networks
S (a) (1E + |B|[|= (r))
As consequence, simple Grönwall argument yields that for any t E [0. f1]:
[t)(t)) - [c) exp(t 2t x 1=11)\12--1)))
In particular, for our choice of r, for a small enough X o d - for any / S min(T,
[a:(1) < (0)
(21)
Note that this implies that & < f1, i.e., T < 11. As consequence. Moreover, we can also show that [a,(t)] > |a;(0){ for any S which implies the first point of Lemma F.2.
2. For the second point, let i E I+ and denote @t(t)= constant sign. Also, for almost any [0,
w.(() a.ll)
Thanks to Lemma F.I, w,(t) E B(0, 1) and a,(t) is of
(
- E D -
dt
Since (t) > 0 for E
(t), Σ8a)
(D,(w,(t), (()),,))))))))))))))))) (,
dt
(.. (D,, Σ8*) D.) (m))) E3*)
inf
inf D. (u;((())
(Dn, E3a) - (w:(1)) (w.)) E3*) 2|1E 11
(D(w:(t)), Σ8*) - (t),
22-1e + 2|
sup
||, D ((xe(())
D (m,D)(w,,1)
(w, E,t)) 1 11
0 22-15 +
d2 log
2 2
n
Solutions of the ODE (1) a² f()² with 5(0) (-a, a) are of the form ((1 a tanh(a(( (0) for some E R.
By Grönwall comparison, thus have
(w:(1),E8*) 2 aaanh( (1+())))
(22)
d2 log
where a= |=11,311 O 12-15 +
n
and (m. a tannl( 5(3).
Thanks to the choice of initialization given by Equation (8), TT. [0,] < 1/ and so (m. E3²) - Moreover,
tanh(z) -1 202 that
1 S a(-1 + 2eat)).
|| 0f 12-18 +
Since
da log n
this yields
d2 log n
2ae at,
1 || + O
12-16 +
The previous inequality can be rewritten
-4a²
-2ae -,, M
1/8
|| + O
A2-45 +
49 log
25

---

Simplicity bias and optimization threshold in ReLU networks
Saa
da log n
2 | (1 + 0
x2-4= +
n
d2 log n
2 8|1381 || + O X2-42 +
n
Using that tanh(x)) 2 - 2e 20, Equation (22) becomes at time T and
n S2(d-).
[o]
2ae aly -ar
d2 log
|EB1 || EE
118118811
O 12-15 +1
)e JE
n
0 12
da log
2--46
π
d2 log
||8
O
x +
3. The same arguments be done with negative neurons.
F.3. Decoupled autonomous systems
In the remaining of the proof, we will focus on an alternative solution (w, a), which is solution of the following differential equations for any t 2
(t) ( and
da:(t) (w. for any E I+.
dt = (t(t) and
(23)
dw,(t)
da,(t)
di (w.(t), for any E I_
dl
where
1 Σ D+(1)
(1), yh Tk:
n
keS
iEI+
(-(t)
Σ
(w)
KES.
TET
and with the initial condition (T( ai(T) wi(T), (T( for any E [m]. We also note in the following w; and the estimations of the training data 24 for any ie E [n]] as:
Eiez, a,(, if E S+ if E S_
ha(xx)
Eier_ (k)
This construction allows to study separately the dynamics of both sets of neurons I4 and I_, without any interaction between each other. As precised by Lemma F.3 below, W,, a, coincide with wis ag as long as the neurons all remain in the sector they are at the end of the early alignment phase.
Lemma F.3. Define T4 = (t 2 E(i, E I x signni
+ sign(xI
and T_ = inf(t 2 - 3(, e I_ x [m], sign(a) w() + -iign<=] }.))
Then for any E [m] and any [T. min T.)); (wi(t), ax())) = (w. at()). Moreover, for any [T, min(I, T-)] and l E [n], (xx) he(e) (xx))
While analyzing the complete dynamics of (w, a), we will see that both T4 and T_ are infinite in the considered range of parameters, thus leading to complete description of the dynamics of (w, a).
26

---

Simplicity bias and optimization threshold in ReLU networks
Proof. Thanks to the definition of T4 and T_, the evolution of (L) (t)) given by 4 coincides with the evolution of
at())) (tt)) given by Equation (23) for [r, (i: 7-)]- The associated ODE is Lipschitz on the considered time interval and thus admits unique solution, hence leading to (t) (w,(t, (:())) on the considered interval. The equality ha(e) (2x) = (zx) directly derives from the ReLU activations and definitions of T4 and T-.
F.4. Phase 2: neurons slow growth
For some E2 > O, we define the following stopping time for any E -}:
T2.0 = it 2
Σ a,))) 2 82).
iEI
Lemma F.4. If Assumption 4.1 holds, for any E (0, 1), there exist X = c3 = (d 8) and n* edd log d) such
that for any À S X², n 2 n*, o E (+. 82 E [A2- sg], with probability 1 ( 2 12,0 < +oc and at this time,
1. neurons in Io are aligned with each other
Vi,j Ios (w,(T2,), (((T,))) = o
A
82
2. neurons in Io are in the same cone as o3 for any t E T2,]::
vi E Io, min (W: (T2,),
Ik |
22(
1 Vd
and
max (T,(0),))) KES_ 9
2b |
-2(
KES.
Va
Proof. In the following, we assume without loss of generality that o = +. Additionally, we assume that the random event I+ A and Equations (17) and (18) hold. First, by definition of 72,4, for any 1 E [T, T2,+]:
||D+ (1) (8)) -
1
Σ
||
VEI
AES+
< 2222E xx [3]=11]]
This also implies with Equation (20) that [[]+ < |,3 || + 28 R [||==|]] Additionally, we have with Equation (17)
that
Σ
(1)+(()
1l S (1) -
-Da Dn 112 + (PDx - D( 1|2
2
$ 0 de2 + d2 logn
(24)
Then for any k S4, E I4 and 1 E [T, 1. long (w: axx) 2 (l), |
(t),
Ik
) (t), w (t)) (t), 26
dt
||
||
2 (Bn( |
25210 ||| [|]] || (t) (t), ||>>||
Ik
(B*),
The
2x
|||=||²] (1) (8* I| + 282 pI ||| (w:)).
(25)
|
|
As (B)), zト 2 a (Equation 19), thanks to Lemma F.2 and the third point in Assumption 4.1, and for a small enough o = (d $), minnes. (T) of a = ( 8) and < <
|
Vā
||
O())
= 2
( Equation (27) then implies for a small enough choice
T
1
min
min (Wx(7),
(26)
KES+
|| ||
Vd
Similarly, we can also show
max 1E(1,2,-
KES
max (T),
|
= - 2
27

---

Simplicity bias and optimization threshold in ReLU networks
which implies the second point of Lemma F.4. Actually, we even have for this choice of parameters the more precise inequality (for the same reasons) that for any & E S4, I+ and i [7, 72,+]
(Wa (T).
2k
(r.(*)) 2k
- O(dez).
(27)
|
We now simultancously lower and upper bound the duration of the second phase 72. T2. For any t E T2,-
1 1
idEies
KEI, a;(1)²
(1)
(w, (+)
2
dt
VEI+
Σ
(t)?
Σ (yk Rooce) (w:)))
(28)
1EI+
KES+
Eies ya mull=xll(
E2 EKES, || /1?
Σ (t)
VEI
Note that E[ v=llaalll 2 (d- for any E T2.4]:
[|||=
Using Chebyshev inequality, we thus have for small enough choice of E3
2/a
d EiEI, ar(1)² di 2 2(1) Σ a((1)?
VEI
Grönwall comparison then directly yields 72.+ < 00.
We now want show that the neurons w; are almost aligned the end of the second phase. For that, we first need lower bound the duration of the phase. Note that Equation (28), with Equation (26), also leads for any t [T, T2,+1 to
1 d EdEEE, a²())
Σ
at))
Σ
yk (w;()), mx)
2 dt
n
iEI
KES
Σ
at (t),
(())
IEI+
Note that by continuity, Eiez, a²( E2. As ΣiEr, a.(7)² < 2-44, thanks to Lemma F.2, a Grönwall inequality argument leads to the following E2 22
82
+2. 2
In
(29)
2 Dr,
22-46
For any pair of neurons i,j I,, we consider the evolution of the mutual alignment: d (t), (t))
(t) + w, (1) (t), wy (t))
= (ID. (( (1) - ((2))) (1 - (w, w,
(30)
Moreover, Equation (27) leads to the following alignment between (1) and (3*) for any / E [T,m2,]]
(D (P
Σ KESt
yu (Zx, (x))))
Dn((*)
Σ KESt
yk (Zt,
[D.( (
(8*) -O(dez)
(Dn(3*),
[D*(3*)]]
= -1')( -O(dez).
28

---

Simplicity bias and optimization threshold in ReLU networks
Equation (30) then rewrites for any / 72,+]
(w,(t)) w (t) 2
(1- (w,(w((),,)()))
dt
Moreover, thanks to Lemma F.2, a simple algebraic manipulation yields (w)(F),,,(T) 2 - -0(x xx
de log
Grönwall inequality then yields, for the considered range of parameters,
(, 2 (1 (W: (T)))
de log
2 o (des Inf (n(2)
2 0 X +
Y2
82
A
2 0
A
(2 4e (dex))
The second inequality comes from the bound +2,+ in Equation (29). The third one comes from the fact that I 3 and 82 (n()) = O()) Noticing that 2 4c 2 1 finally yields the first item of Lemma F.4 for a small enough e7 = Od #).
F.5. Phase 3: neurons fast growth
The third phase is defined for some E3 > 0 and ög by the following stopping time, for any E -+:-}:
33. inf(t 22. (t) Bn.llssn.o I E3 or fi Io, So, (w;(t),
Ik
I 63),
where Bo(t)= = Σ (t))),,(t).
TE,
Lemma F5. If Assumption 4.1 holds, for any E E (0, 1), there exist X 0(2). E = o(d n = (d log d).
ag = -1),, 83 = ( and E3 = (1) such that for any X I X² n 2 n", ao -E + E2 E [A=-se, et] and
T3 < too and E3 [ eg], with probability 1 O de + +
1. neurons in Io are in the same cone as of* for any t [T, T2
any
Vi E Io:
min kES:
(W:(1),
Tk
2 283
and
max (wi(l), kES_
Ik |
S -28-.
In particular, [13. Bn,all
S3 by continuity.
Proof. Similarly to the proof of Lemma F.4, assume that +, that the random event I4 A 0, Equations (17) and (18) and the first and second items states in Lemma F.4 all hold. We can first show that for any E [T2 Ts,+h.
Σ IET
at)) 2 E2.
Indeed, recall that the output weights a; evolve for any t E [r2,4, (3,++ as
(t) dt = (wi(t),D+(t)
(w((),,,((B)))
Σ
(a(e)(2)) (2x))
n
RES+
(w(()) .R) + Σ (wi(l). -oom)))o( -o (0 KES
)
2 ai(!)
Σ
(31)
KES.
IEI+
'A similar manipulation can found in (Boursier Flammarion, 2024, proof Lemma 5).
29

---

Simplicity bias and optimization threshold in ReLU networks
The last inequality comes from the fact that Exesy ||
From then, note that 1 E Exes, (w, (B* 2 n. (d during this phase. Moreover, using Chebyshev inequality, we can show for any that with probability at least
O(d).
1 0 z4m
2.
(32)
SES+
Taking small enough = (6s)), Equation (32) holds with probability 1 1- ( that for any t E [T2,+: +3,+] and E I+:
d
and, along Equation (31), this implies
(t. ai()) (((d)) -0 2 ac(1)) iEI+
dt
In particular, there exists ((2) such that if EiEz, ai( r, all the ac(() are increasing. Moreover thanks Lemma F.4, Esez, ) E2. As 8z 0 3 phase,
choose E Oed 8) small enough that during the third
Σ EET
a;(1) 2 82.
(33)
Now note that by definition of B,,+.
(+) - 1 Σ
Juit [+) - J4YR
11 KES+
-=+ BBn+))
(34)
As consequence, (1) evolves as follows:
d14(1)
I+ ) (+()))
dt
VEI
Σ Iaa Σ Cm(emm(O) (3+(() BB,+))
SEI+
sEI+
In particular, this implies:
xdllB((t) Bm.+ 1...
(t)
En,+ (3+(t) BBmit)
(35)
2
dt
dt
= -(44((1) - (Ti,))Tnn Σ
a
Σ [m(()ma(0)) VEI,
-
Em,+ - -n,+)- (36)
VEI,
The matrix S/ (Ix, (1) (c()) 1//2 symmetric, positive definite. Thanks to Equation (33), its smallest eigenvalue larger than E2dmia (En,+): where Amin denotes the smallest eigenvalue of a matrix. Using typical concentration inequalities on the empirical covariance (see e.g. Vershynin, 2018, Section 4.7), with probability
Y llpp o
1 - -O)), |En,+
With the fourth point in Assumption 4.1, we then have for a large enough
n² = (d log d) and with probability 1 - (H),
0(1) > Amax (2n++) 2 Amin (2n,+) 2 2(11)
Amin
Ammaxe (Emit)
Amax (E)
- log v
(37)
and
+0
Amin (m,+)
Amin (2)
where \maxx) denotes the largest eigenvalue.
30

---

Simplicity bias and optimization threshold in ReLU networks
(37) holds in the following, so that the smallest eigenvalue of - [I, Ia Ez, wi(()w;(!))) Eni
Assume Equation
is larger than a term of order 82. As a consequence, Equation (36) yields
[ Bn,+lB 12 I (((e)))1++())) Pn,+ 1 Ii.....
2 dt Since the third phase ends if [1) - B,,+1e.,, becomes smaller than e3.. this yields:
T3. T2.+ 0 In( E3 =
(38)
Now recall that for any i,j E I+,
d(1 - (w,(t), wt)()) = (D)+ + +w (w.(),,((1))
dt
S 21)10(10) - (w.)))(()).())).))))))
Notice from Equation (34) and the previous discussion that (1)l2 0(1). As consequence, simple Grönwall
inequality with Equation (38) yields that for any t E [+2 T3.+1
(w.(t), (t)) 2 1 (1 (2,+) Wj (22+ exp((t (0(1)
A
-
2 0
exp
0
ln
E2
62
E3
2
The second inequality comes from the value of (1 (T,,+),), wy (T2,)))), thanks to Lemma F4. The last one comes from
choice of Eg for large enough oxp ((1)
In particular, this last inequality can be used to show that for any i.j E I4 and E T3. =w,())((*)=))))
In particular, this yields for any E I+ and E [T2.t:
8+(1= Σ (()
(39)
JEI+
= Σ
as(()²
(wi(t) + o(*+)))
(40)
JEI
Since |l,(1)l2 = 1 1- -(A3-e) this last equality actually yields the following comparison for t E [3,4+
(1) & at < (1+ of 311
(41)
jEI,
In particular, since (t)l22 O(1,, this yields ΣjEr, at) = O(1).
From there, for any FĄ S4 and E I4. (w:(t), (x) evolves follows during the third phase
((),(t), Tia (t), | (D+ (t), (t)) (t),
Fk
dt
(Bn, - (t)) En,+ |
0 (w:(t), ||
1 ] (8m (t),
Ia (BBntt - -3+ | ) o (W,())) ||
+ {(En++
|
2
Note that
Em1 Σ M-TK--
Bn, 33
RES,
For that, decompose WryyW usy with uez 1 w, and show that cr, 1 1- -(t--) and ||,, o(2--)).
31

---

Simplicity bias and optimization threshold in ReLU networks
This then yields, thanks to Equation (37)
d (w)((,),
2 18
1 (+),,,
24 La (Bn, (Bm+ (1+m,+-+(0)) (+)))) (EEnt |
2k
dt
|
1
(((()(())
| ) (42)
Σ KESI
7:20 11
(Wa(t),
From there, thanks to the third point of Assumption 4.1 and Equation (40):
(0 (+(,),
2 c
(mm)())
(A*
1-2c
(43)
|
Vd
|
Additionally, using the fact that ||, - 3_ (1) |T... decreasing over time and smaller than 11Bn,+11r., + 0(22-1e) at the beginning of the second phase,
((En, - Ia (pntt - (0)) ||= 2
Jk )
In,
Ia
1|3 - (t)l22
2 op
1
11 - (+)15)....1 -0(1
d+ log
[ Tallop V
2
Amin (Em,+)
n
0 d + log
1
|Σ -Iallop V Amin (En,t)
+ +O(A2-de]) (18u++1l2n+++
1
Amax (mn+))
1B, 100-111-0222000 -0(x21e +
0
+ log n
|Σ -alloop V
2
Amnnn Em,+)
1
Amax (Em+t)
15--Tallog 18
2
Amiu E En,+)
d + log n
0 22-15 + +-11 Σ Th l2 +
KESt
Now using Equation (37) and the fourth point of Assumption 4.1, note that
Amax (Em,+) Amin (En,+)
d d log
2 0
71
So that the previous inequality yields
12-15 022211 11
Ia ((En,t 2 (Bn,t - (t)),
2 Tellop 113" O ||aa|||
It
Σ marall +
+ log n
(44)
KES.
Finally, thanks to Equation (32), 1 Exes, nhZall s z' with probability at least 1 0 (m and (44) in Equation (42) finally yields for the third phase: (t), |
Using Equations (43)
E La 119 ①
x 1-2r +
d + log n
I6 0 (W)(())
dt
2vd
n
||
Thanks to the fourth point of Assumption 4.1, 2/a enough and n²* (dd log d) large enough so that the previous inequality becomes, with probability at least 0 ((
| Iall op 1pl || l
0, so that we choose X", +()) small
(!,,(!),
o
(wt(t).
Fk
|
simple Grönwall argument with the second point of Lemma F.4 then implies that for any 1 [r,, Ta,+), E Z, and
A E S4.
(t).).
Ω
32

---

Simplicity bias and optimization threshold in ReLU networks
Since the term Q vd
here does not depend dg, we can choose dg O
small enough that
(w.
2k
2d3.
We can show similarly for & E S_, so that point in Lemma F.5 holds, which concludes the proof.
F.6. Phase 4: final convergence
The last phase is defined for some 64 >> by the following stopping time, for any E (+,
T4.- = inf(t 2 -3,0 (1) (T 2 E4).
Lemma F... If Assumption 4.1 holds, for any (0, 1). there exist X ( E O(d n' O(d log d).
Og (1), 03 0 ((4)) E3 = (2)) and 8 = (E3) such that for any X S X", n n², o (, 62 [12-1s, [6
and E [A Ez], with probability 1 0 42 + 2 14.0 too and
1. neurons in Io are the same cone as of for any Z T3,0i
Vi E To, min (w:
| ) > 0 and
max (w:(t), kES-
F < O.
KES.
|
2. lo 8(t exists and 40 (t) Sn.o-
Proof. Similarly to the previous phases, we assume that +, that the random event I+ * 0, Equations (17), (18) and (37) and the statements of Lemma F.5 all hold.
Define in the following the positive loss L, for any 04 E RR(++1)xI+ by
2
1
Σ
a; (w;, rk) yk
2n
KES+
NEI+
Note that the autonomous system given by Equation (23) actually defines a gradient flow over I44, i.e., for 0+ = (a;, wi)kEI+
d
The main argument for this phase prove local (0+)
dt
ineguilllttii inequality:
V 11 2 (11) (04) Ln,+)
(45)
for any 0_ such that Σ a.w, - (Ts,)) S Ea.
IET
1
where
L
((Bn,t,2k) - -yx)² 2x)
2n
KES
Indeed, we can lower bound V (1+)lla for any such 04 as follows
ØL,
(v+)
VL (1+ +) N
Ow:
NEI
Σ 2,(1)
[D+(1)11
iEI
Aminn (Emst)
Σ 2112
118 - 11.
VEI,
33

---

Simplicity bias and optimization threshold in ReLU networks
where 3+ = LiEz, a, The last inequality comes from Equation (34). Note that for small enough choice of e3 0(1) and E4 - ( Liez, a.(1)² s2(1) in the considered set. Moreover, Equation (37) implies Amin (Mn,)) = S?(1,, so that
(VL+ 2 2(1)1l In,+11
(46)
On the other hand, a simple algebraic manipulation yields for any 04:
L+(0+) - Lin
1
Σ
(()) - un) ((n,+,:2)) - -11)²
2n
RES+
1
2v - [2k (3+ - -Bn+t + - ( Bm.t)
2n
RES+
(x,,+Bnn+ (3+ - Br,+) T En,+ (β+ - Bnit) + Xn,+ (X,, Bn,t - y): n
where Xn,t is the S+1 d matrix, whose rows are given by a& for k E St. By definition of the OLS estimator 8n,++
X x 3n, = 0, so that
-+ Ln, - Sn, 1B
(47)
Combining Equation (46) with Equation (47) finally yields the Polyak- Lojyyk--ojasiewiiz inequality given by Equation (45).
From there, this implies by chain rule for any & E (4,++
dL4 (1)) -1]VL+
dt
- (Q())((L Ln,t).
By Grönwall inequality, this implies for some v = ((1), for any & [T3,+: T4,+]
(+)+(()) I,,- S (1+ ( (0 (T3,+)) Ien+)e - u(t--rs)))
- 3 r(t (t-Taa+),
(48)
2
The last inequality comes from the fact that the end of the third phase, || (t) In,11E.. E3-
We bounded by below the norm of VL (0+ (s)), but it can also easily be bounded by above
(V
Σ at()) + (10, [0)(1)1 VEI+
$ 2Amax (Sm,t) Ea((0 VEI+
[13+(1) -m,+
S OL)
L.n+)
From there, the variation (t) can easily be bounded for any t [4,,41 as
(t) OTT 1)12 <
(0+())s
r3.1
/
pl -
s <0(1)es E3
< O(Ea).
(49)
Moreover, note that
- = Σ
(ai(!) - a. + Σ(mi(t) (aTTa.)))(((TT
It
I
In particular,
|1 (t) - (T( 5 Σ la.(t) a (TT
Σ (wa w1(78.+))l20(73.++
34

---

Simplicity bias and optimization threshold in ReLU networks
Σ (a.(t) - a,(Ts,+))²
(Ts,(Ts,+) +
l|wa(t) (s,(T,+)
Σ I
as(Ts,+))
I
- ((1)) d(r,++)ll2
- Ole3)-
can thus choose sg respect Eg such that the previous inequality ensures for any t [T8, T4,+]:
(1)) and 64 - (e$)) small enough such that Equation (46) still holds, but E4 large enough with
(1) (r(,++)) S
64
2
In particular, this implies that T1.+ = +o0. Since 9+ has finite variation (Equation 49), this also implies that lom (t) exists. The same holds for (t) by symmetric arguments, so that lim ((1) exists. Moreover, Equa- tions (47) and (48) imply that
lim (() Bn,+
This yields the second point of Lemma F.6.
now remains to prove the first point of Lemma F.6. Note that for any t 2 Ta,+ and E I,:
[Iw;(1) (T )l12 2 7..
S 0(33).
Thanks to the first point of Lemma F.5, choose 5g 0 (2) small enough so thatfor any / 2 r3, and i I+:
min (w,(t),
0 and
max (w:(t), kES
2k |
0,
kES,
||
which concludes the proof of Lemma F.6.
Proof of Theorem 4.1. We can conclude the proof of Theorem 4.1 by noticing that we can indeed choose E, E2, E3, 54 such that for any S X = ) and n 2 n* (d. log d), with probability 1-0 0 d' + 2 the statements of Lemmas F.2 and F.4 F6 all simultancously hold. In particular, the stopping times T4 and T- defined Lemma F.3 are infinite. Lemma F.3 then implies that for any T, 0(t) @t)). From then, Lemma F.6 implies Theorem 4.1.
35


Reviews:
[{"review_id": "HCbAz7hzyQ", "reviewer": "Reviewer_2Jhy", "summary": "This paper studies the simplicity bias in regression to a two-layer ReLU network using gradient flow. The author shows that despite overparameterization, the network may converge toward simpler solutions rather than merely interpolating the data, leading to a drastic improvement in test loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful feedback.\n\n> In lines 72 and 73, is there any reference to support this claim?\n\nYes, this phenomenon is extensively discussed by Raventos et al. (2024). In particular, Figure 4 and the discussion at the bottom of page 7 suggest that the model does not interpolate, even when trained indefinitely.\n\nFor diffusion models, Kadkhodaie et al. (2023) indicate that models are trained until convergence, yet interpolation ceases beyond a certain number of training samples. However, their experimental details do not specify the exact duration of training.\n\n> The connection to real-world large-scale models remains abstract\n\nAssessing the causes of non-convergence in real-world setups—beyond the analysis in Section A.5 for our toy model—would require extensive experiments that warrant a separate study. As reviewer TN6U noted, our primary contribution is theoretical, making such empirical investigations beyond our current scope.\n\nThat said, Raventos et al. (2024) provide strong empirical evidence supporting the relevance of our work to ICL. Their experiments suggest that after a certain number of task examples, the model stops reaching the ERM solution (modeled as a Nadaraya-Watson kernel with poor generalization) and instead converges to a spurious local minimum that generalizes well, aligning with the optimal estimator. Furthermore, their findings indicate that prolonged training does not improve performance, reinforcing our claim.\n\nA similar pattern is observed in Kadkhodaie et al. (2023) for diffusion models: the training loss remains low with few samples but increases beyond a certain threshold, while the test loss continues to improve.\n\n> The organization of the paper is somewhat unclear.\n\nIn response to the different reviewers’ feedback, we will make the following revisions regarding the clarity and organization of the paper:\n\n- we will include an informal statement of Theorem 4.1 at the end of the introduction\n- emphasis on the goal of Section 3 (see more below)\n- we will add an explanation of how Proposition 3.1 relates to and complements Theorem 3.1\n- we will summarize the key implications of Theorem 4.1 at the beginning of the discussion section\n- we will elaborate on our work’s connections to related literature, particularly regarding the double descent phenomenon. Additionally, we will compare the NTK/lazy training regime with the feature learning/mean-field regime.\n\nWe appreciate the reviewers' valuable suggestions and look forward to refining our work accordingly.\n\n> the mixed use of $\\mathbb{E}_ {X,y}$ and $\\mathbb{E}_{\\mu} $ is confusing.\n\nWe appreciate this feedback. The notation will be clarified (and unified) in the revised version. \n\n> Is Proposition 3.1 a consequence of Theorem 3.1?\n\nAs stated after Proposition 3.1, “Proposition 3.1 relies on the tail bound version of Theorem 3.1 and continuity arguments.” While Theorem 3.1 alone does not directly imply Proposition 3.1, its tail bound version plays a crucial role. This distinction is why we did not present Proposition 3.1 as a corollary. We will add further clarification in the revised version.\n\n> for what choices of $\\mu$ does the extremal vector in Equation (6) exist?\n\nExtremal vectors exist for any distribution $\\mu$ since they correspond to critical directions of the continuous function $G$ (defined on line 196). However, if $G$ is non-differentiable, the definition of extremal vectors should be adapted to account for subgradients. Equation (6) assumes differentiability, which holds for distributions $\\mu$ that are continuous with respect to the Lebesgue measure (or, more specifically, when the marginal of $x$ is continuous).\n\n> What is the relationship between Section 3 and Section 4? \n\nThe first step in the proof of Theorem 4.1 relies on the tail bound version of Theorem 3.1. Specifically, Section 4 requires a characterization of extremal vectors in the finite-data setting, which is facilitated by first studying their infinite-data counterparts (Equation (6)) and then applying the tail bound version of Theorem 3.1 to establish their proximity.\n\nWhile Proposition 3.1 is not used in Section 4, it supports the idea that early alignment behavior in the infinite-data case is not significantly different from the large but finite $n$ case. Theorem 3.1 bounds $|D_n-D|$, but additional assumptions are needed to directly conclude that the extremal vectors of $G_n$ are close to those of $G$. Proposition 3.1 addresses this gap.\nWe believe that Theorem 3.1 (and its tail bound version) and Proposition 3.1 are broadly applicable and may be useful in future work, as suggested by Reviewer TN6U. The paper is structured accordingly: Section 3 presents a general theoretical result that can be used beyond this work, while Section 4 applies it to a specific setting. We will clarify this connection in the revised version.", "claims_and_evidence": "I have a question: In lines 72 and 73, you claim that learning useful features stops before full interpolation in modern architectures (even with prolonged training). I assume this statement refers to diffusion models and large language models. Is there any reference to support this claim?", "methods_and_evaluation_criteria": "Yes, although two-layer ReLU neural networks and gradient flow are oversimplified models used to make the problem tractable, they are accepted by the machine learning theory community.", "theoretical_claims": "No, but the results seem reasonable to me.", "experimental_designs_or_analyses": "Yes, I checked the experiment details, and they make sense to me.", "relation_to_broader_scientific_literature": "This work contributes to the growing body of research on the implicit bias of overparameterized models, particularly in gradient-based optimization of neural networks. It builds on prior studies analyzing the simplicity bias in two-layer ReLU networks and extends these ideas by examining early training dynamics. However, clearer connections to empirical findings in large-scale deep learning, such as diffusion models and large language models, would strengthen its contribution.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. Theoretical contributions provide insights into the simplicity bias of gradient-based optimization in overparameterized models.\n2. The analysis of early training dynamics is well-motivated and connects to the broader study of generalization in deep learning.\n\nWeaknesses:\n1. The connection to real-world large-scale models, such as diffusion models and LLMs, remains somewhat abstract and could benefit from empirical validation or discussion of practical implications.\n2. The organization of the paper is somewhat unclear.", "other_comments_or_suggestions": "Please unify the notation in Theorem 3.1, as the mixed use of $E_{X,y}$ and $E_{\\mu}$ is confusing.", "questions_for_authors": "1. You showed that the empirical gradient approximates the expected gradient with an error rate of $O(\\sqrt{\\log n/n})$ in your theorem 3.1 in Theorem 3.1 and analyzed the critical direction/extremal vectors of the early training dynamics near some \"good\" expected gradient in Proposition 3.1. I am wondering about the relationship between these two results. Is Proposition 3.1 a consequence of Theorem 3.1?\n2. I am also curious about the existence of the extremal vector in Equation (6), as it depends only on the distribution $\\mu$. This suggests that the extremal vector exists only for sufficiently \"good\" $\\mu$. A natural question is: for what choices of $\\mu$ does the extremal vector in Equation (6) exist?\n3. What is the relationship between Section 3 and Section 4? I understand that Section 4 considers a special case of Section 3, where the data distribution $\\mu$ follows a linear model. Are the lemmas in Section 3 useful for proving the lemmas in Section 4? Additionally, what is the connection between the implications of Proposition 3.1 and Theorem 4.1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wdHZNuH8yv", "reviewer": "Reviewer_2wgP", "summary": "The paper theoretically studies how overparameterized networks converge to simpler generalizing solutions (as opposed to interpolating training data) when there are sufficiently many training samples. They do this by studying early alignment, where networks align their weights to the directions of the data early in training before changing in magnitude. They show that early alignment corresponds to the true loss as the number of training samples increases (reaching an optimization threshold), corresponding to a few key directions and thus yielding a simplicity bias which they show persists throughout training.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful feedback.\n\n> Atanasov et al. (2021) seems particularly relevant since they also study early alignment and how it relates to feature learning.\n\nThank you for pointing out this reference. We will include it in the revised version, as it provides valuable insights into the early alignment phenomenon.\n\n> It might be helpful to readers to have some high-level explanations of the theoretical parts as well as some visuals to help with the intuition behind the theory. There could also be more emphasis and discussion on the simplicity bias. I suggest bringing another one of the simulations from the supplementary into the main text for more experimental support of the theory. It could also be interesting to link this to double descent and “data-double descent” (Henighan et al., 2023) in a discussion. In general I suggest having a more extensive discussion relating the results of this work to existing literature.\n\nWe appreciate these suggestions. Our discussion section already connects our results to the literature on convergence to global minima, implicit/simplicity bias, and benign & tempered overfitting. We thank the reviewer for pointing out Henighan et al. (2023), which we will discuss in the revised version. Our setting does not exhibit a data double descent phenomenon, as the observed test loss consistently decreases with the number of samples, whereas double descent is characterized by an intermediate bump in the loss curve. However, the toy experiments in Henighan et al. (2023) illustrate a similar phenomenon: for a sufficiently large number of training points, the training loss remains high while the model learns optimal features. It remains unclear though whether this high training loss stems from an underparameterized regime (i.e., the model lacks sufficient capacity to memorize the data) or if optimization fails to reach the ERM in their setup. \n\nGiven the breadth of related topics, we initially omitted certain discussions (e.g., NTK and double descent) that we felt were less central to our study. However, based on the reviewer’s feedback, we will add a section in the appendix relating our work to these topics.\n\nAdditionally, we will move Figure 3 (unless another figure is preferred by the reviewers) from the appendix to the main text, utilizing the allowed extra page.\n\n> Other work studying early alignment has done so in the context of feature learning and the neural tangent kernel (Jacot et al. 2019). Does data interpolation correspond to the “lazy” learning regime, or would you consider this to be something separate? Can you discuss the connection of this work to feature learning?\n\nThere might be some confusion here. We distinguish between feature learning and the NTK/lazy regime, as they involve fundamentally different training dynamics (see Chizat et al., *On Lazy Training in Differentiable Programming* for an in-depth discussion). Our study specifically focuses on the feature learning regime with small initialization, as indicated by our initialization choice (Equation (2)), where both the inner and outer layers scale as $\\frac{1}{\\sqrt{m}}$.\n\nIn contrast, in the NTK/lazy regime (corresponding to large initialization scales), theory predicts that interpolation should occur at convergence, which is contrary to our main result. However, empirically demonstrating this interpolation in our toy model (with large $n$) is computationally challenging, as it would require an extremely large number of parameters. We will clarify this distinction and add a discussion on NTK vs. feature learning regimes in the appendix.\n\n> Do other (smooth) activations exhibit early alignment and do they learn generalizing solutions, or are they constrained to data interpolation?\n\nOur experiments in the appendix include results with the smooth GeLU activation, where we observe similar early alignment behavior. While theory predicts that with an infinite number of neurons, interpolation should eventually occur, it remains unclear how many neurons are required in practice. Our experiments suggest that this number is quite large.", "claims_and_evidence": "The claims made appear clear and well-supported.", "methods_and_evaluation_criteria": "The methods seem fine.", "theoretical_claims": "I did not check the correctness of any proofs.", "experimental_designs_or_analyses": "The experimental design seems well-done and there are additional experiments included in the supplementary to support the theory in the main text.", "relation_to_broader_scientific_literature": "The paper shows theoretically how the number of training samples impacts the early alignment phase, leading to a simplicity bias and show that this persists through training. This helps elucidate how the number of training samples, parameters, and data dimensionality relate to generalizing versus overfitting (data interpolating) solutions and the corresponding underlying mechanism.", "essential_references_not_discussed": "Atanasov et al. (2021) seems particularly relevant to this relevant since they also study early alignment and how it relates to feature learning.", "other_strengths_and_weaknesses": "Strengths: The paper is well-motivated, well-written, and studies an important topic in deep learning theory. The theory appears sound and the experiments compliment the theory well and support its generality. \n\nWeaknesses: The paper is quite dense and it’s difficult to follow the main points at times. The results and their applications could be emphasized more in a discussion that highlights their significance.", "other_comments_or_suggestions": "It might be helpful to readers to have some high-level explanations of the theoretical parts as well as some visuals to help with the intuition behind the theory. There could also be more emphasis and discussion on the simplicity bias.\nI suggest bringing another one of the simulations from the supplementary into the main text for more experimental support of the theory.\nIt could also be interesting to link this to double descent and “data-double descent” (Henighan et al., 2023) in a discussion.\nIn general I suggest having a more extensive discussion relating the results of this work to existing literature.", "questions_for_authors": "1.\tOther work studying early alignment has done so in the context of feature learning and the neural tangent kernel (Jacot et al. 2019). Does data interpolation correspond to the “lazy” learning regime, or would you consider this to be something separate? Can you discuss the connection of this work to feature learning?\n2.\tThe authors state “Loss of omnidirectionality is specific to the (leaky) ReLU activation and does not hold for smooth activations.” Do other (smooth) activations exhibit early alignment and do they learn generalizing solutions, or are they constrained to data interpolation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8bqY4NSr2j", "reviewer": "Reviewer_Z2XR", "summary": "- In the context of two-layer ReLU networks, the paper theoretically explores the issue where trained models got stuck in spurious local minima of the training loss as the number of training samples exceed a certain threshold. \n\n- It is demonstrated that networks might converge towards simpler solutions rather than interpolating training data, a phenomena so called directional alignment.\n\n- It is pointed out that this type of simplicity bias is indeed beneficial and enhances the generalization of trained models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful feedback.\n\n> the theorems and propositions could be rephrased in a more accessible and less mathematically rigorous manner to improve clarity \n\nWe we will enhance the clarity of our results in the revised version. Specifically, we will include an informal statement of Theorem 4.1 at the end of the introduction. However, we do not find it feasible to do the same for Section 3, as it requires prior knowledge of extremal vectors. More precisely, we will make the following revisions regarding the clarity and organization of the paper:\n\n- **Clarification of key results**: We will include an informal statement of Theorem 4.1 at the end of the introduction. \n- **Emphasis on the goal of Section 3**: This section aims to provide a simple characterization of extremal vectors (up to $\\sqrt{\\frac{d}{n}}$ terms) when the number of training samples is large. This serves two purposes: (1) forming the first step in the proof of Theorem 4.1 and (2) offering a general framework that may be useful in future research, as noted by Reviewer TN6U.\n- **Additional clarification of key results**: We will add an explanation of how Proposition 3.1 relates to and complements Theorem 3.1.\n- **Discussion of main implications**: We will briefly summarize the key implications of Theorem 4.1 at the beginning of the discussion section.\n- **Expanded discussion in the appendix**: We will elaborate on our work’s connections to related literature, particularly regarding the double descent phenomenon. Additionally, we will compare the NTK/lazy training regime with the feature learning/mean-field regime.\n\nWe appreciate the reviewers' valuable suggestions and look forward to refining our work accordingly", "claims_and_evidence": "The theoretical claims are supported by mathematical proofs.", "methods_and_evaluation_criteria": "An experiment with a toy model is illustrated to demonstrate the theoretical result.", "theoretical_claims": "I did not check the correctness of the proofs due to the time limit.", "experimental_designs_or_analyses": "The experiment on the training of an overparametrized two-layer neural networks show that the behavior of training and test loss are indeed consistent with the desired regimes of generalization.", "relation_to_broader_scientific_literature": "The paper advances the previous work on early alignment phenomenon by further indicating that this bias is beneficial to the generalization capability of the model.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "- The paper makes original contributions on top of (Boursier and Flammarion, 2024)'s work.\n- The paper seems technically sound to me, even though I did not carefully check the proofs of the theoretical claims.", "other_comments_or_suggestions": "Maybe the theorems and propositions could be rephrased to a more plain and less mathematically rigorous language for a higher clarity.", "questions_for_authors": "I don't have other questions for the authors", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejvO3ZcelM", "reviewer": "Reviewer_TN6U", "summary": "This is a theoretical paper seeking to explain the phenomenon that in some situations overparameterized models, once the number of noisy training samples is large, fail to interpolate the training data and instead converge to a solution that minimizes the test loss. This was observed with in-context learning and with diffusion models. In this paper, the architecture is overparameterized two-layer ReLU networks, and the focus is on training them from a small initialization by gradient flow on noisy data which is labelled by a linear teacher and which satisfies some further simplifying assumptions. The main result confirms the motivating phenomenon, and its proof provides further insights, pinpointing an early-phase alignment of neurons as the principal cause. Another contribution are concentration bounds for the extremal vectors that drive the alignment process, and the assumptions here are less restrictive than in the main result. The paper also reports and discusses numerical experiments in setups related to but extending the setting of the main theoretical result.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful feedback.\n\n> The gap between the theoretical setting in this work and the empirically observed phenomenon with in-context learning and diffusion modes is large, and it is not yet clear whether and how the properties of the training dynamics identified here are related to what actually happens in those practical situations. (E.g. it might be that case that the stability issues studied by Qiao et al. in NeurIPS 2024 are at play there to a greater extent than the neuron alignment in early training.)\n\nWe agree that it remains unclear whether empirical observations in more complex architectures are better explained by our theoretical results or those of Qiao et al. (2024). Evaluating this in real-world settings would require extensive experiments beyond the scope of our primarily theoretical contribution, as also noted by the reviewer.\n\nThat said, Section A.5 in the appendix suggests that in our toy model—while not necessarily representative of more complex architectures—non-convergence appears to be driven by early alignment rather than stability issues.\n\nWe view our work as an alternative perspective to that of Qiao et al. (2024) in explaining non-convergence. In practice, it is likely that both phenomena contribute to the observed behavior in real-world settings.\n\n> In the References, check whether papers cited as on arXiv have been published, and consider providing a clickable link for each item.\n\nWe will update the references accordingly.", "claims_and_evidence": "Full proofs of the theoretical results, which are the paper's main contributions, are provided in the appendix.  In addition, the key proofs are sketched in the main.  For the numerical experiments, sufficient details are provided so that it should not be difficult to reproduce them.", "methods_and_evaluation_criteria": "The experiments explore several settings that go beyond the assumptions for the theoretical results, and those include considering the GeLU activation and the Adam optimizer.  Although it could have been interesting to investigate real datasets, diffusion models, and in-context learning in order to link the theoretical results more firmly to practical situations, I think the level of experimental exploration that the authors chose to perform makes sense for this theoretical work which seems to be the first in this direction.", "theoretical_claims": "Yes, and I am convinced that they are correct.", "experimental_designs_or_analyses": "Yes, as far as possible.", "relation_to_broader_scientific_literature": "The paper builds on Boursier and Flammarion 2024, and on a number of recent works in the literature that investigate the training dynamics of one-hidden layer ReLU and other homogenous neural networks from a small initialization.  However, as far as I know, the link between neuron alignment in early training and generalisation at the expense of interpolation is for the first time established rigorously in this work.  Moreover, the theoretical results in this paper are not obvious and do not follow easily from the previous works.", "essential_references_not_discussed": "I am not aware of any.", "other_strengths_and_weaknesses": "The paper is well written and the main result is accompanied by a detailed discussion.  The theoretical results are non-trivial, and their proofs are provided in the appendix.  The concentration bounds result may indeed be useful for future work.  The experimental results are interesting, and their discussion is informative.\n\nThe gap between the theoretical setting in this work and the empirically observed phenomenon with in-context learning and diffusion modes is large, and it is not yet clear whether and how the properties of the training dynamics identified here are related to what actually happens in those practical situations. (E.g. it might be that case that the stability issues studied by Qiao et al. in NeurIPS 2024 are at play there to a greater extent than the neuron alignment in early training.)", "other_comments_or_suggestions": "In the References, check whether papers cited as on arXiv have been published, and consider providing a clickable link for each item.", "questions_for_authors": "I do not have any at this point.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper shows that overparameterized two-layer ReLU networks can avoid interpolating training data when the number of samples exceeds a certain threshold. Instead, due to an early alignment of neurons with specific directions, the network develops a simplicity bias that persists throughout training and leads to better generalization. This behavior is analyzed theoretically and supported by experiments.

The reviewers agree that the paper makes a significant theoretical contribution, noting that: 1) the insights into early training dynamics and simplicity bias are interesting and novel, 2) the theoretical results are rigorous, non-trivial, and supported by experiments, and 3) the paper is fairly well-written, with a clear motivation and potential for further impact. A few concerns were raised, particularly regarding the connection to large-scale modern deep learning models and the accessibility of the theoretical content. While most of these points were addressed in the rebuttal, the paper would benefit from including high-level explanations of the theoretical results, along with visualizations to help build intuition around the key concepts.

I therefore recommend acceptance and encourage the authors to consider the feedback and update the final version of the paper accordingly.
Decision: Accept (poster)