Title: Emergence and scaling laws in SGD learning of shallow neural networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: scaling laws

stochastic gradient descent

shallow neural network

multi-index model
Abstract: We study the complexity of online stochastic gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data: $f_*(\boldsymbol{x}) = \sum_{p=1}^P a_p\cdot \sigma(\langle\boldsymbol{x},\boldsymbol{v_p}^{\star}\rangle)$, $\boldsymbol{x} \sim \mathcal{N}(0,\boldsymbol{I_d})$, where the activation $\sigma:\mathbb{R}\to\mathbb{R}$ is an even function with information exponent $k>2$ (defined as the lowest degree in the Hermite expansion), $\lbrace\boldsymbol{v_p}^\star\rbrace_{p\in[P]}\subset \mathbb{R}^d$ are orthonormal signal directions, and the non-negative second-layer coefficients satisfy $\sum_{p} a_p^2=1$. We focus on the challenging extensive-width regime $P\gg 1$ and permit diverging condition number in the second-layer, covering as a special case the power-law scaling $a_p\asymp p^{-\beta}$ where $\beta\in\mathbb{R}_{\ge 0}$. We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective, and explicitly identify sharp transition times to recover each signal direction. In the power-law setting, we characterize scaling law exponents for the MSE loss with respect to the number of training samples and SGD steps, as well as the number of parameters in the student neural network. Our analysis entails that while the learning of individual teacher neurons exhibits abrupt transitions, the juxtaposition of $P\gg 1$ emergent learning curves at different timescales leads to a smooth scaling law in the cumulative objective.


Peer reviews:
comments: SUMMARY:
The authors study the scaling laws exponents for the MSE for learning 2 later neural networks with even activations of information exponent larger than 2 and power law weights in the second layer. The analysis starts by looking at the population loss and showing that the different hidden units decouple. The MSE is then the sum of the error on each of the hidden units. Having a large separation of scale on the weights in the second layer will imply an effective power-law scaling in the MSE.

STRENGTHS AND WEAKNESSES:
The paper is clear and well written. The separation between the learning of the different hidden units is well supported by the proof, and the final result is very intuitive.
I believe the biggest weaknesses of this work lie in the poor applicability of the results and in the limitation of the model under study. Common activations used in practice do not have high information exponent, and it's not clear to me that the same additivity property that the authors exploit will translate to arbitrary functions. I am unsure about the significance of this work. On one hand, there are not many works on power-law scalings for linear-width 2-layer networks, yet the activations choice is so restrictive that the problem is essentially reduced to a stack of generalised linear model, studied in a limit where they interact very weakly.

QUESTIONS:
1. Would there still be additivity in the tasks if you were to consider the gradient flow on the empirical loss?
2. Could you sketch what you expect to happen for lower information exponents?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a precise analysis of online stochastic gradient descent (SGD) for training two-layer neural networks on isotropic Gaussian data, in the challenging “extensive-width” regime $P\\gg1$. The target function is  \n\n$f^*(x) = \\sum_{p=1}^P a_p \\sigma ( \\langle v_p^\\*, x \\rangle ), x \\sim \\mathcal{N} (0,I_d)$,  \n\nwhere the activation $\\sigma$ is even with information exponent $k^\\*>2$, the signal directions $v_p^\\*$ are orthonormal, and the second-layer weights $a_p$ may follow a power law $a_p\\asymp p^{-\\beta}$. The authors identify **sharp transition times** at which each teacher neuron is recovered, and—by overlaying these abrupt emergences—derive **smooth neural scaling laws** for the cumulative mean-squared error (MSE) as a function of sample size, training steps, and width.\n\nKey results include:\n\n- **Complexity bounds** (Theorem 2.1) quantifying when SGD exits a long “search phase” and enters a rapid “descent phase” for each feature.  \n- **Scaling-law exponents** $\\alpha = \\frac{1-2\\beta}{1+\\beta}$ characterizing how the MSE decays as a power law in compute, data, or width under power-law weight tails.  \n- **Simulations** (Section 2.4) confirming the theoretical slopes for Hermite-activated networks and illustrating the compute-optimal frontier.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- **Rigorous theory connecting emergent dynamics to scaling laws.** Bridges sharp feature-recovery transitions with smooth empirical power laws.  \n- **Extensive-width regime treatment.** Handles $P\\gg1$ and allows diverging condition numbers, covering realistic over-parameterization scenarios.  \n- **Sharp, non-asymptotic bounds.** Provides concrete rates for plateaus and descents in terms of problem parameters $(d,\\beta,k^*,\\eta)$.  \n- **Empirical validation.** Figure 2 demonstrates close alignment between idealized theory and SGD simulations, including error bars and absolute loss curves.  \n\n## Weaknesses\n- **Gaussian and orthonormal assumptions.** All results hinge on $x\\sim\\mathcal{N}(0,I_d)$ and exactly orthonormal $v_p^*$, limiting applicability to more structured or correlated data.  \n- **Even activation with high Hermite order.** Requires $\\sigma$ be even with information exponent $k^*\\ge 3$, excluding common odd or non-polynomial activations (e.g.\\ ReLU, GeLU).  \n- **Learning-rate constraints.** Ensuring martingale fluctuations remain small demands $\\eta\\ll d^{-2}a_p$; such small rates may be impractical for moderate $d$.  \n- **Idealized two-layer setting.** Extensions to deeper architectures, non-additive targets, or realistic neural networks remain unaddressed.  \n- **Limited exploration of learning-rate schedules.** Corollary 2.3 requires knowing the total number of SGD steps in advance; adaptive or decaying schedules are not analyzed.\n\nQUESTIONS:\n## Questions for the Authors\n1. How might the scaling-law exponents change if $x$ comes from a non-Gaussian or anisotropic distribution? What barriers exist to extending the analysis beyond $\\mathcal{N}(0,I)$?  \n2. Can the techniques handle ReLU or GeLU activations, which are neither even nor finite-Hermite? Which parts of the proof would break?  \n3. The theory requires $\\eta$ to scale inversely with $d^2 a_p$ to control fluctuations. In practice, did you find reasonable rates that match theory, or is there a gap?  \n4. Could a simple decay schedule (e.g.\\ $\\eta_t\\propto t^{-\\gamma}$) recover the anytime scaling law without knowing the total number of steps in advance? What are the main technical hurdles?  \n5. Do you foresee obstacles to applying your emergent-plus-scaling-law framework to multi-layer or residual architectures? What conceptual changes would be required?  \n6. Figure 2 shows matching slopes between idealized and empirical loss curves for $\\sigma = h_4$. Have the authors tried other Hermite orders or common activations (e.g. ReLU)? How consistent is the predicted exponent $(1 - 2\\beta)/(1 + \\beta)$ in those cases?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper studies SGD in a very simple context and provides strong results characterizing scaling laws of SGD as a function of all the main things of interest (# training samples, amount of training, # of parameters).  A core analysis result is that scaling laws emerge from superposition of neurons with different timescales of learning dynamics.

STRENGTHS AND WEAKNESSES:
Strengths:
-- the motivation for knowing the forms of these scaling curves is very clear
-- the analysis presented is very complete and clear
-- the P>>1 regime is often not studied so this fills in an important gap in the literature
-- emergence result is very interesting. 

Weaknesses:
-- the setting is so simple it seems to me quite unlikely the methods will generalize to real-world situations.  so it's hard to know how applicable results will be to real situations.  
-- there is no real attempt to measure empirically if similar outcomes arise in more complex real-world situations, to which the theory can at least be compared

QUESTIONS:
-- The main thing I'd want to see is some kind of empirical investigation in a much more real-world situation, e.g. like Resnet-18 learning imagenet categorization, or a GPT-2 learning process.   Is there some way to compare the theory to the empirical results in those kinds of cases?  I'm not stuck on that particular evaluation, but I'm open to anything that would take these results out of such a simple highly restricted domain of assumptions.  

-- Is there some way to indicate how the scaling laws would change in other cases, like deep convnets or transformers?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the online SGD dynamic for two-layer neural networks, in the limit where the width $P\\gg 1$, data is generated for a Gaussian distribution, and labeled through another, fixed, teacher network, with orthogonal weights.\nIf the target function is hard enough (information exponent $>2$), the authors are able to prove the convergence and sharp transition of student neurons to teacher ones.\nMoreover, by assuming a scaling on the second layer of the teacher, the paper predicts a scaling law of the population loss minimum, compared with the \"total compute\" measured as the product of time steps with the width of the student.\n\nSTRENGTHS AND WEAKNESSES:\n- The main theorem provides a neat result about the separation of the times where the neurons are learnt, that is unprecedented in the literature, to my best knowledge. Being able to characterize this for $P\\to\\infty$ is very surprising and remarkable.\n- The formal math prediction of the scaling law is matched by numerical experiments, in Figure 2. It would be nice to append the same plot for different values of $\\beta$ though.\n- The setting is a bit artificial for two reasons: the assumption $IE>2$ is strong enough to exclude any reasonable or commonly used activation function, and the scaling laws emerge only when the teacher is crafted ad hoc to have power-law decaying weights in the second layer.\n\nQUESTIONS:\n- Can the mean field limit be connected to this result in some weight? I know the classical _mean field regime_ \\[1,2\\] is usually done for finite $d$, but there are some works that consider the limit $P\\to+\\infty$ before going $d\\to+\\infty$, such as \\[3\\]. I guess here you are considering the limit in the opposite order, but maybe there is still a connection.\n- You say that \"online SGD exhibits a long plateau\" (line 131) for $IE>2$. Isn't the plateau present also for $IE=2$? Does the separations of timescales appear also in this case? I expect not, so there is no scaling law for $IE=2$, right?\n###### Rereference\n> \\[1\\] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences\n\n>\\[2\\] Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-\nparameterized models using optimal transport.\n\n> \\[3\\] Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro. From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}