Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
Authors: Leonardo Defilippis
Yizhou Xu
Julius Girardin
Vittorio Erba
Emanuele Troiani
Lenka Zdeborová
Bruno Loureiro
Florent Krzakala
Venue: ICLR 2026 Oral (2026)
Keywords: Scaling laws; Neural networks; LASSO and matrix compressed sensing; Random matrix theory; Approximate message passing; High dimensional Statistics
Primary area: learning theory
TL;DR: We derive a phase diagram of scaling laws for diagonal and quadratic neural networks via a bridge to LASSO and matrix compressed sensing, predicting both generalization and the emergence of power-law weight spectra.

Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with matrix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish a precise link between these regimes and the spectral properties of the trained network weights, which we characterize in detail. As a consequence, we provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance, yielding an interpretation from first principles.

Published as a conference paper at ICLR 2026
SCALING LAWS AND SPECTRA OF SHALLOW NEURAL NETWORKS IN THE FEATURE LEARNING REGIME
Leonardo Defilippis" 1, Yizhou X Julius Girardin', Emanuele Troiani?, Vittorio Erba², Lenka Zdehorová", Bruno Loureiro', and Florent Krzakala
Departement d' I'Informatique, École Normale Supérieure, PSL & CNRS
"Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL. Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
ABSTRACT
Neural scaling laws underlie many of the recent advances in deep learning, ye their theoretical understanding remains largely confined to linear models. In this work, we present systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with ma- trix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish precise link between these regimes and the spectral properties of the trained network weights, which characterize in detail. As a consequence, we provide a theoretical validation of recent empiri- cal observations connecting the emergence of power-law tails in the weight spec- trum with network generalization performance, yielding an interpretation from first principles.
INTRODUCTION
A central development in modern deep learning has been the recognition that neural network gen- eralization does not improve unboundedly when training data, model size, or compute are scaled in isolation. Instead, extensive empirical evidence reveals the presence of performance bottlenecks un- less these resources are increased together (Kaplan al., 2020; Brown et al., 2020; Hoffmann al., 2022). Characterizing these trade-offs,, and in particular predicting the resulting neural scaling laws, has emerged as a fundamental challenge for deep learning research, with significant implications for the design of efficient and resource-conscienn models.
Our goal in this work is to investigate this question in the context of shallow neural networks. More precisely, consider the following supervised empirical risk minimization (ERM) problem for the class of two-layer neural networks fa:; W a) = a (WWx bb:
min W.a
(np f(xpp W, a2)a + 2 + +aa))
(1)
where W E Rxxd and E RP are the first- and second-layer weights, respectively. Although substantial progress has been achieved recent years, our current understanding of scaling laws for the generalization performance of the ERM minimizer in eq. (1) remains largely confined to the random features regime (Bahri et al., 2024; Maloney et al., 2022; Paquette et al., 2024; Atanasov al., 2024; Bordelon et al., 2024; Kunstner & Bach, 2025). In this setting, the problem reduces to a kernel method, where scaling behavior has been classically studied, and is known as source and capacity conditions (Caponnetto & De Vito, 2007: Cui et al., 2021: Defilippis al., 2024).
In this work, we move beyond the random features regime and investigate neural scaling laws for the ERM minimizer in eq. (1) in the teacher-student setting. That is, assume that the target task
These authors contributed equally this work

---

Published as a conference paper at ICLR 2026
is generated by teacher network of the same architecture
= fa W a²) VA
where ا-= Lid. NO Ia) denotes the dataset and {6 iid (o 1) an additive Gaus- sian label noise with variance A 2 O. The statistics of the target weights W a" will be specified later. Our main goal in this paper to characterize the scaling-law and bottleneck behaviors of the excess risk
R(W. B [(( a") f(x; W. aa)
(3)
associated with the minimizers W, á of eq. (I). We will consider two specific classes of shallow neural networks. Thanks to exact mappings to classical problems in signal processing, these models admit mathematical characterization, enabling end-to-end analysis of the optimization problem in the feature learning regime.
Diagonal networks and LASSO. The first architecture diagonal neural network with p d, diagonal first-layer weights W = diag( w), linear activation and no bias (b 0):
f(: W a) a (w ④ x)
(4)
vd
While the expressivity of this architecture is the same as that of linear model, the reparameter- ization creates an effective implicit regularization that allows for feature selection and has made this setting popular among theoreticians. Indeed, adapting an argument by Neyshabur et al. (2015); Soudry et al. 22; Pesme & Flammarion (2023) (see Appendix A), the resulting empirical mini- mization problem is equivalent to the LASSO problem with parameters 0 a,w;/v and objective
i = arg min BCRd 2
1
Σ (un - 0 xx))²² A||0111.
(5)
1=1
In other words, the ERM problem for diagonal two-layer linear network trained with & weight- decay can be understood through the performance of LASSO.
Quadratic neural network and matrix compressed sensing. The second architecture is that of an over- parameterized two-layer network with (centered) quadratic activation,
2
P
w
x
w
12
Ia
f(;; W, a)
Tr
S
(6)
Vd
Vd
where W W pad E Rddd and the normalization is taken for convenience. In this case, we fix the second-layer weight a of the model to be an all-one vector, but the target network may have arbitrary second layer weights. This class of quadratic neural networks have recently gained in popularity as simple models for non-convex tasks (Sarao Mannelli et al., 2020: Arnaboldi et al., 2023; Martin al., 2024; Ben Arous et al., 2025). The ERM problem in eq. (1) for this architecture can be mapped to sparse estimation setting (Gunasekar et al., 2017: Maillard & Kunisky, 2024; Bandeira & Maillard, 2025; Xu al., 2025; Erba et al., 2025), namely matrix compressed sensing (or low-rank matrix estimation):
$ = arg min Sz0
Σ - Tr(SZ))) + A||S
R. T and ||. |l. denotes the nuclear norm. We refer again to Appendix A for the explicit mapping. Thus, the performance of a quadratic network trained with weight decay can be analyzed via low-rank matrix estimation with nuclear norm regularization.
where Zμ,
These two equivalences underline the central theme of this work: by mapping neural network train- ing problems to sparse vector and matrix estimation tasks, we can leverage the rich thcoretical tool- box developed for LASSO and compressed sensing, and in particular approximate message passing and its high-dimensional state evolution (Donoho et al., 2009; 2013; Javanmard & Montanari, 2013; Berthier et al., 2020b; Erba et al., 2025). This bridge not only enables precise predictions for gen- eralization error and scaling exponents, but also provides a principled understanding the weight spectral distribution in neural networks.
2

---

Published as a conference paper at ICLR 2026
Power-law/quasi-spasee targets. To study scaling behavior, adopt the classical assumption of a target with a power-law spectrum. as considered for instance in (Caponnetto & De Vito, 2007; Steinwart et al., 2009; Spigler et al., 2020; Cui et al., 2021: Bordelon et al., 2024; Ben Arous et al., 2025). In the language of compressed sensing, this corresponds to the notion of quasi-sparsity (Negahban & Wainwright, 2011; Raskutti al., 2011), 1), where the signal is not exactly sparse but its coefficients decay according to a heavy-tailed distribution. This makes the setting natural and relevant to both the machine learning and signal processing communities. Concretely, the case of diagonal linear networks we assume effective weights
0 i.d. ((0 di
0- = W'
(8)
while for quadratic neural networks we assume that
S = V Σ oz wo
(9)
is rotationally invariant with eigenvalues (vdi This setting was recently studied as well in Ben Arous et al. (2025) (who considered noiseless target (A 0) and obtain one of our scaling exponents). In both cases we fix > 1/2 to ensure square-summability of 0* and S
1.1 MAIN RESULTS
1. Phase diagram and complete characterization of excess risk rates for power-law targets. We provide a sharp characterization of the excess risk achieved by empirical risk minimization (I) for both diagonal linear networks and quadratic networks in the regime n, d >>>>> 1 with p d. under a power-law design for the target function and varying regularization strength X, summarized in Figure 1. Our results uncover a striking universality between the settings, including transition from benign to harmful overfitting. Exploring the extent of this universality beyond the setting here an interesting avenue for future work. We also derive the risk rates under optimal regularization X, and show that detimally-reuulrizzer ERM achieves the Bayes-optime rates- previously known only for the diagonal case (Raskutti et al., 2011). These findings are of independent interest for sparse vector and low-rank matrix estimation.
2. Spectral behavior the learned weights. We characterize, across all phases, the spectral prop- erties of the trained network weights. The learned spectrum reflects the implicit trade-off between signal, noise, and regularization, and exhibits phenomena directly connected to feature learning. Remarkably, the resulting spectral behavior mirrors observations in modern deep learning practice (Martin & Mahoney, 2021a; Thamm et al., 2024).
3. First-principles explanation of spectra-generalizattion connection. We provide a clear inter- pretation of the spectrum and its relation to generalization. Building on Result 3, which decomposes the error into underfitting, overfitting, and approximation terms, we show that each of these com- ponents is directly connected to the spectral statistics of the weights. In doing so, we provide a mathematical theory for the empirical observations of Martin al. (2021) and Wang al. (2023) for the spectral statistics of weights in large-scale trained networks.
4. Non-asymptotic validity of state evolution. Our derivations rely on approximate message pass- ing (AMP) and its state evolution equations, which are rigorously valid only in the proportional asymptotic regime with fixed ratios n/d (or n/d") and fixed A. We extend these equations heuristi- cally beyond their proven setting, covering arbitrary scalings of d, X. Through extensive numeri- cal experiments, we demonstrate that state evolution remains accurate down to constants across the whole parameter space, including far beyond proven guarantees. This surprising robustness, already established in ridge regression (Cui et al., 2021; Cheng & Montanari, 2024; Misiakiewicz & Saeed, 2024; Defilippis et al., 2024), suggests broader conjecture: the AMP framework, and related tools from spin glass theory, may provide predictive power well outside their standard asymptotic as- sumptions. We hope this will further motivate work on non-asymptotic control of AMP (Rush & Venkataramanan, 2018; Miolane & Montanari, 2021; Li & Wei, 2022; Reeves, 2025).
Together, these results provide comprehensive theoretical and empirical understanding of scaling laws for feature learning in simple network models.
1.2 FURTHER RELEVANT WORK
Scaling laws- - A large body of work has studied scaling laws in the lazy regime, where the fea- tures remain fixed. This includes kernel methods (Caponnetto & De Vito, 2007; Spigler al., 2020;
3

---

Published as a conference paper at ICLR 2026
Cui et al., 2021), random features (Defilippis et al., 2024; Atanasov et al., 2024; Bahri et al., 2024; Maloney et al., 2022; Paquette et al., 2024; Kunstner & Bach, 2025), and neural tangent kernels (NTK) (Bordelon et al., 2020). Bordelon et al. (2024; 2025) analyzed how scaling laws change for linear networks when both weights are trained, and Worschech & Rosenow (2024) explicitly solves the dynamics of linear network obtain the scalings. Our goal in this work is to go beyond linear networks and the lazy regime and analyze scaling laws in the presence of genuine feature learn- ing. Two recent works (Ren et al.. 2025; Ben Arous al., 2025) analyze settings related to ours, but with important differences. Both consider two-layer networks with sublinear width, orthogonal first-layer weights, and power-law decaying second-layer weights. Ren et al. (2025) study activation functions with large information exponents, which is orthogonal to our setting, while Ben Arous et al. (2025) focus on quadratic activations with a specific SGD dynamics. Both works, additionally, consider noiseless targets and unregularized training (A = A 0). Here, we study empirical risk minimization with weight decay À > 0 and noise A 2 O.
Spectral properties of learned weights
A growing literature investigates the distribution of weight spectra in trained neural networks, with particular attention to the emergence of heavy-tailed eigenvalue distributions in both weights and activations (Mahoney & Martin, 2019; Martin et al., 2021: Martin & Mahoney, 2021b;a; Thamm et al., 2024; Wang et al., 2023; Zhou et al., 2023; Hodgkinson et al., 2025). Despite these empirical observations, a precise theoretical characterization of the learned spectra and their relation to generalization remains elusive. Recent progress includes analyses of the spectrum after a single or few gradient steps (Dandi al., 2024; Moniri et al., 2023; Cui et al., 2024; Dandi et al., 2025; Kothapalli et al., 2025), well as results showing convergence of SGD in mean-field models to spectral distributions reminiscent of those we obtain (Olsen et al., 2025). Our description of the spectrum of the trained weight provides analytic characterization of this phenomenon, and provides an interpretation of these properties from first principles.
AMP and State Evolution - Our analysis relies on approximate message passing (AMP) and its state evolution (SE), which has become a central tool for studying high-dimensional inference problems with structure (Donoho et al., 2013; Bayati & Montanari, 2011; Javanmard & Montanari, 2013; Berthier et al., 2020b; Zou & Yang, 2022; Feng et al., 2022; Gerbelot & Berthier, 2023; Dudeja et al., 2023; Erba et al., 2025). It has also been applied to learning problems beyond sparse recovery, such as kernel methods and learning rates (Cui et al., 2021; Loureiro et al., 2021). In this work, we use the state evolution equations of AMP heuristically, to analyze quasi-sparse models beyond their rigorously proven asymptotic regimes (typically assuming fixed ratio n/d). While recent advances in non-asympiin con-asympo control (Rush & Venkataramanan, 2018; Miolane & Montanari, 2021; Li & Wei, 2022; Reeves, 2025) provide reassurance, finer control of the limit is still required for a fully rigorous justification. Our experiments nevertheless show excellent agreement between SE predictions and numerical results across regimes, suggesting that AMP may be predictive well beyond its standard assumptions.
Compressed sensing - Quasi-sparse settings, where coefficients decay with a power law in Fourier or wavelet bases, have long been studied in statistics and signal processing. This is natural since most real-world signals are not exactly sparse but have heavy-tailed coefficient distributions (Mallat, 1999). Classical work on LASSO and matrix compressed sensing analyzed lp co-cootrlle targets, deriving minimax bounds on error and sample complexity (Raskutti et al., 2011; Negahban & Wainwright, 201 Our results extend this line of work by providing the full phase diagram across all regularization strengths and data scales. For instance, the optimal LASSO rate of Raskutti et al. (2011) arises from setting À = Ō / n/dd) (here is up to logarithmic factors).
2
MAIN RESULTS
2.1 UNIVERSAL ERROR RATES
In this section we discuss the excess risk rates associated to the two problems introduced above. Our analysis is based on a deterministic characterization of the risk R(W, a) Rn.d at large n, d > 1, which is discussed in Section 2.4. We only consider the noisy case (A > 0), and refer Appendix for the noiseless case. In order to highlight the correspondence between the two neural network models, we express the results in terms of the effective sample size neff as follows:
n
for diagonal network
Theff =
n/d
for quadratic network
(10)
4

---

Published as a conference paper at ICLR 2026
Surprisingly, this definition will be enough present both cases in unified manner.
Result 1 (Excess risk rates), Under the setting of Sec. for A > 0 and Reff > 1, the excess risk satisfies
θ
1+11(227
+ (nett/ /d)
if 1 < Beff < d
and
X & d
θ
(x-2/3)
if Heft ~ d and X <<1
(d)
O (d/neett)
if Reff >> d and
X 4 Baft
(11)
θ (2a2-12 /noott) 2-1/7)
if max (V "att 1
Heff
+ (2=d2) /n(ttt
if
Th EX
and Raand = (1) otherwise, where p(1) = - log(t) in the diagonal network case and p(t) = 225 in the quadratic network case. Notice that in both cases the p term is monotone increasing with Dett/ / d, and dominates the error rate when Net - d. Additionally, in the diagonal network case, the first rate holds up to logarithmic
that we specify in eq. (86) in Appendix C.2.
These rates are summarized in Figure 1. For small (fixed) regularization < 1/Vd, with d fixed and Reff increasing, the excess error moves from an initial plateau (Phase Ia), driven by data scarcity, to fast- (Phase IV), where Ra = @ + +1((2))), ) matching the minimax rate in (Raskutti et al., 2011 I; Donoho et al., 2011) As Reff approaches d, the estimator begins to fit the noise, and we observe a harmful overfitting (Phase V), in which the excess risk is dominated by the non-universal scale (arising from overfitting the noise as in Result 3). This transition, characteristic of the under- regularized and under-sampled regime (1 < Nell < d, X
Txx ) happens at
(log d) d
24-1 for diagonal network
Reff
cross
(12)
for quadratic network
The excess risk reaches its maximum around Bef d with R d ~ X-2/3, This non-monotonicity of the risk at interpolation is reminiscent of the double descent behavior (Belkin et al., 2019; Mei & Montanari, 2022), and extends previous findings (Bartlett et al., 2020; Wang et al., 2024) non- linear models. For Neftf > d, the excess risk then enters second fast-decay Phase Vla, with rate proportional to d//eet: this the fastest decay we observe, provided neft >> d?r (Phase Vlb). For larger regularization strength À > 1/Vd, the excess risk decay is described by the upper part of the phase diagram in Figure 1, eventually crossing to the lower part when Neatf dx=.. In particular, if A >> d/2,/22 we observe that increasing Neff. the excess risk is initially in plateau (Phase Ib), induced by the strong regularization with respect to the sample size. For n ~ \2 it crosses into a which transitions to a faster rate (Phase III), still influenced by the large regularization, for neff ~ >/2 with R Rm = ((Ad/naat) )²). Phase recovers the rate in (Negahban & Wainwright, 201 1, Corollary 2). The excess risk eventually transitions to the fast-decay Phase VIb for nef 2d, where the effect of the regularization becomes negligible due to the large sample size. These cross-overs are reminiscent of the ones observed for kernel and random feature ridge regression respectively in Cui et al. (2021); Defilippis et al. (2024).
slow rate Phase II with Rn ()d /
We observe that there are region boundaries along which the rates are discontinuous (red lines in Figure 1). At the boundary Neff (dd) we observe the aforementioned crossover between harmful overfitting and fast decay, with an interpolation peak emerging. At the boundary d < Teff < der 1+1/(27) (which is much lower) when and X = @(Vnatt n/tt// /d), the excess risk jumps from d/ nee to noff increasing the regularization.
As corollary of the above results, we can immediately estimate the behavior of the optimal regu- larization Aopt and the associated optimal ERM rates.
Corollary (Optimal regularization and optimality of ERM). The optimal regularization satisfies
SOly Teft/ dd), if A 0 and (1 < Reef < n or (nett, d) (ô(%/nat / (d), if A and n cross & neft < d2n
>> d
Teeft
(13)
where Ô is up to logarithm factors in the argument. The excess risk rates in the optimally regularized
case matches the large Neft, d > 1 Bayesian risk (BO(D) = E(R(W, (|)(D) ~ RBO (for the
Reg.a
See section 2.4 for formal statement.
5

---

Published as a conference paper at ICLR 2026
À Teatt h /2
À = Dall /dd //27
lb
11
I1I
la. lb: R
A
II: R
Add /neff
2-1/7
Regularization
A hatl
MI: R 22 8 /
VIb
Vla
p
IV: R x eif
1+1/(27)
IV
la
V: R x p( /d)
Vla, Vlb: R d/nett
O(d-)
(1)
Rydff
Cross
(dd)
(dd))
Effective number samples net (n for diagonal, n//d for quadratic)
Figure 1: Excess risk rates of Result function of and X(n, d), with sketch of the corre- sponding spectral properties of the learned weights (Result 2). Red lines represent discontinuous phase boundaries.
diagonal network, up to logarithmic factors), given by
(( 1+1/(27)) O (d/neet)
if A and 1 Neltf dey if A>0 and
(14)
Teff > de
and Ruan d (1) otherwise. Again, in the diagonal network case, in the first regime the rate holds up to logarithmic factors that we specify in eq. (86) in Appendix C.2.
Corollary 1 shows that by appropriately tuning the regularization allows avoid the harmful overfitting phase in the noisy case and reach Bayesian optimality. Interestingly, the noisy rate O( -1+1((2)²)) in the regime < Reff < d coincides with the classical minimax rate for high-dimensional linear regression over an b--bll with q 1/7 (Raskutti et al., 2011; Donoho et al., 2011). Corollary 1 not only recovers the well-known result that properly regularized LASSO achieves this minimax rate, but also extends it to additional regimes and to the matrix case, revealing cross-over between the minimax rate and faster (dd/ned) rate.
2.2 SPECTRA OF THE LEARNED WEIGHTS
Our second set of results concerns the structural properties of the learned weights. that are given by soft thresholding function applied to a noisy version of the target's weights. Notice that for diagonal neural networks, the weights 0 can be seen as diagonal matrix (modulo sign), hence they coincide with the eigenvalues of W.
Result 2 (Spectrum of the learned weights). For the diagonal network case, there exists constants 5(n, d, X) and e(n, d, A) (specified in Appendix C.1.2) such that the empirical risk estimator (1) satisfies (in distribution)
0, aa(0; + 82zz))
(15)
where 2; (0,11) 1), and (z(z;a)= = 0) -)- 0) is the soft-thresholding function. For the quadratic network case, there exists constants ö(n, d, X) and e(n, d, X) (that are obtained from (20)) such that the spectrum of the empirical risk estimator (7) satisfies
v(x) Fys (Ac)ōo(z) - #á(x + Ac)
(16)
8o represents a Dirac mass at 1 is the indicator function of the set and μs represents the spectrum of S* + 8Z with its cumulative function Fus. where Z GOE(d) (i.e. symmetric matrix with N (0, 1/d) elements up to symmetry).
Result 2 characterizes the learned weights in both settings: they are noisy, soft-thresholde versions of the target spectrum. The parameter quantifies the noise from the label noise and finite sample estimation of the target weights, while de sets the cutoff below which singular values vanish due to regularization. For any n, d, X, the spectrum consists of a spike at zero, possibly a bulk near zero, and few outliers aligned with the top eigenvectors of the target.

---

Published as a conference paper at ICLR 2026
lb
11
III
Eigenvalue density
10
Simulations
10P
Th: Bulk
A == va
Th: Spikes
10-1
10-2
la
IV
Vla
Elgenvalue density
10a
10y
= == 1/d
10-1
10-2
2
Eigenvalue
Eigenvalue
Eigenvalue
Figure 2: Comparison between spectra from simulations and theory across different training phases. Blue: eigenvalue histograms after training. Purple/orange: theoretical predictions for bulk and spikes, respectively (16) (for clarity, spike histograms
shown separately). Notice that theory also predicts a spike at zero, which we do not plot for visual clarity. All panels use d d= 800 except III, for which have d 400. Bottom row: X L/d with 100, 1.94 10*, 1.28 10 Top row: x= d with n 800, 6.4 109, 2 x 107. We discuss the phenomenology in Section 2.3.
2.3 INTERPRETABILITY, AND A "UNIVERSAL." ERROR DECOMPOSITION
The spectra depends n. d, only through the functions 6, e, leading qualitative structure shared by both models. Our theory predicts eight distinct spectral phases (Figure 1) which are closely connected the risk rates in Result I, Focusing now the quadratic network, the result provides an interpretation of the risk in terms of the weights spectrum.
Result 3 ("Universal" error decomposition of feature learning). Let {s: S* be the eigenvalues of St in non-increasing order. Consider the following two cases.
(i): Under-regue render-reularizattion Assume that the constants 8(n, d, X) and c(n, d, A) in Result 2 satisfy Ac < 28 and there exists cutoff (8) < satisfying (K(6) 8. Then the excess risk reads
Rn.d 82 822 ano (dx) (x
)= (K))) (8)(28 xt)
K(8)
5
(+)++
apernnitennos arcer for jer foatanes
desoneed
where use (dx) = (2TT)
-El-2 dx denotes the Wigner semi-circle law.
(ii): Overlregulai Assume that the constants 8(n, d, A) and e(n, d, A) in Result 2 satisfy
Xe 2 28 and there exists cutoff K. K(8, (8)) & satisfying (K(,))))
8² de O. Then
(x(0,)e)
K(8,)c) Σ
82
d )
R
Σ
(18)
K(
i=1
undtefitting (not learned (eatures)
approximation evor for learned features
Equations (17) and (18) have clear interpretations, The first component of the overfitting term sponds to the second moment of the bulk spectrum, representing the power of the learned noise and thus quantifying the degree of overfitting. The second part of the overfitting term is proportional to the square of the bulk size, but always remains subdominant compared to the first term. As shown in Equation (18), the overfitting term diminishes with increasing regularization strength.
The underfitting term measures the mean power of the unlearned spikes, indicating how many fea- tures are lost due the cutoff K Xe) (which depends the noise and regularization). The

---

Published as a conference paper at ICLR 2026
approximation error term reflects the average enror in the learned spikes, which depends the ef- fective signal-to- to-soise ratio %t and the effective regularization λc. Notably, when the effective noise 3
5 is zero, the approximation error increases with regularization; conversely, when the regularization is zero, the approximation error increases with noise A In general, however, the approximation error arises from a non-trivial interplay between the effective noise and regularization.
s.
This decomposition is "universal" in that it does not depend the target spectrum, dataset size, or regularization, holds for all A 2 and applies across different spectral phases. While derived here for quadratic networks, similar expressions hold as well for diagonal networks. See Appendix C.1.2. Extending it to broader architectures is an interesting direction for future work.
Based on the error decomposition in Result 3, give an interpretation of the rates in Result 1 in terms of the weight spectral properties (see Figure I for illustrations and Figure 2 for experiments); the bulk corresponds learned noise, the spikes hidden by the bulk are the unleamnd features, and the outliers are the learned features. This provides mathematical theory from first principles for the observations of (Martin & Mahoney, 2021a; Martin et al., 2021), whose terminology (e.g. "bleed- out", "rank collapse".. .... we borrow in the following, We begin by the case of large regularization, considering an increasing number of samples.
lb (Rank collapse); All eigenvalues are zero. Data scarcity and strong regularization imply that the ERM estimator is zero. Result 3 then gives trivial risk (rr(S"))\/d
II (Outliers): The spectrum contains approximately Nout (x//// 1/7 outliers, while the
remaining eigenvalues are zero. Spikes are shifted by "I In this regime, some features learned with noise, while others are lost due to over-regularization. Result implies that the risk
Add
O @ Eiz (Vdi (1)))))))))
determined by the number and shift of the spikes, yielding
@ (Add9/2n)22- /n)///>),. since the error from the shift is of the same order.
111 (Heavy-tail); The spectrum is a perturbed version of the target spectrum with a heavy-tail p(x) x 1-7,/, Regularization shifts the bulk leftward by = 4 yielding R = θ ((Ade/ /n)
Ad
As more eigenvalues emerge from the spike at zero, more features are learned and the risk decreases. Strong regularization suppresses any spurious bulk of small eigenvalues, as well as some of the smaller spikes. Consider now the case of small regularization.
Ia (Rank collapse): The spectrum resembles a small portion of semi-cirele law along with many zero eigenvalues. Perhaps surprisingly. the contribution of the bulk is negligible even for vanishing regularization. Neither features nor noise are learned, so the risk remains R = Tr((S*))1/d
IV (Bulk Outliers): The spectrum exhibits Nout (Ad/4n) 11/2 outliers and a bulk with cigenvalues of order θ ((A7d (AA=d//n)//a))) /n)1/3a). As in Phase II, the risk decreases more spikes emerge from the bulk. Since the bulk contribution is sub-leading, Result implies that the risk scales as
the average power of the unlearned spikes: R Eiz (Vdi- 7-- θ ((d/n)'=*)
(Bulk Outliers): Similar to Phase IV, but the risk is now dominated by the bulk of eigenvalues of order + ((A& /n)) The ERM estimator approaches interpolation and begins to fit noise.
Although the number of outliers (Nou = Ad/4n 4n) continues to increase and the bulk range shrinks, the bulk's 's second moment grows. Altogether, Result 3 implics the risk increases.
Interpolation peak (Bulk + Outliers): The spectrum is dominated by a large semi-circle bulk of order 2 1/33). There may still be
spikes if X > d33/2, but their contribution is negligible. Even if the model learns features, the noise is overwhelming, and Result 3 implies the risk is dominated by the bulk second moment R = (
outliers and a bulk of small eigenvalues of order / /Ad2/4n The smallest outliers merge at the bulk boundary, creating a bleed-out effect. The risk decreases as more outliers emerge. The spikes are perturbed by ((vAd /n), and Result implies the risk scales R ( /n), since this dominates over the unlearned features.
VIa (Bulk Bleed-out + Outliers): The spectrum contains (Ad/4n) - -122
VIb (Heavy-tail): As in Phase III, the spectrum is perturbed version of the target (a heavy-tailes bulk) with perturbations O(AA/2 The risk decays with the perturbation R ( /n).
8

---

Published as a conference paper at ICLR 2026
Therefore, for the under-regularized case, as the number of samples is increased, the bulk keeps shrinking as the spikes pop out. However, as the shape of the bulk and the number of zero eigen- values changes, the risk changes non-monotonically. In other words, the model learns an increasing number of features, but the influence of noise leads to a non-monotonic behavior in the risk. Fur- thermore, Result shows that regularization only affects the first term of eq. (17) and may increase the last two terms. Thus the optimal regularization strategy is to truncate the bulk, setting the first term zero, leaving the second unchanged and minimally increasing the third. If the bulk contribu- tion is negligible, weaker regularization may be chosen (i.e. in phases IV and Vlb). This reasoning explains Corollary
Finally, we should note that although phases Vla and VIb exhibit similar risk decay rates, only VIb achieves the Bayes-optimal rate. In the regime & Nett < d, the optimal performance is reached in phase II with À / neet/ /d. The corresponding spectral density shows transition from outier-dommnated (zero eigenvalues+spikes)) to heavy-tailed behavior, which supports the argument of Martin et al. (2021) that heavy-tailed spectra are associated with superior generalization.
Furthermore, it can be observed that the regularization effect essentially involves setting smaller cigenvalues to zero, which can be achieved through a pruning approach. The following result indi- cates that such a pruning method can attain optimal error rates without the need to manually adjust the regularization strength. This pruning method is widely used in practice (Yang et al., 2020; Yao et al., 2024; Kim & Yoo, 2025) and here we obtain its optimality.
Corollary 2 (Optimality of pruning). Consider the problem eq. (7) with A > O" and À < V
We obtain our praned estimation $ by setting the eigenvalues of S to be X - ReLUU( (25 - (28- xe)) and denote the asymptotic excess risk after pruning to be R, where ö(n, d, X), e(n, d, A) are the same as Result 2. Then we huve R = @(RBO) except around the interpolation peak n = (dP))
Corollary 2 further suggests that overfitting is not an inherent barrier to generalization. The model has already learned all possible features (given a fixed dataset), and the key challenge lies in distin- guishing these features from the spurious noise acquired during training.
2.4 NON-ASYMPTOTIC STATE EVOLUTION
The results of Sections 2.1, 2.2 and 2.3 build on the theory of state evolution and approximate mes- sage passing algorithms (Donoho et al., 2009; Javanmard Montanari, 2013; Gerbelot & Berthier, 2023), whose formal guarantees hold in the high- dinensiinnsii limit Reff d + with fixed ratio nett/ /d and constant strength A. In this regime, state evolution allows to characterize the asymptotic risk and the spectrum of the weights in both the neural networks models under consideration. both for the empirical risk minimizer and for the Bayes-optimal estimator (see Appendices C and D).
Non-rigorous analyses in the ridge regression literature have employed asymptotic formulas to estimate excess risk rates under source & capacity conditions, recovering classical results while also identifying new regimes in striking agreement with finite-size numerical experiments (Bordelon et al., 2020; Cui et al., 2021; Simon et al., 2023). The validity of these formulas beyond proportional asymptotics was subsequently established through non-asymptotic multiplicative bounds, thereby placing these rates on rigorous grounds (Cheng & Montanari, 2024; Misiakiewicz & Saced, 2024; Defilippis et al., 2024). Motivated by this line of work, we derive our results under an analogous assumption: namely. that the state evolution equations for LASSO and matrix compressed sensing remain valid beyond proportional asymptotics. This assumption numerical evidence, depicted in Figure 3 and Appendices C, F, and by rigorous results on the con- vergence rates for the LASSO state evolution (Miolane Montanari, 2021)'. Figure also confirms the theoretical decay rates of the excess risk across all phases, with state evolution and simulations in excellent agreement (see Appendix F for details on the implementation). Nonetheless, estab- lishing non-asymptotie multiplicative bounds for LASSO and matrix compressed sensing remains a challenging open problem. Our results provide both motivation and supporting evidence for this direction, which we leave for future work. For conciseness, we only present the conjecture regarding quadratic networks, and refer to Appendix C for the conjecture concerning diagonal networks.
supported both by extensive
Conjecture 1. Let 0, A 2 0 and consider n, >>>>> 1 sufficiently large. Then with probability at least 1 - o (1) - oa(1) both the excess risk associated to the empirical risk minimizer eq. (7)
When A = 0 there is no overfitting and thus no need for pruning.
Howeeerr we need to extend their results finite sample analysis (Rush & Venkataramanan, 2018)

---

Published as a conference paper at ICLR 2026
A= = 1/d
==1
A= vd
Quadratic network norn. Excoss risk
100
-2+
-2+1
1 +
D 100
10-2
D 200
D 400
D
10-4
Diagonal network
norm. Excess risk
-2+
10a
-2+ !
10-2-
10
10a
10a
10
10
10
10
10
10
Dat
Figure 3: Excess risk in simulations (dots, d = 100, 200, 400, 800) versus non-asymptotic state evolution (solid lines) as a function of Reff (Neff = n for the diagonal case, Reff = n/d for the quadratic case) with X = 1/d, vd and = 0.5. The risk is rescaled such that it starts at 1. We find excellent agreement, despite state evolution being rigorous only in the asymptotic limit Pett /ett/d = ((1) with * 1. Black lines indicate the decay rates of the excess risk predicted by Result 1, again showing good agreement. The networks are trained in PyTorch using LBFGS as detailed in Appendix F.
and the Bayes-optimal risk (i.e. either RBO or R(W, a)) satisfy R - Rn,d = Rn.d a(1). More precisely, for the Bayes-optimal case we have RBO Q* with Q' = Tr[(S*) [3] and q given by the fixed point of the following equation
4n/d
Di 4
q = A+ 9)
1 1 1 +
(1)r7(z)'dz
(19)
2 3
where M1/ denotes the spectrum of S*
+ 1z Z with Z ~ GOE(d). For the ERM, Rn,d
74
2n 82
A where 6 is given by the fixed point of the following equation
2
2
4 2 ((,
b(, = 1000 Ma(x)) dz) (20)
Q' + 52
=1 -1- (11-xc8))](5xx),
62
with pa denoting the spectrum
5* S* aZ with Z ~ GOE(d).
3 CONCLUSION
We studied a theoretical framework for scaling laws in shallow networks with feature learning by mapping them sparse vector and low-rank matrix estimation. This allowed to derive a com- prehensive phase diagram for the excess risk scaling laws, uncovering a universality between di- agonal and quadratic networks. Our analysis provides first-principles explanation of the weight spectra- generalization connection: underfitting, overfitting, and approximation errors correspond directly to distinct spectral features, yielding firm foundation for empirical observations of heavy- tailed weight spectra and their link to generalization.
There are many natural extensions of this work, such as exploring additional structures present in the data (e.g., non-trivial covariances (Wortsman & Loureiro, 2025)), extending beyond two- layer networks and quadratic activations (Barbier et al., 2025), providing a rigorous proof of the state evolution conjecture following Miolane & Montanari (2021). Moreover, our current work only analyzes the global minimum, so we should also look at compute scaling laws of GD/SGD (Ben Arous et al., 2025) well as the implicit biases of SGD towards heavy tails and its relation to generalization (Gurbuzbalaban et al., 2021; Simsekli et al., 2020; Hodgkinson et al., 2022). We hope these results will motivate further progress toward systematic theory of neural scaling laws.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDD
We would like to thank Murat Erdogdu, Surya Ganguli, Antoine Maillard, Pierre Mergny and Denny Wu for insightful discussions. BL and LD were supported by the French government, managed by the National Research Agency (ANR), under the France 2030 program with the project refer- (PR[AT]RIE-PSAL and "ANR- 25-CE23-560
(MAPLE). as well as the Choose France CNRS AI Rising Talents program. We also acknowledge funding from the Swiss National Science Foundation grants SNSF SMArtNet (grant number 21 212049), OperaGOST (grant number 200021 200390) and DSGIANGO (grant number 225837). This work supported by the Simons Collaboration on the Physics Learning and Neural Computation via the Simons Foundation grant (# 1257412 (FK) and # #1257413 (LZ))..
ences
REFERENCES
Luca Arnaboldi, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD. arXiv preprint arXiv:2305. 18502, 2023.
Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592 2024.
Benjamin Aubin, Florent Krzakala, Yue Lu, and Lenka Zdeborová. Generalization error in high- dimensional perceptrons: Approaching bayes error with convex optimization, Advances Neural Information Processing Systems, 33:12199-12210, 2020.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27))e2311878112 2024.
Afonso S Bandeira and Antoine Maillard. Exact threshold for approximate ellipsoid fitting of ran- dom points. Electronic Journal of Probability,300 30:1-46, 2025.
Jean Barbier, Francesco Camilli, Minh-Toan Nguyen, Mauro Pastore, and Rudy Skerk. Statistical physics of deep learning: Optimal learning of multi-layer perceptron near interpolation. arXiv preprint arXiv:2510.22 24616, 2025.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117445):00633-3077)) 2020.
Mohsen Bayati and Andrea Montanari. The dynamics of message passing on dense graphs, with applications compressed sensing. IEEE Transactions on Information Theory, 57(2):764-785, 2011.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biai-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-1558 2019.
Gérard Ben Arous, Murat A Erdogdu, N Mert Vural, and Denny Wu. Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws. arXiv preprint arXiv:2508.03688 2025.
Raphaël Berthier, Francis Bach, and Pierre Gaillard. Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. Advances in Neural Information Processing Systems, 33:2576-2586, 2020a.
Raphael Berthier, Andrea Montanari, and Phan-Minh Nguyen. State evolution for approximate message passing with non-separable functions. Information and Inference: A Journal of the IMA, (1):333-79 2020b.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp. 1024-1034. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2026
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. Proceedings of the 41st International Conference on Machine Learning. 2024. arXiv preprint arXiv:2402.01002.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning improve neural scaling laws. Journal of Statistical Mechanics: Theory and Experiment, 208508):084002, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners Advances in neural information processing systems, 33:1877-1901, 2020 2020.
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematicx, 7(3):331-368, 2007.
Chen Cheng and Andrea Montanari. Dimension free ridge regression. The Annals of Statistics, 52 (6):2879-29)) 2912,, 2024.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 14:10111-1014, 2021.
Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue Lu, Lenka Zdeborová, and Bruno Loureiro. Asymptoties of feature learning in two-layer networks after one gradient-step. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 9662-9695 PMLR, 21-27 Jul 2024.
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. Journal of Machine Learning Research, 25(349): 1-65, 2024.
Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue Lu, and Bruno Loureiro. random matrix theory perspective on the spectrum of learned features and asymptotic generalization ca- pabilities. In Yingzhen Li. Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan (eds.), Proceed- ings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pp. 0224-2233 PMLR, 03-05 May 2025.
Leonardo Defilippis, Bruno Loureiro, and Theodor Misiakiewicz. Dimension-fres deterministic equivalents and scaling laws for random feature regression. Advances in Neural Information Processing Systems, 37:104630-104699, 2024.
David Donoho, lain Johnstone, Arian Maleki, and Andrea Montanari. Compressed sensing over ba-balls: Minimax mean square error. In 2011 IEEE International Symposium on Information Theory Proceedings, pp. 129- 133. IEEE, 2011.
David Donoho, Arian Maleki, and Andrea Montanari. Message-passing algorithms for com- pressed sensing, Proceedings of the National Academy of Sciences, 189445) 2009.
David I. Donoho, Matan Gavish, and Andrea Montanari. The phase transition matrix recovery from Gaussian measurements matches the minimax mse matrix denoising. Proceedings of the National Academy of Sciences, (10021):8405-841) 2013.
Rishabh Dudeja, Yue M. Lu, and Subhabrata Sen. Universality of approximate message passing with semirandom matrices. The Annals Probability, 51(5):1616-16833 2023.
Vittorio Erba, Emanuele Troiani, Lenka Zdeborová, and Florent Krzakala. The nuclear route: Sharp asymptotics of ERM in overparameterized quadratic networks. NeurIPS 2025, 2025. arXiv preprint Xiv:25055.1795.
12

---

Published as a conference paper at ICLR 2026
Oliver Y. Feng, Ramji Venkataramanan. Cynthia Rush, and Richard J. Samworth. A unifying tutorial on approximate message passing. Foundations and Trends@ in Machine Learning, 15(4):335- 536. 2022. ISSN 1935-8237. doi: 10.1561/2200000092. URL http://dx: dox. 1561 /220//000000222
Cédric Gerbelot and Raphaël Berthier. Graph-based approximate message passing iterations. Infor- mation and Inference: A Journal of the IMA, 12(4))2562-2628, 2023.
Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Sre- bro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017.
Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference Machine Learning, pp. 3964-3975. PMLR, 2021.
Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference Machine Learming, pp. 8774-8795. PMLR, 2022.
Liam Hodgkinson, Zhichao Wang, and Michael W Mahoney, Models of heavy-tailed mechanistic universality. ICML 2025, 2025. arXiv preprint arXiv:2506.03470.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya. Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in neural infor- mation processing systems, 55:30016-300333 2022.
Jiaoyang Huang. Mesoscopic perturbations of large random matrices. Random Matrices: Theory and Applications, 7(02):1850004 180004,, 2018.
Adel Javanmard and Andrea Montanari. State evolution for general approximate message passing algorithms. with applications to spatial coupling. Information and Inference: A Journal of the IMA, 2(2): 144, 2013.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Hyeonjin Kim and Jaejun Yoo. Singular value scaling: Efficient generative model compression via pruned weights refinement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17859-17867, 2025.
Vignesh Kothapalli, Tianyu Pang. Shenyang Deng, Zongmin Liu, and Yaoqing Yang. From spikes to heavy tails: Unveiling the spectral evolution of neural networks. Transactions on Machine Learning Research, 2025.
Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under Zipf's law. arXiv preprint arXiv:2505.19277 2025.
Gen Li and Yuting Wei. non-asymptotic framework for approximate message passing in spiked models. arXiv preprint arXiv:2208.03313. 2022.
Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Mare Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher- student model. Advances in Neural Information Processing Systems, 34:18137-18151, 2021.
Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning, pp. 4284 4293. PMLR, 2019.
Antoine Maillard and Dmitriy Kunisky. Fitting an ellipsoid to random points: predictions using the replica method. IEEE Transactions Information Theory, 70(10):7273-7294 2024.
13

---

Published as a conference paper at ICLR 2026
Antoine Maillard, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising. Journal of Statisti- cal Mechanics: Theory and Experiment, 2022(8)008330,, 2022.
Antoine Maillard, Emanuele Troiani, Simon Martin, Florent Krzakala, and Lenka Zdeborová. Bayes-optimal learning of an extensive-width neural network from quadratically many samples. Advances in Neural Information Processing Systems, 37:82085-821335 2024.
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Alexander Maloney, Daniel Roberts, and James Sully. solvable model of neural scaling laws. arXiv preprint arXiv:2210. 2022.
Charles H Martin and Michael W Mahoney. Implicit self-rege regulargulaiittoonn in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, (2(165):1-11 2021a.
Charles H Martin and Michael W Mahoney. Post-mortem on deep learning contest: a simp- son's paradox and the complementary roles of scale metrics versus shape metrics. arXiv preprint 0tt1:21066007344, 2021b.
Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state- of-the-art neural networks without access to training testing data. Nature Communications, 12 (1):4122, 2021.
Simon Martin, Francis Bach, and Giulio Biroli. On the impact of overparameterizzttoon on the training of shallow neural network in high dimensions. In International Conference Artificial Intelligence and Statistics, pp. 3655-3663. PMLR, 2024.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4))667-766, 2022.
Léo Miolane and Andrea Montanari. The distribution of the lasso. The Annals of Statistics, 49(4): 2313-2335, 2021.
Theodor Misiakiewicz and Basil Saced. A non asymptymptoiic theory of kernel ridge regression: deter- ministic equivalents, test error, and GCV estimator. arXiv preprint arXiv:2403.08933, 2024.
Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. ICML 2024, 2023. arXiv preprint arXiv:2310.07891.
Sahand Negahban and Martin J. Wainwright. Estimation of (near) low-rank matrices with noise and high-dimenn -igh-dimessionll scaling. The Annals of Statistics, 39(2):1069-1000 1097, 2011. ISSN 00905364, 21688966.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on learning theory, pp. 1376-1401. PMLR, 2015.
Brian Richard Olsen, Sam Fatehmanesh, Frank Xiao, Adarsh Kumarappan, and Anirudh Gajula. From SGD to spectra: A theory of neural network weight dynamics. arXiv preprint urXiv:2507.12709 2025.
Elliot Paquette, Courtney Paquette, Lechao
Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute- optimal neural scaling laws. Advances in Neural Information Processing Systems, 37:16459- 16537, 2024.
Scott Pesme and Nicolas Flammarion. Saddle-to-saddie dynamics in diagonal linear networks. Ad- vances in Neural Information Processing Systems, 36:7475-7505, 2023.
Sundeep Rangan. Generalized approximate message passing for estimation with random linear mixing. In 201] IEEE International Symposium on Information Theory Proceedings, pp. 2168- 2172, 2011. doi: 10.1109/ISTT:2011.000 .6033942.
14

---

Published as a conference paper at ICLR 2026
Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Minimax rates of estimation for high- dimensional linear regression over Eg ba-ba.s.. IEEE transactions on information theory, 57(10): 2011.
Galen Reeves. Dimension-free free bounds for generalized first-order methods via Gaussian coupling. arXiv preprint arXiv:2508. 10782, 2025.
Yunwei Ren, Eshaan Nichani, Denny Wu, and Jason D Lec. Emergence and scaling laws in SGD learning shallow neural networks. arXiv preprint PrXXi:2504.19955 2025.
Cynthia Rush and Ramji Venkataramanan. Finite sample analysis of approximate message passing algorithms. IEEE Transactions on Information Theory, 64(11):7244-7286, 2018.
Stefano Sarao Mannelli, Eric Vanden-Eijnden, and Lenka Zdeborová. Optimization and gener- alization of shallow neural networks with quadratic activation functions. Advances in Neural Information Processing Systems, 33:13445 33:13445-13455, 2020.
J.W. Silverstein and Z.D. Bai. On the empirical distribution of eigenvalues of class of large di- mensional random matrices. Journal of Multivariate Analysis, (262)):75-112, 1995. ISSN 0047- 259X. doi: https://doi.orge 10. 1006/jmva.. 1995.1051.
James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R Weeeese. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. Transactions Machine Learning Research, 2023.
Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization neural networks. Advances in Neural Information Processing Systems, 33:5138 11:5138-1135 315111,,111111 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent separable data, Journal of Machine Learning Research, 19(70): 1-57, 2018
Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher stadder-student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):
2020.
Ingo Steinwart, Don R Hush, Clint Scovel, et al. Optimal rates for regularized least squares regres- sion. In COLT, pp. 93-93, 2009.
Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix theory analysis of neural net- work weight matrices. In digh-diimesiooal Learning Dynamics 2024: The Emergence of Struc- ture and Reasoning, 2024.
Yutong Wang, Rishi Sonthalia, and Wei Hu. Near-interpolator:: Rapid norn growth and the trade- off between interpolation and generalization. In International Conference on Artificial Intelli- gence and Statistics, pp. 4483 4491. PMLR, 2024.
Zhichao Wang, Andrew Engel, Anand D Sarwate, Ioana Dumitriu, and Tony Chiang. Spectral evo- lution and invariance anear-width neural networks. Advances neural information processing systems, 36:20695-20786 2023.
Roman Worschech and Bernd Rosenow. Analyzing neural scaling laws in two-layer networks with power-law data spectra. arXiv preprint arXiv:2410.09000 2024.
Arie Wortsman and Bruno Loureiro. Kernel ridge regression under plwer-law data: spectrum and generalization. arXiv preprint wrXiv:2510.04780. 2025.
Yizhou Xu, Antoine Maillard, Lenka Zdeborová, and Florent Krzakala. Fundamental limits of matrix sensing: Exact asymptotics, universality, and applications. COLT 2025, 2025. arXiv preprint mixx:22033.11122
15

---

Published as a conference paper at ICLR 2026
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and sin- gular value sparsification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 678-679, 2020.
Xinhao Yao, Xiaolin Hu, Shenzhi Yang, and Yong Liu. Enhancing in-context learning performance with just svd-based weight pruning: A theoretical perspective. Advances in Neural Information Processing Systems, 37:38820-38851, 2024.
Yefan Zhou, Tianyu Pang. Keqin Liu, Michael W Mahoney, Yaoqing Yang, et al. Temperature bal- ancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 01363542-63572 2023.
Qiuyun Zou and Hongwen Yang. concise tutorial approximate message passing. arXiv preprint arXiv:2201.07487, 2022.
A THE BRIDGE FROM SPARSE ESTIMATION TO NEURAL NETWORKS
A.1 EQUIVALENCE BETWEEN DIAGONAL NETWORKS WITH & WEIGHT DECAY AND LASSO
The first equivalence was discussed in number of papers (Neyshabur et al., 2015; Soudry et al., 2018; Pesme & Flammarion, 2023). We consider the diagonal two-layer network with parameters w E IRd and effective predictor
0 == u W,
f(x)
0,0i
1=1
trained with squared loss and & weight decay on both layers:
1 Elly
min M.wERd
+ (1ul1 + (|=1())
(21)
Alternatively, may also consider diagonal two-layer ReLU network with two branches per coordinate:
f(x) = Σ ui (wwz)) + u, o (-wwwz))) 0(2 = mmxxf2, 2,, 0}.
Using (2) - o (-2) = 2, each pair of branches along coordinate induces an effective linear
coefficient 0, such that
f(x) = 1
1=1
We know show the reduction of this problem to the LASSO one:
Step 1. Lower bound via AM-GM. For each coordinate we have
u + w
2 2/m 2|0,|
(by AM-GM, with a = u²; b= w
Summing over i gives
|| + ||lm| z 2|(01
Therefore, for any factorization with u O 0.
lly - x011 + 2 [1|211 2 + (
2 1l - x0123 + A|0011.
(22)
16

---

Published as a conference paper at ICLR 2026
Step 2. Tightness.
For any 0, choose factorization
u, sign(B; 10.12/2
w; = 10,11/2.
Then of wp- 04), so that
" + w = =, new; = 02.
Hence equality holds in (22), and the regularizer becomes
2 マ (1111 1l 2|111|2-
Plugging back into (21), obtain the exact equivalence
min
lly - x (1ul + 11=)) = min
|1
10 +
A|10||
BRRa
2
which is precisely the LASSO loss.
A.2 EQUIVALENCE BETWEEN QUADRATIC NETWORKS WITH E WEIGHT DECAY AND MATRIX COMPRESSED SENSING
Again the equivalance has been discussed in a number of work (Gunasekar et al., 2017; Maillard & Kunisky, 2024; Erba et al., 2025; Bandeira & Maillard, 2025)
We consider the two-layer quadratic network with centered activations
1 1 VP
[ [ww] (w]xx²² x=)² [ m)²]
fa;; W)
Centering is equivalent to learning (and absorbing) the constant offset via a bias term, and can also be naturally implemented in practice by batch/layer normalization applied after squaring.
This network can be written as
f(x) Tr[SZ]. WWT ž 0 and Z. Σ., Σg Ex Thus the network corresponds where = /p exactly to a PSD matrix sensing model with centered measurements Z. Centering removes only constant offset, which in practice would be absorbed by bias term or handled automatically by batch/layer normalization. Moreover, weight decay on W induces a trace penalty on S, since
|W vātr(S).
so that training equivalent to trace-regularized PSD) matrix sensing.
Following the universality results for matrix sensing (see, e.g., Maillard & Kunisky (2024); Bandeira Maillard (2025); Maillard et al. (2024); Xu et al. (2025); Erba et al. (2025)), the analysis can be simplified by replacing the empirical sensing operators Z by i.idd. Gaussian symmetric matrices with matching covariance structure. In particular, for I; ~ No Ia), the centered measurements distributed rank-one Wishart fluctuations, which are asymptotically equivalent, in the sense of state evolution and AMP analysis, to Gaussian measurements with the same variance. Hence, without loss of generality, we may study the trace-regularizee PSD matrix sensing problem with Gaussian measurement operators
yu = Tr[SG,] + Eus
G₂ ~ GOE(d).
B NOISELESS SETTING: ERROR RATES AND SPECTRAL PROPERTIES
In the noiseless case A have the following rates. Result 4 (Noiseless rates). Under the setting of Section for A = and ne# >> 1, the excess risk satisfies
1-27 θ oof
if Neft < d and
X <
21/2
4 (122
if neff > d and
X <
Waaf
(23)
T141/2
+ ((xc2/2) //oo//ott))
) 1/11 if max T
Batt
E
Belf
d74172
2472
17

---

Published as a conference paper at ICLR 2026
À = Rell ha/////2
A = nortdd //2t
la
11
111
la.lb: x 1
Regularization
II: (Ad- /noit
2-1/7
IV
À
1/0
MII: X2 n
lb
IV: e > Wall 1-29
((1)
(d)
Effective number samples nef (n for diagonal, n/d for quadratic)
Figure 4:
Excess risk rates of result 4 for the noiseless task (A 0)) the corresponding spectral
properties of neural networks.
and (() otherwise. In the diagonal network case, the first rate holds up to logarithmic factors that we specify in eq. (92).
Note that as in the least-squares case, the noiseless rates are faster (Aubin et al., 2020; Berthier al., 2020a; Cui et al., 2021). Perhaps more surprising, they also universal, with no difference between the quadratic and diagonal networks. Moreover, the error rate Q 1-27) Ben Arous al. (2025) as in their setting there no regularization.
eff
agrees with
Corollary (Optimal regularization and optimality of ERM, noiseless setting). The optimal regu- larization satisfies
(m 7+ d d-1/2), 0+
A 0 and
1 < Netf & d Teeff
oppt
(24)
A 0 and
d
where θ is up to factors in the argument. The excess risk rates in the optimally regularized case match Bayesian risks, given by
Rust d (Nopt) = (RRBO) (ost = - (n 1-21)
if
0 0 and 1 1 Dett < A - 0 and Nefff >> d
(25)
if
and Rigird O(1) otherwise. Again, in the diagonal network case, the first and third regimes the rate hold up to faaritttmicfacttoo that we specify in eq. (92).
The phase diagram for the noiseless case is simpler, and exhibits just a subset of the phases. Specif- ically, phases II are the same as the noisy cases. For the other two phases, we have
III (Heavy-tail): The spectrum is perturbed and shifted version of the target's spectrum. The perturbation is of order OAdd /n3/2), and the shift is of about Ad 'n Thus the risk is mainly caused by the shift, which gives an error of @(X²d (n²)). When À 0, the spectrum is the same as the target's, s, so the risk is zero.
IV (Bulk+Outliers): The bulk is of size (dh 100-3//00///1///3// /n/5)/5), and there are approximately z outliers. As n increases, the bulk shrinks, all spikes come out of the bulk by one, and the risk keeps decreasing. The contribution of the bulk is always negligible, so Result 3 implies that the risk is proportional the total power of the unlearned spikes: R (((d/n)21-1) Note that this is also information theoretically
1 Eiznna(Vdii-a -2)2 optimal, because with n data points, we could learn at most uncorrelated directions.
The Result 4 confirmed by Figure 5.
18

---

Published as a conference paper at ICLR 2026
A. 1/d
vd
Quadratie relawork
109
1-27
-2
-2+
100
Excese
10-4
= 200
- 400
10-8
= 800
27
24
Dingonal netwark
10a
Exeeess
10
10-x
10"
10
10'
10"
10'
10"
102
10
Ret
Der
Petf
Figure 5: Excess risk in simulations versus non-asymptotic state evolution (solid lines) as a function of Netf The setting the same as Figure except that A 0. Black lines indicate the decay rates of the excess risk predicted by Result 4, again showing good agreement.
C DERIVATION DETAILS DIAGONAL LINEAR NETWORK
Considering the reparametrization defined in Section and detailed in Appendix A, mapping empir- ical risk minimization with L2 penalty on a two-layer diagonal lincar network LASSO regression, in this section we study the supervised learning problem
( 00) Vdl
+ 211011 }
θ arg min OERd
Up
(26)
with By ~ N(, Ia), and
( 0*) + VAC..
S, 1),
1,
(27)
Vd
0¹ N(O,A),
Ai = dijAi,
i= d
(28)
We also define the parameter Q' = d Tr A
d=000 ((2>). Without loss of generality we choose
A A2 ... Aa- The excess risk is defined as
1 El 0 x )²)
R(O)
(29)
In sections C. and C.3 we derive the state evolution equations 48) for the excess risk of the ERM estimator eq. (26) and (107) for the Bayes-optimal estimator in the high-dimensional limit d co with n and À fixed. Then, in sections C.4 and C.2, assuming the excess risk equations holds for arbitrary scaling between dimensions and regularization, we derive the Results in Section 2.
C.1 GENERALIZED APPROXIMATE MESSAGE PASSING AND STATE EVOLUTION
Our theory is built on the analysis of Generalized Approximate Message Passing (GAMP) algo- rithms, tailored for Bayes-optimal estimation and (convex) empirical risk minimization. In this section provide an overview of the derivation of the expressions for R and the LASSO R in our setting from the GAMP framework.
Consider the matrix X E Raxd, with i.i.d. Gaussian components Xij NO 1), the vectors b' E Rdd wf E R", and the functions (known as denoisers) Ja Rdd -> RRd and R Ra, with t > 1. The generic form of GAMP (Donoho et al., 2009; Rangan, 2011) is given by
wh = X (bb) Ug 1 (w) b++1 X (w(w") ) (b').
(30)
(31)
19

---

Published as a conference paper at ICLR 2026
The terms ae and are known in the statistical physics literature Onsager terms, and they defined as
1 Ut d
0 2 ab;
Gt
ge(w):
(b)
(32)
dwl
For separable denoiser functions" one track statistics of the iterated vectors b', w', leveraging well-known results from Bayati & Montanari (2011) 1); Javanmard & Montanari (2013), through the so called state evolution.
C.I.1 GAMP FOR CONVEX OPTIMIZATION
Consider the problem of minimizing the empirical risk with loss E(y, z) convex in the second argu- ment and convex penalty r(0),
arg min BER
fly" 0 x²') + Σ r(0)):
(33)
is possible to design a GAMP algorithm whose fixed points are solutions to the problem defined in (33). A detailed discussion this approach can be found in Feng et al. (2022). Define the functions
J(w, y. v) w
Jw, y. prox (w),
g(w,y.o) v.
(34)
((b pr or
(35)
where the proximal operator of convex function f is defined
1 2 -0)²
prox y (x) = arg min SER
f())
x))²
(36)
Then, Proposition 4.4 in Feng et al. (2022), guarantees that, given a fixed point (w, b) of the GAMP algorithm eq. (30,31) with denoiser functions 9: (w) = g(w, y, vz) and fe(b) fb, az), the vector 0 : (z(b) is the unique minimizer of (33).
As mentioned, in the high-dimensional limit n, d + oo, with n/d fixed, we can track statistics of the iterated variables through a of state evolution equations. We stress that the following result hold for the considered linear target function y ((0*)", \v/Vd.
Theorem (Bayati & Montanari 2011); Javanmard & Montanari (2013), informal). Define
G' (e 2, 2,))
gt+1
[ ( +' " e ) (oee>>((((()) 7 0 ('), 0*) -B
n' ((z,,) (gg((r 2,]]
m=+1
n ( [wg 1 vz)
oft
V.. I(Vi't 0 6*)]
(37)
where & NO, Ia) and
(1)) (3)~~N (())
Q- m² " ))-
(38)
Then the iterated vectors wi and b' the GAMP algorithm (30,31), with denoiser functions functtoonn (34,35), respectively converge weakly to the Gaussian vectors S2 Vq - m/w + m X (with up ~ N I,)) and B'- = - + T. &, in the sense that, for any uniformly pseudo- of order k. deterministic Φ (IRd x RR Rd
IR,
②(b w bi wl
bD) - EO Ω "T Ω2 ....,, S l''
(39)
"f Rd Rd is separable if Vi E {1....,d} df [f E R$), f((b),, for some scalar function
TR
20

---

Published as a conference paper at ICLR 2026
The previous theorem readily implies that, in the high-dimensional limit, de = 6t, and, given 0 f(b', .z)..
1 ( 00'00' 0r) = m²,
z 21001 =d
1
(40)
and the generalization error of the estimator ((x)) f(e, 0²) (10', (0', a/) Vdd
(0',x) x)
(0' (0',0) x0)
R(O') Pr
Q" -2m' + q
(41)
vd
vd
LASSO regression In the case of LASSO, with ((y,z) = (y 22 and r(0) \101, we have that
f(b,a) (b),
(42)
1 Itv"
where (b) max(b X, 0) max(mmaa-b - X, 0) denotes the soft-thresfolding function. The state evolution equations this setting read
q'
n(4+Q* -m 2m
d(1+0))²
ml+1
As erfe
V2(1m9)AAit0")
(43)
TlI+ef n of
jt
ylt1
1 don
Σi=
erfe
(2((+))(\++t)))
and
1
1+
((m")²A, + +2)
(44)
d(mntt)
122
/2((m²))2 +
1
d
2X
q' 1(2 (A)
(45)
d(int)2
2m
At convergence, substituting the equation for e into the equation for one for m, introducing the pa- rameter A nt and leveraging eq. (41), one obtains that the excess risk for LASSO regression i v 2d in this setting is given by
R(O)= (v)
d A erf
+ +(A+ (ā+20) erfe (ZZA,
(VEA++A)
J.A. + A
//A
2v
+ A -
(46)
ny
with Å A + Rn,(())) and
V
n
erfc
(47)
2d
d'
i=1
AA +
For the specific choice of covariance A; di this becomes
R(O)= ~ Rn)(
27
erl
V
+ (A + 20²2 erfe
(
v
ni 24 + A
Vni 27
im
2v
V nii
Å v² /nni (211-)+4))
(48)
n
and
d Σ
À n
crfc
V
= n
(49)
F/
2d
d
V Vni 27 +
A
1=1
21

---

Published as a conference paper at ICLR 2026
Conjecture 2. Define R(O,X, X) the excess risk eq.(29) of the LASSO estimator 0 eq. (26) with regularization strength x. Then, there exists > such that, for any n, >> C, with probability 11) oI(1),
R(O, X) (v))) = Rn,alv(A)) on,a(1),
(50)
with Rv,a(v) defined in eq. (48) and (A) solution of eq. (49).
C.1.2 SPECTRAL STRUCTURE OF THE ESTIMATOR
Theorem I readily implies Result 2. Given the unique fixed point (w, b) of GAMP, the minimizer
of eq. (5) is given by 0 mh S (b), which satisfies, distribution
0 ~ ST ( 5azz),
(51)
with Eg A/m, da valm and
NV, 1). Note that Ea / 2d/n and
d
2d
00.1 ~ max
07 + A
V
0
(52)
n
The random variable variable un satisfies
0 + op(vA.)) u,
if nA > då, if nA < då,
Ui ~ N 0. (D.A + 4
(53)
2, A (1+
where we say that u, corresponds to the signal component 0ni in probability when the signal variance A, dominates over the effective noise variance Åd/n; conversely, the effective noise dominates ug.
Therefore, the ensemble {ur (14))lnAssd constitutes a "bulk" i..d. Gaussian variables, representing the combined effect of label noise and the limited number of samples. In fact, if the sample size large enough, namely n >> d'A,7 the effect of the noise becomes undetectable. We refer to the remaining ) as "spikes", representing the components of the true signal 0 that
want to learn. Therefore, the scale is such that nAÁ, då represents the number of "learnable" components. Note that, in the diagonal linear network setting, the noise bulk has no sharp boundaries and the signal components (or spikes) are themselves random variables. We adopt this terminology in parallel with the quadratic network case, but when nλ; is comparable to d', the signal and noise contribute equally to the variance of u;, so a spike cannot be distinguished from the bulk. For clarity of exposition we will nevertheless classify indices according whether nA, larger or smaller than då, acknowledging that this introduces am asymptotically negligible ambiguity near the crossover. The 7th components of the LASSO estimator is then obtained by soft- coft-thresholding the variables u, with the parameter сa represents a cutoff that induces sparsity in the estimator, setting the smallest values to zero. Note that the cutoff depends on the regularization strength À only through v.
At this level, we can distinguish the following scenarios: terms of number of data,
spikes + bulk
only spikes
nA,- & dÂ
nAa > dÂ
not components can be learned
all components can be learned
in terms of thresholding strength,
weak 2 E max(nAa/d,A) A)
strong
extreme
max(nAa/d, Á) & u² & nA1/d
22 > nA //d
cutoff below spikes: all learnable signal is learned
cutoff between spikes:
cutoff above all spikes: nothing learned
signal is partially learned, noise is filtered
22

---

Published as a conference paper at ICLR 2026
Error decomposition We can interpret the excess risk expression in eq. (46) in light of these considerations on the interplay between signal, noise bulk and regularization, by splitting the sum at the relevant scales identified in this section, namely & such that nA,, dv² and i such that nAig då. Note that the extreme regularization case v² > v is trivial, as the excess risk R E iAs/d, i.e. is dominated by the underfitting of all the signal components.
1. Weak thresholding: the excess risk can be decomposed as R ~ Runder + Rover + Rapp Underfitting term (signal components not learned)
d
Runder d
A erf
V VÁ
A
(54)
d
+1
11+1
Overfitting term (learned noise)
1 Rover
(A+
A 20 (++20²)
crfc
2v
A
-IA
(55)
n
v
V
27
d i
Åd
E
STea
24
(56)
Approximation term (learned signal)
(
n
+ (3 (A+222) 202) erfe (V*A+ + A
v
Rapp
d
A erf
VEA++
Va +
A
(57)
2v
Th A AA VV
(58)
2. Strong thresholding: the excess risk can be decomposed as R RJ Runder R Rapp Underfitting term (signal components not learned)
Runder 1 Are
Ais
(59)
VnA
d
1++1
Approximation term (learned signal)
n As erf
V' V + A
+ (ā+20² erfe
(
v
Rapp
d
71
W/ + A
(60)
4v
2y
M ( /(4++4)
(61)
n
In Section C.2 we observe that these are the relevant scales for computing the leading order terms of the excess risk and its scaling laws. Moreover, estimate the values of and Å as functions of n, d, X. These results, together with the interpretation provided in this section, lead to the identifica- tion of the phase diagram regions in fig. and the phases descriptions in Section 2.2.
C.2 LASSO SCALING LAWS
From eq. (48),
R
1
Σ
(x.) + f2(x,) fz(x,)),
(62)
23

---

Published as a conference paper at ICLR 2026
with F; 1/(2r) and
fi(x) 2x erf
(63)
2x-27 + A
fa(x) (Å +2222) -222)erfc erfe
(64)
x 2x-21 +
1²
fz(x) V UV
2- + å exp
(65)
x- 2y
We observe that the leading order of the functions changes scale around æ ~ Å -11((27),
For 2y > Å
fi(x) = x erf (VX")
(66)
(2(x) ~ (À + 22²²) erfe (ux7)
(67)
fs(x) 2
v -/Vx vx 7 exp (-v2x22)
(68)
For x 2y < Å
[1(x) ~ - -ver erf (vÀ -1/2)
(69)
f2(2) ~ (Å 22²² erfe (V -12)
(70)
2 - 2/12 exp (-~²Â-1) V
fs(x)
(71)
Note that the scale r; 22 Å corresponds precisely to the detectability threshold of signal compo- nents observed in Section C.1.2 for the LASSO estimator and in Section C.4 for the Bayes-optimal estimator.
We compute the excess risk R from eq. (48) at leading order as function of the parameter v. by splitting the sums at the crossover scales. Note that this procedure corresponds to the decomposition of the excess risk into approximation, overfitting and underfitting errors that we have identified in Section C.1.2. Afterwards, solving the self-consistent eq. (49) for we derive the Results in Sec- tion 2.1.
The three main regimes are
- 22 2>
max
(r
<
< x
max(nd 27 2τ, Δ) < v2 & n,
(72)
V max( Å)
2 max(nd 27 Å)
which are the extreme, strong and weak thresholding phases we have identified in Section C.1.2, for the specific choice of covariance A, di-27 In what follows we will often use the expansions
2
erf(x <<< 1)
r + o(x),
(73)
VT
=²²
erfo(r > 1) IV/ π
1
+
o(x o(x-1)
(74)
2r22
4x4
Extreme thresholding For 2² n, the dominant term is given by underfitting
1 ( (xx)) ~ <(2>),
(1))
(75)
24

---

Published as a conference paper at ICLR 2026
while the remaining terms are negligible
min/d (1/00)11/000)))
A
f2(2z) - fs(x;)
Σ
i e
(76)
MV VVRT n
1=1
exp(- (ap(-12/n))
(77)
Vy/T
As expected, in this regime the large softtthresholding parameter forces the components of the esti- mator to zero, and R ((27).
Strong thresholding If instead max(nd-2x, Å) < v² < n, defining in == [(//22)//(e)) and
' == (n/ (m/A) (/())) J, we split the sums into four terms:
approximation error:
2
(1x) - f3(1,)
11 exppp_pt/2/11 /n)))
mm
2
17
n TK
1=1
-1+1/(27)
11
+ 1
2v2 Å
ii)
fa(
erf
(0e(vvvv//e v/vn)
i=1
i=1
underfitting error:
d Σ
11 -1+1/(27)
iii
f1()))
27
24
V2
i +1
202
n
-1+1/(27)
+ A
1/7n
141/(227) +
((2)
11
-141/(27)
min(d,is)
dd 5/2
iv)
Σ i=
fa(x;)
fa(xz)
Σ
25/2 /n 1|2nnd 21
- /
n
VVTT
ny3
Laplace 2
1+1/(27) v 1/7 + Assd 2g)
d/s/2
D2
eV7(27-1 -
ny
exp
A
where in the last step we have approximated the (Riemann) sum by an integral which we solved using the Laplace's method, that is (informally)
[[
Molo) dor
Ma11
2T
zo arg max g(x), g²(x) S Ovr [a. xE(a.b)
V (xo)(
(78)
The dominant term is R = (1/22))1+11221)
Weak thresholding Finally, if 12 < max(nd 27, Å), split the sums into the following four terms:
approximation error:
min(1 .)) [1( fz(2;) (1 + 7) vn
2y3
(1/Ssnd 2n)
(1+7)/(27)
1 Iaann 27 snjatt>)
min(d.ig)
ii) fe(x.) =(20² + min (1-111(22)) -1/(27)) -1/(2))
(79)
25

---

Published as a conference paper at ICLR 2026
overfitting + underfitting errors:
d
d
2v
ii
f1(25) 11 AAnd 2x]
22
V
ÂT
I miminnd.. +1)
imiat!
-1+1/(27)
21
[Bind221]
VÂn A
d
(2r² + ②) ( n -111/22)) A 11(27)
iv)
Σ
fa(1)) fz(xz) x1 ا [b
(= +1)
2 uv & Texp(-1> (app(-u2110 -
d
1 A>nd-2y)
n (80)
The dominant term is therefore R x (2v² + Åjd/n.
Note that, for Ó(å), even if the cutoff is larger than the noise variance, is not large enough to completely filter all bulk terms. We will verify that this is the case Phases IV and V of Fig. 1, for < vn/d and n << d. The leading order terms in the excess error decomposition are equivalent to the ones we have computed in the previous paragraph for the Strong thresholding case. However, we must take into account the overfitting contribution from the bulk components ned exp v2
d
(term iv), which is otherwise negligible for u²/ Á larger than any polylogarithmic function of n. d.
We can now proceed with the solution of the self-consistent equation (49), in order to derive the closed-form expressions for the excess risk scaling laws.
Noisy setting Since A > 0, then, provided R 0(1). Á = ((1) Let n >> d. Eq. (49) readily implies that X d/n, as the second term the left-hand side is bounded by 1. Therefore, for n >> d,
(1),
x n/Vd.
-141/(21)
R-
+
x2d
n²
max nnd (nd- 1/2 vn/d) & <>< < max (nd- /ndd)
(81)
+
d
max
A2d
Let instead < d. The second term on the left-hand side of eq. (49) is
Ev exp
(-v2/n).
v>> n,
1 d
erfe
v
R
2
1/2-))
(12-1/7) (1 + 0(1)) + erfe (1) 1 1<
V x 27
1
V VEI.
(82)
For À > Vn/d, this term is subleading and v x A/d/n. One can observe that eq. (49) does not have positive solution in this regime if << therefore we exclude this case Hence, E vn/d,
erfc
V
n
è 3/A x n/d
(83)
A
d
d
log n
(84)
where we used
1
=² =a
2a 2x22
(202) 2
2 log(2a?) se loga,
(85)
with Wo denoting the principal branch of the Lambent W function. Note that, in the under- < d, even small regularization strength À leads large effective
parametrized regime
26

---

Published as a conference paper at ICLR 2026
regularization y of order vlog d.
We can conclude that, for n < d
(1),
A n/vd,
1+1/(24)
θ
?
max (nd (nd- /n/d < <x<n/VV (86)
x2d
-141/127)
+
(ogld//))
+ log(d/n)
A & vndd
Noiseless setting If A = O, then Á = R. Let n > d. Similarly to the noisy setting, it is
straightforward to show that x Av d/n in eq. (49), implying 2 Ayd/n. Therefore, for n >> d
(1),
-141/(21)
R
θ
x
アー < X2 <
n d
(87)
θ
dd
X² 2
n²a
Let instead < d. In this regime, the Bayes optimal generalization enror R x n n- > nd -2y,
which implies Å = R >> nd-22, Similarly to the noisy setting, the second term on the left-hand side of eq. (49) is
exp (-v²/n),
v² > n,
d
erfe
V
2 n (12-1/7) (1 (()) + erfe (%)
R < V²
d
x 27
R
d
1
12 < R.
(88)
v² >> n, this term is negligible and v X d/n. For R < v² < n, if this term is negligible with respect to n and also this case x X d/n with R = O(n² ((nn/(Ad)) ((d)) as the n >> regime. The condition 2 > R becomes
x2d
X² 2-
111/221-11(22)
X n- 1d/2
(89)
22-1/7
For consistency, we verify that the second term on the Ihs of (49) is negligible with respect to n/d:
n ((2))
2-11 <<
n d v kn.
(90)
d
Finally, if X > n 1---1-111/ 1/2, repeating the same computations of the noisy setting, we find 2² a
Rlog(d/n) and
R
n (Rlogd//n)
12)/(22))
R log(d/n)
(1 log(d/n)
R2/(27)
(log((/m)
-1+1/(2>)
(91)
R R 27 220 (1g(()()))))) (1 (1- log(d dd/n)
-2y
(92)
R =6 22+11)
(93)
In conclusion, for n < d,
((1),
x² zt
4 x
n ~ 2 x "
(94)
④ n 27+11
27

---

Published as a conference paper at ICLR 2026
Interpolation peak Around n d the excess risk exhibits interpolation peak that diverges as À + O+. In this section we show that in this regime R x X-2/3 The self-consistent equation (49) becomes
A
erf
(95)
d
ni-27 A
As have done in the previous paragraph, it is easy to verify that strong effective regularization 2² >> Å results in the right-hand side being ()1), and to the inconsistent solution X. Hence,
taking v << Å (weak thresholding regime),
2V2μ ~
T X 2V2 A
(96)
VTÀ
2 x XV A and the excess risk becomes, asymptotically,
R + V
R+A
UVR+
(97)
VT
SWR+AR
A
(98)
4
n²
*R++
A
(99)
256
A 4/3
R 22/3
(100)
C.2.1 ADDITIONAL NUMERICAL SIMULATIONS
In this section, we include additional numerical experiments, visualizing the Results in Sections 2.1 (Fig. 8) and 2.2 (Figs. 6,7). The remarkable correspondence between simulations and all results derived from state evolution equations further supports our Conjecture 2.
IV
10
Stamlation
100
Thoor
10-14
10
10:5
&,1
Via
300
20
10
20
30
40
30
60
[5]
Figure 6: Comparison between spectra from simulations and theory across different training phases. Blue: LASSO estimator's 's components (in absolute value) histograms after training. Red: theoretical prediction eq. (51). All panels use d d= 5000 and À == 1-2/2 The sample size is n 200 for Phase IV, n 4000 for Phase V, 6000 for Phase Vla and phenomenology in Section 2.3.
10 for Phase IVb. We discuss the
28

---

Published as a conference paper at ICLR 2026
10
11
10
2.1 2.5 3.0
20
30
18]
[8]
Figure 7: Comparison between spectra from simulations and theory across different training phases. Blue: LASSO estimator's s components (in absolute value) histograms after training. Red: theoretical prediction eq. (51). All panels use d == 5000; Phase lb: n = 200, X = 2n/Vd; Phase II: n = 4000, À 10; for Phase III: 4 10 À = 3. We discuss the phenomenology in Section 2.3.
Figure (left) shows the transitions between phases along the vertical lines of Fig. For 35 the excess risk moves from Phase IV, which is independent of regularization, into Phase and soon (A ~ 35/v200) enters the plateau region of Phase Ib. For n 300 and n = 500, the excess risk starts in the fast-decay region IVa, reaches its minimum at À = //ndd (when the soft- thresholding cutoff reaches the edge of the noise bulk, see Section 2.2), then crosses Phase II and enters the plateau Phase lb. Finally, for = 3000, the excess risk begins in Phase IVb, the fastest decay regime, since the noise bulk negligible for n >> d2,, then grows as crosses Phase III and Phase II before reaching the plateau in Phase Ib. Notably, for d 200, observe the interpolation phenomenon when À < vn/d 1, with a peak that grows as the predicted A -2//3 in the limit A + 0+.
101
A 700
30
360
lbsk
Rove lbck
30"1
10:1
107
19
202
20
Regolnication aussualla
Figure 8: (Left) Excess risk of the LASSO estimator, as function of the regularization strength X. with = 200, A 0.5, = 0.75. Dots represent numerical experiments, while lines the solution R, of state evolution equations 48. The curves correspond to the crossovers between rates observed in Fig. 1. (Right) Excess risk of the LASSO estimator, as function of the sample size n, with d 100. A 0.5, y 1. The regularization Aopt has been chosen the minimizer of the theoretical excess risk R,,a and its value accordance with Corollary 1. Dots represent numerical experiments, while lines the solution of state evolution equations 48.
C.3 BAYES-OPTIMAL EXCESS RISK
The Bayesian predictor Ono is given by the expected value of the target function with respect to the posterior distribution P(O P(O DD) Applying Bayes' theorem, the posterior distribution in this setting
29

---

Published as a conference paper at ICLR 2026
reads.
d II [ II N Yui
d X B5 ==1
1
P(OID) = ((D)
(101)
Vd
i=1
= O)..
(102)
= where, recalling the notation X = (01,.... .,, knn² E Rexd for the covariate matrix, (y1...., Un)" for the label vector and A diag (A1, Ad) for the weights' covariance,
=
VX y.
= A A-'+
x
(103)
vdA
dA
Therefore, the Bayesian predictor is gB0 0 x and its excess risk is given by
R = E L 0- - (x))²
(104)
= E| 0 |0
(105)
Tr V.
(106)
Leveraging a classical result from random matrix theory Silverstein & Bai (1995), we have that, in the high-dimensional limit n/d + co, with fixed ratio n/ d,, the excess risk concentrates around RBO, solution of the fixed-point equations
RBO
(107)
A
d 1
A + RBO
d
(108)
27T q"'
where eq. (108) is the excess risk for the specific choice of covariance A, di- The same equations can be derived from the state evolution of Layes-GAMP.. i.e. the GAMP algo- rithm tailored compute marginals of the posterior distribution and the Bayes-optimal predictor. The interested reader can find more detailed discussion in Appendix D of Rangan (2011) 1).
C.4 BAYES-OPTIMAL SCALING LAWS
From eq. (107)
1 BBO
1
(109)
9 1=1
(9 1/27))22
A RBO
Our goal is to derive the leading order of RBO the asymptotic regime n, d > The crossover scale at which the leading behavior of the sum's argument changes is given by q 1/(22) = iq = [g>/(2)). ig < d, we can split the sum at this relevant scale and retain the leading term for each part"
RBO_
1
Σ
(110)
1 (g-
1/(291227
1 ( (q- 1/(27)8)25
(Q)/(n) ++1
(0
1
qi 27
(111)
(9//2-)) +1
2 (+11(127
1+1/(2
(112)
27
2y
1+7/
(113)
2
WWe stress that, throughout the manuscript, the notation & denotes equality up terms that asymptoti- cally negligible.
30

---

Published as a conference paper at ICLR 2026
where we approximate
(d+1)
d
dp 1/(21)
27 dx -
Σ i 27
I (ig + 1) 27 +
27
x dr
(114)
1041
ig+1
d
9 1+1/(22))
0i ( (1-141/27)).
(115)
27 -
1++1
If instead iq >> d
d
RBO
2
(116)
1=1
Noisy setting. Since > 0, assuming RBO = =1), the parameter q = and the Bayes-optimal generalization error
(0(n-141/(r1)
< d
RBO
(117)
((d/n))
n deT
Noiseless setting
When A 0, the parameter & n/ RO and the Bayes-optimal generalization
епоо
2++1),
< d >> d.
RBO
(118)
00
D DERIVATION DETAILS QUADRATIC NEURAL NETWORKS
D.I BBP SIMPLIFICATION
To derive our results, we make the following assumption, which for the moment we do not control rigorously. We assume that for the sake of simpliying the equations that S" has sub-extensive rank, iee. it has eigenvalues for great simplification of the density 148 (the spectrum of S' + 8Z, where Z GOF(d), which can be computed with BBP-like techniques (Huang, 2018)
Jizı for < 1, and zero otherwise. This technical assumption allows
K
ps(x)) = (1-
(a(x)>(2
K ) (Meo + +(1))(=/0)/ + 7
Σ 1=1
8(x fo(Vdi )),
(119)
where K & d is the largest integer such that vdk-> 2 8. fs(x) + 8²/x is the BBP-like shifting function. We then send - 1 a posteriori, after finding that the enror does not depend explicitly on For the rest of the section, let define ā == n/d2.
D.2 BAYESIAN ESTIMATOR
In this section we solve eq. (19) for > 0.
Phase I: Large-samples Phase I > d2++1 The first equation of eq. (19) gives q O(n/d²), > 77' so all the spikes are outside the bulk. We then have
and thus d -
(1 cd
4n²
(1/v/(z)
(oc(x))drr= =1
d11
(120)
3q
3
by eq. (119), and thus eq. (19) gives
22A
25 ==
(121)
A + 2(Q"
which gives RBO Q* q
Art
((P/n).
An
31

---

Published as a conference paper at ICLR 2026
Phase II: Under-sampling
Phase is d < < d We rewrite the integral in eq. (19)
4T²
/
4
/ /(us)'dr = 4m² I dmvl))((()))))
(2/vg(o)'
(122)
3q
3
where v denotes the spectrum density of Z vQ0". The first equality by change variables and the second quality is from (Maillard et al., 2022, Lemma C.1). H [r] denotes the Hilbert transform of v.
We further denote 8 : Vqd. The first equation of eq. (19) suggests that q (n/dP), and thus 5 ((vn/d). For d < d we have d= 3 < < Vd, according to eq. (119) is composed of a semicircle part (denoted as v0 = (1 K/ k/d)(uso + D), where ú is calculated in Appendix D.6) and a discrete part ((0; f1( 1=11 where K = 81/7). Thus we have
/ [n(())(([\(=))> = d
2
Σ
drvn (2) (p)(())
T(x xz)
i=1
K
K
+ d2 (
drv((())
H(m](1z))
+
Σ
x2(x nz (x - Tzj)
d
Td( 3ia)
mml
im1 in.
(123)
Denote the right side termss as I to L.. For the first term we have
1 3 dopp (1) I =
(1 (1- ( E 82 821-27)
a--3//))
1
(1-
(124)
472
i=k
where we use eq. (200). Then estimate the leading orders of the other three terms. For the second term, we have
K
K
12 ~ -
- drppac (x)
I
-
(125)
2T
Iz
t=1
where we use H [Mar] term we have
for E [- and the fact that 2ipscc is an odd function. For the third
2n
K
K
)
13 = θ
θ
Σ
1
2(8211
(126)
i==1
4 is composed of three terms. The first term is
K
1
Σ
K
221+1
(1+ dr 0(81/70
27
(127)
d
82d
where we use the fact that H [Hsse] ~
for r > 1. The second term of 1 is
K 2
K
H(vo((z;)
p
T(0; - xii
nd. xz (xz - xz)
k
+
Tdf2
[z(0T - xx;)
x;(1;; - xz))
(128)
+===1
k
K
Σ
-(8821d-2))
nd x
nd
W
32

---

Published as a conference paper at ICLR 2026
The last term of 4 can be written
K
22
d
I - 21) - xz))
d
7 ddd
mny² 2
$,1.==
1.1.==1 if...
m,,
(129)
where we assuming m, n < i to obtain the leading term. For a fixed i, we sum over m, n1 - 1),
- (i
-1,1,
K i, which gives
K
22+2
22+ HK - H(-1) 20 k2rt3 f 22 log 1 1- 2
dx.
(130)
Pl
P.q
i=1
where we denote Hi Ep=1 p = logg + +(1). Thus have
-
(131)
d3
(rz - 22) 2i)
For < n < d++1 and y > we have 82//d-1 << 1, and thus
2
/
(132)
Taking it back to (19), we have
2āA
4n
2a
θ
dd-
(133)
A + 2(Q'
dA
which gives
1- #
dA
RBO = Q" - -q=
(134)
4n
Phase III: Not enough data Phase III n & d, and thus < 1. By eq. (119) and Appendix D.6, there are outliers and the first-order correction reads
4m²
/
1 ed
(a))dd== -
Σ
(8i
-
(135)
3
Taking it back to eq. (19), we have
26A
4āQ²
2
(136)
A
2(Q' 9)
A
2(Q
9)
and thus RBO = Q* - q = Q', where we use Q' = ((24).
D.3 ERM
In this section we solve eq. (20) for A 0.
Phase Trivial phase The first case of phase I is 8 > vd and 0 < 2 d < In this case the spikes are covered by the bulk and the cutoff is close to the boundary of the bulk. Thus we have
J(8 = S (n (x/6)) - dc) 82 7/2
(137)
105T
where : 5. Then eq. (20) reduces
{
4ad 8 = 82 /6/5
(138)
Q - 2
A + 22d
82
82
16: 5/2
15a
33

---

Published as a conference paper at ICLR 2026
where we use < 1 and keep only the leading term. Thus we have
A - Q"
R = 2āo"
(139)
2
and de x 20- = 2a (Q* + A/2). Then the condition 6 > vd gives
n 16
(2Q +
(140)
The condition 0 0<t< gives
2 c < 4ād, and thus
2Q
(141)
4
d
The second case of phase I is de > 2 max 6, vd). In this case both the spikes and the bulk are below the cutoff, and thus (66, λε) Then eq. (20) reduces to
{
4ad é =0
(142)
Q' + A + 2002
6²
2
which gives
:= 2ād
Q"
(143)
2
1. and = 45
The condition de > 2 max(66, Vd) reduces to
2Q*
4n
max 81
(144)
4
d3/2
To conclude, Phase I is
1 R R=Q', if n (2Q" + A)d or x>
4n
(145)
16
d3/2"
Phase II: Overrregularization phase The second phase is max(6, d 1/2) < de < vd. In this case we only need to consider the spikes outside the cutoff, have
1 7
82
[66
Vdi- xe)2
(146)
where K is given by vak- + vak-7 de O. Thus we have K ~ (Vd/Ae)t/> satisfying
1 < K < d. By keeping only the leading terms, we have
29-1
7 +
1
Ae
2
((8 0² +
7-
27
Vd
vd
1
,>,
(147)
where we use
K Σ
K1-7
+
((7)
(148)
1-y
1=1
and thus Q" >011 j-29 ((27). Taking it into eq. (20), we have
465 =
20-1
(149)
Q' +
A +
2
6²
=Q
de
27-1
va
which gives
4a and
A 2y
2r-1
2-
d3/2
: 2
X
(150)
27 1
Vd
2n
4n
The condition max(6, 1122) & de <<< Vd gives
max (v
V1 n
n
3/2
<
(151)
34

---

Published as a conference paper at ICLR 2026
Phase III: Intermediate over-regularization phase Phase IIII is 8 < de < d In this case all the spikes are above the cutoff, and all the bulk is below the cutoff. Thus we have
od
8² Vid-7
J(6 XE)
Σ
Xe)2 ~ Q- ) + X²E2
(152)
1=1
by eq. (148). Then eq. (20) simplifies to
48d =0
(153)
Q" + - 228
5 = =0- 222:
which gives 1 46
and
A d2d
= 280²
(154)
2
16n
The condition 5 < de < d reduces to
V <
(155)
d3+3/2
which further requires n >> dr+11
Phase IV and V: Benign and harmful overfitting phase Phase IV and V are d < n < de and
0 < 2 - de < 1. d < i < vd. In this case the cutoff is close to the boundary of the bulk and a part of the spikes are outside the bulk. Thus we have J6, λε) = J (6, λε) + J2 Xe), where
J2(o,Ac)= S'¹ (cc(o//)/8 dc) 82
16/7/2
+ 2/2
(156)
105
with A a constant, :: := 2- de and
18/ (8/va)- -1/
8²
(6,
Σ
(Vdi-t 7 +
Xx)²
Vdi
2 Q" + ( 8 2-4
+2 dc 1 2 d
1
2yy11 1 + +Ac/5)2+22
+
Vdl
8 1-
1 5
1 + 27
2dc
117)
vd
(157)
where use 5 >> 7 to obtain the first line and use eq. (148) for the second line. Then eq. (20) simplifies to
4 = 82
16/ 15x
+8277/2
32
+ 9A)
105
(2 - 1/7) (2-1/7) +c((1) (2-1)))+(((()) (1²) 2-
(2))
+
+4+2 2
4
+
1
2y-1
1-7
147
1+27
Q* //4 200
6²
=Q' +0"
/6
17/2
16
+ +AA
15x
105
2-
+
+44
+4+2
2
+
+ (()))
(1)
17
147
1+27
(158)
where we use f < 1 to drop the smaller terms. We also use the shorthand (²(() == - 4+ The second line subtracted by the first line gives
2
4
1-
147
A
1 4
1 1-7
4
5 2-1
82 7/
288
(159)
2
24 -1
1
1+7
1+
27
7
7
Vd
105T
35

---

Published as a conference paper at ICLR 2026
Therefore, at the leading order we have ò' 2
V
A
and
46
8² 16t5/2
52
2 480²
=
(160)
15x
where we assume λő
<
Then we have
== 228²
A
2478
dA
1- 27
+
A
15x
2/5
n
2/5
(161)
2
4,3 + 472 -1
4n
4
d
The condition d <nn < d and A8 < 1.0 <t<
1, d 7+ < 8 < V
reduces to
X
n
and d d<n< << d.
(162)
d
where we use 82/5/2 ~ A
A8 Note that under this condition we further have t < 1 as 5 >> 1.
Interpolation peak The interpolation peak is at ā 1 and A < 1. The first case of the interpo- lation peak max (ee, d 1 < dd. Then J(ō, dc) = (1)) + J2(6, Xe), where
1 cd (1) = d
8²
8
2-4
ReLU(Vdd di
2 Q' + +c())
(va
(163)
Vii
by eq. (177) and
J2(6, Ac) = 82 See
2 Msc(z)(2 - -x/8)²
82
8
deδ.
(164)
2
3π
de
Then eq. (20) reduces to
{
4 8² = 8²
8 Xed C()) 1/7)
(
5
2-1
3n
Vd
(165)
2-
Q' + 3/4 + 200
8a
== + 602 + C(7)
(vd)
By using ā 4. we obtain
3T
3 A 2
1/3
X
2/3
8
A -1/3
(166)
8
32
2-1
as the leading order solution, where the
8 vd
term is ignored because
6 Va
& 1. Then we have
A
2/3
R = 228²
2 2
2/3
(167)
2
32
The condition max(Xe, d 1-/2) < 8 < vd reduces to
d 3/2
1
(168)
The second case of the interpolation peak is max(le, vd) < 8, which gives [6,, de) Similarly we can obtain the solution
8
deo.
3
A
3T
2
3x
1/3
A
2/3
Q
A
2/3
8
Q*
+
-1/3
1/3
(169)
2
8
2
and thus
A
3x
2/3
A
4/3
R
288
R 2
Q*
+
X
22/3
(170)
8
The condition (Xe,)ee vdl < reduces to
A E 3/2
(171)
36

---

Published as a conference paper at ICLR 2026
Phase VI: Large-sample phase The first case of Phase is > de and λe < d < d 7+ In this case the cutoff is almost and all spikes are outside the bulk. Then we have
cd
8²
J(o, Ae) 23 00
Vidi
2 Q" + 81 4c)
(172)
d ==1
dii
2
and thus eq. (00) simplifies
406
8(1 + 4c)
(173)
Q'
+
200
6.
Q" + 28 + 4c),
which has a solution
A
1
8²
(174)
48
4e
Then we have
R
2ād
(1 4c)
(175)
8ā
for σ >> 1 and the condition Ae < < d
reduces to
X de
n
and n n> d++1
(176)
d
The second case of Phase VI is d < the cutoff is almost 0 but only part of the spikes are outside the bulk. Thus we have J Xc) =
< d and max (c, d=1+ < 8 vd. In this case
J1 (6) + _ where
cd Σ
]((6 d
ReLU(Vdi
7
)
Vdi-
1=1
(8/va)
8²
(Vdi +
(177)
Vdi
2-
=
C(+) C(+)
where C(r) ;= 1-4, is constant. In the second line we use d >> d 1+12 In the third line eq. (148) and only keep the leading term. Then eq. (20) reduces to
2-
4 - -2 C()) 1/7) vo
(178)
( ( 2-4
Q' + 2 + 255 =
Q"
C(7)
(vd)
The second equation subtracted by the first equation gives
2-
A
-88 C(:)(1-11)
280
(179)
As n >> d, at the leading order we have 8 m
A 16
Then one can verify that if n < dr+1 we have
< 82, which suggests that
R = 2
(180)
86
In this case we also have
4 and thus the condition max(Xe, < & vd reduces to
n and & < < & dr+1
(181)
d2
To conclude, Phase VI is
Ad
R
if
n>
d and
X & V
n
(182)
8n
d2
37

---

Published as a conference paper at ICLR 2026
D.4 UNIVERSAL ERROR DECOMPOSITION OF FEATURE LEARNING
In this section we derive Result 3.
(i) When part of the spikes are outside the bulk and part of the spikes are inside, and the regular- ization does not cut off the bulk (e.g., in phases IV and V), we can rewrite the SE as
{
4 Q'+ A + 2602 2
= ( Ac) + J2 Xc)) (1 Ac (JI Ac) + J2(6, Ac)),
(183)
where the auxiliary functions are defined as
K(8)
8²
J1(0, = d Σ
(8:
XE)2
(184)
84
and
L (xo(z)(( Xe/0)². Sxe/8
J2 Xε) =
02 82
(185)
Recall we are considering general model with s; denoting the i-th eigenvalue in a descending order and K (6)) < d satisfying (K(8) = 6. The excess risk is given by
A
R = 2802
Q' + (50g Acdie (), + Ac))
(186)
Then we have
K(6)
K(8)
(00, Aedsc- 10,)(
(K xe)2 +
d
1
2
d
Σ
(1
8
8;
Xe)
20
8:
Xe)
d
Σ
(8;
82
81
Xe)
1=1
K(
K(6)
d
1 (K'()) - xe)2
+
7
1
Σ
82
Xe)
82
82
X6)
d
1
Σ
2
8i
8;
8:
1=1
1=1
(187)
and
(80g + Xedre - 1)J2(6, Ac) =8² =
2e/8 Mao( (=x - 2E/6)2.
(188)
= Now we obtain eq. (17) by using Q= Ed 87
(ii) When the regularization cuts off the bulk and part of the spikes (e.g.. in phase III), we can rewrite the SE as
4882 8² [6, Xc) (1 - (1,
c -5 _
(189)
+++ 2252
where the auxiliary function is still defined as
1 (6,)
82
(5, = d 1=1
Σ
(s, +
xc)2
(190)
8;
but the cutoff becomes &K(S,At)
+
82
de = 0. The excess risk is given by
#k(h)20)
A
R: 2
=
Q' + (50s + dedxe (o,
2
2
K
(6,)c)
82
28
K(8,Ac)
82
Q+ +
(8i +
Xe)
- Xe)
Σ
(8; +
xe)²
d
8i
(191)
1=1
1=1
(((.Ac)
7
1
82
Xe)
82
(81
+
82
x6)
+ d.
1
Σ
8
8i
8i
8;
i= K (6.Ar)++1
which gives eq. (18).
38

---

Published as a conference paper at ICLR 2026
D.5 OPTIMALITY OF PRUNING
In this section we derive Corollary 2.
Following the calculation in (Erba et al., 2025, Appendix A.4.4) find that the excess risk after pruning can be written as
R Q= + (d0s + XEdse 1)J1 Xe) Ae=
(192)
where 8 is still the fixed point of eq. (20). Then we can obtain the following risks.
Phase la: We have J1 (6, Xe) = 0 and thus R = =
Phase IV and V: As the contribution of the bulk disappears, we obtain
24 473 + 4,
dA
(193)
4n
Phase Vla: As the contribution of the bulk disappears, by (179) have
R = (()) - 117)) (
8
1-1/7
4(1
7)(1
7)²
dA
1-
(194)
vi
2
-1
4n
Phase VIb: We have
ed
8²
Ji( (ddi
Xe)²
Vdi
(195)
which gives R (1+114
2cx
D.6 PERTURBATIVE EXPANSION OF THE BULK
In this session we discuss how to obtain the correction of the bulk in eq. (119). Consider H Z + Σ 'aaaaa with { smaller than 1, where Z GOE(d) and {wikez are uniformly sampled from the unit sphere. Its resolvent be expanded as
1 Tr(2 H)-1 d
m=(²):=
+ 1 k
2 mzz
Σ
Az (2
2)- + Σ
Tr(2 - oz, z)- - Z)-1.
7
f,j=1
(196)
For the first-order correction we have E doo (2 - = Eiz de m2z(2). For the second- order correction we have
Tr(z - - 2) - 11 (z - 2)-
1-=
Tr(; - -
- -)- + Tr(= -3-3 Σ X
(197)
ddd
(2)
This gives a correction on the spectrum ((x) = شرت ple (x) :0)). Note that the first term is an odd function and the second term is even function, and the resolvant of GOE is given by
شنا im( + 10)m' (a +
mz (x + :0)
iptee (x):
(198)
39

---

Published as a conference paper at ICLR 2026
we have
4x²
/
4m
/
(xcc())da 4m² i p
un
un(z)) do
Ms
3
1
x
2n²² / (xc( + Mac )dx
7
j=1
(199)
1
47²
/ [nalx))
x
7
3
Σ X²
E=1
This correction the leading term only if Σ higher-order terms are of the same order, but their sum converges for |=| > 3, which gives
< E X However, if i-,, all the
4m²
vu(2)) = 1-6 /
(
11
Σ x
(200)
3
d
2m
instead.
D.7 NOISELESS SETTING
D.7.1 BAYESIAN ESTIMATOR
In this section solve eq. (19) for A For A 0, the first equation of eq. (19) gives
i 20 RBO-
Phase Perfect recovery
We can verify that = and RBO solution of eq. (19) for
A 0 because
4T²
/ (1 =1
lim 40000
(201)
3
Then a = 9 is the perfect recovery threshold, which suggests that
RBO =0 if n d2
(202)
Phase II: Under-sampling if < & d, eq. (132) combined with eq. (19) gives
ād 1/27
2a θ d
(203)
MMSE
and thus
RBO θ ((1))))))
(204)
Note that for this case we still have 51/1d-1 < 1, so the derivation of eq. (132) is valid.
Phase III: Not enough data nn < d, eq. (135) combined with eq. (19) gives
2ā
2ā = Q' Q* ---
(205)
and thus RRO = Q* q - Q"
D.7.2 ERM
In this section solve the equations in Appendix D.3 for A 0.
40

---

Published as a conference paper at ICLR 2026
Phase I:
Trivial Phase
The results in Phase I of Appendix D.3 holds also for A 0, which gives
Phase
4n
R Q" if n Q'd
A>
(206)
d3/2"
Phase II: Over-eegularization phase
The analysis of Phase Il in Appendix D.3 also holds for
A 0, which gives Phase II:
27
/3/2
if max (vo
63/2
1
drt d
< <>
(207)
24 1
4n
n
Phase III Intermediate over-regularization phase III in Appendix D.3. Eq. (153) solves
Phase III is given by the analysis of Phase
1 5²
2d
(208)
46'
32n
for A = 0. Thus we have
d2d
R = 2ā0²
(209)
16n
The condition å < de < 1 gives
A
and >>> d²
(210)
d/+3/2
Phase IV: Under-sampling phase
Phase IV is given by the analysis of Phases IV and V in Ap-
pendix D.3. Eq. (159) reduces
2ā0²
((s
6
1
4
7-1 +
82
7/2
(211)
27
147
1 27
7
105m
for A = 0, where 55/2 ss 4ā and assume À < ād. Then the second term on the right side is negligible since 2/2 << ā. Its solution is
8=2 2- (66
1
4
1
7-1
4
d
+
(212)
27
1+7
14 27
7
7
n1
which gives
R = 2882 = 2-227+
(6-
4
+
7-1
+
4
2y
d
27-1
(213)
27
1+7
1+ 27
7
N
The condition d d de and \ ad, 0 <
2
1,
< 1
< 8 & Vd reduces to
d' 3/2
and de
(214)
nº--
where we use 82/5/2
458
M 2
0. Note that we further have 552 ~ 4ā 28 << 1 satisfied.
E COMPARISON WITH L2 REGULARIZATION
In this section we compare the scaling laws we have obtained for ERM to the ones of ridge regression for linear model, proving in particular their sub-optimality. The ridge estimator is defined as
Ô ridge arg min BERd
(n (0, x,))) 1110
(215)
This be mapped both the diagonal network and quadratic network case, depending the choice of æ. For simplicity, we assume x ~ (O, Ia).
Cheng & Montanari (2024) readily implies the following.
41

---

Published as a conference paper at ICLR 2026
Theorem 2 (Excess risk rates for ridge regression). Assume that y (0" .z) VDc with Ç ((.01) 1) and B|]001|3] = (1). For n, d > 1, the excess risk associated to the estimator defined in (215) satisfies
if n d
1
X2,
if n > d
and
vdjn <\<L
Ra.d =
(216)
Ad/n, AX-1/2
if n >>< d
and
X < vd/n x<
if n= d and
Proof. The excess risk concentrates with high probability, for n, around the following deter- ministic expression (Cheng & Montanari, 2024)
nv²2
R
El
(217)
(11 + 2)
n21 + 2)2 d
with the unique non-negative solution of
"
A 1 1
(218)
d
1+
Therefore
d/n+>,
if n <d,
V == θ
X
if n > d,
(219)
VA,
if n d
and
λ 1.
Substituting into (217), the result follows.
Therefore, cannot obtain non-trivial risk for < d with Lا regularization.
F NUMERICAL DETAILS
The state equations (20) can be iterated in more numerically convenient form that depends on an extended set of parameters (g, m, E, q.q. m, Σ) as presented in Erba et al. (2025), Appendix A.4.4. Then, we can compute and e as 8 V m and e = 2/m. The limiting distribution of S* is not easy compute exactly d co, and for finite d the contribution of the spikes the state evolution is the sum J1 Ac), whose upper limit depends on the width of the bulk 6. The state equations include derivatives of this sum, where delta spikes difficulty of representing them. Having obtained results that do not match the experiments with these approximations, we resorted computing the integral by Monte-Carlo procedure. The overlaps m, Σ are computed using finite size samples of matrices and their eigen-decomposition each step of the state evolution
neglected due to the numerical
M
/ q Z m S* = O diag(()......................... OT)
(220)
dd can be taken diagonal matrix, since this amounts to rotation of M, which does not affect the distribution of Z by rotational invariance. One can then apply the spectral denoiser described in Erba et al. (2025) and compute the overlaps using the reconstructed matrix M = O diag( (2,, O²) where 0 Σ ReLu(r; 2A) the denoised eigenvalues. Finally, the order parameters can be computed
where GOE(d) and S* Σg 2- diag(1,2
vd
m ++ JE Tr((S*)TM)) d JE Tr[M M]
(221)
y = EE Σ1_ ((n Eiej
Σ
The expectation is taken over Nsamples ~ 10 samples for d 10 independently for each order parameter, for total of 3esples sampled matrices per iteration of the state evolution.
Since the ERM problem is convex we resort to using LBFGS with Wolfe line search. We used the Porchc implementation of the optimiser, taking of evaluating the network efficiently at
42

---

Published as a conference paper at ICLR 2026
each pass. For the specifics of the implementation refer the code repository https: // github. www.oo//eeeeeeee SPOC-group Quadrat inneratiitettt. wwwtteeeeeeaaeeeeee Convergence is typically achieved with precision of at least 10 8 in a few hundred iterations. The main challenge is in storing the dataset in memory. For each run we used up to 1800 gigabytes of RAM on nodes with 2 Intel Xeon 8360Y CPUs. Our total computing cost (including intial explorations) is around 250000 CPU hours.
43


Reviews:
[{"review_id": "t8MU3s9Rgh", "reviewer": "Reviewer_Bf9w", "summary": "This paper studied the neural scaling law and analyzed the spectra of the trained weight matrix of two-layer diagonal networks and quadratic neural networks. Their teacher network has the same architecture as the student model, but with certain noise. By transferring diagonal networks to the LASSO problem and transferring quadratic neural networks to matrix compressed sensing, the authors can utilize  Generalized Approximate Message Passing (GAMP)  and its state evolution to compute the final excess risks and the spectral distributions of trained weight matrices. This heuristic computation works well for a power law target, a general ridge parameter, input dimension, and effective sample size. As a result, this paper showed how the scaling of the sample size and regularization affect the generalization, and how different scalings lead to different spectral behavior of the trained weight matrix (e.g., rank collapse, bulk+spikes, heavy-tailed distribution).", "questions": "1. What is $\\varepsilon$ in Figure 1? Figure 1 is nice and very important for the presentation. However, it is a little bit messy in the x and y axes. There is a line $\\lambda = \\sqrt{n_{\\text{eff}}/d}$, but when $n_{\\text{eff}} = 0$, why do you have $\\lambda = 1/d$? You should explain the red lines in the caption of the figure.\n\n2. For VIa and VIb, both risks decay fast, but the spectra of weights behave differently. One is bulk+spikes, and the other is heavy-tailed. Does this mean that we do not need heavy-tailed spectra to get a good generalization error? Martin & Mahoney, 2021a, and Martin et al., 2021, argued that networks with heavy-tailed spectra exhibit optimal generalization capabilities.\n\n3. In results 2, does the trained weight still have independent entries? $\\hat{\\theta}_i$ are independent for different $i$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, and its results are of interest to the deep learning theory community. To the best of my knowledge, this is the first paper that theoretically explains the relationship between the spectral behavior of learned weights and generalization errors, as well as the 5+1 phases for trained weights observed by Mahoney and Martin in 2019.", "weaknesses": "The neural network architectures are too specific. I understand that for general two-layer neural networks, the AMP is too complicated to analyze. However, it would be better to explain the difficulty, if there is any other simple architecture that can still work beyond quadratic or diagonal networks, and what the expected results are for general cases. It would be insightful to provide such information or discussion.\nAdditionally, there is a lack of training in dynamic analysis. This paper only used the state evolution to analyze the solution of the LASSO and matrix compressed sensing, but did not study the scaling of the training time or computational scaling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMgt3bV8eY", "reviewer": "Reviewer_d5QU", "summary": "This work extends neural scaling law theory  by analyzing quadratic and diagonal neural networks in the feature learning regime, using tools from matrix compressed sensing and LASSO to map out how excess risk scales with data, model size, and weight decay. The authors identify distinct scaling regimes, plateaus, and spectral signatures (like power-law tails in the weight distribution) and show how these phenomena predict generalization, providing a first-principles explanation for trends observed empirically in large neural networks and the correlation between power-law tails in the weight distribution and generalization in neural networks.", "questions": "In addition to the issues mentioned in the Weaknesses section, I may also have the following questions.\n\n1. In line 473, what does the first  $R$ represent? In addition, what is the relationship between the $R_{n,d}$ of ERM mentioned here (line 479) and the $R$ that appears in equations (11), (14), and (17)?\n\n2. There are also some papers [2][3][4][5][6][7][8][9][10] that discuss the relationship between the spectrum of the weights / features / data and generalization or neural scaling law. Could you provide some comments and discussion on these papers?\n\n\n\n\n[2] Bartlett, Peter L., et al. \"Benign overfitting in linear regression.\" Proceedings of the National Academy of Sciences 117.48 (2020): 30063-30070.\n\n[3] Simsekli, Umut, et al. \"Hausdorff dimension, heavy tails, and generalization in neural networks.\" Advances in Neural Information Processing Systems 33 (2020): 5138-5151.\n\n[4] Hodgkinson, Liam, et al. \"Generalization bounds using lower tail exponents in stochastic optimizers.\" International Conference on Machine Learning. PMLR, 2022.\n\n[5] Wang, Yutong, Rishi Sonthalia, and Wei Hu. \"Near-interpolators: Rapid norm growth and the trade-off between interpolation and generalization.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\n[6] Wang, Zhichao, et al. \"Spectral evolution and invariance in linear-width neural networks.\" Advances in neural information processing systems 36 (2023): 20695-20728.\n\n[7] Dandi, Yatin, et al. \"A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities.\" arXiv preprint arXiv:2410.18938 (2024).\n\n[8] Worschech, Roman, and Bernd Rosenow. \"Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra.\" arXiv preprint arXiv:2410.09005 (2024).\n\n[9]  Kothapalli, Vignesh, et al. \"From spikes to heavy tails: Unveiling the spectral evolution of neural networks.\" Transactions on Machine Learning Research (2025).\n\n[10] Arous, Gérard Ben, et al. \"Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws.\" arXiv preprint arXiv:2508.03688 (2025).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The paper gives a unified phase diagram of generalization error across sample size, width, and regularization for diagonal and quadratic networks (true feature learning, not just kernels).\n\n2. Exploring the theoretical relationship between the weight spectrum and generalization is an important open problem, and this paper provides an explanation for it in certain special neural network architectures. It analytically characterizes the trained weight spectra (bulk, spikes, heavy tails) in every regime and links these spectral features directly to underfitting, overfitting, and approximation error. \n\n3. It explains benign vs harmful overfitting and double-descent–like peaks from first principles, and shows how optimal regularization suppresses noisy bulk without destroying learned signal.", "weaknesses": "**If you address the issues in the Weaknesses and Question sections, I will increase the score.**\n\n1. The writing of the paper could be improved. Some of the notation is not explained very clearly (see  Questions sections), and also it seems the paper does not include a conclusion section.\n2. The current setting is limited to special two-layer neural networks, which allows the study of their excess risk minimizer to be reduced to matrix compressed sensing and Lasso problem. I would like to know whether, for more general two-layer (or even multi-layer) neural networks, there is a similar way to reduce the problem. If this is difficult, do you think it would be possible instead to study the training dynamics of the weight spectrum under (stochastic) optimizers in order to explore the relationship between heavy-tailed weight spectra and training dynamics?\n3. In the paper, you assume that the spectrum of the target network weights follows a power-law distribution (which is  heavy-tailed ). Is this assumption necessary[1]? and how much does this heavy-tailed structure affect the final weight spectrum of the student network?\n4. Minor: It looks like in Figure 3, the third plot in the first row should be labeled with the exponent $-2 + \\frac{1}{\\gamma}$.\n\n\n[1] Gurbuzbalaban, Mert, Umut Simsekli, and Lingjiong Zhu. \"The heavy-tail phenomenon in SGD.\" International Conference on Machine Learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ka4jPS9xRC", "reviewer": "Reviewer_jRAX", "summary": "This paper provides a theoretical framework for neural scaling laws in shallow neural networks that go beyond the random features (lazy training) regime by explicitly modeling feature learning. It focuses on two analytically tractable architectures:\n\n1) Diagonal networks — equivalent to LASSO (l1-regularized linear regression).\n\n2) Quadratic networks — equivalent to matrix compressed sensing (low-rank estimation with nuclear norm regularization).\n\nThrough these mappings, the authors derive:\n\nA complete phase diagram (Fig. 1, p. 6) of generalization excess risk as a function of sample size, model dimension and weight decay.\n\nA universal scaling law showing crossovers between plateaus, fast- and slow-decay regimes, and double-descent–like interpolation peaks.\n\nA connection between scaling phases and spectral properties of the learned weights, yielding a theoretical explanation for the empirically observed power-law weight spectra in modern deep networks.\n\nThe analysis combines approximate message passing (AMP), state evolution, and random matrix theory, yielding results that match both Bayes-optimal rates and extensive numerical simulations (Figs. 2–6, pp. 7–25).", "questions": "1) Can the spectral–generalization decomposition (Eq. 17) be extended to multi-layer or nonlinear activations?\n\n2) How sensitive are your phase boundaries to non-Gaussian inputs or correlated features?\n\n3) Is there a principled path toward proving the non-asymptotic AMP conjecture (Eq. 19)?\n\n4) Does the observed \\lambda^{-2/3} interpolation scaling have a connection to known double-descent exponents in overparameterized deep models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Novel analytical unification: \nThe paper builds an elegant bridge between feature-learning neural networks, LASSO, and matrix compressed sensing, allowing exact asymptotic characterization of generalization in previously intractable regimes.\n\nComprehensive phase diagram: Figure 1 neatly summarizes all scaling regimes and their transitions. It captures empirical phenomena (plateaus, overfitting peaks, fast decays) reported in large-scale scaling-law studies (Kaplan et al. 2020; Paquette et al. 2024).\n\nSpectral–generalization link: The “universal error decomposition” (Result 3) expresses the generalization error as an explicit functional of the weight eigenvalue spectrum, providing the first first-principles explanation of heavy-tailed spectra linked to generalization quality.", "weaknesses": "Restricted architecture scope: Results hold for shallow (two-layer) diagonal or quadratic networks; extensions to multilayer or non-polynomial activations remain speculative. It’s unclear how much insight transfers to modern deep transformers or CNNs.\n\nDependence on heuristic AMP validity: The central results (e.g., Eq. 19) assume non-asymptotic validity of state evolution, supported empirically but not proven. The authors acknowledge this (Sec. 2.4) as a conjecture, so the paper’s main theorems are partially non-rigorous.\n\nTarget-model assumptions: The teacher–student setup with Gaussian inputs and power-law eigenvalues is mathematically convenient but idealized. Real data distributions or nonlinear activations could alter scaling transitions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RFUAt5Q8D8", "reviewer": "Reviewer_XduY", "summary": "The paper obtains precise limits of the test loss and weight spectrum for two models of shallow (linear/quadratic) networks. A complete phase diagram is given as a function of the regularization intensity and effective sample size.", "questions": "- Do the L1 models that you consider outperform their L2 counterparts in general or in specific settings? And if so by how much? Since you seem to consider true functions that are not exactly sparse but only have a certain decay, I would imagine that the advantage of the L1 approach would appear for sufficiently steep decay of the true function spectrum?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper gives a very complete answer for the scaling laws of the regularized global minimum in these two simplified models. These networks implicitely add a L1-type of bias, comparison to previous work on random feature models/kernel methods which are in a L2 setting. The tools for the L2 setting are now well established (Random Matrix Theory etc), but the L1 case was to my knowledge less studied. \n\nThe paper is very clear, and the authors have done a great job of explaining the different phases and relate them to behavior of the weight spectrum, which helps a lot to not be lost in the many big formulas.", "weaknesses": "The paper only studies diagonal linear networks and quadratic networks with fixed second layer weights, which is basically a shallow fully-connected linear network with symmetric weights. These are very simple models, far from anything used in practice, but one has to start somewhere.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)