Title: The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural networks

quadratic activation

global minima

nuclear norm

regularisation
Abstract: We study the high-dimensional asymptotics of empirical risk minimization (ERM) in over-parametrized two-layer neural networks with quadratic activations trained on synthetic data. We derive sharp asymptotics for both training and test errors by mapping the $\ell_2$-regularized learning problem to a convex matrix sensing task with nuclear norm penalization.  This reveals that capacity control in such networks emerges from a low-rank structure in the learned feature maps. Our results characterize the global minima of the loss and yield precise generalization thresholds, showing how the width of the target function governs learnability. This analysis bridges and extends ideas from spin-glass methods, matrix factorization, and convex optimization and emphasizes the deep link between low-rank matrix sensing and learning in quadratic neural networks.

The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks

Abstract

We study the high-dimensional asymptotics of empirical risk minimization (ERM) in over-parametrized two-layer neural networks with quadratic activations trained on synthetic data. We derive sharp asymptotics for both training and test errors by mapping the ℓ 2 -regularized learning problem to a convex matrix sensing task with nuclear norm penalization. This reveals that capacity control in such networks emerges from a low-rank structure in the learned feature maps. Our results characterize the global minima of the loss and yield precise generalization thresholds, showing how the width of the target function governs learnability. This analysis bridges and extends ideas from spin-glass methods, matrix factorization, and convex optimization and emphasizes the deep link between low-rank matrix sensing and learning in quadratic neural networks.

Modern machine learning relies heavily on training highly over-parameterized neural networks, which often generalize well despite having far more parameters than data points [Zhang et al., 2021]. While it is known that large non-linear networks can approximate many functions [Cybenko, 1989], it remains unclear what these models actually learn in practice, and why training succeeds so often. In particular, we lack a precise understanding of how the structure of the data and the target function affects learnability, and how many samples are needed. Developing such an understanding remains a central theoretical challenge.

A promising route toward addressing these questions is to study models that go beyond the linear case-already well understood, eg. [Lee et al., 2018, Ji and Telgarsky, 2019, Pesme et al., 2021]-while remaining simple enough for rigorous analysis. Two-layer networks with quadratic activations are then the next natural candidate that captures some nonlinear behavior while still allowing for mathematical treatment [Gamarnik et al., 2019, Arjevani et al., 2025, Maillard et al., 2024]. On top of that working with synthetic data helps isolate the core mechanisms of learning and generalization [Mei et al., 2018, Aubin et al., 2020, Mei and Montanari, 2022, Xiao et al., 2022, Damian et al., 2024], free from the confounding factors of real-world datasets.

In this paper, we thus study learning by empirical risk minimization with quadratic networks from data that is also generated by a quadratic network. Consider a dataset D = { x µ , y µ } n µ =1 where the data x µ ∈ R d are standard Gaussian x µ ∼ N (0 , I d ) (though our results allow for some universality) for µ = 1 , . . . , n , and the labels y µ ∈ R are generated by an unknown target function f ⋆ ( x ) . We aim at learning this unknown function using a quadratic neural network ˆ f ( x ; W ) with a number of

hidden unit m ≥ d :

where σ k ( u ) = u 2 - || w k || 2 /d is the (centered) quadratic activation, and where we collect the first-layer weights w k ∈ R d for k = 1 , . . . , m in the matrix W ∈ R m × d . We learn W by empirical risk minimization of the square loss with ℓ 2 regularization (or equivalently, weight decay):

Given the structure of the model, the considered quadratic neural network can represent any centered positive semi-definite quadratic form of the input data-but no more. In particular, functions involving higher-order nonlinearities cannot be captured and are effectively treated as noise by the learner. To focus on the regime where generalization is possible, we therefore choose a target function f ⋆ that lies within the expressivity class of the model (1) (we will also refer to the model as the student while thinking about the target as the teacher ):

where ξ µ ∼ N (0 , 1) is an additional Gaussian label noise. As will shall see, our result will depend on W ⋆ only through the spectral density of S ⋆ = ( W ⋆ ) T W ⋆ / √ m ⋆ d ∈ R d × d , which needs to have a well-defined limit as d →∞ .

We will work in the high-dimensional limit d →∞ with extensive-width target and quadratically many samples, i.e. the joint limit d, n, m ∗ , m → + ∞ with

Our contributions. We provide an exact characterization of training and generalization in overparameterized two-layer neural networks with quadratic activations, in the high-dimensional limit with Gaussian data. Our main result (Theorem 1) gives closed-form expressions for the training loss, generalization error, and spectral properties of the global minima of the regularized empirical risk (2), in the regime κ = m/d ≥ 1 , κ ∗ = O (1) , and λ > 0 .

Our solution to this problem connects three previously distinct lines of research that turn out to be related: the geometry and training dynamics of quadratic networks in teacher-student setups [Sarao Mannelli et al., 2020, Gamarnik et al., 2019, Martin et al., 2024a, Arjevani et al., 2025], recent advances in high-dimensional Bayesian analysis of networks with extensive width [Maillard and Bandeira, 2023, Maillard et al., 2024], and the role of implicit regularization in matrix factorization [Gunasekar et al., 2017] and its connection to matrix compressed sensing and nuclear norm regularization [Recht et al., 2010, Fazel et al., 2008].

Concretely, we map the non-linear estimation problem for W in (2) to the linear one of estimating the matrix S = W ⊤ W/ √ md ∈ R d × d , where, remarkably, the ℓ 2 regularization on W translates into a nuclear norm regularization on S . This reveals that the learning dynamics implicitly favor solutions ˆ f corresponding to narrow neural networks. We then study this equivalent matrix model by rigorous tools based on approximate message passing [Berthier et al., 2020, Gerbelot and Berthier, 2023] and their relation to convex optimization [Loureiro et al., 2021b]. Our main result is Theorem 1, an analytical prediction for the test error achieved by the ERM, where importantly the error does not depend on the value of κ as long as κ > 1 , thus describing potentially massively over-parametrized models. Theorem 1 allows us to answer a range of questions as a function of κ ∗ (see Section 3).

Location of the interpolation threshold: How many samples n are needed to have only a unique global minimum of (2) in the limit λ = 0 + , both in the noiseless and noisy case?

Generalization performance: How many samples n are needed to achieve perfect generalization with zero label noise?

Low-rank limit: What is the generalization performance in the limit κ ∗ ≪ 1 corresponding to low-rank target functions?

In this paper, we provide new sharp results focusing on the extensive width case, encompassing the full range of target widths and focusing in particular on the setting 0 < κ ∗ < 1 , previously unexplored.

Finally, we remark that while (2) is a priori non-convex, it has no non-global local minima [Arjevani et al., 2025], implying that if a gradient-based algorithm converges to a minimum at all, it must converge to a global one. Hence, our main result Theorem 1 provides a closed-form characterization of the behavior at convergence to minima of gradient-based algorithms for any strictly positive regularization λ .

Further related work. Deriving exact formulas for two-layer networks in the teacher-student setting in high dimensions is a classical problem that has been extensively studied over the past decades (e.g., [Gardner and Derrida, 1989, Bayati et al., 2010, Thrampoulidis et al., 2018, Barbier et al., 2019, Montanari et al., 2023]). The case of a single hidden unit, or single-index model, has been widely explored and, for quadratic activations, reduces to phase retrieval [Mondelli and Montanari, 2018, Lu and Li, 2020, Maillard et al., 2020]. The case of a finite number of hidden units falls into the broader class of multi-index models, which has seen a surge of recent interest [Seung et al., 1992, Aubin et al., 2018, Damian et al., 2022, Bietti et al., 2025, Troiani et al., 2025, Bruna and Hsu, 2025, Montanari and Urbani, 2025]. When the number of directions becomes extensive (i.e., of order O ( d ) ) and for general activations, the problem has been studied for linearly-many samples n = O ( d ) [Li and Sompolinsky, 2021, Cui et al., 2025, Pacelli et al., 2023, Bordelon and Pehlevan, 2023, Camilli et al., 2025]. It becomes significantly more challenging in the setting we consider here of quadratically-many samples n = O ( d 2 ) . This more difficult regime has been explored empirically [Cui et al., 2025] and heuristically [Barbier et al., 2025] in a few specific settings, but an asymptotically exact sharp characterization is still lacking.

Significant progress, however, has recently been made for the quadratic networks in the Bayesoptimal, information-theoretic setting . In particular, [Maillard et al., 2024] computed the Bayesoptimal performance in the setting considered here, a result later rigorously proven in [Xu et al., 2025]. These works build upon recent advances originating in the context of ellipsoid fitting [Maillard and Kunisky, 2024, Maillard and Bandeira, 2023], and leverage a Gaussian universality principle that also underpins our approach [Goldt et al., 2022, Hu and Lu, 2022, Montanari and Saeed, 2022, Dubova et al., 2023]. This allows to study the problem as a matrix denoising task [Bun et al., 2014, Troiani et al., 2022, Pourkamali and Macris, 2023a,b, Semerjian, 2024]. For empirical minimization instead, landscape properties have been studied in [Du and Lee, 2018, Venturi et al., 2019, Soltanolkotabi et al., 2019, Gamarnik et al., 2019, Sarao Mannelli et al., 2020, Arjevani et al., 2025], and gradient descent on the population loss has been discussed in [Martin et al., 2024b].

The concept of double descent (see e.g. [Belkin et al., 2019, Nakkiran et al., 2021]) has reshaped our understanding of the bias-variance trade-off, revealing that increasing model complexity can, counterintuitively, improve generalization. While many of these analyses have focused on linear or kernel regimes [Gerace et al., 2020, Geiger et al., 2020, Belkin et al., 2020, Mei and Montanari, 2022], our work instead provides sharp asymptotics for non-linear two-layer nets.

Our analysis for two-layer networks hinges on a mapping to a matrix compressed sensing, or low rank-matrix recovery , problem with nuclear norm regularization [Fazel et al., 2008]. The number of samples required for perfect recovery has been discussed in [Donoho et al., 2013, Oymak and Hassibi, 2016, Amelunxen et al., 2014]. Additionally, we leverage tools from Approximate Message Passing (AMP) theory. AMP has become a tool of choice for the rigorous analysis of statistical inference problems and algorithms in high-dimension. It often allows to provides sharp asymptotic characterizations of both Bayesian and empirical risk minimization procedures in many models [Bayati and Montanari, 2011, Donoho and Montanari, 2016, Rangan et al., 2016, Loureiro et al., 2021b, Vilucchio et al., 2025]. In this work, we use AMP with non-separable denoisers [Berthier et al., 2020, Gerbelot and Berthier, 2023] to prove rigorously our main formula, that can also be derived by the heuristic replica method, a non-rigorous technique from statistical physics [Mézard et al., 1987, Mezard and Montanari, 2009].

1 Setting and notations

We consider a dataset of n samples D = { x µ , y µ } n µ =1 , with x µ ∈ R d and y µ ∈ R , constructed as in (1). We remark that the assumption of Gaussian data could be relaxed in the same spirit as in [Xu

et al., 2025, Assumption 2.2], and denote by E the average of the training set. We will assume that the empirical spectral density of the matrix S ∗ = ( W ∗ ) T W ∗ / √ m ∗ d ∈ R d × d converges to a limiting distribution µ ∗ with finite first and second moment as d →∞ with κ ∗ = m ∗ /d = O (1) , and call Q ∗ its second moment. As our running example, we will focus on the Marchenko-Pastur (MP) target case, where the weights w ∗ are such that the limiting distribution satisfies µ ∗ ( x ) = √ κ ∗ µ M . P . ( √ κ ∗ x ) , where µ M . P . is the Marchenko-Pastur distribution [Marchenko and Pastur, 1967] with parameter κ ∗ (the asymptotic spectral distribution of A T A/m where A ∈ R m × d has i.i.d. zero-mean unit-variance components), and Q ∗ = 1+ κ ∗ . For example, this is the case for w ∗ with i.i.d. components extracted from a distribution with zero mean and unit variance, but we stress that our results hold also for deterministic targets, as well as targets with different spectral distributions.

We learn the dataset by empirical risk minimization on the loss (2), and unless stated otherwise, in this paper we will always consider learning with m ≥ d , typically for m>m ∗ . We will measure the performance of the empirical risk estimator using the test error on the labels

where the average is over a new test sample x with the same distribution as the training samples (notice that for the sake of the test error we do not add any label noise on the test label). The Bayesoptimal (BO) test error, i.e. the minimum test error achievable by any estimator (on average over the joint realization of the dataset and the target weights) is a natural baseline. In the high-dimensional limit (4), the BO test error has been characterized in [Maillard et al., 2024]. We will always consider the high-dimensional limit (4).

2 Main theorem

Our main technical result is the characterization of the properties of the global minima of the empirical loss (2) in the high dimensional limit in terms of training and test error.

Theorem 1 (Asymptotics of ERM (2) , informal) Consider the setting of Section 1 with κ ≥ 1 . Define ˜ λ = √ κλ and µ ∗ δ = µ ∗ ⊞ µ s . c .,δ , where ⊞ is the free convolution and µ s . c .,δ = √ 4 δ 2 -x 2 / (2 πδ 2 ) the semicircle distribution of radius 2 δ for δ > 0 . Call ( ¯ δ, ¯ ϵ ) ∈ R 2 + the unique solution of

Then, for all values of α, κ ∗ , λ > 0 , ∆ ≥ 0 and κ ≥ 1 any global minimum ˆ W of (2) satisfies

Moreover, if µ ∗ has compact support, then the empirical singular value density of ˆ W/ 4 √ md satisfies

where { σ i } d i =1 are the singular values of ˆ W/ 4 √ md , I is the indicator function and F δ is the c.d.f. of µ ∗ δ .

Sketch of the proof. The proof, detailed in Appendix A, proceeds as follows. We use a reduction of a regularized version of the minimization (2) to the one of a positive semi-definite matrix estimation:

where we defined S ∗ = W ∗ ( W ∗ ) T / √ m ∗ d and X ( x ) = ( xx T -I d ) / √ d , and where we have added a Frobenius norm penalty of amplitude τ . Notice that (9) is strongly convex for λ, τ > 0 , so that

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Sample complexity

α

=

n/d

2

10

-

3

10

-

2

10

-

1

10

0

Test error

λ

= 0

.

4

λ

= 0

.

2

λ

= 0

.

1

λ

= 0

.

05

λ

= 0

.

025

λ

= 0

.

01

λ

= 0

+

Bayes optimal

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Sample complexity

α

=

n/d

2

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

Theory

CVX (d = 50)

CVX (d = 100)

LBFGS (d = 300)

ˆ S is unique. Under the mapping S ( W ) = WW T / √ md , we have ˆ S = S ( ˆ W ) for any (Frobenius regularized) solution of (2), due to the uniqueness of ˆ S . The second part is to use the Gaussian universality principle, that allows to replace each matrix ( xx T -I d ) / √ d by a random Wigner matrix, following closely the steps of [Maillard et al., 2024, Xu et al., 2025]. This in turns reduces the problem to a rank-penalized matrix recovery problem with GOE( d ) sensing matrices:

The problem can then be studied in various way. For instance with the heuristic, but powerful, replica method technique from statistical physics (see for instance [Maillard and Kunisky, 2024]). To provide a rigorous approach, we use instead a suitable Approximate Message Passing (AMP) algorithm with non-separable prior [Berthier et al., 2020, Gerbelot and Berthier, 2023], designed in such a way as its (unique) non-trivial fixed point is also the fixed point of projected gradient descent on (9) [Bayati et al., 2010, Montanari et al., 2012, Loureiro et al., 2021b], which by the convexity of (9) coincide with the unique ˆ S . We then write the state evolution of this AMP algorithm which gives us all the relevant characteristics of the minimizer. Simple manipulations of the state evolution equations, in the limit τ → 0 , lead to the characterization (6). □

Theorem 1 provides an asymptotic result for train and test error, as well as a characterization of the singular values of the optimal neural network weights. It is of independent interest for the matrix compressed sensing problem (10) and extends directly to any strictly convex matrix problem .

A remarkable observation is that the ℓ 2 regularization over the weights W naturally translates to a nuclear norm regularization in the equivalent matrix problem (the convexification of the minimum rank regularization), naturally favoring model weights configurations with an effective lower width (i.e., implementable with fewer hidden units): a weight decay in W thus implies a low nuclear norm of the matrix S .

Theorem 1 also implies that the properties of the global minima do not depend on the network width m , as long as m ≥ d . This means that a neural network with width κ ≫ 1 will achieve the same test error as a much narrower network with κ = 1 , when trained on the same data, provided that the regularization strength is appropriately matched. Theorem 1 thus described both mildly and massively over-parametrized models since for very large κ the number of learnable parameters will be not only much larger than the target function, but also can be much larger than the number of samples.

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Singular values of

ˆ

W/

4

√

md

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Singular value density

α

= 0

.

3

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Singular values of

ˆ

W/

4

√

md

α

= 0

.

55

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Singular values of

ˆ

W/

4

√

md

α

= 1

We remark that our results can be extended to the setting where the second layer weights of the student are binary ± 1 with at least d positive and d negative weights, see Appendix G.

Illustration of Theorem 1 and the behavior of gradient descent. In Figure 1 we compare the asymptotic results of Theorem 1 with numerical experiments at finite size run directly on the equivalent convex loss (9) (using the solver CVXPY [Diamond and Boyd, 2016, Agrawal et al., 2018]). We notice that despite the theoretical results being valid in the high-dimensional limit, they are in excellent agreement with simulation at sizes as moderate as d = 50 . Details on the numerical experiments are given in Appendix F.

The mapping onto a matrix problem implies immediately that Theorem 1 describes also the performance of gradient descent at convergence. Indeed, [Arjevani et al., 2025, Theorem 12] implies that if m ≥ d , then the original minimization problem (2) has no local minima (in the language of [Arjevani et al., 2025], it has no spurious valleys, which implies in our case absence of local minima). We verify this claim experimentally at finite size by running gradient descent (GD)

initialized as w t =0 ki ∼ N (0 , ζ 2 ) independently, with ˆ w GD = w T , where T = 10 4 is a fixed stopping time (we check that by time T GD has convincingly reached convergence) and η a tuned learning rate (in Figure 1 right we used LBFGS for better convergence). We present our experiments in Figure 1. Again, we observe a very nice agreement, surprisingly precise in the region of small test error (notice the log-scale on the vertical axis in the left panel). We further test our results by comparing the empirical spectra with our theoretical prediction of equation (8) in Figure 2, observing a nice match.

Relation with kernel ridge regression. While the equivalent model (9) is linear in the quadratic feature space x → x i x j , out result departs significantly from the analysis of kernel ridge regression, random features regression and other linear methods studied in the literature (see e.g. [Gerace et al., 2020, Bordelon et al., 2020, Loureiro et al., 2021a, Mei and Montanari, 2022]). The crucial difference is in the regularization, which in (9) is a non-separable regularization including the positive semi-definite constraint , that takes into account the full spectral distribution of the target µ ∗ , while in the literature only the case of ℓ 2 regularization in feature space was usually studied explicitly (a separable regularization that takes into account only the second moment of the target Q ∗ , disregarding for example the low-width of the target function). Among the consequences of this, we find an interpolation threshold that, unexpectedly, depends on the structure of the target function (Result 1).

Narrow students κ ∗ ≤ κ < 1 . We remark that Theorem 1 applies as is also to models with m<d for certain values of the parameters, i.e. for all choices of α, κ ∗ , λ, ∆ such that m/d = κ > 1 -F ¯ δ (¯ ϵ ) . This is a consequence of the fact that the global minima of (2) for κ ≥ 1 have rank F ¯ δ (¯ ϵ ) (see (8)), and thus are also global minima of the (2) for κ < 1 under the above condition. Lower than this value of κ , the rank constraint imposed by the structure of the student network affects the global minima of the unconstrained (9), leading to non-convexity and to the failure of Theorem 1.

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Sample complexity

α

=

n/d

2

0

.

0

0

.

4

0

.

8

1

.

2

Test error (∆ = 0)

Optimal

λ

λ

= 0

.

4

λ

= 0

+

Bayes

optimal

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Sample complexity

α

=

n/d

2

1

2

3

4

5

6

Test error (∆ = 0

.

5)

λ

= 0

.

1

λ

= 0

.

05

λ

= 0

.

02

λ

= 0

.

01

λ

= 0

.

005

λ

= 0

.

001

λ

= 0

.

0001

λ

= 0 +

Optimal

λ

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Sample complexity

α

=

n/d

2

0

.

00

0

.

05

0

.

10

0

.

15

Train loss (∆ = 0

.

5)

3 Main results and consequences of Theorem 1

In this section we focus specifically on the Marchenko-Pastur target case (see Section 1, details on the computation of µ ∗ δ in Result 1 are given in Appendix E). We believe that our results generalize (qualitatively for the learning curves, and as is for the thresholds and the low-rank target limit) to any target such that µ ∗ has a finite spectral gap, i.e. such that λ min = min { x ∈ supp( µ ∗ ) | x > 0 } satisfies λ min > 0 , with mass in zero equal to max(0 , 1 -κ ∗ ) (i.e. rank of the associated target matrix ⌊ min( κ ∗ , 1) d ⌋ ) and second moment Q ∗ = 1 + κ ∗ .

Learning curves. In Figure 3 we show the test and train error of any global minimum of the loss function (2) given by Theorem 1, both the noiseless ( ∆ = 0 ) and noisy ( ∆ = 0 . 5 ) scenarios. Details on the numerical experiments are given in Appendix E.

In the noiseless case (Figure 3 left), we plot the test error as functions of the sample ratio α for a finite regularization value ( λ = 0 . 4 , blue line), in the limit of vanishing regularization ( λ → 0 + , yellow line), and for the optimal choice of regularization (dashed line). These are contrasted with the Bayes-optimal performance as derived in [Maillard et al., 2024] (purple line), clearly illustrating the effect of regularization on the perfect recovery threshold, marked by the vertical gray line as per Corollary 1.

In the noisy case (Figure 3 center and right), we show both the test and train losses as functions of the sample ratio α for multiple values of λ (solid lines), including the minimum regularization interpolator limit ( λ → 0 + , yellow line) and the optimally regularized scenario (dashed line). Notably, we highlight the specific region of α where the non-regularized training loss reaches zero (vertical gray line, as per Result 1), which coincides with the emergence of a cusp - the double descent peak [Belkin et al., 2019, Nakkiran et al., 2021] - in the test error as λ decreases.

Interpolation threshold. We define the interpolation threshold α inter ( κ ∗ , ∆) as the value of the sample ratio at which the set of PSD matrices S for which the loss (9) is zero for λ = 0 (we call those the interpolators of the training dataset) shrinks to a single point. For α > α inter ( κ ∗ , ∆) , (9) thus has a single global minimum with positive training loss if ∆ > 0 , or with both zero training and test loss if ∆ = 0 . For α < α inter ( κ ∗ , ∆) instead, the non-regularized λ = 0 loss admits many global minima (all PSD interpolators), while the limit λ → 0 + has still a single global minimum with zero training loss, and minimum value of the regularization norm. While we stated Theorem 1 for λ > 0 it also holds, at least at an heuristic level, for α > α inter ( κ ∗ , ∆) and λ = 0 by directly plugging λ = 0 in (6). The theorem does not hold for λ = 0 and α < α inter ( κ ∗ , ∆) while it can be adapted to the limit λ → 0 + by a careful rescaling, which we comment upon in Appendix B. A

direct consequence of Theorem 1 is a theoretical characterization of the position of the interpolation threshold α inter ( κ ∗ , ∆) .

Result 1 (Interpolation threshold) Consider the setting of Section 1 for a Marchenko-Pastur target. Then, the interpolation threshold α inter ( κ ∗ , ∆) satisfies

where ∂ 1 denotes derivative w.r.t. the first argument, and J was defined in (6) . Additionally, we have

Result 1 is derived by considering the solution of the minimization problem (6) and setting λ = 0 directly at large α - where there is a unique minimum of (9) - and then looking for the smallest value of α consistent with a unique minimum assumption. Equivalently, this is when the asymptotics training loss reach zero (see Appendix C). We plot (13) in Figure 4. As discussed above, the interpolation threshold does not depend on the width of the student.

It is interesting to contrast our results with the ellipsoid-fitting problem , where one seeks a PSD matrix S such that a dataset of points { x µ } n µ =1 lies on the surface of the associated ellipsoid. This problem, recently studied in a high-dimensional limit similar to (4) [Maillard and Kunisky, 2024, Maillard and Bandeira, 2023], predicts that fitting is possible as long as α < 1 / 4 . We recover this same threshold, α inter = 1 / 4 , in two extreme cases: when noise dominates ( ∆ → + ∞ ) or when the target is vanishingly small ( ∆ = 0 , κ ∗ → 0 + ). In all other scenarios, however, the interpolation threshold is strictly larger, reflecting the structural advantage of the data over random labeling. Unlike linear ridge regression or kernel methods, the interpolation threshold here is nontrivial and intricately linked to data structure. Notably, it is not simply determined by the ratio of samples to parameters. Despite the effective number of parameters scaling as O ( d 2 / 2) , interpolation can occur well before this count if the target function is sufficiently narrow ( 0 < κ ∗ < 1 ). This effect arises from the PSD constraint in the equivalent matrix problem (9), marking a fundamental departure from classic kernel theory.

We note that special cases this threshold has been considered in the literature, for κ ∗ ≥ 1 and ∆ = 0 in [Gamarnik et al., 2019], and our result agrees with these works. An analogue of this threshold (still for ∆ = 0 ) has been considered in [Sarao Mannelli et al., 2020] for the case of low-width target m ∗ = 1 , where the interpolation for the model with κ ≥ 1 happens at n = 2 d . The authors conjecture without a solid theoretical justification that for generic m ∗ the interpolation happens at n = d ( m ∗ +1) -m ∗ ( m ∗ +1) / 2 , which, however, is not compatible with our results. We conclude that their conjecture is incorrect in the regime m ∗ = O d ( d ) , as it predicts interpolation for values lower than the rigorous 1 / 4 random labels threshold [Maillard and Bandeira, 2023, Maillard and Kunisky, 2024] as well.

Strong recovery threshold. Adifferent phenomenon can be studied in the noiseless case ∆ = 0 and λ → 0 + , i.e. what we call the strong recovery : the value of the sample ratio α strong ( κ ∗ ) = n strong /d 2 such that, for α > α strong the test error of the global minima of (2) is zero, and for α < α strong is strictly positive at ∆ = 0 . Notice the difference with the previous paragraph, where the interpolation peak was concerning the training error vanishing in the ∆ > 0 case, while we here consider the perfect test error at ∆ = 0 . This limit was studied in the context of matrix model (9) in the minimalnorm interpolation setting [Donoho et al., 2013, Amelunxen et al., 2014, Oymak and Hassibi, 2016]. Here we re-derive their result independently as a consequence of Theorem 1.

Corollary 1 (Strong recovery threshold) Consider the setting of Section 1 for Marchenko-Pastur targets with ∆ = 0 , λ → 0 + and κ ≥ 1 . Define for x ∈ [ -2 , 2] the incomplete moments of the semicircle distribution

and call ¯ c the solution of the equation

10

-

2

10

-

1

10

0

10

1

10

2

Target width

κ

∗

0

.

00

0

.

25

0

.

50

Interpolation

α

inter

(

κ

∗

,

∆)

∆ = 0

∆ = 0

.

01

∆ = 0

.

1

∆ = 0

.

5

∆ = 1

0

.

0

0

.

5

1

.

0

Target width

κ

∗

Thresholds at ∆ = 0

ERM Str.

Interp.

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

Rescaled sample complexity ¯

α

=

n/

(

dm

∗

)

10

-

2

10

-

1

10

0

10

1

Test error for

κ

∗

→

0

BO, ∆ = 0

∆ = 0,

λ

= 0

+

∆ = 0

, λ

= 1

∆ = 0

.

005

, λ

= 1

∆ = 0

.

01

, λ

= 1

for 0 < κ ∗ < 1 . Then,

Informally, the strong recovery threshold presented in Result 1 is derived by expanding (6) for δ → 0 + within an appropriate scaling ansatz, as the test error is proportional to δ 2 (Theorem 1, noiseless case ∆ = 0 ), and finding the value α = α strong such that the expansion is consistent. We provide all the details in Appendix C. We plot (16) in Figure 4, comparing with the strong recovery threshold of the BO estimator [Maillard et al., 2024] and with the interpolation threshold Result 1 for ∆ = 0 .

Small target rank limit. Finally, we consider the small target width limit κ ∗ → 0 + for κ bounded away from zero. This is an example of large over-parametrization, as the ratio between student and target widths m/m ∗ diverges. In the limit κ ∗ → 0 , i.e. m ∗ ≪ d , we define the rescaled sample ratio ¯ α = α/κ ∗ = n/ ( m ∗ d ) . The natural baseline to compare this limit against is given by the analogue limit for the test error of the BO estimator given in [Maillard et al., 2024].

Result 2 (Test error in the κ ∗ → 0 limit) Consider the setting of Section 1. Consider the limit κ ∗ → 0 + for any fixed κ > 0 , ¯ α = α/κ ∗ and ¯ λ = λ √ κ ∗ . Then, if ∆ = 0 and ¯ λ → 0 + the test error satisfies

In Appendix D we provide analogous expressions, albeit less explicit, for generic regularization ¯ λ > 0 and noise ∆ ≥ 0 . We just state that for all ¯ λ > 0 and ∆ ≥ 0 , and for α < ¯ α weak ( ¯ λ, ∆) the test error equals one, where

and that for all ¯ λ > 0 and ∆ > 0 and for ¯ α → + ∞ he have

Result 2 provides a closed-form expression for the test error in the κ → 0 limit. It is derived by a direct expansion of (6) at small κ , and it involves a BBP-like phase transition [Baik et al., 2005] in the spectrum of ˆ W . We notice that Result 2 holds for any extensive student width κ > 0 due to the fact that the rank of the global minima of (2) is vanishing as O ( κ ∗ ) .

̸

Figure 4 right shows the test error in the κ ∗ → 0 limit for a selection of parameters. We see that the solution at λ, ∆ = 0 as a non-trivial behavior, tracking closely but not exactly the λ → 0 + , ∆ = 0 curve until an angular point is reached. We observe that for ¯ α < ¯ α weak ( ¯ λ, ∆) , the empirical risk minimizer achieves BO performance, even though that is given by the trivial prior-average estimator. Moreover, for ∆ = 0 and ¯ λ → 0 + , the empirical risk minimizer achieves strong recovery for ¯ α = 3 in accordance with Corollary 1. Finally, we remark that for fixed ¯ λ > 0 and ∆ > 0 , the test error goes to zero as 1 /α , which is the Bayes-optimal rate.

4 Discussion and limitations

In this paper, we presented a closed-form asymptotic characterization of learning a one hidden-layer quadratic neural network target with an over-parametrized architecture, in the challenging highdimensional regime of extensive widths m ∗ = O ( d ) target functions and quadratically-many samples n = O ( d 2 ) . We presented the asymptotic learning curves and values of relevant thresholds, as well as numerical experiments (including gradient descent algorithms) illustrating that our theory applied also accurately to very moderate sizes.

The main limitations of our setting are the restriction to random input data (which can be relaxed to a certain extent, see [Xu et al., 2025] for a theoretical direction hinging on Gaussian universality, or Loureiro et al. [2021b] for an empirical direction modeling real data by matching to Gaussian mixtures with appropriate first and second moments), the choice of quadratic activation, the single hidden-layer architecture and the requirement that the width of the target function satisfies m>d (imposing no non-convex rank constraint in (9)). All such elements present highly non-trivial technical challenges, and we hope our work will spark progress in these directions.

Acknowledgments and Disclosure of Funding

We thank Yatin Dandi, Cedric Gerbelot, Antoine Maillard, Simon Martin, Yizhou Xu and Nathan Srebro for insightful discussions. This work was supported by the Swiss National Science Foundation under grants SNSF SMArtNet (grant number 212049), SNSF OperaGOST (grant number 200390), and SNSF DSGIANGO (grant number 225837).

References

Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Amelunxen, M. Lotz, M. B. McCoy, and J. A. Tropp. Living on the edge: phase transitions in convex programs with random data. Information and Inference , 3(3):224-294, September 2014. ISSN 2049-8764, 2049-8772.

Yossi Arjevani, Joan Bruna, Joe Kileel, Elzbieta Polak, and Matthew Trager. Geometry and optimization of shallow polynomial networks. arXiv preprint arXiv:2501.06074 , 2025.

Benjamin Aubin, Antoine Maillard, Florent Krzakala, Nicolas Macris, Lenka Zdeborová, et al. The committee machine: Computational to statistical gaps in learning a two-layers neural network. Advances in Neural Information Processing Systems , 31, 2018.

Benjamin Aubin, Florent Krzakala, Yue Lu, and Lenka Zdeborová. Generalization error in highdimensional perceptrons: Approaching bayes error with convex optimization. Advances in Neural Information Processing Systems , 33:12199-12210, 2020.

Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Annals of Probability , pages 1643-1697, 2005.

̸

Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences , 116(12):5451-5460, 2019.

Jean Barbier, Francesco Camilli, Minh-Toan Nguyen, Mauro Pastore, and Rudy Skerk. Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation, January 2025.

Mohsen Bayati and Andrea Montanari. The lasso risk for gaussian matrices. IEEE Transactions on Information Theory , 58(4):1997-2017, 2011.

Mohsen Bayati, José Pereira, and Andrea Montanari. The lasso risk: asymptotic results and real world examples. Advances in Neural Information Processing Systems , 23, 2010.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

Raphaël Berthier, Andrea Montanari, and Phan-Minh Nguyen. State evolution for approximate message passing with non-separable functions. Information and Inference: A Journal of the IMA , 9(1):33-79, March 2020. ISSN 2049-8764, 2049-8772.

Alberto Bietti, Joan Bruna, and Loucas Pillaud-Vivien. On learning gaussian multi-index models with gradient flow part i: General properties and two-timescale learning. Communications on Pure and Applied Mathematics , 78(12):2354-2435, 2025. doi: https://doi.org/10.1002/cpa.70006.

Erwin Bolthausen. An iterative construction of solutions of the tap equations for the sherringtonkirkpatrick model. Communications in Mathematical Physics , 325(1):333-366, 2014.

Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Journal of Statistical Mechanics: Theory and Experiment , 2023(11): 114009, 2023.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024-1034. PMLR, 2020.

Joan Bruna and Daniel Hsu. Survey on algorithms for multi-index models. arXiv preprint arXiv:2504.05426 , 2025.

Bun, J. P. Bouchaud, S. N. Majumdar, and M. Potters. Instanton approach to large n harish-chandraitzykson-zuber integrals. Phys. Rev. Lett. , 113:070201, Aug 2014.

Francesco Camilli, Daria Tieplova, Eleonora Bergamin, and Jean Barbier. Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime. arXiv preprint arXiv:2505.03577 , 2025.

Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. On double-descent in uncertainty quantification in overparametrized models. In International Conference on Artificial Intelligence and Statistics , pages 7089-7125. PMLR, 2023.

Hugo Cui, Florent Krzakala, and Lenka Zdeborová. Bayes-optimal learning of deep random networks of extensive-width*. Journal of Statistical Mechanics: Theory and Experiment , 2025(1):014001, January 2025. ISSN 1742-5468. Publisher: IOP Publishing.

George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989.

Alex Damian, Loucas Pillaud-Vivien, Jason Lee, and Joan Bruna. Computational-statistical gaps in gaussian single-index models. In The Thirty Seventh Annual Conference on Learning Theory , pages 1262-1262. PMLR, 2024.

Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory , pages 5413-5452. PMLR, 2022.

Yatin Dandi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro, and Lenka Zdeborová. Universality laws for gaussian mixtures in generalized linear models. Advances in Neural Information Processing Systems , 36:54754-54768, 2023.

Jairo RL de Almeida and David J Thouless. Stability of the sherrington-kirkpatrick solution of a spin glass model. Journal of Physics A: Mathematical and General , 11(5):983, 1978.

C De Dominicis and Imre Kondor. Eigenvalues of the stability matrix for parisi solution of the long-range spin-glass. Physical Review B , 27(1):606, 1983.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

David Donoho and Andrea Montanari. High dimensional robust m-estimation: Asymptotic variance via approximate message passing. Probability Theory and Related Fields , 166:935-969, 2016.

David L. Donoho, Matan Gavish, and Andrea Montanari. The phase transition of matrix recovery from gaussian measurements matches the minimax mse of matrix denoising. Proceedings of the National Academy of Sciences , 110(21):8405-8410, May 2013. ISSN 1091-6490.

Simon Du and Jason Lee. On the power of over-parametrization in neural networks with quadratic activation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1329-1338. PMLR, 10-15 Jul 2018.

Sofiia Dubova, Yue M Lu, Benjamin McKenna, and Horng-Tzer Yau. Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime. arXiv preprint arXiv:2310.18280 , 2023.

Maryam Fazel, Emmanuel Candes, Ben Recht, and Pablo Parrilo. Compressed sensing and robust recovery of low rank matrices. In 2008 42nd Asilomar Conference on Signals, Systems and Computers , pages 1043-1047. IEEE, 2008.

Oliver Y. Feng, Ramji Venkataramanan, Cynthia Rush, and Richard J. Samworth. A Unifying Tutorial on Approximate Message Passing. Foundations and Trends® in Machine Learning , 15(4):335-536, 2022. ISSN 1935-8237, 1935-8245.

David Gamarnik, Eren C Kızılda˘ g, and Ilias Zadik. Stationary points of shallow neural networks with quadratic activation function. arXiv preprint arXiv:1912.01599 , 2019.

Elizabeth Gardner and Bernard Derrida. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General , 22(12):1983, 1989.

Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment , 2020(2):023401, 2020.

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning , pages 3452-3462. PMLR, 2020.

Cédric Gerbelot and Raphaël Berthier. Graph-based approximate message passing iterations. Information and Inference: A Journal of the IMA , 12(4):2562-2628, 2023.

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning , pages 426-471. PMLR, 2022.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems , 30, 2017.

Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69(3):1932-1964, 2022.

Adel Javanmard and Andrea Montanari. State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference: A Journal of the IMA , 2(2):115-144, 2013.

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In 7th International Conference on Learning Representations, ICLR 2019 , 2019.

Florent Krzakala and Lenka Zdeborová. Statistical physics methods in optimization and machine learning. Lecture Notes , 2024.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations , 2018.

Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Phys. Rev. X , 11:031059, Sep 2021.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems , 34:18137-18151, 2021a.

Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions. Advances in Neural Information Processing Systems , 34:10144-10157, 2021b.

Yue M Lu and Gen Li. Phase transitions of spectral initialization for high-dimensional non-convex estimation. Information and Inference: A Journal of the IMA , 9(3):507-541, 2020.

Antoine Maillard and Afonso S Bandeira. Exact threshold for approximate ellipsoid fitting of random points. arXiv preprint arXiv:2310.05787 , 2023.

Antoine Maillard and Dmitriy Kunisky. Fitting an ellipsoid to random points: predictions using the replica method. IEEE Transactions on Information Theory , 70(10):7273-7296, 2024.

Antoine Maillard, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Phase retrieval in high dimensions: Statistical and computational phase transitions. Advances in Neural Information Processing Systems , 33:11071-11082, 2020.

Antoine Maillard, Emanuele Troiani, Simon Martin, Lenka Zdeborová, and Florent Krzakala. Bayesoptimal learning of an extensive-width neural network from quadratically many samples. Advances in Neural Information Processing Systems , 37:82085-82132, December 2024.

VA Marchenko and Leonid A Pastur. Distribution of eigenvalues for some sets of random matrices. Mat. Sb.(NS) , 72(114):4, 1967.

Simon Martin, Francis Bach, and Giulio Biroli. On the impact of overparameterization on the training of a shallow neural network in high dimensions. In International Conference on Artificial Intelligence and Statistics , pages 3655-3663. PMLR, 2024a.

Simon Martin, Francis Bach, and Giulio Biroli. On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages 3655-3663. PMLR, April 2024b.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75 (4):667-766, 2022.

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of twolayer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018.

Marc Mezard and Andrea Montanari. Information, physics, and computation . Oxford University Press, 2009.

Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications , volume 9. World Scientific Publishing Company, 1987.

Leon Mirsky. A trace inequality of john von neumann. Monatshefte für mathematik , 79(4):303-306, 1975.

Marco Mondelli and Andrea Montanari. Fundamental limits of weak recovery with applications to phase retrieval. In Conference On Learning Theory , pages 1445-1450. PMLR, 2018.

Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. In Conference on Learning Theory , pages 4310-4312. PMLR, 2022.

Andrea Montanari and Pierfrancesco Urbani. Dynamical decoupling of generalization and overfitting in large two-layer networks. arXiv preprint arXiv:2502.21269 , 2025.

Andrea Montanari, YC Eldar, and G Kutyniok. Graphical models concepts in compressed sensing. Compressed Sensing , pages 394-438, 2012.

Andrea Montanari, Feng Ruan, Basil Saeed, and Youngtak Sohn. Universality of max-margin classifiers. arXiv preprint arXiv:2310.00176 , 2023.

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 2021.

Samet Oymak and Babak Hassibi. Sharp MSE Bounds for Proximal Denoising. Foundations of Computational Mathematics , 16(4):965-1029, August 2016. ISSN 1615-3375, 1615-3383.

Pacelli, S. Ariosto, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo. A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence , 5(12):1497-1507, December 2023. ISSN 2522-5839. Publisher: Nature Publishing Group.

Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems , 34:29218-29230, 2021.

Marc Potters and Jean-Philippe Bouchaud. A first course in random matrix theory: for physicists, engineers and data scientists . Cambridge University Press, 2020.

Farzad Pourkamali and Nicolas Macris. Bayesian extensive-rank matrix factorization with rotational invariant priors. Advances in Neural Information Processing Systems , 36:24025-24073, 2023a.

Farzad Pourkamali and Nicolas Macris. Rectangular rotational invariant estimator for general additive noise matrices. In 2023 IEEE International Symposium on Information Theory (ISIT) , pages 2081-2086. IEEE, 2023b.

Sundeep Rangan, Philip Schniter, Erwin Riegler, Alyson K Fletcher, and Volkan Cevher. Fixed points of generalized approximate message passing with arbitrary matrices. IEEE Transactions on Information Theory , 62(12):7464-7474, 2016.

Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review , 52(3):471-501, 2010.

Stefano Sarao Mannelli, Eric Vanden-Eijnden, and Lenka Zdeborová. Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions. In Advances in Neural Information Processing Systems , volume 33, pages 13445-13455. Curran Associates, Inc., 2020.

Guilhem Semerjian. Matrix denoising: Bayes-optimal estimators via low-degree polynomials. Journal of Statistical Physics , 191(10):139, 2024.

H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory , pages 287-294, 1992.

Mahdi Soltanolkotabi, Adel Javanmard, and Jason D. Lee. Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks. IEEE Transactions on Information Theory , 65(2):742-769, February 2019. ISSN 1557-9654.

Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. Precise error analysis of regularized m -estimators in high dimensions. IEEE Transactions on Information Theory , 64(8):5592-5628, 2018.

Emanuele Troiani, Vittorio Erba, Florent Krzakala, Antoine Maillard, and Lenka Zdeborová. Optimal denoising of rotationally invariant rectangular matrices. In Mathematical and Scientific Machine Learning , pages 97-112. PMLR, 2022.

Emanuele Troiani, Hugo Cui, Yatin Dandi, Florent Krzakala, and Lenka Zdeborová. Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholds. To appear in ICML 2025 , 2025.

Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research , 20(133):1-34, 2019.

Matteo Vilucchio, Yatin Dandi, Cedric Gerbelot, and Florent Krzakala. Asymptotics of non-convex generalized linear models in high-dimensions: A proof of the replica formula. arXiv preprint arXiv:2502.20003 , 2025.

Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, and Jeffrey Pennington. Precise learning curves and higher-order scalings for dot-product kernel regression. Advances in Neural Information Processing Systems , 35:4558-4570, 2022.

Yizhou Xu, Antoine Maillard, Lenka Zdeborová, and Florent Krzakala. Fundamental limits of matrix sensing: Exact asymptotics, universality, and applications. In Nika Haghtalab and Ankur Moitra, editors, Proceedings of Thirty Eighth Conference on Learning Theory , volume 291 of Proceedings of Machine Learning Research , pages 5757-5823. PMLR, 30 Jun-04 Jul 2025.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We present faithfully our claims in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the main limitations of our setting in the final Section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In the paper we present Theorem 1, for which we provide a sketch of the proof in the main text, as well as all details in the Appendix. All other theoretical results follow from analytic manipulations of Theorem 1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all details of the numerical procedures used to produce the figures in the Appendix, and we provide the code in the SM.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code in the SM.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all details of the numerical procedures used to produce the figures in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We describe error bars whenever they are present.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: All our numerical procedure can be run on common laptops.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work is theoretical in nature, and we respected the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work is theoretical and in a synthetic setting.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work is theoretical and in a synthetic setting far from direct applications.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers associated with special (still publicly released) code libraries used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No assets produced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We use synthetic data that we generate ourself.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We use synthetic data that we generate ourself.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: No LLM was used for the mentioned core aspects of the research.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Proof of Theorem 1

In this section we discuss theorem 1 for the minima of

which, due to the added regularization, is strongly convex for τ, λ > 0 and hence it admits a unique global minimum. Note that while we concentrate on this case, the computation goes through for any (strongly) convex denoiser.

The proof of Theorem 1 is an application of standard approaches in high-dimensional statistics, and goes as follows:

In Section A.1, we show how to map the original learning problem into an equivalent matrix estimation problem. This, in turn, can be reformulated as a vector Generalized Linear Model (GLM) with a non-separable regularization.

In Section A.2, we use Gaussian universality and show that this model is asymptotically equivalent to a Gaussian sensing problem, leveraging recent results on universality in high-dimensional estimation.

The resulting Gaussian model can then be analyzed using the Generalized Approximate Message Passing (GAMP) algorithm. We briefly recall the relevant properties of GAMP in Section A.3.

To apply GAMP rigorously, we exploit the fact that its fixed points coincide with the solutions of the associated convex optimization problem (Section A.3.1). These fixed points are characterized by a set of deterministic equations known as state evolution (Section A.3.2). With appropriate initialization (see Corollary 2), GAMP converges to the correct fixed point (Theorem 5), which corresponds to the global minimizer of our original problem. Hence, the fixed points of the associated state evolution equations describe the properties of the global minima of (1). Notice that this discussion is generic (i.e. allows for generic convex losses and regularizations), as long as the global minimum of the associated matrix problem is unique.

Finally, in Appendix A.4, we describe the state evolution equations that precisely characterizes the asymptotic behavior of the minimizer of the empirical risk (2), which is precisely equivalent to the results given in Theorem 1 in the limit τ → 0 + .

A.1 Mapping to a matrix model

The mapping of the data part of the loss is straightforward. For the ℓ 2 regularization on the network weights we have

The same goes for the part proportional to τ in (20), that become the Frobenius norm of ˆ S . This leads to the equivalent problem:

A.2 Data universality

The universality of the minimal error (or 'ground state energy' in the physics jargon) follows directly from a rich line of work on universality of empirical risk minimization [Goldt et al., 2022, Hu and Lu, 2022, Montanari and Saeed, 2022, Maillard and Bandeira, 2023, Martin et al., 2024b, Xu et al., 2025]. These results establish that, under broad conditions, the asymptotic behavior of the test and

training errors becomes equivalent to those with a Gaussian sensing matrix. Given the maturity of this theory, we will refer directly to these foundational works and provide a concise sketch of the main arguments for completeness.

The universality follows directly from Proposition 2.1 in [Maillard and Bandeira, 2023], who derived it in the related context of ellipsoid fitting. This was adapted to the "planted" case in [Xu et al., 2025, Section 3.1, Lemma 3.3]. The proof strategy is to consider an interpolating model where each data matrix is given by U ( t ) = cos( t )( x µ x T µ -I d ) / √ d +sin( t ) G µ in t ∈ [0 , 2 π ] , from the original model at t = 0 to the one with a GOE( d ) random matrix G µ at t = π/ 2 , and to show that the "time" t does not change the expected loss. We refer to these works (and to [Montanari and Saeed, 2022]) for the detailed proofs, and directly state the universality result:

Theorem 2 (Gaussian Universality of the loss, from [Maillard et al., 2024, Xu et al., 2025]) Let ψ : R → R be a bounded differentiable functions with bounded derivative, then for any finite α = n/d 2 , the minimum of the training loss GS( { Φ µ } n µ =1 ) := arg min S ˜ L ( S )( { Φ µ } n µ =1 ) (9) is universal with respect to the training data Φ µ in the sense that for any such ψ , then

if the matrices ( { x µ x T µ -I d } ) / √ d respects the so-called one-dimensional CLT property [Xu et al., 2025, Assumption 2.2] and if target weights S ∗ and the regularization term of the loss r ( S ) satisfy [Xu et al., 2025, Assumption 2.1].

The required one-dimensional CLT [Goldt et al., 2022, Montanari and Saeed, 2022, Dandi et al., 2023] property is a point-wise normality of the projection of the operator, proven in the case of Gaussian x µ in [Maillard and Bandeira, 2023, Lemma 4.8].

The universality argument can be readily adapted to other quantity as well, in order to show that not only the loss, but also other observables such as the overlaps are universal. We will not repeat the full proof, and directly use Theorem 3 in [Dandi et al., 2023] that can be applied mutatis mutandis 1 :

Theorem 3 (Universality of the overlaps [Dandi et al., 2023], informal) For any boundedLipchitz function Ψ : R d 2 × R d 2 → R , we have:

where ˆ S = arg min S ˜ L ( S ) , and S ∗ determines the target function.

This directly can be applied to the overlap Tr[ ˆ S T ˆ S ] and Tr[ ˆ S T S ∗ ] , which are thus universal, as well as to the test error. Notice indeed that

leading to an expression for the test error as a function of the overlaps. Finally, note that one can apply the same argument to show the universality of any finite k ∈ Z + moment d -1 Tr[ ˆ S k ] (hinging crucially on the PSD constraint to claim that all such perturbations are convex). With mild regularity assumptions (for e.g. compact support, which is satisfied by the Marchenko-Pastur target case studied in detail in the main text), this implies the universality of the spectral distribution of ˆ S as well.

A.3 Generalized Approximate Message Passing algorithm and state evolution equations

We review here how to use GAMP to solve asymptotics convex optimization problems. Later, in section A.4.1 we show indeed that the present PSD matrix optimization problem can be written as a vector one, with a non-separable denoiser.

We will work in a vector setting, where one has sensing matrices A ∈ R N × D ∼ N (0 , 1 /D ) (i.e. N samples in D dimension) with rows a µ .

Notice that in this scaling || u || 2 should be always of order O ( D ) .

The generic form of the "rectangular" GAMP algorithm reads [Javanmard and Montanari, 2013]:

The terms b t , d t are usually called Onsager's reaction terms [Mézard et al., 1987], and are tuned in such a way as to erase time-to-time correlations between the iterates ( u t , v t ) and the sensing matrix A , so that from a statistical point of view of ( u t , v t ) it is as if the matrix A was resampled independently at each time-step (conditioned on the previous iterates { ( u s , v s ) } t -1 s =1 ). One has with initialization u t =0 and the convention that g -1 ( · ) = 0 . Here v ∈ R N , u ∈ R D , g : R N → R N and e : R D → R D .

where ∇· f = ∑ d i =1 ∂ i f i denotes the divergence of a function f : R d → R d . We will see later that in high-dimension D ≫ 1 , N = O ( D ) , the Onsager's term concentrate and do not depend on the iterates u t and v t anymore.

A.3.1 GAMPfor convex optimization

Using AMP for studying the minimum of a loss is now a classical approach that has been used in many context, e.g. [Bayati et al., 2010, Montanari et al., 2012, Rangan et al., 2016, Loureiro et al., 2021b, Vilucchio et al., 2025]. This is also discussed in detail the pedagogical review [Feng et al., 2022, Section 4.4] and in the lecture notes [Krzakala and Zdeborová, 2024, Section 12.7.4].

The argument goes as follows: consider the convex optimization problem

with ℓ, R convex in their first argument, and C ⊆ R D a convex set. Define the functions

Consider the GAMP algorithm

where g is applied component-wise to v and y . Then, [Feng et al., 2022, Proposition 4.4] guarantees that fixed points of (29) are solutions of (27) (by a minor adaptation, as they consider separable regularization, but all their arguments generalize directly).

As long as ℓ, R are convex in their first argument, state evolution follows automatically for this choices of g and e .

Additionally, we remark that if the loss is strictly convex , then the optimization problem (27) has only a single global minimum, so that AMP will have a single fixed point coinciding with this global minimum.

A.3.2 State evolution

Thanks to the conditional Gaussianity discussed in Appendix A.3, one can track the statistics of the iterated u , v of GAMP through a set of so-called state evolution equations . The main difference with respect to previous approaches in our case is that the denoising function e is not separable. However, [Berthier et al., 2020, Gerbelot and Berthier, 2023] guarantees that the state evolution that allows to track the performance of AMP remains corrects (under regularity assumptions that are automatically satisfied by e and g being proximal operators).

The state evolution associated to the GAMP

(where again here we consider g applied component-wise to v and y ) is derived by considering that y µ = ϕ ( u ∗ · a µ ) where ϕ is a possibly random non-linearity, and taking into account the evolution of the iterates u and v separately in the parallel and orthogonal directions to u ∗ (and similarly with v and v ∗ = A T u ∗ ). Using [Berthier et al., 2020, Gerbelot and Berthier, 2023] we have:

Theorem 4 (Theorem 1 in [Berthier et al., 2020], informal) Define

where the average E t u is over the randomness in ϕ as well as over ( ω, z ∗ ) ∈ R 2 distributed as and E t v is over z i.i.d. Gaussian N (0 , I D ) .

Then the AMP vectors u t +1 and v t converges weakly to their Gaussian version: U t = m t u u ∗ + √ q t u z (with z random Gaussian as above) and V t = √ q t v -m t v w + m t v A T u ∗ (with w ∼ N (0 , I N ) ), in the sense that, for any deterministic sequence ϕ n : ( R D × R N ) t × R N → R of uniformly pseudo-Lipschitz functions of order k ,

An immediate corollary we shall use is the following:

Corollary 2 (Fixed point initialization) Consider a fixed point of the state evolution equations (31) (we denote the fixed point quantities by fp ). If one initializes an AMP sequence in the fixed point, i.e. by using u 0 = m fp u u ∗ + √ q fp u z (with z random Gaussian as above), then in probability for any time t > 0 and as d →∞ , the sequence of iterates U t and V t remains in their fixed point.

We then use the following theorem, that ensure with the initialization of 2, then the GAMP equation converges to their fixed point:

Theorem 5 (Convergence of GAMP, Lemma 7 from [Loureiro et al., 2021b]) Consider the GAMP iteration with e, g as in (28) , where all free parameters are initialized at any fixed point of the state evolution equations. If the associated loss (27) is strongly convex, then GAMP converges to the fixed point of the loss under study:

Thus, state evolution allows to compute all scalar observables (such as overlaps, errors, etc) on the iterates, which at convergence and under the setting of Appendix A.3.1, are equivalent to the global minimum of (27).

A.4 State evolution for Theorem 1

Now that we have reviewed how to use GAMP to study the fixed point of an optimization problem, it remains to show that the present matrix problem can be mapped to such an equivalent vectorized problem. Consider (9), and write it in the following form:

where S ∈ Sym d is a symmetric d × d matrix, C ⊆ R D is convex and ℓ, R are convex in their first argument.

A.4.1 Mapping matrix-GLM to vector-GLM

We consider the mapping from vec : Sym d → R d ( d +1) / 2 (which conveniently maps the Frobenius scalar product in Sym d given by ⟨ A ; B ⟩ = Tr( AB ) to the standard Euclidean scalar product in R d ( d +1) / 2 ) given by

under the choice of orthonormal basis

Here ( ab ) stands for the ordered pair of 1 ≤ a ≤ b ≤ d , and we denote A ij as the i, j entry of a matrix A , while as A ( ab ) the component of matrix A onto the basis element b ( ab ) . Let us denote d ( d +1) / 2 = D (we will often use D ≈ d 2 / 2 as we will be interested in the leading order in d ).

Under this mapping, we have that w = √ d/ 2 vec ( S ) satisfies

for any S with asymptotically well-defined spectral density. In particular in state evolution u ∗ = vec ( S ∗ ) is such that || u ∗ || 2 /D = Q ∗ . Moreover, we can define the correctly normalized sensing vectors

for all 1 ≤ a ≤ b ≤ d , where we used that X ∼ GOE( d ) , giving

where α = n/d 2 .

Thus, we can pick

where mat () denotes the inverse of vec () , ξ is a standard Gaussian (we are restricting here to the Gaussian noise function ϕ ), and similarly we define ˜ C by a rescaling of C . Finally, we define the equivalent vector problem in dimension D = d ( d +1) / 2 with N = n samples

In particular, the GAMP iteration described in Appendix A.3 for this loss solves (9) modulo the bijection vec () , with generic convex loss, regularization and constraint set C .

A.4.2 GAMPdenoiser g for the square loss

We directly specialize to the case of the square loss, but all this can be generalized to generic losses, see [Loureiro et al., 2021a]. For ℓ ( a, b ) = ( a -b ) 2 we have

The associated state evolution equations read (recall that ϕ ( z ∗ ) = z ∗ + √ ∆ / 2 ξ , including the average over the noise ξ in the activation ϕ in E u )

where we used that N/D = 2 n/d 2 = 2 α .

A.4.3 GAMPdenoiser e for spectral regularization, including PSD constraints and nuclear/Frobenius regularization

Spectral denoising function e . We have (calling Γ = mat ( √ 2 /d r ) and k = ¯ d to avoid confusion with the dimension d )

Now, assume that both C and R are spectral, meaning that they do not depend on the eigenvectors of Z . Then, if we consider the spectral decompositions Γ = ODO T , e ( r , k ) = ULU T , and we parametrize Z = V TV T , then for each T

which, by Von Neumann's trace inequality [Mirsky, 1975] is given by the rotation V such that U T O aligns the eigenvalues of T and D in decreasing order, which if we take the convention that all spectral decomposition are given with eigenvalues sorted decreasingly, gives U = O . Thus, the problem reduces to a spectral minimization

where here we are abusing the notation by writing T i to mean T ii as T is really a diagonal d × d matrix, and similarly for D . Notice that this expression is still general.

Spectral denoising function e for the case of the main text. We now specialize to the case of the main text. If C is the set of PSD matrices and R ( T ) = d 2 ( ˜ λ ∑ i T i + τ/ 2 ∑ i T 2 i ) , then

In the end, this means that if

with eigenvalues D i sorted decreasingly, then

where here ReLU is applied component-wise to the diagonal matrix D -2 ˜ λ I d .

State evolution equations. Now we need to derive explicit expressions for the associated state evolution equations

We revert to the case of general C and R for this, assuming that R is strictly convex over C for simplicity. It is useful to consider the following function

where the limiting procedure and all following integration/derivation/limit exchanges are well defined by the strict convexity of R , and Z β was defined in the last line. Then we have immediately

Moreover (dropping the omnipresent β limit and average E , as well as the inputs of Z for readability)

where in the third line we used Stein's lemma, so that

We just need to compute Ψ for the sake of the state evolutions.

State evolution equations for the case of the main text. If C is the set of PSD matrices and R ( T ) = d 2 ( ˜ λ ∑ i T i + τ/ 2 ∑ i T 2 i ) , then

where D i is the i -th eigenvalue of the matrix S ∗ + √ q u /m u Z , and L i is determined by

giving

where in the last step we used that at leading order the spectrum of S ∗ + √ q u /m u Z concentrates (by assumption in 1) to µ ∗ √ q u /m u defined in Theorem 1.

A.4.4 Final form of the state evolution and observables

Collecting the state evolution equations derived in Appendices A.4.2 and A.4.3, calling m v = m , q v = q , b = Σ , m u = ˆ m , q u = ˆ q and d = ˆ Σ we get the system of equations

where J is defined in Theorem 1. This reduces to the equations presented in in Theorem 1 at its fixed point and under the change of variable δ = √ ˆ q/ ˆ m,ϵ = 2 / ˆ m .

The test error then is given by

The expression train loss instead van be derived by using the fact that the residuals in the data part of the loss can be computed from state evolution as ˆ q/ 16 , and by evaluating the regularization at the spectral density µ ∗ δ (which is just the Tr of a shift and rescaling of this spectral distribution) giving the expression in Theorem 1 in the limit τ → 0 + .

Notice that these state evolution can be also heuristically applied in the case ˜ λ = 0 as long as the problem has a unique global minimum. This can be checked by having Σ < + ∞ , as Σ is a proxy for the inverse of the curvature of the loss at the global minimum [Clarté et al., 2023], which becomes flat when Σ = + ∞ . Notice also that Σ can be used to guarantee, at the heurisitc level, that the minimum of the global loss is unique even without the need of the regularization τ > 0 .

A.5 Replicon condition for Generalized Approximate Message Passing fixed points

For completeness, we discuss in this section the pointwise convergence of the GAMP algorithm when initialized at a fixed point of the state evolution equations. While the convex nature of the problem ensure this convergence [Loureiro et al., 2021b], we discussed here the explicit criterion that can also be directly checked within the state evolution formalism. As noted by Bolthausen [2014], it hinges on a stability condition-known in statistical physics as the replicon or de Almeida-Thouless (AT) condition [de Almeida and Thouless, 1978, De Dominicis and Kondor, 1983]-which can be directly and easily checked from the state evolution equations.

Wethus provide the replicon condition on point-wise convergence of GAMP for generic non-separable denoisers (see for instance Vilucchio et al. [2025] for the separable case).

Consider again the recursion (25)

with the b t , d t given in (26). We want to assess linear stability. To this end, we suppose that the iteration is initialized at a fixed point ( u , v ) (and that at this point we also assume that the nonlinearities g, e and the Onsager's terms are constant in time), consider a Gaussian perturbation ϵ ∼ N (0 , ϵ I d ) of u , and compute whether the iteration converges back to the fixed point after such perturbation. Notice that we can avoid perturbing the Onsager's terms, as in the high-dimensional limit they are independent on the iterates. We have (at first order in ϵ )

where in the second to last step we used the explicit form of the Onsager's term d , and in the last step we suppressed the dependencies of the functions g, e on u , v for clarity and passed in components notation.

The L2 norm of the perturbation u new -u equals (on average over the initial perturbation ϵ )

where we used that E ϵ ϵ i ϵ i ′ = δ ii ′ , and the law of large number to substitute

where in the last step we only kept the term that will contribute to leading order (equality at elading order is denoted by ≈ . This is the generalization to non-separable functions g, e of [Vilucchio et al., 2025, Definition 1].

Thus, the linear stability criterion for GAMP is

In particular, in the case of independent observations we will have ∂ µ g ν = δ µν ∂ µ g µ , and in the case of separable denoiser one would have ∂ i e j = ∂ ij ∂ i e i . Notice also that this criterion is independent of the value of the iterates u , v at the fixed points, as it concentrates onto its state-evolution-predicted value.

In the case of ℓ 2 loss +PSD + nuclear norm regularization, a fixed point of the state evolution describes a stable fixed point of GAMP if

We have checked numerically that this condition was satisfied in our fixed point (in accord with Theorem 5).

B Minimal norm interpolation limit λ → 0 +

Consider (6). We would like to give a prescription to manipulate this equations in order to describe minimal regularization interpolators. We first remark that the limit λ → 0 + of (6) as written describes the case of non-regularized minimization, and thus Theorem 1 will be valid in this case only for α > α interp , when only one interpolator of the training dataset exists.

Consider now the rescaling of (9) given by ˜ λ -1 ˜ L . For any fixed positive ˜ λ , this rescaling does not alter the solution of the minimization problem. In the limit ˜ λ → 0 + instead, the loss first imposes that the global minimum is an interpolator of the training dataset, and then minimizes the regularization within the interpolator set. Thus, rescaling the loss by ˜ λ -1 and then sending ˜ λ → 0 + will describe minimal regularization interpolators, as long as α < α interp in the noisy case (after that threshold no interpolator exists, and the minimization problem is ill-defined) of for all α in the noiseless case (as in this case there exists always one interpolator).

We practically achieve the loss rescaling in (6) by renaming ˜ λϵ → ϵ , and obtaining the new system of equations

Thus, to summarize the learning curves at ˜ λ = 0 + are found as follows.

equivalent to (6) for all ˜ λ > 0 , and then taking ˜ λ → 0 + .

In the noiseless case ∆ = 0 , we solve (67) with ˜ λ = 0 for all values of α .

In the noisy case ∆ > 0 , we solve (67) with ˜ λ = 0 for all values of α < α inter , and we solve (6) for ˜ λ = 0 for all values of α > α inter .

C Derivation of Result 1 and Corollary 1

C.1 Prerequisites

For both Result 1 and Corollary 1 we will need the following results on the spectral density µ ∗ ⊞ µ s . c .,δ , where ⊞ is the free convolution and µ s . c .,δ = √ 4 -x 2 / (2 πδ 2 ) the semicircle distribution of radius 2 δ for δ > 0 . Here µ ∗ ( x ) = √ κ ∗ µ M . P . ( √ κ ∗ x ) , where µ M . P . is the Marchenko-Pastur distribution [Marchenko and Pastur, 1967] with parameter κ ∗ (the asymptotic spectral distribution of A T A/m where A ∈ R m × d has i.i.d. zero-mean unit-variance components), and Q ∗ = 1 + κ ∗ .

We start by recalling that

where and where we used that

by additivity of the mean and variance.

Now we use [Maillard et al., 2024, Appendix D.3] to state that for any fixed x , for δ → 0 + , we have at leading order

This will allow us to compute the last integral in (68) when δ ≪ 1 for some values of s . Indeed, when δ ≪ 1 and 0 < κ ∗ < 1 the spectral density µ ∗ δ is composed by a small semicircle around the origin with mass 1 -κ ∗ , and an approximately µ ∗ -shaped bulk gapped away from the origin. So as long as s lies in the gap between the two bulks, only this semicircle will contribute to (68). For κ ∗ > 1 instead, the bulk at the origin is not present leading to a zero contribution from all the integrals. The case κ ∗ = 1 is more delicate, as there the spectral distribution of the target may be non-gapped (for example, in the narrow target case). We recover this case by a limiting procedure from the two sides.

C.2 Derivation of Result 1

To find the interpolation threshold, we consider (6) for ˜ λ = 0 . To do this, we need to make sure that we are in the α > α inter region, where only one interpolator (noiseless case) or no interpolator (noisy case) exists, giving a single minimum of the training loss. This can be checked by having

where Σ was defined in (59). The interpolation threshold will be exactly at the point where Σ ∝ ϵ diverges.

Noisy case. Consider the (6) with ˜ λ = 0 . We have

We now consider the limit ϵ → + ∞ , obtaining i.e. an equation for δ , which can be plugged in to directly find α . This gives the first part of Result 1. Notice that the value of δ found will be finite, giving also an analytic prediction of the height of the cusp of the test error at interpolation.

For ∆ → + ∞ , we see that a consistent solution is given by δ → + ∞ (starting from (73) at finite ϵ , and then taking the limit). In that case, µ ∗ δ is, at leading order, a semicircle with radius 2 δ , so that

implying that for large ∆ the interpolation threshold converges to 1 / 4 .

Noiseless case. Here we can advance more. In the noiseless case ∆ → 0 , we also know that after interpolation we have zero test error, hence δ = 0 . Thus, we can expand J ( δ, 0) for small δ . Using (68) and (71) we have

implying that the equation for δ is satisfied for δ = 0 , and that

Let us notice that the δ = 0 solution here is valid for all α > α inter , so we could have worked with (6) with ˜ λ = 0 and finite ϵ , and computed ϵ explicitly as a function of α . Doing this shows that ϵ < + ∞ for all α > α inter , and that ϵ → + ∞ exactly at the threshold. This concludes the derivation of Result 1.

C.3 Derivation of Corollary 1

Consider first the case 0 < κ ∗ < 1 . We start from the stationary conditions for (6)

To compute the strong recovery, i.e. the value of α at which the test error is zero, we look for the value of α that is consistent with a solution of (6) in the limit δ → 0 + . We assume the scaling and verify it self-consistently. We have that (71) implies at leading order

where we defined the incomplete k -th moment of the semi-circle distribution as

Similarly

This leads to the equations (at leading order in δ → 0 + )

where we renamed c ˜ λ → c (which will allow to be in the interpolation limit for ˜ λ = 0 as detailed in Appendix B). Define

that is the small regularization limit and noisless limit at the critical scaling. Then the equations read

giving one equation for c , and one for α strong given c . By computing the next-to-leading order, one could access the speed at which α → α strong in δ , invert it and find the critical scalings of ˜ λ and ∆ as a function of α strong -α . In the case ˜ λ = ∆ = 0 we have the simpler equation

which modulo conversions gives the same α strong as [Donoho et al., 2013, 12].

For κ ∗ > 1 , the same equations hold with all M contributions set to zero (see discussion above), giving (again for ˜ λ = ∆ = 0 )

This shows Corollary 1.

D Derivation of Result 2

We start again from (6)

We redefine ˜ λϵ → ϵ and consider the limit κ ∗ → 0 under the scaling ansatz α = ¯ ακ ∗ , δ = dκ ∗-1 / 2 , ϵ = cκ ∗-1 / 2 and ˜ λ = ¯ λκ ∗ 1 / 2 . Notice that Q ∗ = 1 + κ ∗ ≈ 1 in the limit. The equations read

Then

We now use the fact that at leading order for small κ ∗

is given by a semi-circle distribution of radius 2 d centered at zero with mass 1 -κ ∗ , plus a Dirac's delta spike at y = 1 + d 2 with mass κ ∗ if d 2 ≤ 1 , see [Maillard et al., 2024, Appendix E.1].

We then have the following cases in which the spike does not contribute.

No spike d 2 > 1 and c > 2 d , or spike 0 < d 2 < 1 and c > 1 + d 2 > 2 d . Then the integral J does not contribute to the equations. Thus

under the condition

No spike d 2 > 1 and bulk integral contribution c → (2 d ) -. In this case c tends to 2 d with the appropriate rate in order to have order O (1) contribution from the bulk integral in the equations. We have, calling 2 d -c = dt i.e. c = d (2 -t ) for some t > 0

giving

from which we see that the bulk integral contributes to non-diverging order if t 5 / 2 = O ( κ ∗ ) , giving the equations

under the conditions

Thus, we start finding that

We now consider the case in which the spike contributes.

If there is the spike d 2 < 1 and it is the only contribution 2 d < c < 1 + d 2 , then the equations read

This system can be solved explicitly under the conditions above, giving quite an unmanageable expression. It reduces to δ = 0 for ∆ = ¯ λ = 0 .

If there is the spike d 2 < 1 and it contributes along with the bulk (see discussion above and (96)) c → (2 d ) -, then the equations read

These give again unmanageable expressions, but for ∆ = 0 it gives

One can check by explicit solution that the last two cases lead to test error strictly smaller than one. The last two cases can be easily solve numerically, allowing to plot the curves in Figure 4.

Finally, the large α behavior can be found by considering the last case

under the scaling ansatz

and solving the equations perturbatively gives

giving the large α scaling presented in Result 2.

This completes the derivation of Result 2.

E Details of the numerical implementation of Theorem 1

We provide the code for the numerical implementation of the equations in Theorem 1 and the experiments at https://github.com/SPOC-group/OverparametrisedNet .

The numerical implementation of (6) reduces, at its core, to two sub-tasks: the numerical evaluation of µ ∗ a , the integration involved in computing J ( a, b ) and its partial derivatives, and the numerical solution of the equations themselves.

Computing µ ∗ a . Recall that the spectral density µ ∗ a is the free convolution of a rescaled MarchenkoPastur µ ∗ distribution, and that of a semicircle distribution with radius 2 a . In particular, µ ∗ ( x ) = √ κ ∗ µ M . P . ( √ κ ∗ x ) , where µ M . P . is the Marchenko-Pastur distribution with parameter κ ∗ (the asymptotic spectral distribution of A T A/m where A ∈ R m × d has i.i.d. zero-mean unit-variance components). This free convolution can be computed using standard random matrix theory [Potters and Bouchaud, 2020] tools. We report here the procedure as described in Maillard et al. [2024], and summarize here only the steps without providing a derivation.

To compute the spectral density under consideration, first we fix the noise level a and the location x at which we want to compute µ ∗ a ( x ) . Then, we solve the following cubic equation (looking for the solution with largest imaginary part, if all solutions are real then the spectral density at that location is zero)

where z = x -iτ with τ a small positive constant.

The solution g = g ( z ) of the equation is the so-called Stieltjes transform of µ ∗ ( x ) , i.e.

which can be inverted by the Stieltjes-Perron inversion formula, i.e.

In practice, the limit is done by using τ = 10 -12 directly in (106).

Additionally, by imposing that the discriminant of (106) is zero we obtain an equation for x (setting τ = 0 ) that allows to precisely evaluate the boundaries of each of the bulks of µ ∗ a (the distribution has two bulks in the low-noise setting, and one single bulk otherwise, with the splitting threshold depending non-trivially on κ ∗ and a ).

Computing J ( a, b ) . We use the estimate of the bulk boundaries to set-up an efficient integration scheme to finally compute J ( a, b ) . We first compute the bulk boundaries, then intersect the support of the bulks with the integration region ( b, + ∞ ) , and finally for each bulk integrate µ ∗ a ( x )( x -b ) 2 using quad from the Scipy library in Python. This explicit use of the bulk boundaries allows precise evaluation of J ( a, b ) even in the low-noise regime, where one of the bulks has very small support with quite large values for the spectral density.

Finally, we compute the partial derivatives of J ( a, b ) by centered finite differences, with step size equal to 10 -8 .

Numerical solution of the equations. To solve numerically (6), we find it convenient to revert to the state evolution equations presented in Appendix A.4, equation (59). We follow the prescribed iteration, possibly damping the updates with a factor as small as 10 -2 , and declaring convergence when the Euclidean distance between two subsequent updates falls below a prescribed tolerance of at least 10 -4 . It is also convenient to adopt a planting scheme, where the equations for a given value of ˜ λ, ∆ , κ ∗ are solved for one initial value of α (we found that a value around the mid-point between α = 0 and α = α strong works best), and then the rest of the learning curve as a function of α is computed by successively using the solution at a value α as the initialization for that of α + δα , where δα is a small step-size.

F Details of the experiments

For Figure 1 right and d ≤ 100 we used the CVXPY package in Python. All the other experiments are realized in PyTorch. We instantiate the student weights randomly as centered Gaussian variables with standard deviation 10 -3 and the target ones with variance 1 , appropriately changing the functional form of the target. In Figure 1 right we optimize using LBFGS for the sake of efficiency. For Figure 1 left we optimize using the GD iteration

with η = 20 .

G Arbitrary sign in the second layer weights

A key limitation of the network in (1) is that it can only learn targets with positive definite S ∗ . This is because the second layer weights are all positive. Within our formalism we can consider a more generic function with fixed a k ∈ {-1 , 1 } with at least d positive and d negative entries.

with loss where

Our goal will be to show that minimizing over ˜ L W ( W ) is equivalent to minimize ˜ L S ( S ) from (9) over all symmetric matrices S

The argument goes as follows. For any fixed value of a , define the sets of indices P , N as

and additionally, define

The matrix S is then given by the sum S = S P + S N , and we remark that both matrices are symmetric, non-rank constrained and respectively positive and negative semi-definite. The minimization of ˜ L W ( W ) over W can be rewritten as

What we have to show is that (113) and (116) have the same optimal value, and that the minimizer ˆ S of (113) yields a minimizer ( ˆ S P , ˆ S N ) of (116) by taking the positive and negative spectral parts of ˆ S . First, we write where

By the triangle inequality hence R ( S ) ≥ ∥ S ∥ ∗ . This bound is in fact achievable: let S be symmetric, with spectral decomposition

Define its positive and negative spectral parts where

Then S + ⪰ 0 , S -⪯ 0 , and S = S + + S -. Moreover, so

Thus, for this admissible decomposition R ( S ) ≤ ∥ S ∥ ∗ , which combined with the other inequality gives us R ( S ) = ∥ S ∥ ∗ . This proves the claim.

Asymptotics without PSD constraint. We are solving the same problem as (22), just without the positivity constraint:

For this case, the spectral denoising function (48) becomes

where

Similarly, equation (57) needs to be substituted by

The final state evolution will be unchanged by redefining J in (6) as

We illustrate how this changes the theoretical predictions in Figure 5.

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Sample complexity

α

=

n/d

2

10

-

1

10

0

10

1

10

2

Test error

λ

= 0 +

λ

= 0

.

001

λ

= 0

.

002

λ

= 0

.

005

λ

= 0

.

01

λ

= 0

.

02

λ

= 0

.

05

λ

= 0

.

1

λ

= 0

.

2

λ

= 0

.

5

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze a two-layer, overparameterized neural network with quadratic activation, trained by empirical risk minimization (ERM) on data generated by the same model. They show that, in the high-dimensional limit (input dimension $d$, number of samples $n$, network width $m$ all  $\\rightarrow \\infty$ at fixed ratios), the nonconvex ERM problem can be exactly reformulated as a convex nuclear-norm‐penalized positive semidefinite matrix sensing problem. Leveraging Gaussian universality, AMP/state‐evolution, and convex‐optimization tools, they derive closed-form asymptotic expressions for the training error, test error, and singular‐value distribution of the learned weight matrix. Their formulas reveal sharp thresholds for perfect recovery, interpolation, and the double‐descent phenomenon as functions of sample complexity $d^2$, width ratio $\\kappa$, regularization strength $\\lambda$, and noise level $\\Delta$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Exact asymptotics: Delivers fully explicit high-dimensional limits for both training and generalization error—stronger than typical “order-level” bounds.\n\n2. Convex reformulation: The reduction from a nonconvex quadratic network to a convex nuclear-norm problem is conceptually elegant and may inspire similar mappings for other architectures.\n\n3. Universality: By invoking one-dimensional CLT and an interpolation argument, they show their results hold for a broad class of input distributions, not just Gaussian.\n\n4. Numerical Validation: Experiments in moderate dimensions (e.g. $d=50,100,300$) match the theory closely, demonstrating practical relevance of the asymptotics.\n\n**Weaknesses**\n1. Model specificity: Focuses exclusively on centered quadratic activations; it is unclear how to adapt to ReLU, tanh, or deeper nets.\n\n2. Synthetic data: All experiments use rotationally invariant Gaussian-like data; performance on real-world datasets remains untested.\n\n3. Regularization dependence: The sharp formulas require nonzero nuclear regularization $\\lambda>0$; the purely unregularized case is only partially characterized by an interpolation threshold. Actually, many existing works focus on learning theory of regularization-free models that demonstrate the optimization methods can automatically found the parameters with good generalization performance from multiple solutions.\n\n4. Not well-defined notations. For example, authors used semicircle distribution and free convolution terms in Theorem 1 without introduced beforehand. The $\\delta$ in defined  radius of semicircle distribution conflicts with the $\\delta$ function in Eq. (8).\n\nQUESTIONS:\n1. I want to make sure if the training and testing errors in (7) can be always non-negative?\n\n2. Standard GAMP / AMP state‐evolution theory covers separable scalar denoisers. Here your denoiser acts on symmetric matrix blocks. What modifications to the state‐evolution proof are needed to handle this non‐separable, PSD‐projected “denoiser”? Are there additional convergence or stability conditions?\n\n3. In the unregularized case ($\\lambda=0$) you identify an interpolation threshold αinterαinter​. Is it possible that local but non-global minima exist just below $\\alpha_{inter}$? \n\n4. For the low rank setting $\\kappa^\\star <<1$, can you compare learning performance with work [1]? \n[1] Li et al. (COLT 2018) in “Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations.”\n\nLIMITATIONS:\n1. Authors study the well-specified student-teacher model and quadratic activation,  the more realistic setting is considering mis-specified case where student and teacher have different models and general activation functions beyond quadratic and smooth activations such as widely used ReLU-type activation functions.\n\n2. All sharp predictions are exact as $d,n,m \\rightarrow \\infty$. No finite-$d$ error bounds or convergence rates are provided. For moderate dimensions, it’s unclear how large the bias or variance of the asymptotic formulas is.  \n \n3. Limited insight on feature learning. By focusing on a fixed quadratic feature map, the paper sidesteps any notion of learning useful representations from raw data. All the heavy lifting happens in the convex PSD stage, with no mechanism for adapting or discovering features—a key strength of deep networks in practice.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

comments: SUMMARY:
The paper analyzes the training and test error of regularized ERM solutions of the empirical square loss for two-layer networks with quadratic activations. They assume that the teacher network is also a quadratic two-layer network, and allow for label noise. They focus on the high-dimensional limit with quadratically many samples and certain bounds on the widths. They provide closed-form expressions for the training and test losses. They use their result to analyze some more specific settings and properties of the solution.

STRENGTHS AND WEAKNESSES:
The paper analyzes a basic setting and proves strong and highly non-trivial results for the high-dimensional limit. I believe the results are significant and contribute to our understanding of the problem.

Could the authors also compare their results with the following paper? “On the Power of Over-parametrization in Neural Networks with Quadratic Activation”, Du and Li (2018)

Overall, other than the above question and the obvious limitations mentioned in the last paragraph of the paper, I don’t have further comments. Indeed, the paper considers a rather specific setting with many assumptions, but I still believe that a detailed analysis of this setting is of interest.

QUESTIONS:
See the question in the strengths and weaknesses section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study presents an analysis of empirical risk minimization over the quadratic loss for a two-layer neural network with extensive width and quadratic activations. The authors rigorously derive predictions for the generalization error in high dimensions. The results are then analyzed and used to derive thresholds for interpolation and strong recovery. Finally, all results are placed in the context of related literature on different models and compared with previous Bayes-optimal baselines for the same model.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The wide network setting is considered to be challenging.\n- The setting of Theorem 1 is very general: very few assumptions on the spectral density of $S^*$ are required, providing versatility for different applications.\n- This work shows that $\\ell^2$ regularization over $W$ translates into nuclear norm regularization over $S$. This correspondence of regularizations is completely new to me.\n- The rigorous delineation of thresholds for interpolation and strong recovery is an important contribution to understanding networks with extensive structure. The fact that these thresholds differ from the kernel ones when $\\kappa^*$ is small is noteworthy.\n- The results in the paper disprove a conjecture on interpolation thresholds from [Sarao Mannelli et al., 2020].\n\n### Weaknesses\n- The assumption of i.i.d. Gaussian noise is very restrictive, although it is shared with most of the related literature.\n- The quadratic activation assumption is restrictive. However, given the other challenges this model presents, together with the fact that several related works share this assumption, I believe this limitation is not too impactful.\n- Several works on this model have already appeared, especially the Bayes-optimal analyses by Maillard et al. [2024] and Xu et al. [2025]. While it is clear that ERM is a different problem from Bayes-optimal analysis, the two settings are related and share some core analytical ideas. In my view, this slightly impacts the originality and significance score.\n\nQUESTIONS:\n### Questions\n- What are the key technical differences in the proofs that distinguish the ERM setting considered here from the Bayes-optimal analyses by Maillard et al. [2024] and Xu et al. [2025]?\n- Can you comment on the gap between Bayes-optimal performance and the best ERM error in Figure 1 and Figure 2 (left plots)? Does this suggest the presence of a statistical-to-computational gap?\n- In line 268, you mention \"reflecting the structural advantage of the data over random labeling,\" but it is unclear to me how the previous reasoning on $\\Delta$ and $\\kappa^*$ implies this. Could you elaborate further?\n\n### Minor Corrections\n- Line 196: \"then,\" not \"than.\"\n- In equation (13), the cases for $\\kappa^* $  repeat the same condition. I assume the bottom line should state $\\kappa^* \\ge 1$.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper delivers a technically rigorous analysis of two‐layer quadratic networks trained by $\\ell_2$‐regularized empirical risk minimization on high‐dimensional Gaussian data, by showing that the nonconvex learning problem can be exactly reformulated as a convex, nuclear‐norm–penalized matrix sensing task. Through precise AMP state‐evolution arguments, the author obtain closed‐form expressions for training and generalization errors in the limit $\\alpha = n/d^2 = O(1)$, $m^\\star/d = O(1)$ and $m/d = O(1)$, where $n$ is the number of measurements, $d$ is the embedding dimension, $m$ is the number of neurons (i.e., rank of the weight matrix) in the learner's (i.e., student) model and $m^\\star$ is the same for the teacher model.\n\nThe main results are as follows,\n\n1. The calculation of an interpolation threshold $\\alpha^\\star_1$ in the case of an i.i.d. prior on $w^\\star$ (referred to as the MP case), which corresponds to a characterization of the ratio $\\alpha$ for which interpolators collapse to a single solution as the nuclear norm regularization strength goes to $0$. This is a statement about the optimization landscape of the regularized $\\ell_2$-regularized empirical risk.\n\n2. The calculation and a strong recovery threshold, $\\alpha^\\star_2$ in the MP case, beyond which perfect generalization occurs in the noiseless setting. This is shown to be $1/2$ when $m^\\star/d \\ge 1$ (overparameterized setting).\n\nTheir results demonstrate that $\\ell_2$ weight decay implicitly enforces low‐rank structure and also results in test error that is independent of $m$ in the overparameterized setting.\n\nSTRENGTHS AND WEAKNESSES:\nI am not an expert in this area, and am reviewing this paper as a member of the broader audience. Perhaps someone more well versed with the statistical physics analyses of this problem and mean-field approximations for matrix completion in the overparameterized regime may have a different take about this, but overall I thought that the paper was incredibly hard to read. The main result is quite dense, and missing many simple interpretations. It is written for an expert who already understands the setting (which is admittedly very specific). There are several notations (for instance, free convolution) which were not defined anywhere in the paper, which I believe is necessary for the results to be understood in a self contained manner. Overall, I would recommend that the paper be simplified and presented in a top-down approach, and even potentially lengthened for the goal of making it easier to motivate the main result. I believe my review does not adequately address the technical contribution of the paper, which I was unable to grasp fully. However, improving the writing in the paper seems to be a prerequisite to be able to carry this out.\n\nMinor:\n\n1. $\\kappa^\\star$ is denoted $\\kappa_\\star$ in some locations.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe paper is restricted to the case of quadratic activations and a single layer. While I do not view these as weaknesses of the paper, I think it would be nice if the authors addressed what happens if the activation function is no longer quadratic, for instance.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}