Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networks
Authors: Brandon Livio Annesi
Dario Bocchi
Chiara Cammarota
Venue: ICLR 2026 Oral (2026)
Keywords: Overparametrization
Loss landscapes
Signal recovery
High-dimensional learning
Primary area: learning theory
TL;DR: We quantitatively analyze how overparametrization reshapes the high-dimensional loss landscape of a teacher–student setup in random positions, showing it can anticipate and qualitatively alter transitions between successful and failed signal recovery

Abstract: High-dimensional non-convex loss landscapes play a central role in the theory of Machine Learning. Gaining insight into how these landscapes interact with gradient-based optimization methods, even in relatively simple models, can shed light on this enigmatic feature of neural networks. In this work, we will focus on a prototypical simple learning problem, which generalizes the Phase Retrieval inference problem by allowing the exploration of overparametrized settings. Using techniques from field theory, we analyze the spectrum of the Hessian at initialization and identify a Baik–Ben Arous–Péché (BBP) transition in the amount of data that separates regimes where the initialization is informative or uninformative about a planted signal of a teacher-student setup. Crucially, we demonstrate how overparameterization can "bend" the loss landscape, shifting the transition point, even reaching the information-theoretic weak-recovery threshold in the large overparameterization limit, while also altering its qualitative nature.
 We distinguish between continuous and discontinuous BBP transitions and support our analytical predictions with simulations, examining how they compare to the finite-N behavior. In the case of discontinuous BBP transitions strong finite-N corrections allow the retrieval of information at a signal-to-noise ratio (SNR) smaller than the predicted BBP transition. In these cases we provide estimates for a new lower SNR threshold that marks the point at which initialization becomes entirely uninformative.

Published as a conference paper at ICLR 2026
OVERPARAMETRIZATION BENDS THE LANDSCAPE: BBP TRANSITIONS AT INITIALIZATION IN SIMPLE NEURAL NETWORKS
Brandon Livio Annesi Dario Bocchi*.1.2 Chiara Cammarota 1.3
'Physics Department, University of Rome 'La Sapienza', Piazzale Aldo Moro S, Rome, 00185, Italy Instiiutee of Nanotechnology, CNR-NANOTEC, Piazzale Aldo Moro, 5, 00185 Rome RM
National Institute for Nuclear Physies, INFN-Romal, Piazzale Aldo Moro Rome, 00185, Italy [brandon] invi.. fario.boccht chiara. cawma rota rota) romal..rrom.....mmmmmm iit
ABSTRACT
digh-dimmnsional non-convex loss landscapes play a central role in the theory of Machine Learning. Gaining insight into how these landscapes interact with gradient-based optimization methods, even in relatively simple models, can shed light this enigmatic feature of neural networks. In this work, will focus a prototypical simple learning problem, which generalizes the Phase Retrieval in- ference problem by allowing the exploration of overparametrized settings. Using techniques from field theory, we analyze the spectrum of the Hessian at initial- ization and identify a Baik-Ben Arous-Pécht (BBP) transition in the amount of data that separates regimes where the initialization is informative or uninformative about a planted signal of teacher-student setup. Crucially, we demonstrate how overparameterization can bend the loss landscape, shifting the transition point, even reaching the iitormationnthhorette weak-recovery threshold in the large over- parameterization limit, while also altering its qualitative nature. We distinguish between continuous and discontinuous BBP transitions and support our analytical predictions with simulations, examining how they compare the finite-N behav- ior. In the case of discontinuous BBP transitions strong finite-N corrections allow the retrieval of information a signal-to-noise ratio (SNR) smaller than the pre- dicted BBP transition. In these cases we provide estimates for new lower SNR threshold that marks the point at which initialization becomes entirely uninforma- tive.
INTRODUCTION
The geometry of high-dimensional, non-convex loss, risk, cost landscapes plays central role in modern machine learning and data science. Such landscapes hide important structural features of the data into specific local structures and mostly deep configurations. Despite their complexity, the optimization of these landscapes is typically performed using local iterative algorithms, most no- tably gradient descent and its stochastic variants. Understanding the success and limitations of these algorithms remains a fundamental open problem. It has been first observed that in regimes where the dataset large relative the problem dimension N, i.e., at high signal-to-noise ratio (SNR), the landscape can undergo an effective trivialization, becoming nearly convex and devoid of spuri- ous local minima (Fyodorov) 2004 Soudry Carmon 2016; Cai et al. 2022). this setting, each point in the landscape contains a clear directional signal guiding the optimization toward informative minima. Furthermore, it has been widely reported that overparamettrizzatin of the learning model can induce a smoothing of the loss landscape even in regimes with moderate or low SNR, thereby facilitating optimization (Shevchenko & Mondelli] 20200: Cooper 2021). Perhaps more surprisingly, in settings where spurious non -on--nformmatve minima remain prevalent, gradient-based methods often still succeed (Baity-Jesi et al. 2018 Liu et al. 2020 Ros et al. 2019) Mannelli et al. 2019). This apparent paradox has been addressed in series of works on high-dimensional inference prob- lems such as matrix tensor PCA (Sarao Mannelli et al. 2019) and phase retrieval (Sarao Mannelli
"qqual contribution

---

Published as a conference paper at ICLR 2026
et al. 2020a). These studies reveal that gradient flow dynamics can avoid these poor solutions due to the local geometry of high-dimensional basins of attraction, which are typically explored by the dynamics. The high dimensional basins of attraction of gradient flow, although still non informative themselves, develop an instability towards the signal at relatively low SNR. This phenomenon is sometimes referred to as the "blessing of dimensionality". Crucially, the emergence of such insta- bilities at increasing SNR is associated with a qualitative change in the spectrum of the local Hessian in transition known as the Baik-Ben Arous-Péché (BBP) transition (Baik al. 2005).
Alternative learning approaches, mostly applied to signal reconstruction problems, are based on the use of spectral methods (Netrapalli et al. 2013; Montanari & Sun 2018) to define warm start to subsequent local iterative algorithms, with the aim of boosting their performances. Typically, in spectral methods such initial guess is provided by the leading eigenvector of a matrix, which is a function of the input data tailored to the structure of the specific problem (Montanari & Sun| (018
Lu Li 2020; Mondelli & Montanari] 2018: Maillard et al., 2022 Defilippis et al. [025] Kovačevv et al. 2025). Interestingly in some cases, for instance phase retrivval tensor PCA, such ad hoc procedure can be also linked to the risk landscape as the matrix used for spectral initialization corre- sponds to the negative Hessian of a suitably defined cost function evaluated at random configurations and averaged over many of them (Biroli et al.j 2020). Therefore its leading eigenvector represents the direction with the most negative (or smallest positive) curvature found at the initial condition in such averaged landscape. Also in this case, when the SNR increases, the spectrum of such matrix undergoes BBP transition, after which the leading eigenvector develops finite correlation with the signal. A similar phenomenon occurs in the Hessian of the cost function evaluated at individ- ual random configurations (Bonnaire et al. 2025 Arous et al. 2025), but it is less known how this landscape feature is also affected by overparametrization. Moreover, very recent work (Bonnaire) jt al., 2025) has shown that the information contained in the curvature of the landscape in random configurations, for finite input dimensions N, could further automatically help gradient-based meth- ods in finding the deep informative minima. The interplay between the gradient flow algorithmic transitions and the emergence of the signal in the Hessian at the initial condition then defines an effective algorithmic transition in the SNR, which slowly changes with the dimensionality of the data set. This already nontrivial mechanism may be further modified in the presence of overparam- eterization, motivating a deeper exploration of its role in shaping the optimization landscape and the dynamics therein.
In this work, we consider an extended version of the classical phase retrieval problem by focusing on a teacher-student setting based on two-layer soft-committee machines with quadratic activations. The widths of the hidden layers of the student and teacher networks, denoted by and p" respec- tively, are generic and finite, while the dimensionality N of the input samples will be considered very large and diverging, except in numerical tests. When = p' = 1. the setting reduces to the standard phase retrieval problem, which involves recovering a hidden signal from magnitude-only projections. It notoriously results in a non-convex optimization problem with broad relevance in optics (Millane| 1990), signal processing (Bendory et al.] 2017). quantum mechanics (Orl et al., 1994), and which has often served as prototypical example for exploring the interplay between optimization dynamics and high-dimensional geometry (Sun et al. 2018)).
For general p and p², this setting is particular case of what is known in the literature as a multi- index model Li (991]]: Defilippis et al. (0025): Troiani et al. (0024). Here we focus on the case p p", and explore the effect of overparametrriztion on the landscape structure. In particular, we focus on the information contained in the local curvature in random positions of a suitably defined class of loss landscapes spanned by parameter a. We study how it changes with a, p and p" As previously mentioned, the Hessian at initialization could contain more information than expected, which could lower the SNR of algorithmic transitions for gradient-based algorithms, or could be explicitly used in a sort of generalized spectral method.
Our analysis shows that overparametrizattoon generally shifts the BBP transition in the Hessian spec- tra of random configurations toward lower SNR. The corresponding spectral initialization method based on such local Hessian matrices is therefore expected to extract information earlier than in the underparametrizeed case and even gradient- gradient-based learning dynamics is expected to work better with overparametrization in mime--iimeniionll practical implementations of the problem. However, we also obtain that in few very specific instances overparametrization may slightly harm the efficiency of signal recovery obtained through the diagonalization of the Hessian at initialization. We also observe that the nature of the BBP transition changes from underparametrized students to students
2

---

Published as a conference paper at ICLR 2026
benefitting from overparametrization. When overparametrization increases the standard BBP transi- tion tends to be replaced by a BBP transition associated with a discontinuous jump in the amount of information retrieved. The emergence of discontinuous BBP transitions has been only very recently
discussed in association to signal reconstruction in phase retrieval problems (Bocchi et al.] 2025a Bousseyroux & Potters 2024), and previously only conjectured theoretical grounds (Potters & Bouchaud 2020). With this work, we illustrate how they become central when phase retrieval is generalized an overparametrized learning setup. Moreover, strong finite size effects are expected to affect numerical observation of discontinuous BBP transitions (Bocchi et al., 2025a). We high- light this aspect in the results of the signal recovery in the overparametrized cases. Interestingly, we observe that higher overparametrization renders the transition more discontinuous. This effect tends to counterbalance the small shift to higher SNR of the signal recovery transition-in the large dimensional limit-obtained at higher overparametrization in specific instances, effectively reinstat- ing a generalized advantage of overparametrization in realistic applications. Finally, we discuss the for fixed p* and we reobtain the weak recovery algorithmic
large overparametrization limit p transition at an SNR equal to p² /2, already discussed the literature for p' in Mondelli & Montanari (2018) and for p" > 1 in the Bayes optimal setting in Maillard et al. 2024) Troiani et al. (2024).
2 THE MODEL
The teacher-student framework provides simplified yet powerful setting for studying supervised learning. In this setup, a student network 5(x) R- + R is trained to match the outputs of an unknown teacher function y(x) IN + R using a set of M aNV labeled examples. In this model the parameter a, which controls the size of the dataset compared to the input size, acts as the SNR described in the previous section. The examples consist of input vectors [x* drawn independently from a Gaussian distribution N IN IN/N), along with their corresponding teacher outputs yd ) In the case of coft-committee machines with quadratic activations, for given input vector x², the output of the teacher network is:
y(x")
(wi x")
(1)
where pr is the width of the hidden layer of the teacher network and u²² = w/ x" is the pre- activation output of the 1-th teacher node wi. Each teacher node w is independently sampled from the sphere SN-1(VN). Similarly, the output of the student network is defined as:
((x*)
1 P k=1
(Wk
x²)
(XH)
where p is the width of the hidden layer of the student network and X WA x" is the pre-activation output of the k-th student node Wk. Standard gradient descent algorithms iteratively modify the
weights {w 1 to minimize an empirical loss on the training data ) Following previous works (Bonnaire et al. (2025), we define family of normalized quadratic loss functions:
M=aN
M=aN Σ
[x((") )1
Lw
(x"1
(3)
a+ (x*)
p=1
μ=1
where the parameter > 0 controls the strength of the normalization that prevents pathologies due to rare very small or very large teacher outputs. By regulating the conditioning of the Hessian eigenspectrum, the denominator ensures the appearance of finite left edge, an essential feature for our analytical analysis focusing on an isolated eigenvalue exiting from the left.
Instead of studying the dynamics of the learning process, for which we only provide preliminary results in Appendix [ we focus here on the structure the loss landscape itself. In partieular, are interested in the local curvature of the empirical loss at initialization, which is governed by the spectral properties of its Hessian matrix H E xpN This can be seen as a block matrix, comprising of p² blocks of N x N matrices Hyg' defined as
02
aN Σ
aN FP
(Hor )is O(w,); O(
(Lu (.. Σ
3

---

Published as a conference paper at ICLR 2026
F# 2 p N + &og [ - (2p) 1 Ei=l (uft))
where
(5)
a a -i (uf))
where the pre-activations ( and (uf) are random iid variables (V,0
Let (h,) denote the eigenvalues of H. In the large-N limit, the spectrum consists of a continuous
"bulk" component, described by the density
PN Σ 8(A h4),
(()) lim
(6)
-00 P.V
along with finite number of outlier cigenvalues. From these. can extract information about the geometry of the loss landscape, such as the presence of directions correlated with the signal. This procedure can be connected to a broader class of techniques known spectral methods (Lu & Li. 2020; Mondelli & Montanari, 2018 Maillard et al. 2022). These approaches are based on
constructing matrices of the form
D Σ T()(x*)) (x*)?
where T" R - R is an appropriate pre-processing function, to study the simple phase retrieval problem (which in our notation corresponds the p p" case). The leading eigenvector, i.e. the eigenvector associated with the largest or smallest eigenvalue, depending on the sign convention, then computed provide an estimate of the underlying signal. This estimate can be used directly as proxy for the signal or serve as an initialization for a subsequent descent-like optimization algorithm.
Whether this spectral reconstruction successfully aligns with the true signal depends on the signal- to-noise ratio a. This phenomenon is captured by the Baik-Ben Arous-Péché (BBP) transition (Baik) et al. 2005), which describes phase boundary in the spectrum: only when the signal-to-noise ratio exceeds a critical threshold BBBP does a leading eigenvalue detach from the bulk of the spectrum, allowing its associated eigenvector to carry non-trivial information about the signal. Below this threshold, the spectrum remains uninformative, and the leading eigenvector fails to align with the teacher.
Interestingly, for p² the forms of matrices H and D are similar, the main difference being that the pre-processing function T only depends the labels y(x²)) while the factors F depend both on the labels and the student outputs (xx"). For the right function T however, H can be mapped into D by averaging over the student weights w. Spectral methods then can be interpreted as extracting information from this averaged Hessian. This was first noted in Biroli et al. (2020), where authors use this perspective to develop a spectral method for a different inference problem called tensor PCA.
In this work, rather than studying the averaged Hessian, we study the spectral properties of the actual Hessian, following the lines of Bonnniire et al.) (2025). We extend this analysis to the more general case of arbitrary student and teacher widths (p. p ). Specifically, in this work we study the BBP transition of the training loss Hessian at initialization, i.c., when the student network weights are randomly and independently sampled from the sphere SN-I(VN) for teacher with generic number of nodes (p" 2 1), and examine the effect of student overparameterization (p > p²). In some sense, overparameterizztion the many student nodes. Indeed, we will show that the limit of infinite overparameterization, the performance converges that of the optimal spectral method found in Mondelli & Montanari (018)) for p" 1.
be viewed implicitly averaging the loss landscape across
To build intuition for how the BBP transition extends beyond the phase retrieval setting, begin by recalling the simpler case. In phase retrieval, for signal-to-noise ratios larger than critical threshold 0BBP single eigenvalue X* separates from the bulk of the spectrum, and its associated eigenvector v* exhibits nontrivial alignment with the signal vector v. This alignment is quantified by the normalized overlap
m
| M
(8)
4

---

Published as a conference paper at ICLR 2026
In the more general two-layer teacher setting, isolated eigenvalues similarly correspond align- ments between student and teacher nodes. The student output can be rewritten as
W W x"; WE BOXN, Wh (wx)4:
ú(x*) (xx*
P
where the matrix W collects the student weight vectors Wi. This expression is invariant under rotations W + OW with O E RexP an orthogonal matrix, implying that the learned configuration is only identifiable up to orthogonal transformations (Sarao Mannelli et al. 2020b Martin et al. 2024 Bocchi al. 2025b). For an eigenvector kl associated with alignment between student node k and teacher node the overlap is defined as:
YIE ΣEO (M
(10)
where Mt = V (W"), W collects the teacher weight vectors w and Vkl E RR is the reshaped eigenvector matrix. This represents the Frohenius norm of the overlap matrix between the eigenvector and teacher weights. Since all student-teacher overlaps are equivalent, they can be summarized by single scalar parameter mm Ak:, I. Further details can be found in appendix
3 METHODOLOGICAL OVERVIEW
In this section we give a brief overview of the analytical methods used to compute the spectrum of the matrix H in the case p p" 1. We refer to the appendix the full details of the computation, and how it can be extended to calculate aBpP. Our analysis is based on field-theoretic techniques first introduced in Zee (1996), but rarely employed in the Statistical Physics/Machine Learning community. The objective is the computation of the Sticltjes transform of the spectral distribution
g(=) lim E.
(11)
N
-I H
This quantity can be cast in the field-theoretic formalism by introducing an N N-dimensional scalar field of and using basic identity for Gaussian integration to write
g(z) lim Ex 1 f dabe 1=T 70 +
(12)
N -00 2
where is the normalization constant. This integral cannot be computed exactly, however we can expand the exponential e = * and take the average of every term with respect to the gaussian measure of the fields & and { ju=z. According to Wick's probability theorem, these averages can be expressed as the sum over all possible pairings between fields of their covariances. For example the average
(Mater )n = (xf=+") ((rvth)) + ((be)))))) + (1iely) (thw)))
(13)
To track such combinations. graphical method due to Feynman is used, and each term is expressed as a diagram. We represent each (Waeyy) with straight black line, and every (af" xf") with a double blue line. For example, the first term in the expansion be written in diagram form
Weight: N E FP
By understanding the general form of these diagrams, possible to rule out whole family of subdominating ones, and to express the Stieltjes transform function of the sum of certain type of diagrams, that are called in Physics --Particle Irreducible (1PI) diagrams. we call (2)) the sum of all such diagrams, then get the final expression
g(2) E(²)
(14)
5

---

Published as a conference paper at ICLR 2026
The problem of computing the Stieltjes transform is thus reduced that of understanding which IPI diagrams are dominating. We will see in the appendix that although the number of such diagrams is infinite, their contribution can be summed analytically, and convenient elegant form for (2) can be derived.
4 RESULTS
In this section we present the main analytical predictions for the BBP thresholds and their compari- with finite-N simulations. All derivations, including the field theory techniques used compute the bulk distribution and outlier eigenvalue, are deferred to the Appendices Annd B
4.1
ANALYTICAL BBP TRANSITION
0.7
0.5
000
0.5
9p00
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
3.0
1.5
2.0
Figure 1: Overlap between the signal estimate and the true signal as a function of a for continuous BBP (Left) and discontinuous BBP (Right), with p and p* 1.
The critical value ONBP is analytically determined by imposing the condition
(aBBP) (aBBB))
(15)
where A_ denotes the left edge of the bulk spectrum and X, denotes the outlier eigenvalue. Depend- ing on how the eigenvalue spectrum p()) vanishes near its left edge when equation 15 satisfied,
the nature of the BBP transition can be one of two types, either continuous discontinuous (Boc-
chi et al., 2025a Bouchbinder et al. 202I Potters & Bouchaud 2020). A more detailed analysis of discontinuous BBP transitions and their finite size effeets is addressed in [Bcchhi al. (2025a). For the sake completeness here we re-discuss some aspects in relation to their application to our teacher-student learning problem.
In the continuous BBP transition, the overlap m between the eigenvector associated with the outlier eigenvalue and the signal(s) continuously grows from 0 finite values as the signal-to-noise ratio increases above the threshold aBp.. This case corresponds to sharp edge of the spectrum, where the eigenvalue density vanishes with a square-root singularity:
p()) x 2
(A 1
(16)
In contrast, if the BBP transition is discontinuous the value of the overlap immediately jumps from 0 to finite value as soon ABBP. This occurs when the left edge of the spectrum is smooth, with the density decaying exponentially as
p(A)
& A-
cxp
for some constant A > 0.
(17)
(A
x
In Figure we show the behavior of the overlap in the two different scenarios. While their difference clear in the large dimensional limit N - oc, distinction between the two types of transitions is also visible for finite system sizes, as discontinuous BBP transitions are characterized by a strong anticipation of the transition at N finite, which we diseuss in Section 4.2
Depending on the student and teacher number of nodes p. p², as well as the normalizing constant a, either type of transition can occur. In what follows, we present results for p" 1, but verify in the

---

Published as a conference paper at ICLR 2026
appendix that that varying p" does not qualitatively alter the overall picture. Figure shows the critical ratio ΛBBP, revealing two principal effects. First, OBBP (a) shows non-monotonic dependence on at fixed p. with its minimum placed at critical value a.(p) where the transition changes from continuous (left) to discontinuous (right). In other words, for given (and p"), the most convenient allowing for earliest recovery transition is the where the BBP transition is at the verge of becoming discontinuous. Second, while increasing at fixed a generally reduces ΔBBB,, we again observe a critical threshold Pe(a) beyond which the transition becomes discontinuous. Note that in its vicinity (see middle inset of Figure 2) non-monotonic dBBP( (p) behavior is sometimes visible: the subsequent small increase of (BBP(() with p. i.e. increasing overparametrization,
at odds with the general expectation of the benefits of overparametrization in smoothening the landscape to let the signal emerge. However, as we will see in Section 4.2 this weak effect obtained in the infinite dimensional limit can be masked by strong, non-trivial finite-size effects, reinstating a general advantage of overparametrization for all practical purposes.
p=1
2.5
p=3
p=4
2.0
BBP
1.5
1.0
0.5
0.4
0.7
1.0
1.3
a
0.4
0.7
1.2 dhcontinanss
1.4
contimuoos
1.4
Proce discommnoous
1.4
1.2
1.0
0.8
P
Figure 2: QBBP as function of for p" = and several values of p. The point at which the curves start increasing almost linearly is the point in which the transition becomes discontinuous. The dashed line shows (BBP(a) in the large overparametrization limit where the transition is always discontinuous. The insets show QBBP indicate the transition continuous, while blue points that is discontinuous. The red crosses are estimates of og. "finite-N" estimate of the transition described in section 4.2
function of for three fixed values of Here, red points
Infinite overparamecrization limit infinite overparamettrzaaiion. p xo. This limit taken after the N oo limit, so we remain in the regime where p < N. In appendix B we show that in this limit the BBP transition is always discontinuous, with a threshold given by
Before analyzing finite-size effects, we consider the limit of
pa + 1)
p=00
(18)
BBP
2
For any value of p", the minimum of a occurs at a = and is equal to p² /2, matching the cnommatioo-theorttic weak recovery threshold identified in Maillard et al. (2024). Note that setting is far from being Bayes optimal as in Maillard et al. (2024) since the overparametrized student, by definition, does match the teacher structure. Yet, this result shows how powerful

---

Published as a conference paper at ICLR 2026
overparametriizatoon can be, as in the large overparametrizzttonn limit, even simply extracting spectral information from the Hessian at random configurations, the optimal weak recovery threshold can be achieved.
4.2 NUMERICAL SIMULATIONS AND FINITE N BBP TRANSITION
In this section we compare the BBP thresholds calculated above with the empirical BBP threshold obtained in simulations of finite-dimensional problems with p p" = 1. For a fixed value of a, and over a range of values of a, we generate the Hessian H. for several values of the dimensionality N of the problem, look its eigenvalue spectrum, and count the number of times the eigenvalue associated to the eigenvector with maximum overlap with the signal is the smallest. In theory. this frequeney, which we denote with the letter Ø, should go from limit at BBBP. We perform this experiment for values of for which the transition is both continuous and discontinuous.
discontinuously in the N oo
0.6
a 15
0.7
200
N 200
0.8
Frequency
800
1600
Frequency
N 800
1600
0.7
dgpP
0.5
0.6
0.4
0.5
0.4
1.00
1.25
1.50
1.75
1.00
1.25 &
1.50
1.75
a
Figure 3: Comparison of BBP transitions for p= = the fraction of times the eigenvector with the maximum overlap with the signal corresponds to the smallest eigenvalue. On the left a value of a for which the transition continuous, on the right value for which it is discontinuous. The vertical blue lines show prediction for the BBP threshold, while for the discontinuous case the red line shows our estimate of cg-
On the y axis we plot Φ, defined as
We observe that when the transition is continuous, the predicted BBBP threshold matches the point which the curves for different values of V intersect, while when it is discontinuous the threshold evaluated in the large N-limit is always above this point. That is, in the discontinuous case, our N + 5o prediction for OBBP greatly overestimates the finite N behavior. The explanation this phenomenon is related the shape of p(A) near the edge. When the transition is continuous the left edge sharp and for a finite matrix then the typical deviation the smallest eigenvalue of the bulk from the left edge is of the order of N 2/3, When the BBP transition discontinuous the left edge is smooth and, consequence, for finite N matrix it is much harder to sample this tail of the eigenvalue distribution and the smallest eigenvalue of the bulk will larger than the N oo edge by distance of the order of 1/ log(N) > 2/3 (Bocchi al., 2025a). Therefore the tails of the eigenvalue distributions for finite dimensional problems are much shorter than expected and allow the BBP eigenvalue to exit the bulk earlier. Unlike continuous BBP transitions, here the BBP eigenvalue retains finite amount of information about the signal at the transition point and con- tinues so for a < (BBP,, least long as it remains the smallest eigenvalue of finite-N matrices. These strong finite-size effects and the residual information explain why the observed al- gorithmic threshold (Figure right panel) lies below the predicted QBBP in the discontinuous case. To corroborate this explanation carried by the BBP eigenvalue for a < OBBP must decrease following a square root behavior until its vanishing at a smaller ogg. Calculating the values of m² for various values of > QBBP, is
conjecture that the residual information
about the teacher
'The conjecture based the fact that for smaller N the domain where g is not real, due to the singularity arising the denominator Equation equation 42 shrinks as Cyinn. increases Craz > 0 decreases. As a consequence is possible to approach closer the point where da vanishes and triggers the square root de vanishing of it is the when approaching the continuous transition.
8

---

Published as a conference paper at ICLR 2026
1.4
0.33
0000000000
naP
0.25
1.2
BBP
0.3
1.0
1.1
Bga p= Og OgBp p=
3
0.8
0.0
0.5
1.0
1.5
Figure 4: OBBP (solid) and au (crosses) for two different values of p as a function of a.
possible to perform linear fit and obtain og from the intersection of its extrapolation to lower with the x axis. The result this analysis is shown in Figure 4 The inset shows how the value of the threshold is extracted. On the right side of the plot we the predicted discontinuous BBP transition as a function a for a couple of p and p" = 1 and the extrapolated signal to noise ratio ag at which it is expected that the BBP eigenvalue will completely loose its information about the teacher. As we can see, this threshold lies below the corresponding predicted discontinuous BBP transition and slightly below the empirical transition for the finite dimensional version the prob- lem as shown in the right panel of Figure 3 The expectation is that Og must represent a lower bound for the empirical transition at finite N described above as there is no hope that the lowest eigenvalue carries finite correlation to signal when even the BBP eigenvalue has lost it and that the empirical transition will very slowly move to the higher a as N increases as the bulk cigenvalues will populate the tail progressively hiding the BBP eigenvalue. We also reported the estimated threshold oo for the loss of information the BBP eigenvalue in the middle and left insets Figure 2 Note that it appears always monotonically decrease with p, finally supporting the intuitive principle that overparametrizzttoo should favor learning.
CONCLUSION AND DISCUSSION
this work, we presented a theoretical analysis of the loss landscape at initialization for teacher- student setup with quadratic activation, considering networks with generic, but finite, number of nodes, both for the teacher and the student. We investigated whether it possible to extract infor- mation about the teacher simply by looking at the spectral properties of the Hessian initialization, which reflects the curvature the loss landscape in random configurations, without using iterative algorithms like gradient descent. the high-dimensional data limit where both the input dimension N and dataset size M diverge while maintaining finite signal-to-noise ratio a M/N 0(1), obtain that small a the initial Hessian contains no information about the teacher, while larger one or more Hessian's 's principal eigenvectors develop finite correlation with the teacher in a phenomenon called BBP transition. This approach resembles that of spectral algorithms (Mondelli & Montanari] 2018 Kovačević et al.) (025). which employ matrices that for some inference prob- lems can be seen Biroli et al. 2020) as Hessians averaged over many random choices the student weights, to recover signals via spectral analysis. Nevertheless, isolate the effect overparametrization on this signal recovery transition. We complemented our theoretical findings with numerical simulations, fully characterizing this phenomenon for both finite and infinite N. Our analysis leads to the following key results:
approach makes it possible to
The BBP transition varies qualitatively with overparameterizztion and choice of loss. De- pending on the number of student nodes and the loss function's 's normalization constant a, the transition be either continuous discontinuous, The key difference between the two cases lies

---

Published as a conference paper at ICLR 2026
in the overlap behavior at the transition: in the continuous case, the correlation with the teacher increases smoothly from zero when a increases, while in the discontinuous case, the outlier eigen- vectors exhibit a finite overlap with the teacher immediately at the transition. Note that larger over- parametrization is systematically associated to a discontinous BBP transition for signal recovery from the spectra of the Hessian at initialization. This result comes as the first practical application of the concept of a discontinuous BBP transition-very recently introduced and discussed in Potters & BBuchaud (2020); Bocchi et al. (2025a) association with overparametrization for a machine learning problem.
Overparameterization tends to anticipate the transition, with notable exceptions. Increasing (i.e., overparameterizing the student) for fixed a generally lowers QBBP, so that larger networks need less data to develop informative modes. Yet, for each fixed normalizing constant a, there exists critical student size (a(a) beyond which the transition becomes discontinuous. Near this threshold, (B)P(P) can be non-monotonic and its precise shape depends on a. On the one hand, the overall trend confirms the generally established intuition that overparametrization is beneficial to learning, even extending it to the possibility to retrieve information about the teacher at initialization. On the other hand we observe notable, despite of small entity, exceptions such behaviour. Surprisingly the entity of such exceptions ends up being further mitigated by finite size correction in empirical observations, reinstating general advantage of overparametrizatioon for most practical purposes.
The large overparametrization limit achieves optimal performances. In the limit of infinite overparameterrizatoon (p - oc), information about the teacher emerges through discontinuous transition for all values of a. Intuitively, highly overparameterized student can reproduce the teacher's weights an infinite number of times. This idea, discussed in other contexts (Biroli et al., 2020), helps explain why heavily overparameterized models can avoid overfitting and achieve better generalization, as if accessing an average view of the loss landscape. In our setting, the averaged Hessian at initialisation has a similar form to the spectral matrices used in (Mondelli & Monta- nari 2018). However signal recovery via this spectral analysis, examined (Bocchi et al.] 2025a). does not quantitatively match the BBP transition in the large-p limit, since it requires a stronger signal-to-noise ratio. Finally, as a - O, the BBP threshold CBBP in the large overparameteriza- tion limit converges to the informatiion-hheorett threshold for weak recovery- surprising result showing how strongly overparameterization can reshape the loss landscape to reveal the hidden sig- nal. Remarkably, simple spectral analysis of the Hessian initialization, far from Bayes-optimal conditions, suffices to match the weak recovery threshold in optimal settings.
Finite-size correction affects the discontinuous BBP transition. We compared the predictions for the BBP transition at different values of p. p", and a with its numerical estimation for prob- lems with finiteddimensional datasets, for several value of the dimensionality N. We obtained very good agreement in the case of continuous BBP transition but we observed a strong mismatch in the case of discontinuous BBP transitions. As also discussed in general in Bocchi et al. (2025a), we argued that these effects must be very strong- logart-ogaaittmicccc in N. They are due to the smooth nature of the spectral edge, which finite-N matrices fail to properly sample, and the large amount of residual information of the leading eigenvector even below the transition point. The undersampling of the tails gets stronger the lower N and it allows the BBP eigenvalue to emerge earlier than the predicted BBP threshold, resulting in numerical signal-recovery transition much lower than the predicted BBP transition. We also extract a lower bound ag to the numerical transition evaluat- ing the signal-to-noise ratio where the extrapolated overlap of the leading eigenvalue vanishes as square root. The empirical transition is expected to slowly move from Ogp to higher a approaching the BBP transition only in the large N limit. Finally, surprisingly og is found to decrease with p so that in the accessible finite-N cases overparametrizatton turns out to be effectively advantageous for the empirical signal-recovery transition even when the predicted discontinuous BBP transition gets anomalously postponed. As suggested in Bonnaire et al. (2025), the fate of standard gradient descent should be influenced by the interplay between the emergence of the signal in the Hessian at initialization and the gradient-flow algorithmic transition. The latter can be predicted by evaluat- ing the signal-to-noise ratio which threshold states develop an instability toward the signal, also through a BBP transition. Understanding how overparameterizztton affects this second transition, both quantitatively and qualitatively, remains a very interesting open problem.
Finally, although our analysis focuses on the case of networks with quadratic activation functions, we show in appendix [ that a qualitatively similar behavior is expected to hold in the more generic case with arbitrary activation functions.
10

---

Published as a conference paper at ICLR 2026
Acknowledgements We thank Giulio Biroli for early discussions the field theory approach to RMT, and Lenka Zdeborova for insightful exchanges. We acknowledge financial support from the European Union NextGenerationEUU (PNRR) Fund, Mission 4, Component 2. In particular, we acknowledge support from Investment 1.1 Project 202234IKKBW "I "Land(e) sccapes::: Statistical Physics Theory and Algorithms for Inference and Learning Problems", CUP B53D23003885006 (PRIN 2022). This work was also supported by Investment 1.3 - FAIR Foundation, Extended Part- nership "Future Artificial Intelligence Research" (Project Code PE00000013-FAIRS and by Invest- ment 1.4 National Research Center in "High Performance Computing, Big Data and Quantum Computing" (Center Code CN00000013-CNI, ICSC).
REFERENCES
Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, and Aukosh Jagannath. Local geometry of high-dimensional mixture models: Effective spectral theory and dynamical transitions. arXiv preprint wrXiv22502.15655, 2025.
Jinho Baik, Gerard Ben Arous, and Sandrine Peche. Phase transition of the largest eigenvalue for non-null complex covariance matrices. Annals of Probubility, 33, 09 2005. doi: 10. 10.1214/ 0091 17905000000333
Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gérard Ben Arous, Chiara Cam- marota, Yann LeCun, Matthieu Wyart, and Giulio Biroli. Comparing dynamics: Deep neural networks versus glassy systems. In International Conference on Machine Learning. pp. 314-323. PMLR, 2018.
Tamir Bendory, Yonina C Eldar, and Nicolas Boumal. Non-convex phase retrieval from stft mea- surements, IEEE Transactions on Information Theory, 64(1):467-484 2017.
Giulio Biroli, Chiara Cammarota, and Federico Ricci-Tersenge How to iron out rough landscapes and get optimal performances: averaged gradient descent and its application to tensor pea. Journal of Physics A: Mathematical and Theoretical, 53(17): 174003, 2020.
Dario Bocchi, Giulio Biroli, Chiara Cammarota, and Federico Ricci Tersenghi. Discontinuous bbp transitions. preprint, 2025a.
Dario Bocchi, Theotime Regimbeau, Carlo Lucibello, Luca Saglictti, and Chiara Cammarota. On- line learning for soft committee machines. preprint, 2025b.
Tony Bonnaire, Giulio Biroli, and Chiara Cammarota. The role of the time-dependent hessian in high-dimensional optimization. Journal of Statistical Mechanics: Theory and Experiment, 2025 (8):083401,
2025. 2025.
Eran Bouchbinder, Edan Lerner, Corrado Rainone, Pierfrancesco Urbani, and Francesco Zamponi. Low-frequency vibrational spectrum of mean-field disordered systems. Physical Review B, 103 (17):174202, 2021.
Pierre Bousseyroux and Mare Potters. Spectral initialization for dimensionl phase retrieval with biased spatial directions. arXiv preprint arXiv:2403. 15548, 2024.
Jian-Feng Cai, Meng Huang, Dong Li, and Yang Wang. Solving phase retrieval with random ini- tial guess is nearly as good as by spectral initialization. Applied and Computational Harmonic Analysis, 58:60-84, 2022.
Yaim Cooper. Global minima of overparameterized neural networks. SIAM Journal on Mathematics of Data Science, 3(2):676-691, 2021.
Cirano De Dominicis and Irene Giardina. Random fields and spin glasses: field theory approach. Cambridge University Press, 2006.
Leonardo Defilippis, Yatin Dandi, Pierre Mergny, Florent Krzakala, and Bruno Loureiro. Optimal spectral transitions high- sighdiimessiona multi-index models. arXiv preprint arXiv:2502.02545, 2025.
11

---

Published as a conference paper at ICLR 2026
Yan V Fyodorov. Complexity of random energy landscapes, glass transition, and absolute valuee of the spectral determinant of random matrices. Physical review letters, 92(24):24060, 2004.
Filip Kovašević, Yihan Zhang, and Marco Mondelli. Spectral estimators for multi-index models: Precise asymptotics and optimal weak recovery. arXiv preprint arXiv:2502.01588, 2025.
Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414): 14):114433662222 1991.
Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems, 33.8543-855 8552, 2020.
Yue M Lu and Gen Li. Phase transitions of spectral initialization for high-dimensional non-convex estimation. Information and Inference: A Journal of the IMA, 93)):077-54,, 2020.
Antoine Maillard, Florent Krzakala, Yue M Lu, and Lenka Zdeborová. Construction of optimal spectral methods in phase retrieval. In Mathematical and Scientific Machine Learning, pp. 693- 720. PMLR, 2022.
Antoine Maillard, Emanuele Troiani, Simon Martin, Florent Krzakala, and Lenka Zdeborová. Bayes-optimal learning of an extensive-width neural network from quadratically many samples. Advances in Neural Information Processing Systems, 37:82085-82133 2024.
Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborova. Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models. In international conference on machine learning, pp. 4333 4332. PMLR, 2019.
Simon Martin, Francis Bach, and Giulio Biroli. On the impact of overparameterizztion on the training of shallow neural network in high dimensions. In International Conference Artificial Intelligence and Statistics, pp. 3655-3663. PMLR, 2024.
Rick Millane. Phase retrieval in crystallography and optics. Journal of the Optical Society of America A, ((3):394-411 1990.
Marco Mondelli and Andrea Montanari. Fundamental limits of weak recovery with applications to phase retrieval. In Conference On Learning Theory, pp. 1445-1450. PMLR, 2018.
Andrea Montanari and Nike Sun. Spectral algorithms for tensor completion. Communications on Pure and Applied Mathematics,
2018.
Praneeth Netrapalli, Prateek Jain, and Sujay Sanghavi. Phase retrieval using alternating minimiza- tion. Advances in Neural Information Processing Systems, 26, 2013.
Arkadiusz Orl, Harry Paul, et al. Phase retrieval in quantum mechanics. Physical Review A, 50(2): R921, 1994.
Mare Potters and Jean-Philippe Bouchaud. A first course in random matrix theory: for physicists. engineers and data scientists. Cambridge University Press, 2020.
Valentina Ros, Gerard Ben Arous, Giulio Biroli, and Chiara Cammarota. Complex energy land- scapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions. Physical Review X, 9(1):0 1003, 2019.
Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, and Lenka Zdeborová. Who is afraid of big bad minima? analysis of gradient-flow in spiked matrix-tensor models. Advances in neural information processing systems, 32, 2019.
Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Complex dynamics in simple neural networks: Understanding gradient flow in phase retrieval. Advances in Neural Information Processing Systems, 33:3265-3274, 2020a.
Stefano Sarao Mannelli, Eric Vanden-Eijnden, and Lenka Zdeborová. Optimization and gener- alization of shallow neural networks with quadratic activation functions. Advances in Neural Information Processing Systems, 33:13445-13455. 2020b.
12

---

Published as a conference paper at ICLR 2026
Alexander Shevchenko and Marco Mondelli. Landscape connectivity and dropout stability of sgd solutions for over- parrrraaaaaeezeed neural networks. In International Conference on Machine Learn ing. pp. 5773-8784 PMLR, 2020.
Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint UrXiv:1605.08366 2016.
Ju Sun, Qing Qu, and John Wright. A geometric analysis of phase retrieval. Foundations of Com- putational Mathematics, 18(5): 11--199, 2018.
Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.
Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, and Flo- rent Krzakala. Fundamental limits of weak learnability in high-dimensional multi-index mod- els. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024.
Anthony Zec. Law of addition in random matrix theory. Nuclear Physics B, 4744)))77667444 1996.
LLM Disclosure LLMs were employed in this work to review the manuscript, correct grammati- cal errors, and polish the writing to improve clarity and readability.
A FIELD THEORY APPROACH
In this technical section, we give an overview of a Giardina 2006) used to derive self-consistent equation for the Stieltjes Transform of the Hessian, defined as
approach (Zec 1996 De Dominicis
1 TrG(2) lim Ex R Np
1
g(2) lim N
Ex N Np
Tr -I
(19)
N
The eigenvalue spectrum density can be obtained via the Stieltjes inversion formula:
p(A) lim -00 π
Im Img(A - ic).
(20)
Furthermore, we will build on the same formalism to obtain self-consistent equation for the outlier eigenvalue X", when exists.
The first step is to use the rotational invariance of the teacher weight vectors. Without loss of gener- ality, we fix them to lie along the first p" canonical directions w = VNeg. for g. (1,,..,,") where the VN ensures the correct normalization. With this choice, the teacher pre-activations re- duce to uf" VNaf' ~ (0, 1). separate the parts of the matrix where the coefficients Fag are correlated to the components of the vectors x!", we permute the rows and columns of H so that the first pd rows and columns of each block Hog' are grouped together in the top-left corner
submatrix with elements (HL))--4 En FK F ayy""" zyt for q. E (I.....,p... q.. 'qg
(l.....p"). H. should be understood block matrix, consisting of (p*)² blocks, where each block is a p x matrix (H The final shape of this permuted H is
H-
H 7b
H = (HL
H E RPP xpp" 7tt E (NNN==''xx(N-2 H E RPP x(N-x
---ppp )p
(21)
In section A.1 we calculate the spectral distribution of its bulk eigenvalues p(A). It is standard Random Matrix Theory result (see for example exercise 2.4.3 of Tao (2012)) that the spectral dis- tribution does not change if we remove a number of rows and columns whose Frobenius norm is o(N). Since every element of H is (1). the Frobenius norm H is clearly O(1). while by the strong law of large numbers the Frobenius norm of H O(VN). For the purpose of computing the spectrum bulk then
simply discard them, and compute directly the Stieltjes transform of the matrix H. We will that this leads to great simplification.
13

---

Published as a conference paper at ICLR 2026
instead we focus the pp" outlier eigenvalues, which not captured by the distri- bution p()). In this case we cannot simply ignore the other blocks the matrix H. Indeed, if we divide the resolvent matrix ((2) in blocks in the same way,
In section A.2
G-GG
G GT G)
G
G
G E XPF GERWW-''PXNN-') G ReP x(N-P)),
(22)
we have that the top left corner G encodes precisely for these outlier eigenvalues. Since G and H are related by an inverse, G will depend on all four blocks of H. We will calculate exactly using field theory.
A. SPECTRUM BULK
Following Zee 1996), the starting point for the calculation of the Stieltjes transform of 7. is to use basic identity for Gaussian integration to write
(2)) = lim Ex Tr N
Np
2I(NN-p HD
II dopbee Ene (1ba) (xTix 20 (10--")) H))aa( Der tu Σ /b/l1 q=1
1 No
lim
Ex
(23)
N 80000
Z
Here, introduced p (N p" -dimensional scalar fields t/ and denoted by
[II H²)
the
99'
qq'-th block of the matrix =I Hº. The next step is to get rid of the normalization constant Z using
the replica trick 1 limm- zn and introducing
replicas of the scalar fields { Ja=1
d - Enw (2)) ) th Σ
g(=)
lim lim Ex
N N n-0
Np
1
9
= lim lim N n-0
+ Σ Np
(24)
N,N
This integral cannot be performed analytically, as the x appear the covariance matrix of the fields
<b@- However, if expand the exponential e - E (*"") "H t5..
reduced computing the average of every term with respect to the "bare" measure, namely the measure that appears in equation 24 with H." set to zero. Since for every element of the Hessian in the bulk
aN (H )ig = Σ
Fli Th
(25)
11=1
Wick's probability theorem-according to which every higher order moment can be expressed function of second moments-to compute each term of the expansion. To wield the power of Feynman diagrams we identify two fields, one for yi and one for af" which in accordance with Zee (1996) we call "quark" and "gluon" fields. Their bare propagators, g¹, are defined as the correlations of the fields in the "bare" measure. We will represent the former as straight lines and the latter as double lines.
Fe is independent of 2 gg'
a, i,
a, 8, g
Quark z:
i, 14
i,
gluon
Ex N
The interaction between the two fields can be read off H³. and can be represented with a vertex of the following kind and its corresponding weight
14

---

Published as a conference paper at ICLR 2026
8
J.
Weight: FM
a.
0, Jo
Note that although we will have consider also propagators between different blocks (vi, ( which don't depend on the index i. From now on we will use Go to indicate the x matrix formed by these elements, and with Go quark Squart P the diagonal p matrix that contains the bare quark propagators the diagonal. The Stieltjes Transform can be then obtained from g(z) = TT (z).
interested in the propagators (())))))))))))))))))) to derive self-consistent equation
Let us begin by examining the first set of diagrams. Since the contribution identical for any index we may, without loss of generality, fix 1. In the diagrams shown below, propagators associated with a, i = will be left unlabelled. When propagator corresponds to a generic i or a, we will explicitly annotate it the line as a reminder that the corresponding index must be summed over. For each diagram, we will also indicate its total weight. With one vertex we have diagrams:
1.
1 N 2 Σn
FP "g
g
i o
11
i,
2.
1 2N Eogie Fd = .2222 Eng gg6
q
Note that the total contribution of diagrams type is proportional the number of replicas, which in the limit n 0 goes to O. In general, this holds for any disconnected diagram, so in the following we will focus on connected ones. Note also that in diagrams of type the 1/2 factor that comes from the weight of the vertex is canceled by factor 2 that comes from the number of ways in which the vertex can be connected. Other than this 1/2 factor, each diagram also carries an 1/n! factor, where is the number of vertices in the diagram, from the exponential expansion in equation 24 However, this is canceled by a factor n!! that comes from the number of ways the n vertices can be aligned. This cancellation happens at all orders, Let us now consider two vertex diagrams:
will ignore such factors from now
No 2 E
Eusger
FH FP. qq q'
g
40
d
i, ".
No 23 Σ Engi Fu.
g
d'
d"
15

---

Published as a conference paper at ICLR 2026
5.
N 2 Eungr Pag Frr qp
9
d"
The total weight of diagrams of type 4 is o(1), their contribution is negligible in the N + limit. In general, this holds for all diagrams where gluon propagators intersect, so we will exclude these from Diagram is instead obtained by connecting two diagrams type 1 "in series". This a general property of diagrammatic expansion: new diagrams can always be generated combining earlier ones in series through bare quark propagators. If we sum the contributions of all diagrams that are obtained in this way, which in Quantum Field Theory are known as one-partick irreducible (IPI) diagrams, we can exploit this recursive structure to derive a self-consistent equation for the propagator G Let us call D the p x p matrix whose elements Σ> are the sum of all such IPI diagrams connecting fields with block indices q and q', with external bare quark propagators removed (so-called amputated diagrams), Then we can express the sum of all diagrams as
Go G + Go quaark Ed G'Guurri + G Qquark E'
>d Go Gquark +...,
(26)
Factoring out GO quark reveals geometric series, leading
G G = quark (I, GOO quark
GO Gquork
quark
EnG
quark + ..... ((Qguark) 'Ip )ー)
(27)
This gives a self-consistent equation for the matrix G since E depends on G that Physics known as the Dyson equation. This allows us to focus on IPI diagrams from now on. Let us look at diagrams with 3 vertices.
6.
1 No Emiggg Fig FP. Fir
FR Fina o
g
1
d'
i, q"
7.
1 N8 3.4 Euerigg Fb Fgg
Fig Fig
F g
g i, q"
j. g.
As see, diagrams of type are just diagrams of type 3 with diagram of type 1 added to the inner quark propagator. This is a general property that is unique to this field theory: new diagrams can be obtained from previous ones by adding them the inner quark propagators. If the starting diagram is I-Particle Irreducible, then so is the new diagram. Since the sum of all possible diagrams just the propagator, this form of combination can be accounted for by substituting Go to every inner bare quark line.
Indeed, the self-energy can be obtained from the sum of all leading order diagrams mentioned above, where the incoming and outgoing propagators are "amputated", and where every inner bare quark propagator is substituted by propagator Gb qq'" which we indicate graphically with blob.
The first term in the sum is diagram of type 1, which gives a total contribution of x Σ,, qq The second term is given by diagram of type with Go propagator its inner quark line
N' Eni (F"G
9
11
11
99
i
G
16

---

Published as a conference paper at ICLR 2026
Note that the incoming and outgoing propagators are shorter indicate that considering amputated diagrams. The third term is diagram of type 6 where again we substitute propagators Ggg
NY Ens (F"G'F"G'"') qq
9
11 1
P
3
Gamoggim
Note that we do not have to consider diagram of type with propagators because already included in the second term. The next term is just the equivalent of the previous diagram but with three inner arches, from which we can guess the general form of the diagrams that appear in the self-energy.
Summing the weights of these diagrams, we can write in matrix form
Σ F FM + N Σ FP
+...=
PI NP Σ F" (I, - GDF')
N Σ FM (I, F F FFFFGFF +
(In-G'F")
>aEF FLF
(28)
where the last identity we assumed that in the N + oo the sum concentrates to its mean with respect to the dataset distribution.
Plugging this expression into equation 27 we get that the propagator satisfies the self-consistent equation
= zI, aEF (I, F)
(29)
Note that the F matrix is of the form F = axx" + BIp where À is standard Gaussian vector. In particular, it rotationally invariant, so this equation should hold also for O FO where is rotation matrix.
- - FO (I, GOO GO)
(30)
from which
aFF - O"
(31)
This must hold for any rotation matrix O, this implies that the matrix G must be proportional
the identity matrix. If we call g(z) the value on the diagonal, equation 29 can be written in scalar form
1
(=) = a- Σ
ET ((11-9(=)F]+)
EE
(32)
p
g())a
1=1
where the a are the eigenvalues of the F matrix.
A.2 OUTLIER EIGENVALUE
The starting point for the calculation of the outlier eigenvalues is similar, with the exception that have consider the full matrix H calculate the elements of G(z):
P
Gl..% 9 (2) q9'
lim E
dab 3 Ex()") H ()0)()))))))))))))))))))))))))
n-0
n
1=1
X e (.
p -0)
(33)
17

---

Published as a conference paper at ICLR 2026
Here, introduce the notation (z/")= denote for each the pp" -imeennnsii vector with com-
ponents vig for q E {I...., p), ė E {1..... p²) J. and similarly (2b")> )> the p(N - p"") p''-dimeniional vector of components Vhg for E (I,....P),.
p). 1 E {p" + + + N}. The vectors are flattened in
way that the first components of (v6*)< and (uh*)> are respectively {vi 1 and (vi +1)g
the second p components are (1=,, and {4/ (2) ) and
As before, that we cannot average over the first p" components of the x field, since they appear in "gv and we cannot simply use Wick's probability theorem. Indicating this time by a curly line the bare propagator between fields with indexes that belong to (*²)) and with straight lines indexes belonging to (xb")>, we have that the bare propagators are
calculate this integral using diagrammatic expansion, with the only difference
g0
q, 4+ www 9. 9-
En Fou (u(e) 2-00P
a, i,
a, q
i, p
i.
9 Igluon
The vertices are instead given by the old one
j. 1
Weight: 2 FM
a,i,q
a, J,
and
i, g
Weight:
F VN 96 ut.
i,
μ wmmm g'
As before, let us define the self-energy S9.0 as the sum of all IPI diagrams connecting the curly fields with indices q, u. and d'.g' 9h-
Again interpreting Σ as a block matrix, with (x*)² blocks of x p-dimensional matrices Ea.e. then we can write matrix Dyson equation for the resolvent
G G= ( pp- Σ
(34)
We expect the matrix G to be singular when is equal the outlier eigenvalue, self-consistent equation be obtained by imposing the singularity of -1 Σ.
Let us now calculate the self-energy directly, using the same rules we found above, namely that disconnected and gluon-intersected diagrams subleading. As are looking for the self-energy of the "curly" field, we will necessarily need two vertices of the second kind. The first term is given precisely by two such vertices
NO Eni
(F"G
g. 4r ~m 1
p m I.
99"
18

---

Published as a conference paper at ICLR 2026
Next, we can add vertex of the first kind to get the diagrams
1 (F*G*P"G*H")
2.
pa 1.
No
Eui
Ug. u
qq"
3. 9.
pa d.
No Euei (F"GGF"GGF")
u u qq¹
Garia""
Garongua
Diagrams of type however are already counted in diagrams of type 1. We could also add two vertices of the second kind, but we would not obtain 1PI diagrams. The next diagram is obtained by stacking one more arch diagram 2, and again we can guess the iterative form of the diagrams that contribute. Summing the weight of all such diagrams, and remembering that Go [p we get that the self-energy is
Eu.d
1
Σ
oh q-
dh F( (* +..)=
(35)
N
1
Σ
wh " g() (FH)² (I, F" F()())) "]] aEug nhg [(=)) F2 (I, Fg(z))
"J]:
N
where again in the last equation we supposed that the sum concentrates to its mean with respect to the dataset.
Since F is an even function of ug 559.0l is for qo A dc (and in particular, the expectation will be the same for every index (a). Using the fact that Go is diagonal we can write
E9.0. aEu? F (Ip F)())) "]
(36)
Again, F rotationally invariant, so the same equation should hold for rotated F0g((=)))) [1
-- = aEu, 2
(37)
from which
a OT aBu, ((()) - - "1
(38)
For this equation to hold for every rotation matrix O, the matrix E9-a- must be proportional to the identity. Consequently, we obtain Eq. (2)) 04. Imposing the singularity of the RHS of equation 34 we get the equation
(x* A -
=0
(39)
In other words, the pp' outliers all coincide with value X" which can be found by solving the self-consistent equation
- BuffT (F(T Fg((*))7 1 -F
X'
alE Tr +
a p
p
aEE,
2 1
Σ
g(\*)q E(X").
(40)
p 1=1
Combining this last expression with equation g" ==== g(>):
32 one can rewrite self-consistent equation for
Σ (uf 1) - g'
= aF
(41)
g"
i=1
19

---

Published as a conference paper at ICLR 2026
B ANALYTICAL COMPUTATIONS FOR THE BBP
In the large dimensional limit, in order to obtain an expression for the left edge of the spectrum bulk we define the inverse function x(g) of the resolvent g(2), using equation 32 as
=(g) - + E 9
p
c
(42)
gcs
The density of eigenvalues ((A) of the spectrum bulk as a function of = = X will be obtained as the imaginary part of the 9 solution to equation442 In particular the edges correspond to the = - À where non null imaginary part develops. which can occur by means of two different mechanisms. First: the domain of a g real constrained by the requirement that all denominators in equation (2 do not vanish. In particular equation 42 is well-defined for E R (supp(1/c) U {0)), where (upp(1/e) denotes the support of 1/c across all c,, i.e., supp(L/c) U i=1 (1 x supp(c:)) Assuming each
c, has support within [ci,mins Gi,max), we define Cmin = min, (Ca,min) and Cmax = max; max;(Ci,max)
Cmin and Cmax
0 (which will correspond to our case), then
Gmrnn 10.))
ge
(gmin 0) U (0, 9max).
(43)
Crnxx
Second: within this domain it can occur that dz vanishes. Beyond that point the solution to equa- dg
tion 42 cannot be real and non zero imaginary part develops with square-root behaviour. The second case corresponds the standard square-root singularity the edge of an eigenvalue distri- bution.
Sharp edge If there exists a point 9thh (9min 0) at which the derivative of the inverse function
dz
p
cे
alE
(44)
dg g²
P
g-g)
go;)²
vanishes, the left edge of the spectrum can be obtained by applying the inverse function this value, i... nhh = 2(ghh) The corresponding self-consistent condition is
1/2
E
(45)
9 c.
In this case, the left edge of the spectrum is sharp, meaning that the derivative of the eigenvalue density diverges as À approaches the edge from the right. This behavior arises through a standard square-root singularity the edge:
p(A) x (A 20h31/12 2oden
(46)
Smooth edge no solution 9th E (gmin, 0) exists such that the derivative of =[g) vanishes, that is, if equation 45 cannot be satisfied within the domain of definition then the left edge of the bulk spectrum is determined by the boundary of the domain itself. In this case, the edge of the resolvent located
Imin
(47)
Cmin
and the corresponding spectral edge is given by am =(g""). This edge is referred to as smooth, in the sense that the derivative of the spectral density p(A) vanishes as X - Xm In our setting, this occurs via exponential decay of the spectral density near the edge:
A
P(A)
exp
(48)
(A
sm
for some constant A > 0 (see Bocchi et al.) (2025a) for more details about the derivation and the explicit form of the constants), indicating essential singularity at the edge. In the case of
20

---

Published as a conference paper at ICLR 2026
ceacher-student setup, the eigenvalues &;
2
1
3 p
1
P
C1 Pat F"
X
1==1"g
p f=1
(49)
2
1 Σ
1
X
Pat 1 P" Df= ]
2
P i=1
which have the same support C E (-2/p, +00) V [1, [1,p) This implies that for our setting Jmin -p/2. As increase a, for fixed values of p. p' and a, the left edge goes from being smooth being sharp. That is, for a smaller than some value ae. the left edge is obtained from equation 47 while for larger places, age can be determined by imposing the equation
obtained from equation 45 The precise point in which this transition takes
1/2
C
E
(50)
P
22
Depending on the student and teacher number of nodes p. p", as well as the normalizing constant a, either type of transition can occur, as illustrated in Figure In what follows, present results for p" = 1, but we verify section D of the appendix that varying p" does not qualitatively alter the overall picture.
-0447
(0)
-0.475
Op0r
-0.00
-0.400
-0.50
-0.455
-0.500
1.1
(a) p = 1.p" 1,
(b) 2, p
Figure 5: In Panel 5a continuous BBP is shown: the QBBP corresponds the crossing of the
g" (a) and g- (a) curves. In 5b. a discontinuous BBP is shown: the OBBP corresponds to the crossing of the g" (ai) and (min(a) curves.
Infinite overparametrization limit In this paragraph we consider the limit of infinite over- parametrization, p - oc. This limit is taken after the N - oo limit, so we remain the regime where p < N. We begin by computing the critical value below which the transition is discontin- uous. For large we can approximate equation 50 as
-1/2
L p" Σ 44
~
a. E
(51)
2
22a
(1 +a)2
which yields
p" (a + 1)2
a.
(52)
2
We now turn to the BBP threshold for values of such that p=ce (a) (a).. Evaluating equation 41 at g" -p/2, we obtain
pmoo
(a) in the oc limit. This expression will be valid only
BBP
P (uf 1)
=
p=oo
(53)
2
p i=1
1+
21

---

Published as a conference paper at ICLR 2026
which leads to the solution
pa 2
p=
(54)
BBP
Note that for all > O, we have BBP (a) < (r(aa,, implying that the BBP transition is always discontinuous in the p - oo limit. Furthermore, for general p". the minimum of BBBP (a) occurs 0 and is equal to p² /2, matching the information-theereiic weak recovery threshold identified in Maillard et al. (2024). Note that our setting is far from being Bayes optimal setting as in MMillard et (2024) since the overparametrized student, by definition, does not match the teacher structure. Yet, this result shows how powerful overparametrization can be, as the large overparametrizztion limit, even simply extracting spectral information from the Hessian at random configurations, the optimal weak recovery threshold achieved.
p=00
C DERIVATION OF THE OVERLAP
In the general multi-index setting, each isolated eigenvalue may correspond an alignment between a specific student node Wa and a teacher node w/. We denote the corresponding eigenvectors by v E RON and reshape them into matrices V-ki RRx,,, where the first N entries of v form the first row, and on.
We define the overlap matrix:
M = E R
(55)
where W R X is the matrix whose rows are the teacher weight vectors. Ideally, M contains a single non-zero entry m at position (k, 1), i.e., (M = murōk, dr.
However, the output of the student network in equation can be equivalently written
ú(x²) = (xx1) W W
(56)
where W E RRxN is the matrix whose rows are the student weight vectors. This expression reveals that the output is invariant under rotations of the matrix W in the p-dimensional space. That is, for any orthogonal matrix O E Rexp, the transformation W + OW leaves the output function unchanged. A similar argument applies to the teacher network.
This rotational invariance implies that the student and teacher configurations are only identifiable up to orthogonal transformations, provided that the norms of the student nodes are unconstrained. As consequence, the overlap matrix observed in practice takes the form
M kl OV (W*)
(57)
where O RPXP and Ô E RP" are random orthogonal matrice The scalar overlap can then be extracted using the Frobenius norm:
Mal
(M
(58)
Furthermore, due to the symmetry of the problem, all the overlaps will be equivalent:
my = m for ks = I,...,,, 1 1 1, I....,p'.
(59)
We now derive an analytic equation for m. Let us again choose the teacher vectors such that they are the first p" canonical directions, as we already did in the previous section. The starting point is to write the resolvent as
PN A.ss
G(z) (=I H)
(60)
NNte that such orthogonal transformations,, while preserving the output, don't preserve the individual norms of the network nodes. If we apply this rotation to the teacher vector, which is taken to be normalized such that each node on Sw-1(VM) sphere, such normalization property lost. However, since dataset invariant under this rotation, the student cannot infer this normalization from the data, and will align to a randomly rotated teacher OW"
22

---

Published as a conference paper at ICLR 2026
PN where {Ae, U/ firm is the set of cigenvallusseiieenvetoss Without loss of generality, let assume that we are a base which the first eigenvector is aligned with the first teacher. That is, we call é the first canonical direction in pN dimensional space, the overlap is V2 E₁ Multiplying equation 60 on the right and on the left by é,, and taking the limit À - X we get that
m 2
lim of (=lpN - 71) -7)+ E, lim x-x
(61)
x->
X
(=)
The square overlap is the residue of the function 111 at the pole z = x*. Using the relation found in the previous section (2) = (2 -==))) it can be calculated as
m lim 2 X -
(62)
(z)
where remind the reader that
65
E(A) = a
Euf 1 g()cs
(63)
This limit gives the undetermined form 0/0, but can be calculated using l'Hopiaal's rule
m² lim 0,(= X*) - -=(=-E(z) 1-0,3
(64)
D DEPENDENCY ON p" AND UNDERPARAMETRIAATTON
In this appendix we consider cases where pz of 1. In figure 6 we show the equivalent of figure 2 for p" = 2. As we can see the behaviour is qualitatively similar. For the values of p we chose &BBP is always monotonically decreasing function of p, although don't expect this to hold for larger values of p.
(LS
BBB
p=1
D=2
D=4
1
2
Figure 6: QBBP as a function of for p" = 2 for various values of p. In the inset we show QBBP as function of p for a fixed value of 0.5
In figure explore the effect of underparametrizztion, by keeping and p fixed and plotting QBBP as a function of p². Perhaps unsurprisingly we observe that aMaP increases, making the problem increasingly harder the network is more underparametrized.
23

---

Published as a conference paper at ICLR 2026
0.7
60
50
30
10
2
4 5 6
p
Figure 7: annp as function of p" for p and 0.7
E EXTENSION TO OTHER NON-LINEARITIES
In this section consider the extension arbitrary non-linearities of the learning task. We consider a generic non-linear activation function (z)) The teacher and student outputs become
0(x")
0 (w) x") =
(ul))
(65)
1=1
1=1
1 p
1
v(x")
(wl
x"
(N1),
(66)
p 1=1
The expression for the Hessian becomes
aN
1
+ ger -0")
(Hagr)
Σ
Fii qu x 2
Fl 99'
(67)
a + y
Although repeating the theoretical derivation for such a matrix is beyond the scope of this work, let comment that expect qualitatively similar behavior to the outlined above. To support this claim, we show in figure the overlap of the smallest eigenvector with the teacher w" for two non-linear activation functions, (x) (1+ - the sigmoid function and (zx) tanh(r) the tanh function. In the both cases we choose p" = 1 and = 3. Let us note that, while the denominator in the loss function (a + is reasonable choice for positive activation function, it not for a function that can become equal to aa. In the case (xr) = tanh(r) we choose a large studying effectively reduces to the simple MSE loss. As
value of a, such that the loss can see in both cases an outlier that is correlated with the teacher emerges, thus showing that this phenomenon is not restricted to the quadratic activation function.
o sigmoid
tanh
0.4
p-3
N-500
0.4
p= N=500
3 N-1000
0.3
0.3
0.2
0.2
0.1
5
Q
Figure 8: Overlap between smallest eigenvector of the Hessian and the true signal as a function of a for sigmoid activation function (left) and for the tanh activation function (right). For the sigmoid activation a = 1, while to avoid the divergence of the denominator in the tanh case we set a 50. Points averaged over Arans 60 runs.
24

---

Published as a conference paper at ICLR 2026
F CONNECTION TO DYNAMICS
To understand how the static BBP transition at initialization correlates with gradient descent dynam- ics, we performed simulations across different values of p, N, and a, keeping pz = for simplicity. We selected two values of a: 0.1, where the transition is continuous for all and = 2.0, where it is discontinuous. Each simulation ran for 20,000 gradient descent steps, with the learning rate chosen that the average initial update per student component was of order 0.1.
At initialization, the overlaps between student and teacher nodes, defined in equation 10 are all equivalent, and could simply plot their average across all teacher pairs. However, the dynamics can break this symmetry and change student norms, consequently altering their relative importance. For example, condensation can occur: in the simple case of p 2 and p" 1 one student could perfectly align with the teacher (m = 1) and have order one norm, while the other has negligible norm and is uncorrelated (mij 0). Such network can achieve zero generalization error, yet a simple average of the magnetizations would yield (m²) = 0.5. Since network perfor- mance is best captured by the most aligned student, we plot the maximum overlap as proxy for final performance versus
The results, shown Figures and 10, demonstrate that the BBP transition initialization cor- relates with the point where dynamics begin to yield networks that perform better than random guesses, even in overparameterized settings. This earlier transition enables improved performance with fewer samples. The plots also suggest that as N increases, the transition point shifts rightward, approaching the (currently unknown) BBP transition the threshold states, which are the config- urations that ultimately trap the dynamics for large N. Thus, while these preliminary results align with our intuition, a complete characterization the BBP transition at the threshold states remains necessary to confirm this picture.
Continuous BBP Transition: 0.1, p* 1 p
2
1.00
N-256
1.00
2.17
0.75
0.75
0.00
p ==3 3
p 4
1.00
6.75
max((n))
0.30
0.50
5.5
N-254
Ne236 N=1124
N-1022
Figure 9: Maximum squared magnetization m² vs. a for a = 0.1. The BBP transition at intialization correlates with the onset of weak recovery.
25

---

Published as a conference paper at ICLR 2026
Discontinuous BBP Transition: 2.0, p* 1
p
2
1.00
N-256 Newstog
1.00
N-256 TwADsS 1.53
1.00
0.75
0.75
m000(1gg
0.00
p
4
N--024
0.75
0.50
0.50
0.25
Figure 10: Maximum squared magnetization m vs. a for a = 2.0. Also the discontinuous case, the BBP transition correlates with the onset of weak recovery.
26


Reviews:
[{"review_id": "Q5nGQ5ZJ12", "reviewer": "Reviewer_Z5EM", "summary": "This paper discusses a phase transitional behavior of the Hessian at initialization in the overparametrized setting. They discuss the setup where the teacher and student networks are two-layer neural networks with quadratic activation, and the width of the student is possibly larger than the width of the teacher. In classical BBP transition analysis, it is known that there is a threshold where the SNR is larger than this threshold the largest eigenvector aligns with the true signal. They analyze the threshold in the overparametrized student-teacher setup, and show that 1) transition happens either continuously or discontinuously 2) overparametrization decreases the threshold and makes training easier 3) as the width of the student -> infinity, the threshold becomes optimal. The claims are supported with experiments.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. An interesting perspective on neural network training by discussing overparametrized phase retrieval - the theory involves machinery from quantum mechanics (of which I did not fully understand), which shows an interesting link between learning theory and physics.\n\n2. The related works are cited extensively and mentioned appropriately in relevant parts of the manuscript.", "weaknesses": "1. Clarification in terminology is needed. \n - Why would this be a \"loss landscape\" result? Seems to me that the result is mostly on Hessian \"at initialization\" - which to me, it is not natural to understand the result as loss landscape result (of course, Hessians and loss landscape are related, but the training dynamics is not discussed).\n - What does \"bend the landscape\" mean? \n - What is the SNR that is repeated throughout the paper? I assume it would be alpha = M/N, I am wondering if the terminology SNR is used to express such quantity.\n - Using the term overparametrized could be a little bit misleading, because in general the term is used to state that the number of parameters >= number of data points. \n\n2. It is a little hard to understand the technical novelty of the paper. What is the technique that is needed to study the overparametrized setting, which is different from previous approaches? Are there any challenges? \n\n3. The experiments are good in the sense that the discoveries are verified, but the experiments are quite small-scale. $p, p^{*}$ are in the scale of 1,2,3,4. Larger experiments may be helpful. Also, I can see that the threshold becomes smaller when overparametrization occurs, but does that imply that it needs to better training? I don't see direct evidence of it. For instance, it would be helpful if there is an experiment where you apply gradient descent on the actual learning problem and show that overparametrization yields faster convergence/better generalization etc.\n\n4. It would be good if we could see justifications of certain theoretical problem settings. e.g. Why should the activation be quadratic? Why should we train with normalized quadratic loss function?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aNXEmIGPOG", "reviewer": "Reviewer_upYX", "summary": "This paper studies how overparameterization changes the geometry of simple neural networks at initialization.\nThe authors look at a two-layer quadratic network in a teacher–student setup, analyze the spectrum of the Hessian at random initialization, and show that it undergoes a Baik–Ben Arous–Péché (BBP) transition as the sample size (or SNR) increases.\nThe main finding is that wider networks “bend” the loss landscape: the BBP threshold shifts to smaller SNRs, the transition can become discontinuous, and in the large-width limit the threshold actually reaches the information-theoretic weak-recovery limit.\nThe work ties together ideas from random matrix theory, spectral initialization, and overparameterized learning, with clear analytical results and convincing numerical checks.", "questions": "Beyond quadratic activations:\nThe current analysis focuses on quadratic activations, which make the problem analytically tractable. It would be useful to discuss whether similar BBP behavior should appear for other smooth nonlinear activations (e.g., ReLU, erf). How much of the observed “bending of the landscape” is a consequence of the quadratic structure, and how much would persist in more realistic nonlinear settings where the Hessian couples to the input distribution?\n\nInterpretation of discontinuous BBP transitions:\nThe discontinuous BBP transitions are a striking result. Can the authors clarify their physical or algorithmic interpretation? Do they correspond to a first-order–like instability in the optimization landscape, or to a sharp onset of alignment during early training dynamics?\n\nConnection to optimization dynamics:\nSince the paper analyzes the Hessian at random initialization, one may wonder why this spectral transition should meaningfully predict the onset of learnability for gradient descent in practice, especially at finite width. Are the BBP signatures expected to survive after a few optimization steps, or are they quickly “washed out” as the model moves in parameter space?\n\nRobustness beyond Gaussian inputs:\nHow sensitive are the predicted thresholds to the Gaussian data assumption? Would structured or correlated inputs (e.g., non-isotropic covariance, nonzero mean) qualitatively alter the BBP critical point or the continuous/discontinuous nature of the transition?\n\nRelation to implicit regularization and flatness:\nCould the authors connect their findings to the broader literature on implicit regularization in overparameterized models—such as the bias of gradient flow toward flat minima? Does the observed “bending” of the Hessian spectrum have an analogue in the flatness or margin properties of trained solutions, or suggest a theoretical link between curvature at initialization and generalization in the final model?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a strong theoretical contribution. The idea that overparameterization changes the nature of the BBP transition—and that in the large-width limit one reaches the optimal weak-recovery threshold—is both interesting and novel.It deepens our understanding of why wide models are easier to train, and it connects two previously separate lines of work: loss-landscape curvature and spectral initialization.\n\nOriginal and timely topic: the interplay between overparameterization and loss-landscape geometry.\nTechnically clean derivations, connecting to known spectral and phase-retrieval results.\nSolid numerical support and a nice discussion of finite-size effects.\nClear writing and good figures that make a rather technical story accessible.\nConceptually important: shows how widening a network effectively reshapes the curvature of the loss, anticipating information about the teacher signal even before training starts.", "weaknesses": "The analysis is limited to quadratic activations, which makes it less clear how general the conclusions are.\n\nThe field-theory derivations could be compressed; parts of the appendix are a bit heavy \"physics-style\"\n\nIt would have been nice to see a direct quantitative comparison with actual spectral initialization methods to highlight practical implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T88aHOL6Gf", "reviewer": "Reviewer_x8Zk", "summary": "The paper investigates how overparameterization affects the loss landscape at initialization in gradient-based learning. In particular, it considers a teacher-student setup where both teacher and student are two-layer neural networks with quadratic activations, generalizing previous works on phase retrieval to a multi-index setting. By analyzing the spectrum of the loss Hessian, the authors identify the conditions under which its leading eigenvector becomes informative about the teacher signal, and show that overparameterization bends the landscape, shifting the associated transition toward smaller sample sizes (lower SNR). In the infinite-width limit, the transition reaches the information-theoretic weak-recovery threshold. Finally, the authors investigate finite-size effects and compare them with the high-dimensional predictions.", "questions": "1. Could you offer some intuition on how the label noise might qualitately affect the BBP phenomenology observed in this work?  \n\n2. In the conclusion, you mention that understanding the interplay between the emergence of a signal in the Hessian and the behavior of gradient-descent dynamics is an open direction. Do you have any preliminary numerical evidence or intuition on whether the BBP transition identified here corresponds to the point where gradient descent (or flow) begins to correlate with the teacher signal?\nIn particular, do you expect a qualitative behavior similar to the loss of correlation with the informative eigenvector observed in [Bonnaire et al., 2024], or would the overparameterized setting change this picture? Even a qualitative comment on whether such a connection is expected or not would be very interesting, although understandably beyond the main scope of this work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is theoretically and numerically sound. It addresses the important question of how overparameterization affects the learning landscape, offering novel, quantitative results in the specific setting of quadratic two-layer networks. Its originality lies in extending previous analyses of phase retrieval to a more general framework, providing a detailed characterization of the BBP phenomenology and valuable insights on finite-size effects. Overall, the presentation is clear and connects the findings to known information-theoretic thresholds and spectral methods.", "weaknesses": "1. One main weakness is the lack of a methodological overview in the main text. The technical analysis is confined to the appendix, leaving readers without intuition about the derivation. The paper could strongly benefit from a short but insightful methodological summary in the main section, possibly by shortening the (sometimes redundant) conclusion and/or using the additional page.\n\n2. Some relevant references on the multi-index setting are missing. For instance, the critical SNR $p_\\star/2$ has also been derived as the computationally optimal threshold in [1]; spectral method for multi-index models, included the quadratic network studied here, have been investigated and rigorously characterized in [2, 3].\n\n3. The choice of the loss function may appear somewhat *ad hoc* and problem specific. It can benefit from further motivation and discussion.\n\n[1] Troiani et al., \"Fundamental limits of weak learnability in high-dimensional multi-index models\"\n\n[2] Kovačević et al., \"Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery\"\n\n[3] Defilippis et al., \"Optimal Spectral Transitions in High-Dimensional Multi-Index Models\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NvsnQkhwMG", "reviewer": "Reviewer_W5zG", "summary": "The authors consider a teacher student model for finite-width, one-hidden-layer neural networks with quadratic activations (generalizing phase retrieval). \nThey study the Hessian of the loss landscape of the student at initialization, and characterize when a certain overlap between teacher weights and leading hessian eigenvector transitions from zero to finite (either in a continuous or discontinuous way). A positive overlap means that the Hessian at initialization contains easily accessible information on the teacher weights.\nThey compute the sample critical threshold, i.e. the minimum amount of samples M over dimension N such that there is information about the teacher weights in the initialization Hessian, and study the phenomenology as a function of the overparametrisation.", "questions": "It is not apparent to me why one would need p,p* = O(1). What would fail in the derivation for p=O(d), for e.g.?\n\nline 116: I would like to highlight also the following relevant paper https://arxiv.org/abs/2505.17958 where over-parametrized phase retrieval is considered also in an empirical minimization setting (finding the same weak recovery threshold as Maillard et al. 2024) in the complementary setting p^* = O(d), p=O(d) with p^*/d -> 0.\n\nline 143: it is not clear to me why one would normalize the loss by 1/labels. Given that the authors remark that this is an important element of the subsequent analysis, it would be nice to have more intuition here.\n\nline 157: is it \"when\" or \"where\"? If the first, when one would expect such Gaussianity to hold?\n\nline 215: it would be nice to have an explicit definition of m here.\n\nline 254-258: is there an intuition for why exponential decay at the bulk boundary induces a first order transition in the overlap? \n\nline 333 and 351: is it the largest eigenvalue instead of smallest?\n\nline 370 - 400: it is not clear to me how is alpha_0 computed: is it an analytical quantity? A numerical one? Also, it is not clear why it should be a lower bound to the finite-N transition. I suggest clarifying a bit.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors quantify the effects of overparametrisation in a non-convex problem, in particular the effect at initialization. Given the context on spectral initialization and GD dynamics, I find it a nice way to discuss such effects.\n\nThe authors provide an application of the recently discussed discontinuous BBP transition, showing that such behavior is not a totally abstract curiosity.", "weaknesses": "The mismatch between finite N simulations of alpha_BBP in the discontinuous case and the theory should be characterized more precisely, even though it is clear that finite size corrections will be hard to \"eliminate\". \n- Is there a different set of observables (other than that in Figure 3) that could be probed numerically to highlight the transition? Maybe with less important finite size effects? \n- How prohibitive would it be to access finite size corrections from the field theory formalism?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)