Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data
Authors: Guillaume Braun
Bruno Loureiro
Minh Ha Quang
Masaaki Imaizumi
Venue: ICLR 2026 Oral (2026)
Keywords: scaling laws
gradient flow
power-law spectrum
phase retrieval
anisotropic data
learning dynamics
Primary area: learning theory

Abstract: Scaling laws describe how learning performance improves with data, compute, or training time, and have become a central theme in modern deep learning. We study this phenomenon in a canonical nonlinear model: phase retrieval with anisotropic Gaussian inputs whose covariance spectrum follows a power law. Unlike the isotropic case, where dynamics collapse to a two-dimensional system, anisotropy yields a qualitatively new regime in which an infinite hierarchy of coupled equations governs the evolution of the summary statistics. We develop a tractable reduction that reveals a three-phase trajectory: (i) fast escape from low alignment, (ii) slow convergence of the summary statistics, and (iii) spectral-tail learning in low-variance directions. From this decomposition, we derive explicit scaling laws for the mean-squared error, showing how spectral decay dictates convergence times and error curves. Experiments confirm the predicted phases and exponents. These results provide the first rigorous characterization of scaling laws in nonlinear regression with anisotropic data, highlighting how anisotropy reshapes learning dynamics.

Published as a conference paper at ICLR 2026
FAST ESCAPE, SLOW CONVERGENCE: LEARNING DY- NAMICS OF PHASE RETRIEVAL UNDER POWER-LAW DATA
Guillaume Braun
Bruno Loureiro
Ha Quang Minh RIKEN AIP
RIKEN AIP
École Normale Supérieure, PSI. CNRS
Masaaki Imaizumi University of Tokyo & RIKEN AIP & Kyoto University
ABSTRACT
Scaling laws describe how learning performance improves with data, compute. or training time, and have become a central theme in modern deep learning. We study this phenomenon in canonical nonlinear model: phase retrieval with anisotropic Gaussian inputs whose covariance spectrum follows power law. Un- like the isotropic case, where dynamics collapse to two-dimensional system, anisotropy yields qualitatively new regime which an infinite hierarchy of cou- pled equations governs the evolution of the summary statisties. We develop a tractable reduction that reveals a threc-phase trajectory: (i) fast escape from low alignment, (ii) slow convergence of the summary statistics, and (iii) spectral-tail learning in low-variance directions. From this decomposition, we derive explicit scaling laws for the mean-squared error, showing how spectral decay dictates con- vergence times and eтror curves. Experiments confirm the predicted phases and exponents. These results provide the first rigorous characterization of scaling laws in nonlinear regression with anisotropic data, highlighting how anisotropy reshapes learning dynamics.
INTRODUCTION
Scaling laws quantify how the performance of a learning algorithm varies with resources such as training time, dataset size, model capacity. Empirically, losses often follow simple power laws across wide ranges of data and computation, enabling forecasting from a handful of measurements (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022). These regularities naturally raise the fundamental question: when and how do such laws emerge from first principles?
Despite their central role in modern deep learning practice, neural scaling laws remain theoretically poorly understood. A notable exception is provided by linear models, where the scaling of the gen- eralization enror has been thoroughly analysed within the classical kernel literature, encompassing both ridge regression (Caponnetto & De Vito, 2007; Rudi & Rosasco, 2017) and stochastic gradient descent (Yao al., 2007; Ying & Pontil, 2008: Carratino al., 2018: Pillaud- Vivien et al., 2018: Kunstner & Bach, 2025). Recent developments in this direction, driven by the empirical observa- tions of cross-overs and bottlenecks in the context of neural networks, demonstrate that analogous phenomena are already present in linear settings (Cui et al., 2021: Defilippis et al., 2024; Bahri et al., 2024: Maloney et al., 2022; Atanasov al., 2024; Paquette al., 2024: Bordelon et al.. 2024; Lin et al., 2024). By contrast, nonlinear settings, ubiquitous in practice (e.g., functional data, learned feature maps, or embeddings with heavy spectral tails), remain far less understood.
We address this gap in the canonical nonlinear regression problem of phase retrieval:
(z,, w²) + E,
(O, Q),
where w" Rd is the target vector, and has eigenvalues (A:) obeying power law X i-m with a > and noise E. This model captures two core difficulties: (i) a nonconvex landscape, and (ii) strong anisotropy that induces highly unbalanced learning across directions.

---

Published as a conference paper at ICLR 2026
10
10-1.
a=0
a
MSE
10
0.75
a= 1.25
a= 1.5
10-3
10
10a
10a
10a
10c
10
t
Figure 1: Evolution of the MSE during training with online SGD for different spectral exponents (log-log scale). For 1, convergence is markedly slower than the exponential decay seen in the isotropic case, reflecting the difficulty of learning directions associated with small eigenvalues.
Figure illustrates this phenomenon: under the same initialization, noise level, stepsize, and opti- mization algorithm (SGD), the mean-square error (MSE) behaves differently depending on a. Intu- itively, when a larger, directions associated with small eigenvalues X; are harder learn, so the MSE decays more slowly. By contrast, in isotropic designs, all directions progress at comparable rates, causing the error to drop sharply. These observations motivate our central question:
How does the input spectrum govern predict the learning curve from the spectral decay?
governfinite-tiinn finite-time convergence in nonlinear regression, and can we
Concretely, we seek to (i) analyze the mechanisms behind the plateau-drop structure in anisotropic phase retrieval, and (ii) derive scaling laws that quantify MSE decay functions of the spectral parameter a and time L.
1.1 CONTRIBUTIONS.
New phenomena the anisotropic case. We show that anisotropy challenges several aspects of the intuition developed in isotropic settings. In the isotropic setting, the dynamics collapse to a low-dimensional ODE, and the main challenge escaping mediocrity, i.c. the regime where the correlation with the signal w" is vanishing, before convergence accelerates. Under anisotropy, by contrast, the dynamics form an infinite hierarchy of coupled equations. This structural change flips the qualitative behavior, yielding an scapp--coverggence trade-off': escaping from mediocrity can be faster, but convergence to a low MSE is slowed by the difficulty of learning directions as- large is associated with slow
sociated with small eigenvalues, as illustrated by Figure/ decay of the MSE) and Figure (a large a leads quickly to constant order correlation with the sig- nal). Numerical experiments confirm this phase-level contrast between isotropic and anisotropic regimes.
Analytical framework. We first obtain closed-form representation of the dynamics via Duhamel's 's formula. We then introduce a phase decomposition of the trajectory, isolating regimes where different approximations become valid. This allows to analyze the qualitative behavior of the ODE hierarchy phase by phase: (i) fast escape from mediocrity, (ii) macroscopic conver- gence of summary statisties, and (ii) spectral-tail learning of small-eigenvalue directions. The combination Duhamel representation and phase-specific approximations provides a systematic dynamics tractable.
way make
Scaling laws. As byproduct of this analysis, derive explicit scaling laws for anisotropic phase retrieval. These formulas quantify how the eigenvalue decay governs the MSE. Numerical results corroborate the predicted exponents across different spectral profiles.
2

---

Published as a conference paper at ICLR 2026
1.2 OTHER RELATED WORK
Theory of scaling laws. The study of risk scaling in problems with power-law structure is a clas- sical theme in the kernel literature, where it falls under the framework of source and capacity con- ditions. It has been extensively investigated for kernel ridge regression (Caponnetto & De Vito, 2007; Cui et al., 2021), random features regression (Rudi & Rosasco, 2017; Defilippis et al., 2024), and also for (S)GD (Yao et al., 2007: Ying Pontil, 2008; Carratino et al., 2018; Pillaud- Vivien et al., 2018). This line of work has recently gained renewed relevance in the context of neural scaling laws, with linear models emerging as theoretical testbeds to explain the plateaux and cross- overs observed in practice (Bahri et al., 2024; Maloney et al., 2022; Atanasov et al., 2024; Bordelon et al., 2024; Worschech & Rosenow, 2024; Paquette et al., 2024; Lin et al., 2024). More recently, Wortsman & Loureiro (2025) have investigated kernel ridge regression under anisotropic power-law inputs, showing that in some regimes the scaling of the covariates transfer to the features. Sums of orthogonal single-index models have been analysed in the feature-learning regime, albeit still under isotropic input distributions (Ren et al., 2025; Ben Arous et al., 2025; Defilippis et al., 2025). In short, while the linear setting with anisotropic spectra (e.g. power-law decays) is by now well understood, the nonlinear regime has remained largely isotropic. Our work addresses this gap by deriving compute- error scaling laws in a nonlinear model with anisotropic Gaussian inputs.
Phase retrieval and quadratic neural networks. Phase retrieval (PR) is a classical inverse prob- lem motivated by imaging: reconstruct signal from intensity-only measurements; see Dong et al. (2023) for a recent tutorial. A rich algorithmic literature includes spectral initializations and non- convex Wirtinger-flow flow refinements (Ma et al., 2021; Candès al., 2015; Tan & Vershynin, 2019; 2023; Davis et al., 2020). PR can be viewed as learning a single neuron with a quadratic activation, connecting it the broader theory of quadratic networks and their training dynamics (Sarao Man- nelli et al., 2020; Arnaboldi et al., 2023a; Martin et al., 2024; Erba et al., 2025). In this context, Ben Arous al. (2025) have studied scaling laws for one-pass SGD and Defilippis al. (2025) for full-batch ERM, in a quadratic net model where the target coefficients follow a power-law. Most theoretical analyses of PR assume isotropic uub-Gaussian measurements; the impact of anisotropic covariances on the learning curve has received less attention. Our analysis isolates precisely this aspect and quantifies how the input spectrum shapes the three-phase trajectory and the resulting scaling laws.
Non-convex optimization and feature learning. A complementary line of work studies gradient-based training in multi-index and shallow networks, characterizing feature learning, con- vergence phases, and computational computationall-sttittiaaal trade-offs, predominantly under isotropic designs (Saad & Solla, 1995a;b; Goldt et al., 2019; Veiga et al., 2022; Arnaboldi et al., 2023b; 2024; Collins- Woodfin al., 2024; Ben Arous al., 2022; Abbe et al., 2022: 2023; Bietti al., 2025; Dandi al., 2024: Bruna & Hsu, 2025). Results with anisotropic inputs are more limited: some works consider spiked covariances and rely on preconditioned methods (Ba et al., 2023), while others treat a broader but weaker anisotropic regime that does not include power-law spectra (Goldt et al., 2020; Braun et al., 2025). By contrast, we analyze the unpreconditioned gradient flow in a strongly anisotropic closure of the dynamics, leading to regime and show that anisotropy destroys the reduction yields explicit scaling laws. an infinite hierarchy whose
1.3 NOTATIONS
We use II-Il and (--) to denote the Euclidean norm and scalar product, respectively. When applied to matrix, |I=I] refers to the operator norm. Any positive definite matrix Q induces a scalar product defined by (a. v)q = xTT =TQu. Qy. The Frobenius norm of a matrix A denoted by || The d x d identity matrix represented by Ig. The (d -)ddimnnsional unit sphere is denoted by s We use the notation an S b, (or a, Z bn) for sequences (a,)n2) and (bx)221 if there exists constant C such that 0n S Cb, (or On 2 Cb,,) for all n. If the inequalities hold only for sufficiently large n, we write an Obb.) (or an ((b..)). We denote by Cg (I>z;; (2) the Banach space of bounded and infinitely differentiable functions from R20 to l² := {(In))20 : > x < oo), the Hilbert space of square summable sequences. We write for convolution, and for function f denote by f its Laplace transform.
3

---

Published as a conference paper at ICLR 2026
2
PROBLEM SETUP
Data distribution. By orthogonal invariance of the Gaussian distribution, we may assume without loss of generality that the covariance matrix diagonal: = diag((A1.... Aa) Rd We assume a power-law spectrum,
A
1,
that tr(Q)= 1. This assumption reflects the slow, heavy-tailed eigenvalue decay observed in many empirical covariance spectra (e.g.. images, text embeddings, kernel features). From a theo- retical perspective, the exponent provides simple parametrization of anisotropy: it controls the balance between a few dominant directions and long tail of weak ones. It is widely used as a canonical model for scaling laws in learning dynamics. We consider the phase retrieval setting
y (sr,w)) E,
(00, Q),
where the target weights generated by w* [Q// (0, Ia). This construction ensures that |/2 wI
1. For numerical experiments, sometimes adopt the alternative normalization ||=*|| = d. so that all curves start from the same baseline when comparing different decay expo- nents a; both normalizations are of similar order. The noise term & ~ (O, a²) is independent of x. Since our analysis focuses on the population dynamics, label noise only shifts the loss by a constant and can be set zero without loss of generality.
Model. We consider estimators of the form (ar, w)2 for w Ied and the population loss used to train our model
[(w) = E. [(x [<)=)) (x w }9}]]
However, under anisotropy, this loss be misleading: vector that aligns with w only along the directions corresponding to the largest eigenvalues may still achieve a small loss. To evaluate how well actually recovers the signal direction, a more natural metric the MSE defined as
MSE(w, w") = d min || w" 112, w| + w w²]]
Optimization. The learner maintains weight vector w(t) Rd initialized uniformly random on the unit sphere gd-1, Its evolution follows the gradient flow dynamics
(t) = - (v)((w(t)
which can be viewed as the continuous-time counterpart of gradient descent.
Loss GEOMETRY AND EVOLUTION OF SUMMARY STATISTICS
In this section, we describe the main characteristics of the loss landscape and establish ODEs to describe the evolution of the key summary statistics. The proofs are in the appendix, Section B. For convenience, we introduce the following notations: s 1!ll w Qw,, S. == 1| and = (w, w*) w*)q = w Qw".
3.1 Loss SIMPLIFICATION
Although the loss is defined on d-dimensional parameter space, it depends only on two summary statistics: || and (w, w w²) 222 This dimensional reduction makes the geometry of the loss land-
scape transparent, as captured in the following proposition.
Proposition 1. Let x ~ NoO,, Q) Rd, where Q E I is a symmetric positive definite diagonal matrix. Let w, w² E Rdd The population loss can be rewritten as
((w) 3| + 33/ 111 4(w, w w²) 2| ||l" |1
3s² 2
35 4u² 22.
Next, compute the population gradient and characterize the critical points.
4

---

Published as a conference paper at ICLR 2026
3.2 GRADIENT, HESSIAN, AND CRITICAL POINTS
Proposition 2. The population gradient and Hessian of the loss L are given by
V 12sQu 48, Qu SuQu",
(3.1)
V2L(w) 24(QW) + (12s --8848. ((QQ(((Q))"")"
(3.2)
The set of critical points is {0, w", =w" } U {w (u, (0,s,/3).. Among them, 0 is a strict local maximum, =w² are strict global minima, and every w with (u, 8) - (0, 8.33 is a saddle point.
Remark 1. The loss landscape contains no spurious local minima: the only minima are the global optima =w- At the same time, the origin is a strict local maximum, and the remaining critical points are saddles. Nevertheless, the dynamics exhibit a long plateau phase. At initialization, one rypically has u(0) 1/2 O. this low-correlation regime, the gradient is small, so the iterates take long time 1o escape. Even after leaving this regime, convergence remains slower than in the isotropic case.
Remark 2. The anisotropic gradient flow can be viewed as Q-preconditioned version of the isotropic flow. Under the reparametrization z = Q_/2u, define the isotropic loss Li(z) = L(Q=1/22). By the chain rule, 2(t) = (z))(z(t)) Thus, while the critical points coincide with those in the isotropic case, the dynamics differ substantially due the preconditioning by Q.
3.3 DIMENI-DIMENSIONAL STRUCTURE OF GRADIENT FLOW
As shown in Proposition 1, the loss depends only on the summary statistics u(t) and s(t). Control- ling these two quantities already yields a faithful description of the loss trajectory.
A crucial distinction, however, arises between the isotropic and anisotropic settings. In the isotropic case (Q- = I), the dynamics the loss can be expressed entirely in terms two scalars: the signal overlap u() and the energy s()). This leads to a closed, two-dimensional ODE system.
In contrast, as soon as Q * I, the situation changes qualitatively. Proposition shows that the evolution of u(t) and s(t) necessarily involves higher-order weighted overlaps,
(k) (t) = (,)())
84 (k) = || 1.p.
u (t) = (wr(f), w² qa
with the conventions s (t)) u- (t). and s.
The resulting system is an infinite hierarchy of coupled ODEs. Unlike the isotropic case, no finite- dimensional closure exists: the time derivative order-k statistics depends on order-(k+1] statis- tics, and so Understanding the anisotropic dynamics thus requires working with this infinite- dimensional structure.
Proposition The gradient flow dynamics satisfy
(t) = 4), - 3s(()) (t( (tf) + 8Aau(t)uf
i= L=
(3.3)
g(k) (t) (s. 3s(()) (1)+)) 16 16 (()+))((),
(3.4)
(*)) (L) 4 (s+ -3x(()) u (k++)) (1) + 8 u((l) s (++1)
(3.5)
In particular, setting recovers the dynamics of s(t) and u(t)
3.4 THREE- PHARE STRUCTURE OF THE DYNAMICS
The system of ODEs from Proposition 3 involves higher-order moments, preventing closed-form analysis as in the isotropic case. To guide our theory, we first examine the empirical evolution of the key observables u(t) and <t); Figure 2. The trajectories display three-phase structure:
(i) Phase I: Escape from mediocrity, s(t) ( < Ti). At initialization, u(0) 1/Vd, there almost no correlation with the signal. During this "warm-up" stage, the correlation u(t) escapes exponentially from and reaches small but fixed constant u(Ti) 8 > 0. The overall MSE, however, remains essentially unchanged, since only a few easy direc- tions-those aligned with large higenvalues-have been learned so far.
5

---

Published as a conference paper at ICLR 2026
(ii Phase II: Convergence s(t) + 17; <t < T2). Here, the signal alignment strengthens and the summary statistics u(t), approach their limiting values. Two distinct episodes appear:
(Ila) Transition (T) S TD. The energy s(t) crosses the critical threshold 1/3 and
stays above it.
(lIb) Asymptotic convergence (T < < T2). Both u(l) and s(l) approach 1, but conver- gence is slower than in the isotropic case (see Section F), reflecting the difficulty of learning directions corresponding to small eigenvalues of Q
(ii) Phase III: Spectral-tail learning u(t), s(t) se > T2). After u(t) and s(t) have essen- tially stabilized, the MSE continues to decrease as the flow progressively learns directions associated with the small eigenvalues of Q.
This empirical decomposition provides a roadmap for the analysis: by isolating each phase, the dufnite-dimensionn system becomes amenable to tractable approximations.
1.0
4.56 10
0.8
4.55 10-1
4.54 10
u(t), s(t)
0.6
utt)
4.53 10-1
MSE
s(t)
4.52 x 10-1
MSE
0.4
4.51 10-1
0.2
4.5 x 10
4.49 10-1
0.0
100
10
102
103
104
10
106
T17
T2
Figure 2: Evolution of the MSE, u(t) and s(t) under population gradient descent (log-log scale).
Parameters: 2, d 10 7) 10-2 2, T 106
0.05.
MAIN RESULTS: THREE-PHASE GRADIENT FLOW DYNAMICS
The experiments in Section 3.4 revealed a characteristic trajectory with three successive phases: (i) escape from mediocrity, (ii) convergence of the summary statistics, and (iii) spectral-tail learning. We now show that this qualitative picture admits a rigorous derivation from the gradient flow equa- tions. The following theorems make precise the stopping times, plateau behavior, and tail-driven decay observed empirically.
4.1 PHASE I-II: ESCAPE AND APPROXIMATE CONVERGENCE OF THE SUMMARY STATISTICS
The next theorem formalizes escape and approximate convergence, under initialization conditions whose full statement is deferred to Appendix (see Assumption Al).
Theorem (Phases and II). For d sufficienll large, let 2 d (a-a1))/2 Under mild conditions on initialization (e.g. u(0) x d' 1/2,, there exist stopping times Ti O(log d), Ti = Ti + O(1), and T3 = T" + O 2a--2a//ee
(a-1) 2a/(a-1)
log(1/e) such that:

---

Published as a conference paper at ICLR 2026
1.0
0.8
0.6
u(t)
0.4
a =
= 0.5
0.2
= 1.2
1.5
0.0
103
104
10s
106
t
Figure 3: Evolution of the correlation u(t) for different exponent a (d 1000, 17 = 10-
(i) There exists constant 8 0 such that |u(Ti)), s(Ti), (x (Ti)l
6, and the sign of
(2) (Ti) agrees with that of u(TT).
(ii) There exists a constant 8o > such that s(t) > 1/3 + Su for all t 2 TV.
(iil) At time T2, both statistics are close to their limits: |u(T2)), (T2) E [ [1-6,1) 1).
Remark 3. Several features of the theorem are worth noting.
Initialization and mediocrity. The sign of u(t) is fixed at initialization; w.l.o.g. one may as- sume it positive by replacing w, with -w.- Under random isotropic initialization, |u(0) is typically of order d-1/2 (as in the isotropic case), hence the escape mediocrity time satisfies Ti x ((])) O d). In particular, anisotropy does not remove this logarithmic bar- rier. table summarizing the different stopping times can found in Section A.
Anisotropic vs. isotropic. In isotropic models, u(0) directly controls early growth and conver- gence relatively fast; the anisotropic setting, the Volterra reduction shows that post-plateau escape is controlled by linear functional of the initialization (via residue/resollent calculus), and convergence after escape is tail-controlled and slower.
Accuracy vs. time. For Z d da(a-1)f2 stabilization occurs within Ols-2a/(a-1) log(1/s)), while demanding < d (a-1)/2 leads even higher time costs (see Phase III analysis).
Remark 4. The analysis Phase I reveals that the u(()) grows exponentially at a rate depending on a: the convergence is faster when a is large (see Proposition 9 in the appendix, and Figure 3 for numerical illustration).
4.2 SPECTRAL-TAIL LEARNING AND PHASE III
While Phases 1-I1 describe the evolution of summary statistics (u, s), these quantities do not capture how the estimator w(t) approaches the ground truth entrywise. The mean squared error
MSE(() 1 (w:(t) w
is the natural measure of recovery: it is the quantity plotted in Fig. 1, and its log-log decay rates yield the scaling laws highlighted in the introduction. Moreover, the MSE aggregates contributions from all eigen-directions, making it the right observable to expose the effect of the spectral tail.
Up time T2, while the summary statistics are still converging, the MSE shows almost no decrease and stays essentially at its initial level. The following proposition captures this plateau behavior.

---

Published as a conference paper at ICLR 2026
Proposition 4. Let oī Ex=r (w,)? Under the assumptions of Theorem we have MSS((T)) - oo. x
1/3
log d 1/3
d
After T2, progress is governed by coordinates aligned with the smallest eigenvalues. Let e, wi(l) wi be the coordinate eтror and define मे ex Eee e,(T))² The weighted average
Sa(T) = E, TzE 16AsT describes the spectral decay of the error, while the uniform benchmark
Sa(T) =; -16AT admits explicit asymptotics. We also set s.²) \Zw)) Hd,
dd Σj=1 "., 16/ Ha,a. and Fz = (327)1/0.
Theorem (Phase III: spectral-tail learning). Under the assumptions of Theorem 1, the MSE satis- fies for every T 2 0.
MSE(T, + = + O(s)) MSE(T,) Sa(t) (((G)++(5 +0
(4.1)
In particular, Sa(r) = (1+ and Sa(T) admits the asymptotics
1 - 16 T (
d
BaT 1
Sa(T)
1 ['L +
1< F < d,
S exp(-Bat d)
ス d.
Thus, the plateau persists until T2, after which the MSE decays at rate controlled by the spectral tail. The exponent a determines the slope of this decay on log-lop scales, accounting for the scaling laws observed in Fig. 1.
PROOF OUTLINE
We briefly sketch the main ingredients of our analysis, deferring complete proofs to the appendix:
Section C (Phase I), Section (Phase II), and Section (Phase III).
5.1 PHASE ESCAPE FROM MEDIOCRITY
In the anisotropic setting, the summary statistics (u, s) not form a closed system; instead, they are coupled to an infinite hierarchy of correlations. Our strategy is to lift the dynamics to an infinite- dimensional space, solve the system there, and then project back to obtain a closed Volterra repre- sentation for (t), which we can analyze directly.
Step 1: siinite-dimensiona formulation. The core difficulty is the infinite hierarchy of coupled correlations, To tame it, we collect them into the vector Utt) = (z() (t).... E H. ==
C (R2o; e). Let B be the right-shift operator on sequences, defined by (Bx); = 4k+1 and let S: (Bs?)e e, be rank one operator with 8 = (! .T...T E l2 and 01,0,0.... TT.
Then the correlation dynamics (3.5) collapse into the compact operator form
(t) = (4(1-3s(t)) 3s(t)) +8)) Utt).
(5.1)
This is the key structural observation: the entire infinite system is generated by the shift operator B plus a rank-one perturbation S. Applying Duhamel's formula yields the following representation.
Lemma 1. The unique solution E Cg (Rzo; e)) of (5.1) satisfies, for all t 2 0,
U(1) (Be(())) Uo + 8 I 4B -0())))0(()))
0(7)) (B= (2) (T) dr,
(5.2)
where (1) = (e1, U(F) u((), and (()) = 1' 3s(T)) dr.
8

---

Published as a conference paper at ICLR 2026
Step 2: Reduction to Volterra equation. Let 8 >> 0 be arbitrarily small, and let T₁ denote the stopping time such that s(t) S 6 for all S Ti. On this time interval we may approximate ( = t. Projecting the representation (5.2) onto e then yields the Volterra equation
((t) = ao(t) + 8 S. K(t ((t-77 (uuu((
((t-))u(T))
(5.3)
where
ao(t) Σ (i)(0 d ethite
K(t) Σ (f
Applying the Laplace transform gives ú(p) = 1-8K(p) The growth of u(t) is therefore gov-
ap(p)
erned by the rightmost pole of ú(p) (see Section and Chapter VII Gripenberg al. (1990)).
Computing (p) and solving the equation 1 8K()) yields the following.
Lemma 2 (Exponential growth rate). The correlation u(l) grows at rate eltutt where Pore 4X1
is the unique positive solution of
1 1 (-8K() 0.
Step Control of the higher-order correlations. A similar analysis shows that the influence of all higher-order terms is dominated by the leading mode u(t). Without loss of generality, we assume that u(t) grows positively (the case of negative growth analogous). The key point that time Tr, the second correlation u. (2)) (Ti) has the same sign as uTT)). a fact that will be critical in the analysis of Phase Ila.
Proposition 5 (Control of higher-order correlations). For all 2 and all E [0, Tı}, we have
log Ahit + 8 (k) (((r) (1) s X loud 4 -
X (u(t) u(0)). (5.4)
In particular, for d sufficiently large there exists constant ej > 0 such that (2 2 C1.
5.2
PHASE II: CONVERGENCE OF SUMMARY STATISTICS
Since the proof techniques differ, we separate the analysis of Phase Il and Phase Ilb.
5.2.1 ANALYSIS OF PHASE IIA
The first step is show that s(t), which at time Ti is still small (s(2i) = 82), must increase up to the critical threshold 1/3. Indeed, from
8(() 8(1 3s(()) 8 (1)(1) 16
both terms on the right- hand side are positive whenever s(t) s 1/3, s(f) is driven upward and crosses 1/3 finite time.
The second step establishes stability beyond the threshold: once st) has passed 1/3, it cannot fall back below. Close to the boundary, the positive contribution 16u(t)u(²) dominates the negative drift from the first term. A careful comparison shows that s(t) remains uniformly above 1/3 + 8 for some constant 8 > 0. The detailed proof of these two points is given in Section D.I.
5.2.2 ANALYSIS OF PHASE IIB
We now study the error A (t) = u(t) 2 0 for 1 2 Ti through the Volterra equation
A(I) (l( ["] ((((())((T)d A(T)
where the source term be is detailed in Appendix D.2. Since after Tí we have @(t) @(r) < -t - 7), the kernel Ke is decreasing, so the integral contribution diminishes over time.
To control the positive part of be. split the spectrum the tail, which is small but slow to learn, while X; 2 X form the head, easier but requiring more training time when d, is small. Balancing these two effects yields the following result.
cutoff >e: directions with X; < À form

---

log Published as a conference paper at ICLR 2026
Proposition 6. Let 2 d - " and set
C
2a 1 a-1
T2(e) Ti +
480
E
with C sufficiently large. Then (Tb) S. a-1
The condition Z d 2 prevents from being too small. Otherwise the required time becomes much larger; for instance, decreases as increases, so T2(e) is shorter faster spectrul decay. Intuitively, for larger a, the lighter tail means fewer small eigenvalues learn, the ervor contracts more rapidly. In contrast, for a close to the heavy tail creates many slow directions, delaying convergence. This trend is confirmed numerically in Section F.
Remark 5.
d= already forces T Z d Note that the exponent a-1
5.3
PHASE III: SPECTRAL-TALL LEARNING AND THE SCALING LAW
After T2, both u(I) and s() are close to one, so the coordinate dynamics reduce to
(t) 82, (w: w,(());
whose solution shows exponential relaxation each w; toward w,. The resulting MSE is a weighted spectral average Sa(r): under the heuristic т, R d=1 this reduces the benchmark Sa(T), whose asymptotics follow from classical sum cun-integral comparisons and reveal distinct decay regimes. The approximation error is controlled by Duhamel (variation-of-consitie with Phase IIb bounds on 1 u(t) and 1 st).. Grönwall argument shows that these corrections remain small, so the full trajectory closely tracks the idealized one. Altogether, this yields the scaling law of Theorem 2, see Section F.1 for an illustration of the approximation.
representation, together
DISCUSSION
Summary. We analyzed the gradient flow dynamics anisotropic phase retrieval. Unlike isotropic case, which reduces to a two-dimensional ODE, the anisotropic setting induces an infinite hierarchy of coupled equations. Our main contribution is to render this structure tractable via a Duhteel-Volterra reduction, revealing three-phase trajectory: (i) escape from mediocrity, (ii) convergence of summary statistics, and (iii) spectral-tail learning driven by small eigenvalues. This decomposition leads to explicit scaling laws, validated empirically.
Limitations. Our analysis assumes Gaussian inputs with a power-law spectrum, simplifying the theory, but this may not be essential for the three-phase phenomenon. In addition, our quantitative results apply only in the gradient flow limit; extending them to discrete-time SGD remains an open problem.
Future directions. Several extensions are natural. On the theoretical side, deriving scaling laws discrete-time SGD, analyzing finite-sample effects, and relaxing distributional assumptions would strengthen the link to practice. On the modeling side, extending beyond quadratic nonlin- carities to more general single- and multi-index models could reveal new scaling regimes. More broadly, findings raise the question of whether analogous phase decompositions and scaling laws govern the training dynamics wider neural networks, potentially offering a bridge between nonlinear regression models and deep learning theory.
ACKNOWLEDGEMENTS
We would like to thank Elisabetta Cornacchia, Florent Krzakala, Gabriele Sicuro, Simone Maria Giancola and Urte Adomaityte for insightful discussions. BL was supported by the French gov- emmment, managed by the National Research Agency (ANR), under the France 2030 program (PR[ATJRIE-PSAD and MANR-25-CE22 (MAPLE), well as the Choose France CNRS AI Rising Talents program. MI was supported by JSPS KAKENHI (24K02904). JST CREST (JPMJCR2ID2). JST FOREST (JPMJFR2160), and JST BOOST (JPMJBY24A9)
with the project references
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp. 4782-4887. PMLR, 2022.
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: dynamics. In The Thirty Sixth Annual Conference Learn-
leap complexity and ing Theory, pp. 2552-2623. PMLR, 2023.
Luca Amaboldi, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Escaping medi- ocrity: how two-layer networks learn hard generalized linear models with sgd. arXiv preprint arXiv:2305. 18502, 2023a.
Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high-dimensional & mean-field dynamics dimensionless odes: A unifying approach sgd in two-layers networks. In The Thirty Sixth Annual Conference on Learning Theory, pp. 1199-121 1227. PMLR, 2023b.
Luca Arnbooli, Yatin Dandi, Florent Krzakala, Luca Pesce, and Ludovic Stephan. Repetita iu- vant: Data repetition allows sgd to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405. 15459, 2024.
Alexander Atanasov, Jacob A Zavatone-Vert Veth, and Cengiz Pehlevan, Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00552, 2024.
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, and Denny Wu. Learning in the presence of low-dimensional structure: A spiked random matrix perspective. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):2211878112 2024.
Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling. Advances in neural information processing systems, 35:25349 223622, 2022.
Gérard Ben Arous, Murat Erdogdu, N Mert Vural, and Denny Wu. Learning quadratic neural networks in high dimensions: Sgd dynamics and scaling laws. arXiv preprint arXiv:2508.0366 2025.
Alberto Bietti, Joan Bruna, and Loucas Pillaud-Vivien. On learning gaussian multi-index models with gradient flow part i: General properties and two-timescale learning. Communications on Pure and Applied Mathematics, 2025.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scal- ing laws. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR, 2024.
Guillaume Braun, Minh Ha Quang, and Masaaki Imaizumi. Learning single index model from anisotropic data with vanilla stochastic gradient descent. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
Joan Bruna and Daniel Hsu. Survey on algorithms for multi-index models, 2025.
Emmanuel Candès, Xiaodong Li, and Mahdi Soltanolkotabi. Phase retrieval via wirtinger flow:
Theory and algorithms. IEEE Transactions Information Theory, 1985:2995-2007 2015.
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics. 7(3):331-36,, 2007.
Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random features.
Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2026
Elizabeth Collins-Woodiin Woodfin, Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting the high-dimensionss notes: An ode for sgd learning dynamics on glms and multi-index models. Information and Inference: Journal of the IMA, 13(4)iaa 2024.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kemel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131-10143, 2021.
Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborova, and Florent Krza- kala. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (cds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 9991-10016. PMLR, 21-27 Jul 2024.
Damek Davis, Dmitriy Drusvyatskiy, and Courtney Paquette. The nonsmooth landscape of phase retrieval. IMA Journal of Numerical Analysis, 40(4):2652-2695) 2020.
Leonardo Defllipiis Bruno Loureiro, and Theodor Misiakiewicz. Dimension-free deterministic equivalents and scaling laws for random feature regression. Advances in Neural Information Processing Systems, 37:104600-104911 2024.
Leonardo Defilippis, Yizhou Xu, Julius Girardin, Emanuele Troiani, Vittorio Erba, Lenka Zde- borová, Bruno Loureiro, and Florent Krzakala. Scaling laws and spectra of shallow neural net- works the feature learning regime. arXiv preprint arXiv:2509. 24882, 2025.
Jonathan Dong, Lorenzo Valzania, Antoine Maillard, Thanh-an Pham, Sylvain Gigan, and Michael Unser. Phase retrieval: From computational imaging to machine learning: tutorial. IEEE Signal Processing Magazine, 40(1):45-57, 2023.
Vittorio Erba, Emanuele Troiani, Lenka Zdeborová, and Florent Krzakala. The nuclear route: Sharp asymptotics of erm in overparameterized quadratic networks. arXiv preprint arXiv:2505. 17958, 2025.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019.
Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Physical Review X, 10(4):041044, 2020.
G. Gripenberg, S. O. Londen, and O. Staffans. Volterra Integral and Functional Equations. Ency- clopedia of Mathematies and its Applications. Cambridge University Press, 1990.
Joel Hestmess, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Henni- gan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NeurIPS '22, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under Zipf"s law. arXiv preprint arXiv:2505. 2025.
12

---

Published as a conference paper at ICLR 2026
Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter Bartlett, and Jason D. Lee. Scaling laws in linear regression: Compute, parameters, and data. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024.
Junjie Ma, Rishabh Dudeja, Ji Xu, Arian Maleki, and Xiaodong Wang. Spectral method for phase retrieval: An expectation propagation perspective. IEEE Transactions on Information Theory, 67 (2):1332-1355) 2021.
Alexander Maloney, Daniel Roberts, and James Sully. solvable model of neural scaling laws. arXiv preprint arXiv:2210. 2022.
Simon Martin, Francis Bach, and Giulio Biroli. On the impact of overparameterizztion on the training of shallow neural network in high dimensions. In International Conference Artificial Intelligence and Statistics, pp. 3655-3663. PMLR, 2024.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute- optimal neural scaling laws. In The Thihty- Annual Conference on Neural Information Processing Systems, 2024.
Amnon Pazy. Semigroups of linear operators and applications partial differential equations, volume 44. Springer Science & Business Media, 2012.
Loucas Pillaud- Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gra- dient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31, 2018.
Yunwei Ren, Eshaan Nichani, Denny Wu, and Jason D Lee. Emergence and scaling laws in sgd learning of shallow neural networks. arXiv preprint arXiv:2504. 19983, 2025.
Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017.
David Saad and Sara Solla. Dynamies of on-line gradient descent learning for multilayer neural networks. Advances in neural information processing systems, 8, 1995a.
David Saad and Sara Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995b.
Stefano Sarao Mannelli, Eric Vanden-Eijnden, and Lenka Zdeborová. Optimization and gener- alization of shallow neural networks with quadratic activation functions. Advances in Neural Information Processing Systems, 33:13445-13455, 2020.
Yan Shuo Tan and Roman Vershynin. Phase retrieval via randomized kaczmarz: theoretical guaran- tees. Information and Inference: A Journal of the IMA, 8(1):97-223, 2019.
Yan Shuo Tan and Roman Vershynin. Online stochastic gradient descent with arbitrary initialization solves non-smooth, non-convex phase retrieval. Journal of Machine Learning Research, 24(58): 1-47,200 2023.
Gerald Teschl. Ordinary differential equations and dynamical systems, volume 140. American Mathematical Soc., 2012.
Rodrigo Veiga, Ludovic Stephan, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks. Advances in Neural Information Processing Systems, 35:23244 -32255, 2022.
Roman Worschech and Bernd Rosenow. Analyzing neural scaling laws in two-layer networks with power-law data spectra, 2024.
Arie Wortsman and Bruno Loureiro. Kernel ridge regression under power-law data: spectrum and generalization. arXiv preprint arXiv:2510.04756 2025.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learn- ing, Constructive approximation, 26(2):289-315, 2007.
Yiming Ying and Massimiliano Pontil. Online gradient descent learning algorithms. Foundations of Computational Mathematics, 8()):561-596, 2008.
13

---

Published as a conference paper at ICLR 2026
The appendix is organized as follows. Section A contains summary of the different stopping times. Section B contains the proofs of the results stated in Section 3. Section C provides the detailed analysis of Phase I, followed by the analysis of Phase II in Section D and Phase III in Section E. Finally, Section presents additional numerical experiments.
A
SUMMARY OF THE DIFFERENT STOPPING TIMES
The definition and properties of the stopping times are summarized in Table 1.
Phase
Definition
Scaling
Dependence on initial- ization
[u(t) reaches fixed 6 Zi
Ti log(L/|u(0))
Strong: logarithmic in the initial alignment [u(0)] d=1/2
Ia
(f)) crosses 1/3 and stabilizes above 1/3 + 80
Ti - Ti = 0(1)
Only depends on the choice of 5
[u(t)], s() [1 e,
T2 Tí O(E
20
log
Only depends on the ap- proximationg error e, in- dependent of d.
Table Summary of the different stopping times.
B PROOF OF SECTION
This section provides the proofs of the results stated in Section 3.
Proposition 1. Let x ~ No0, Q) Rd, where Q E Rdxd is a symmetric positive definite diagonal matrix. Let w, w* Rd. The population loss can be rewritten as
[(w) = 3 3/1 + 3|| 111 4(w, w") 2 21 1| l12 = 3s + +33 Au² 28
Proof. Expanding the loss gives
((w) = [[ [T) + [[(x w² [xx T 2 )=
Step 1. Fourth moment of a Gaussian linear form. Since x ~ No Q), the scalar x w is
Gaussian with variance Qw || la- Its fourth moment
E (x T w) = 3 (w Qw)² = 3 =/| wl
By the same reasoning,
E [x w = w0 l1
Step Mixed fourth moment.
Expanding the product gives
(x T 2x )²
Σ TEE w.w.wwtt
i.j.k.
For Gaussian x, Wick's 's formula expresses the fourth moment as sum over pairings: (aaaeyaaa. = [[[r,,][[[ree + B[aaao] E[aya]) + [| B(ryaxx)
Since Q = diag A Xa), B[razy] Azdig
Evaluating the three pairings:
Pairing (k, e) contributes
Σ dode wi (wtt²² 1 118-
14

---

Published as a conference paper at ICLR 2026
Pairing (i. E) contributes
Σ
Xidg walwwayz Σ
(w, w²) 1²)
Pairing (i,0)(,k) contributes the same quantity
(w, w²)²
Summing up.
|1 l1 + 2(w, w¹) B[(x T "))(x" "0((""")"]] )² = llul2 11
Step 3. Final expression.
Putting everything together,
((w)= 3|| l8 + wwww ll (1wl1 11 (ww, u, w w²))
|| 11å 4(w, w 1²) 3| la + 3| w ll - 2| 2||w|1
Proposition 2. The population gradient and Hessian of the loss C are given by
V 12sQw 4s, 4s,Qw SuQW",
(3.1)
( 24 (4()) + (12s - 4s.) -sQ))==(QQ""(uu""))
(3.2)
The set of critical points is {0, w", -w" } {w (u, local maximum, = are strict global minima, and every w with (u, 8) (0, s./3) is a saddle point.
Among them, 0 is strict
Proof. The proof proceeds in three steps: computing the gradient, identifying the critical points, and classifying them via the Hessian.
Gradient. To compute the gradient,
the closed-form expression L(w) 35² =3(s,)2 288, 4u2, apply the chain rule and use Vs 2Qw and Vu = Qw" to get
((w)) Vs 2s. V
Vu (1 12s 48.) SuQw",
which is (3.1).
Critical points. Setting V.(w) = 0 and using Q leads to
(12s 4s. ) Suw" = 0.
Case 1: If 12s - 4s, F O, then w must lie in span w". Write cw" then a²s, and u as.. Substituting gives
(12a² 4)ww" = 8aw"
12a
12a =
a E +1,
Hence, we obtain the critical points 0 and -
Case 2: 12s - 4s, = O, then necessarily s. /3. The gradient condition reduces to -Suw" 0, i.c. 0. Thus every with (u, s) (0, s.)) is critical point.
Hessian. Differentiating (3.1), and recalling that V 2Qw and Vu Qw*, we obtain
V2C(w) = 24 (Qw) + (12s 4s.) - 8 (Qw*)(Qu*))
which is (3.2).
15

---

Published as a conference paper at ICLR 2026
Characterization of the critical points. At 0 (so 0, 0),
v C² -ss. 8 (Qw" (Qw")(Q")) (Qw))
which is strictly negative definite since Q > 0 and the rank-one term is negative semidefinite. Thus w = 0 a strict local maximum since [(() > L(+w*) = 0.
At = - (so S., t..),
V= = 24 (Q)=)(QQu)))) + (128,,48.))4 ( (0w = 16
0,
hence both - are strict local (and, by the loss formula, global) minima.
Finally, for any w with (u, s) (0, s,)), the Hessian reduces
(2 = 24(Q (Qw) - =(Qw))(Qu*))
Since Q > w² + O, and Qu = 0, w T Qw fo0 = {[(")) TQw*, we have
w V-[(w) w w [24Qww] QQ SQw" (w*) T Q/w 24(w' Qw)² 8((w*) Qu"1== 3(w"=) 0, (w") (V'))>>((w)))) w = (w²) [24Qua T Q SQw" (w") ()(w)) = =<((w=)TQr) < 0.
follows that ((w( must necessarily have both positive and negative directions. critical point is saddle.
Thus, this
Proposition 3. The gradient flow dynamics satisfy
(t: 4),(s= - 3s(()) (t) + SX,u(t)uf
i= L=
(3.3)
(( = (s. 33(()) (1 (t) + 16 u(t (1)
(3.4)
((()(1) (1) (s. 33((t) u (R+1) 1)+))(())))))) (1) 8 u(() s (++1).
(3.5)
In particular, setting recovers the dynamics of s(t) and u(t).
Proof. We first derive the component-wist ODE, then deduce the dynamics for the summary statistics s(t) and u(t). Recall from the previous computation that the population gradient flow vir(t) V (ww(t) satisfies
(()) 4(s. 3s) 8u Qw".
(B.1)
Component-wise dynamics. yields
With Q =
Xa), taking the i-th coordinate in (B.1)
ii:(1 4(s. As) wi(() + Su Xr uf.
Dynamics for (t). Differentiate (t) = w(t) Qw((t along the flow:
s(t) = 2 it(t Qw(t).
Using (B.1) and defining
(1) = w ((w)(((((t()
() := (())) =ww)'Q"w',
Quw",
obtain
s(1) = 8(s. - -3s) (2) + 16u v (1).
Dynamics for u(t). Differentiate u(t) = w(() T Qw"
i(t) út) Qw"
Using (B.1) and letting s = w T n
w", we get
i 4(s 3s) (2) (t) + Su (2)
These identities establish the stated ODEs for s st) and u(t).
16

---

Published as a conference paper at ICLR 2026
Higher-order moments. w(t) Q"w", leading to
The same computation applies (t)
sorssstssssss)) w(t) T TT @ht((t) and (+)
( (t) 8(s, -3s(t)) 8 ( ( + 16u(t) (t)+
(1) 4(s. 3s(()) u (1) + Su(t) 9,
(k+1)
C ANALYSIS OF PHASE I: ESCAPING MEDIOCRITY
In this section, we provide the detailed proofs for the analysis of Phase outlined in Section 5.1. We begin by formalizing the difinteedimeesiioall ODE in suitable Banach space (Section C.1). We then derive a closed-form solution (Section C.2), show that u(t) satisfies a Volterra integral equation by projection (Section C.3), and analyze its asymptotic behavior. Finally, we study higher-order correlation terms (Section C.4), showing that their dynamics are primarily driven by u()).
C.1
STEP O: PRELIMINARIES ON BANACH- VALUED ODES
Before entering the main steps of the proof, we formalize the Banach space on which the ODE (C.9) is defined. Throughout the following, L(w) refers to the loss function in Proposition 1. Recall that we define (tt) = (z() (t), u (2) (t), (t),...) . where u (²*) (t) w(t) Q"." We show that
UEH = Cτ (R20; where 2² {(ax))=11 Ex of < oo) is the Hilbert space of sqmarel sequences. Thus U is bounded, smooth (2-valued function Rzo.
Lemma 3 (Coercivity of the loss). The loss function in Proposition is coercive, that is |il L(w)
Proof. By the Cauchy-Schward inequality
((w) 3| + 31 Ilg 4(w, wn 0 2||w| l/ II2
2: 31l + 3/| 61| [1lll11ve ||00=112 = 3(1 w la - 1lw*118)
from which it clearly follows that wil-so L(w) co.
Lemma 4 (Local Lipschitz continuity of the gradient). Consider the function Z : Rd - Rd defined
by
Z(w) 3 // 12 Qw - || ||00 llg Qw -2( Qw" 'Qw"
(C.1)
For all w1, w2 E Rdd
[1 - (w2)
s 3|1Q |3| well (1|w)] wall((1uu.]
+ || |I)| || w + ||w2|12 + ||0=111
(C.2)
In particular, for ||
- M. ||w2ll < M, with M >
[ (ww2)ll s 3(3M22 ||=01|+)|1 w2ll
(C.3)
Proof.
[Z(ww) ((w2)ll
< 3||
[| |||1||//2/ 11]0u - |/2 12 130l +||
Qwi - Qwell + 2][(w] w2) T@ Qw"l
< 3/ /w.112 - 1|w & ||Qu,] + 3/ [Q Qwzll
+ 11:1111 || - w2|l + 2 w2
||
< 3 w/ - | |1@
+
+ w/2 12
||w] - w
+ || 10 ||| w2|l + 2/ w 1|@
||w 33 ||0=//)||00 [
+ 3/
|| ||
[|/==|11119 11 + 2||Qw"
< 3|[[|||[[w] - w2][[([]m] |1|l ||
||
||w"||]]]
17

---

Published as a conference paper at ICLR 2026
We recall the classical Picard-Lindels Theorem for the existence and uniqueness of the solution of the initial value problem
a f(t, x(lo) Zu.
(C.4)
where E C(U; U C Rd+1 is open subset, and (to, xo) E U, e.g. Teschl (2012) (Theorem 2.2, Lemma 2.3).
Theorem 3 (Picard-Lindelôf), Let f E RU; k 2 O. where U C Rd+1 is an open subset and (to, zo) E U. Assume that f is locally Lipschitz continuous in the second argument, uniformly with respect to the first argument, i.e. VV C CU V compact, 3L L(V) > 0 such that
((ttz,))(((taap S -||z1 - z2l1, v(t,z1), (t, e V.
(C.5)
Then there exists > 0 such that the initial value problem (C.4) possesses unique solution x² (t) Ck+1 (I; R where Ito to +
E
Lemma -Well-posedness of the gradient flow). The gradient flow on Rd
(t) (ece)==vV(((((( VV)((tt) ((0) = wo E IRd
(C.6)
has unique solution w C (Rzo; Ywo Rd.
Proof. By Lemma 4, the gradient (w is locally Lipschitz on F4. thus by Picard-Lindelöf Theorem there exists a unique local solution w E (T: Rd) on I T= [- for some >> O, since E C (RR+1;; IRd). The interval can in fact be extended to of TR. By differentiating [(w(t)) have, by the chain rule,
dl(w(t)) =
= -VV
di
Thus L(w(tt) is decreasing along the flow w(t) and therefore w(t) always lies in the level set Tuwo (C) = (w E IRd L(w) S [(wn)]. Since L(w) is coercive by Lemma 3, Lww, (f) must necessarily be bounded. Thus Picard-Lindelöf Theorem can be repeatedly applied to extend to all of R20. Since L(w) is bounded below by zero, the limit lim,-voo [(w(t)) necessarily exists. Let Bg (0, M) be the smallest ball in Rd, centered at the origin, such that Long (f) C Bg (0, M), then [[w)( < M VI 2 O. Thus w()) E (TR2o ).).
Lemma 6. Let w be the unique global solution of the gradient flow (C.6). Define U R20 + R by U(t) = (t) (())))))))))........)))))))))))) (t)(()).... where u (kk) (t) = w(t) wh, Then E Cp (Rzoi 8))
Proof. By the assumption that tr(Q)= and A 2 2 Aa O, we have ||| d For each fixed f, we have
u (t)p (ww(t) (wwt))Qh"")) 22 > |10(())11 100112
||
S $11w(0)||||w=] Σ1101 = 1|w(())) ||0
< 00.
1 ||
Since w is bounded, for each fixed I, we thus have U(I) E R, and
||
|| = sup [[U()]] S || -0 ||""
t20
-1 1
As w is bounded and smooth, so is U, hence U E (2200 E).
18

---

Published as a conference paper at ICLR 2026
C.2 STEP 1: MAANITEDIMMENIONALL ODE FOR THE MOMENT SYSTEM
Contrary to the isotoopic setting where the dynamics only depend on two ODEs involving u(t) and s(t), in the anisotropic setting, higher order correlations (*() and s appear. We first derive ODEs for higher-order correlation terms and then solve the infinitedimmensional ODE system.
Differentiating u (*) w(() Q w. along (3.1) gives, for l: 2 1,
i (t) = 4(s. 3s(()) u (t) + 8 Bu(()s+ (k+1)
(C.7)
Using the normalization s. 1, we obtain
(t) = (01-3s(t))un (1+))))))))))))))))))))))))))))) ( +8t())ss (k+11)
(C.8)
)" E R By Lemma 6, we have Infnite-dimeesionall ODE. Let (t) (t), u (2) (t)....)
U E H. C% (R2u: (²). Consider B l2 l the right- operator on sequences e², defined by (Bx) == Fk+1 for x (Ix)NeN l², Let S = (Bs ej l2 + la be a rank one operator with s = (s s. ....) E L2 Bs = (s (2) ....) ) E e and e1 = (1, 0, o....) is the first basis vector in the canonical orthonormal basis for L Then (C.8) admits the equivalent formulation
((t) = (4(1 - 3S--3s(t)BB+53 (t).
(C.9)
This is an infinit-ddimensional ODE, with U belonging to the Banach space C5 (Rzo: 2) of smooth, bounded functions with values in the Hilbert space 62, In the following, we seek to solve (C.9).
We recall that on a Banach space B, with L(B) denoting the Banach space of bounded linear opera- tors on B, the exponential operator e is well-defined AA L(B), with
A#
C(B L(B),
(C.10)
k!
k=0
||||
||o ||
ellAll < 20.
(C.11)
k!
km0
Lemma (Shift exponential), For any E IR and x E e,
4m
(c
Tk+m
(C.12)
m!
m=0
Proof. Because B is bounded on L² with |B] < 1, the exponential series for BB converges in operator norm for every O E R, and (C.12) follows from ""x x having coordinates (Bmrr) Ik+m-
Proposition 7. Define the function θ R2o + R by
((t) = / (1 - 3s(()) dr.
(C.13)
The initial value problem on t
Ú(t) = [4(1 11(1-3811 3s(()) B + 88) U(t), U(0) = Uo E et,
(C.14)
has unique solution U 0 (Rzoi f2, satisfying, Vt 2 0.
U(t) = e (BB( Uo + 8 %." (B(00) SU(() dr.
(5.2)
Equivalently, since S (Bs eT. (5.2) can be written as
U() = e (BB(()) Uo + 8 L 4B (0()) (Bs u (T) dr,
where (1) (1) = ((,,U(())
19

---

Published as a conference paper at ICLR 2026
To prove Proposition 7, apply the following result ODEs in Banach space.
Theorem 4 (Pazy (2012), Theorem 5.1). Let B be a Banach space. Assume that A(t) B - B is bounded linear operator E T) 0 S T < DG, and that the map - A(I) is continuous in the uniform operator topology, with ma ||A(!)) < co. Then Vua B, the initial value problem
{
du = A(t
0 to sts <0
aF u(46) up B.
(C.15)
has a unique solution u E C¹ ((to: T): 1))
Proof of Proposition Let a : Rzo + R be defined by a(t) 3s(()= = (tT)))((t..
Since E C 2o: Rd) by Lemma 5, we have E C (Rzo; R), C (R2o; with
|a(t)| < + [|
||l| - 1 3| ///l.
la(t)) - aa(t)) = (1(((t)) Qw((t)) - - Qw((z)) (((((t)))Qu(())
s 3| (1)((t))) - ((t2) (t) + 11w((t))
Let A R200 - [(E2) be defined by A(t) = 4a(t) + 85, then A(t) : l + l is a bounded linear
operator vtt 2 0, with
||(!II < 4||B|| (1 + 3| ||((() 8||S|
|| S B|| (1 + 3| ||])==]]]))))) + 8|S]
[|A(t)) - A(t)) < |2| B |0||| -
(t)]l + [|x((t)) S 24 |[2]] -
-
Thus A(t) is continuous in the uniform operator topology and sup:20 ||)(t)] < co. By Theorem 4, the initial value problem (C.14) has unique solution U C³ ([0 ([0,7]: vo < T < co. Since s(t) smooth, follows immediately that U E C² ((0, v0 < < cc. Consider the following function U R2o - l satisfying
utt) = (Be( Uo + 8 L BB (B(( SU(T) dr
e 410()) Un + Se (n(()) / (n SU(T) dr,
where we recall that @(t) = So d(r) dr = So (1 38(T)) dr, with é(t) = a(t). Differentiating (t) on both sides gives, via the product rule,
I e (B) sU())
((t) = 4a(() 4880) Up + 32a(() B 488())
= ((())BBU(++SUUU)
which is precisely the differential equation in (C.14). Since E Cg (R>o: R). U E C (Roz0 (2). Therefore, the unicity of solution implies that U must be the unique solution of (C.14) and we must have U E Cg (Rzo: e
follows that
C.3 STEP 2: VOLTERRA REDUCTION
Taking the first coordinate of (5.2) yields a Volterra equation for u(t) (t)
Proposition 8 (Volterra equation). Define
Σ (40())) m
ee(t)= = (e (B)))0U(0)),
a (1+m) (0).
(C.16)
m!!
Ke(t)= = (e 1Be( (33)0)Bs22,, = (4)()))
m
(2+m)
8.
(C.17)
m!!
m20
20

---

Published as a conference paper at ICLR 2026
Then satisfies
u(1) as(l) + S/' ke((-7))(f)dds
(C.18)
Moreover,
ne Σ wi (O)w3 X 20.611
Ke(() = E(w))² c0
(C.19)
Proof. Equation (C.18) is the first coordinate of (5.2). The spectral forms follow by expanding ml++m) (0) Σ:
A +m w; (0)wī and s (2+m) Σ, (w;)²
>x? 2 and summing the exponential series.
To analyze the growth u(t), we employ the Laplace transform together with residue calculus; (Gripenberg al., 1990, Chapters 2, and 7) for detailed exposition. We first illustrate the method in the ideal case (tt) t. and then explain how the analysis extends to the approximated
C.3.1 IDEAL CASE: @(t)
Fix > 0 (e.g. b b= b= 4if 1) and consider the sealar Volterra equation of the second kind
(t ao(t) + 8 f." Kolt 5) (s)
Ko(t) [ww))
c bdat
0, (C.20)
where A = max; A and the series defining Kτ (hence Rs) converges for Rs > bd, Define
(w:)) 2
Do(s) 1 1-8K- 8K)(s) 1 Σ
Rs > bY1-
bλ;
Lemma 8. Assume (wT)2>} > O. There is unique real p(b) bā1 such that
(w
Da(p(b)) = 0
(C.21)
bb;
Moreover, the map b + p(b) is strictly increasing and continuous.
Proof. For real > 6b,, define
(wf) )a
Fo(p) =
P bā;
Then F6 is continuous and strictly decreasing on (bλ1, oc). Since (wt)² > 0, we have
lim Fo(p) = +00, oābA1
lim Fo(p) = 0.
Hence there exists unique p(b) E (0) such that Fo(p(b)) 1, equivalently Ds(p(b)) = 0. Now let b2 b, and fix p > b2d1. Since p - b2d1 < p- b1 for all i, we have Fog (p) Fb, (p).
Evaluating at p = p(b,) gives
1= Fb, (p(b))) < Fb,
Because Fb, is strictly decreasing, this implies p(b2) p(b.).
Continuity follows because (b, p) + (p( is continuous on sets {p 2 bb1 + o} and each Fb is strictly monotone, so the inverse graph p(b) depends continuously on
21

---

Published as a conference paper at ICLR 2026
Proposition 9. Fix b and let p(b) bb1 be the unique real zero of
Do(s) = (w:) >
bd;
Set 66 p(b) - -by1 0. Then, for all 20
c(b)
as(()
el(b)e (1 (11))
(C.22)
vd
where the constant
c(b) vdāo(p(b)) ([((()))
(3/2))8/2
satisfies |c(b) x
5b
In particular, if (p(b) 0, then c(b) > O.
Proof. The proof consists in solving the Volterra equation in the Laplace domain, and then applying Laplace inversion together with residue calculus to characterize the solution in the original domain.
Step 1: Laplace inversion.
For Res bλι we have
(s()) Ds(s)'
A (0 w
(s(s)
ao(s)=
bλ;
1=1
By Bromwich inversion, for any on > p(b),
1
tatio
at 0e(s)
(1)
ds.
2Ti
D2(s)
JOR-io
Step 2: Contour shift and residue extraction. Fix o. (bλ1, p(b)) and T > 0. Consider the rectangle with vertices OR + iT' and σ. + il' The only singularity of (s)) in the strip (a. Res < R is the simple pole at p(b) (zeros of D, are real and interlace the poles bλ.). Introduce the notation
6, = p(b) - bλ1 >
for the gap between the top pole and the dominant zero
On the horizontal edges + if with E on) we obtain
Xm (0) 5
1 Σ (ow < T
1 - ||x(0)]l2 Σ (wt) [(wx)²2?)
1/2
[)((s)] <
by
T
where we used Schwwrzhcccc..... and [1)(1))
Moreover, since
(w)) 223
= DT( Σ x bλg + iT
uniformly in x E [o., as + 0c,
we have inf inf (rn) DD(r + T 2 for all T large enough. Therefore the horizontal contribu- tions are O1T and vanish as T + xc.
By the residue theorem, we obtain the exact identity (valid for all 0):
(t = ae (p(b)) (b(b) (t), (pbb))
1 ( +
(s(s)
Ro,(t) = 2ti -100
ds.
(C.23)
Do(s)
22

---

Published as a conference paper at ICLR 2026
Step 3: Coefficient bound.
Let 8 := \e((p(b) - bd.). Then
ae a = Σ wi w0)
Under a standard isotropic initialization (e.g. w00) ~ Unif(S or w(0) ~ NO, Ia/d)), condi-
tional on w this is centered with variance d 2, (w) )2 hence
(2)
x vd Σ (10)))09)) 1/2
1
Σ (2o (xot))17) 1/2
3,
ke(p(b))
vd 66
6, ōgvd
Next,
(w,)2
Db(((b)) =b- (p(b) 6A,)²
8 (w
85
where we used 1 = (w; bA:) and
Hence
Te (o(())
(s. (3/2)3322
(C.24)
56 V.
Step 4: Remainder bound on Res = o.. Since o, E (b)) and all zeros of D, are real, Ds has no zeros on the line Res G., and (z iy) + as by co. Thus
mz(o.) = inf Da(a. + ig) > 0.
yER
Moreover, for G. + iy.
[0 + iy)l =
Σ wi (0)z7
A bA:
Σ (w))
X?
iy
Vd
(a. bb, )2 y²
by the same isotropic-inittalzzatiie scaling as in Step 3 (applied to the fixed direction depending on
y). Using a. bλ, 2 o. bàj and the standard integral fgl(a? + -/2 x²) 1/2 dy z
T/ r/V one obtains
C(sl2). .)) eat
C(s (2) b) of (s )3/2
Ro. (t) s Vdmm(o.)
bb1
With the midpoint choice p(b) - 62/ this gives
RR. (t)
(b./2)t
(C.25)
ue ((bb))/D((((l)) eP(D)
mz(o.)
Step 5: Uniformity for t = O(log d). From the relation,
1 1 8 (w,) 22
8(w7))22 p(b) - bλ
8(w))
((b) Py:
8b
so 8, > X2, If (wt) 2 ke 0, then ds 60 SkØ, constant independent of d. Taking t = c loog in (C.25) yields exponential decay in d. Combining with (C.24) gives the claim with e(b) (d (cbb))(/D(((((b)) and |c(b) S (o (2) (2))33/2/..
C.3.2 GENERAL CASE
The ideal case O(t) t served as a benchmark where the growth rate and coefficient could be computed explicitly. In the general case, O(t) is only an approximation of t, and the kernel Ke becomes perturbation of the ideal one. The key point is to show that the dominant pole of the Laplace transform is stable under this perturbation, so that the same exponential growth persists up to small shift in the rate and coefficient.
23

---

c(0) Published as a conference paper at ICLR 2026
Proposition 10. Denote by p(4) > 4x1 the unique real zero of F4 (4)1,00) Write F(p) 1 8 Ke(p) and let Perve be the rightmost real zero of Then:
Perue p(4) S 8.
ii Let c(6) /druu( 1' F' (Prrue). We have [c(6)| S (3/2) (Ptrue 421). and
m(t)
Orract (1+
Vd
Proof. Let AK: == Ke K be the difference between the ideal kernel and the true one.
Step 1: Pointwise kernel perturbation. Since (1 35)t - ( S t and E" - 1 S x for x I 0,
(etr -
edAill s 4A, (t e(t) e S 125 X te
Multiplying by (ac²) ?2 and summing.
(K((t) & 12A ŌtK(t)
(t20).
(C.26)
Step 2: Laplace control of the perturbation. For Rp > 4h2.
AK (p) = L e -M *A((t)d dt,
AK(P) I / E -( [tK()) dt.
Using (C.26) and differentiating under the integral sign (dominated by tk (t)e (MRp))). (t))-(Mp)))
Ak(p) S 12\1 5 ( te tet) - 12A1 5 ('))..
Hence, for F(D) 8Kolp) = 1 ((k(p) AK())),
1F(p) Fa(p)
[k(p) s 16 0 (-))
Rp> 4h1.
(C.27)
Step 3: Unicity of the dominant zero. Let po p(4): then Fi(ro) -8R co > 0 and
F2 is strictly increasing on (4X1, co). By continuity, pick > such that on = P - r,Po + r]
(1) 2 ca/2,
Mi == sup (p)) < <x.
(C.28)
PET
From (C.27),
sup Fp() - -F((p) S 96 M1 8 =: E6- pEl
Choose &e small such that Eg S (co/4)r r for all 8 (0, 8g). Then
Frp 2 (a(potr) E8 2 (co/ (c)/4)r > O,
F(Po-1 r) S F res S (co//4)r < 0.
Because F'(p) -8 K8 (p) = 8 le te -e (e(() dt > 0 on (411, oo), there is a unique zero Perue E (Po - r. r. + r) and it the rightmost one. Moreover,
P p(4) <
Es 2
S 8,
proving i).
Step Contour decomposition and leading coefficient. to Rp = E (4X1, Phrne) give the exact identity
Bromwich inversion and a contour shift
u(t) = de ae (Perue) eoencct + (prrue)
1 - t
(p) F(p)
e.
dp,
(C.29)
2 Jo. -00
24

---

Published as a conference paper at ICLR 2026
since the only singularity crossed is the simple pole at p Pirue- For the coefficient, set 22 X - 4),). Then
1/2
(Ptrun)))) Σ (oc())acoo
S
1 Vd
Put
1
Vd
Ptrue 4
where the X d-1/2 scaling follows under isotropic random initialization (e.g. w(0) (0, Ia/d) or w(0) ~ Unif(S "), and is encoded in Assumption A1. Further, F" (Ptrue) =
8 Jo Preal Ke (1) di > 0, and continuity from the ideal case implies F" ('(ttro)) 2 16.. 12y for all
small Therefore, under Assumption A1
ae (Ptrue) F (Puru)
(s.2))3/2
(C.30)
vd (Perue 4
Step 5: Vertical-line remainder. a uniform gap infy Foa. + iأ)l 2 Ptrue Also,
On Rp os monotonicityy/oonveiity and F (Perue) > 0 yield
1/2
X2
Tāe + iz)l 2
Σ (wo))
Va
(o. + y²
by the same isotropic-iiiiaaiizaiion scaling as above (applied at the fixed value y). Estimating the dy integral in (C.29) by Cauchy-Schware in y and using Su (a"+y") - ग and
J Vo
dy
S π V/AAA
02 12 2
8²
we obtain
1
tice
āe
coot
eet
dp
2Ti
Ja. ic0
F(p)
Vd
Ptrue o.
Choose the midpoint 0, Ptrue
A/2 with Pirue
4)1. Then
R S
( - A/2)t
Vd
Step 6: Conclusion. From the ideal dispersion relation,
22
)
1=8 Σ
p(4) 4h1 > 8Ke-
p(4)
4h,
p(4) 4h1
By i). for small 8 the true gap obeys A Ptrue - 4h1 2 4ng =: Ao > 0. Hence for t = c clog d,
R
S (Ao/22)
do 0.
(Ptuuer))/F
ePtenct
Putting this with (C.30) yields the stated asymptotic with c(6) vd F' F' (Ptrue), continu- ous in and bounded away from 0 as ö - when àe (p((p) (((4) >
This proposition motivates our assumption on initialization.
Assumption A1 (Initialization), We assume that u(0) x 1/1/2, s(0) d= and åe (ptru))) X d-2/2 Moreover, we assume that wi is of constant order and non-zero.
Remark 6 (On the initialization assumption). The assumption in Al is natural under random ini- tialization. Indeed, if w(O) ~ (O, Ta/d) (or uniform on the sphere) independently of w., then conditionally on w. have
A
do (ptrre))) = Σ (r(0)ut ~ N(O. Σ (wz)) (u?)
2
Perue 4
25

---

Published as a conference paper at ICLR 2026
This variance is of order 1/d. Hence ae (Prru)) is of order d with large probability, that the prefactor in Proposition 10(ü) is indeed nontrivial.
The quantity u(0) also fluctuates the order d for Gaussian initialization, but establishing deterministic relation between u(0) and ăe (Prrue) is delicate, as the two depend differently on the spectrum and w., This explains why Al is stated as mild probabilistic assumption rather than a deterministic condition.
C.4 STEP 3: BOUNDING HIGHER- ORDER STATISTICS AND POSITIVITY OF u
Our goal in this subsection is to show that the second moment u (2) (T)) is strictly positive.
For 2 1, define
(K))&(t) == (e ( (1) = (40(t)) T
(k+m)
E(w)) 821 1.
m!
m20
Lemma 9 (Kernel domination). For all l 2 and 1 LE {0,,Ti}
(K))x(1) S 1 (e((,
Ko(l) = Σ
(40()))
(1+m)
m!
m20
Since (tt) 2 0 for 1. S Ti, all coefficients (10(1)) m ml are nonnegative. For each m 2 0,
Proof.
(k+m) Σ (wtt x
( (x
X (1+m)
Multiplying termwise by the nonnegative coefficients and summing over m yields the claim.
Lemma 10 (Kernel positivity). For all 2 1 and E Ti).
(K))x(t) 2 2 (wf)²
Proof. When ( 2 0, all coefficients (40(1))* in the definition of (K))& are nonnegative. In particular, the m = 0 term contributes (K))&(t) 2 s Finally, s+ (k) (w (
Proposition 11 (Control of higher-order correlations). For all k > and all E [o,Ti we have
X logd cdre + (x I (((()) (t)) s xt lond eldit + k-1 At-1(((t) (u(t) z(()).
-
(C.31)
In particular, for d sufficiently large there exists constant e 0 such that (( (Ti) > C1-
Proof.
Work on [0, Ti] where O 2 0 (so Lemma 10 applies), and recall Lemma 9, We have w.h.p.
(k+)m))))))))) (0)) < k log
d for all k 2 1, m 2 0.
(C.32)
By Duhamel's formula, for each k > and E [o,
(40(I))TT (++m) (0) + 8 S (K))))((-T)u(() w())
(1) =
m!
20
From (C.32) and (1) < 1,, we obtain
(4) TM
log d
(40(1) mn!
n (8+m) 5
X
m
m20
log 111)())
d
log
< 1/
ahit
d
26

---

Published as a conference paper at ICLR 2026
Furthermore, by Lemma 9, we have (Ko) X so
[. (Ko))(t-r) & (ut) -
Combining with the homogeneous upper bound yields the right-hand inequality in (C.31),
Lemma 10 gives (t s.a for [0, f], hence
( (Ko))(t-r)u(T)))
2/4)) S/ u(() dr.
Combining with the homogeneous lower bound gives the left-hand inequality in (C.31). Finally, under the positive growth of a assumed in Step 3 (e.g. a(t) 2 Sael
elt for large t), together
with rate gap p 4)1, the integral term 8s!2) (2) Jo dominates the homogeneous remainder
A eATT = O(Vlog for large enough. Hence, there exists e₁ > 0 such that (2 (Ti) 2
log d
D ANALYSIS OF PHASE II
In Section D.I, we show that s() crosses the threshold 1/3 within O(1) time and remains above thereafter. Once s(t) > 1/3, the key quantity ((t) appearing in the Volterra equation begins to decrease and eventually becomes negative. This marks a qualitative shift in the dynamics of the system. We leverage this change Section D.2 derive a convergence rate for the summary statistics.
D.1
PHASE: II:: CROSSING THE s(1) = 1/3 THRESHOLD AND IRREVERSIBLE GROWTH
Recall that at the end of Phase I, there exists an absolute constant 8 >> such that
u(Ti) > 6, s(i)) 2 6,
> 8. u (2 (Ti)
a high level, the behavior of s(t) in this regime is governed by simple mechanism. While s(t) < 1 1/3, both terms in (D.1) are nonnegative, so s(t) is pushed upward and necessarily crosses the threshold 1/3 in finite time. Once s()) has crossed, the positive mixed term 16 16 outweighs the negative contribution of the first term near the boundary, which prevents s(l) from falling back. Thus s()) remains bounded away from 1/3 uniformly after crossing. The next proposition makes this precise.
Proposition 12 (Crossing and stability beyond the 1/3-threshold)) Let Ti be the stopping time from Theorem I. There exist constants 5 and C > O such that:
1. There exists Ti E (Ti, Ti + C] with
(
1 2
3 + 8.
2. For all t t> TV,
s(t) 2 3
+
Proof. From Ti onward we first prove that u (1) and then u(t) stay strictly positive. While s() < 1 1/3, this makes both terms in é nonnegative, so s reaches 1/3 in finite time (Part A). Next, we work in thin band [1/3, 1/3 v/.. show that for small enough the drift of is still uniformly positive, so s reaches 1/3 +-7 in finite time, and that the vector field points inward at s = 1/3 + n, preventing any return (Part B).
Recall
s(t) = 8(1 - 3s(1) (() + 16 16
(D.1)
i(t) = 4(1 4(1-3) 4(1-3s(t)) (t) + 8
3s())) ! u(t), ((3(()) = 4(1 4(1-3s(t)) a (t) + ( u(t).
(D.2)
27

---

Published as a conference paper at ICLR 2026
Part A: Pre-band positivity and finite-time reach of 8 1/3. Fix := T1 2 0 and define Or (T) = (Ti + (Ti). Writing (t) == (;),), we have
(t) 4(1 - (t) (x(w))) (M(
Multiplying by e (4t) and integrating from 7i Ti + yields
aT(Ti + +) (at)) f. (t) (e)) {x(1))² u(Ti + ds.
Multiplying by X and summing gives
u 22 (Ti + Σ 13 Ma +8 S Kx(T - 8) u() + +8) ds,
(D.3)
(r)
with
( 2 0. Kx (c) [(u)) (1)
On the pre-band interval where S 1/3, we have Or (T) 2 00
so ed2. (+) 2 1. Thus
a0 (T) 2 Σ (T)) = (T) > 0.
By (D.3) and Kx 2 0.
(t) 2 (2) (Ti) for all Ti S := inf(t 2 T1: -(t)=1/33,
(D.4)
While s S 1/3, from (D.2) and (D.4),
i(t) 2 85 u(())
Ti st
so u(( 2 (TT)) there. Finally, (D.1) gives on (s S 1/33
8(() 2 16 v (1) 2 (6 v (Ti)
ke 0.
Hence
(2) (Ti)
1/3
T1/3 - Ti
(D.5)
16 (T)
Kg
Thus & reaches 1/3 in finite time.
Part B: Crossing the band [1/3, 1/3 - n] and no return. used in-band. Since Q < A and [|w([[]] < M,
We record uniform bounds that will be
5 (2) S (1))2(1)12 < ST
ST == X²M²
(D.6)
and S stt) < 5max with 5max == M2, By Cauchy Schwar,,,
(1) & / s()) < Smax s (3)
=: C2, (1() (0))=((0000)))0 (() V (4) <
Smax (4)
8*
=:
Cz.
(D.7)
Fi E 1/6 On {1/3 < < 1/3+ nn we have 1- 3s E 1-3 0). From (D.2) and (D.7),
ù 2 - 12n C2 + 85 (2) u,
(1(0) 12n C3 8s (3) u.
As in a linear comparison argument, if
n و 3 0/ (Ti)))))))))))))n))))
1 5 2 (0)
2 u(Ti),
(D.8)
then throughout the band one has u(()= 2 u (T:) and n (2 2 (Ti)
28

---

Published as a conference paper at ICLR 2026
Now, from (D.1), (D.6), and these lower bounds,
s(t) 2 -241 -2 S2
+ 16 (Ti).
Choosing
7 S min { u( (Ti) 35
2 25 ()
(TTi), C (Ti), f.
(D.9)
C2
we get s() 2 8 (T ( (Ti) > 0 across the band. Thus overshoots to 1/3 + 1 in time at most (8u((T ()()))))))))))))))))))) ), and at any last contact with & = 1/3 + 1j the same inequality shows $ > so the vector field points inward. Therefore, cannot return below 1/3 + 1).
Together with (D.5), this completes the proof: 8 reaches 1/3 in finite time, then crosses to 1/3 + in finite time, and never falls back below.
D.2 PHASE IlB: APPROXIMATE CONVERGENCE OF THE SUMMARY STATISTICS
After the entrance time Ty, the error ((t) == 1 --) satisfies
A(t) = be(t) + ["' Ty
(o(t,T) A(T) ((T) T2T,.
(D.10)
with
d Ko(I,T)= Σ
8 8 (wt)e an (600)))))))) he(1) = Σ Awa wi (Ti) 42 (01))-)))))
1=1
(t(t) holt()) S" K)(
We will repeatedly use the following facts:
i 12 A(r 2 0 and (t, 7) 2 0 for 2 Tí.
(ii (t) 3s(() S 8g 0, hence 441(((1) -(()) s 4004 fort Ti
The first fact results from Lemma 11. For cutoff A₂ let
I= = (;: < Act,
I= f
Define
:= Σ de (wf)²
(2) Σ
(w))?
s(T)) Σ d u. (T(())) $1,
ArcAc
and the head alignment term
S>())) = Σ A
< ( E do (1))))
Σ Au
I 5/ s(Ti).
A.ZAD
AzZAo
A,ZA
Proposition 13. For all 1 2 Ti and any cutoff do. >0 0.
A(t) < T(X.) + (T(A))((T)
(D.11)
Proof. From the Volterra equation (D.10) and the definition of be.
(1) = - ho)) [" () 1) dr.
Since 0 < A(T) and Kg 2 0, the integral is nonpositive. Hence
A()) < he()))+-
29

---

Published as a conference paper at ICLR 2026
Now split the spectrum into I< fi d de) and In {i: A 2 Act- Using O I ((T))
and (t) E -s. we obtain
- he))++ z Σ xxw))) +
Σ (i + VEI
Σ
Adloe wi (TT)) 450 d (t-Ti).
IEZ
REZ
The first sum equals T().). By Souchy-Schwarz
Σ
Alw & ((())s((T)
KET
The last sum is bounded by Sz Sz(Aeee 450de (t- (t-Ti), Combining these estimates yields (D.11).
Corollary (Time to reach accuracy Fix (0, 1) and choose de 0 such that
E T():) S 4'
(((e)
E s
4 = T())) S min
f E²
(D.12)
4
16 sT)
Then
log (482(Ne)
T2(E) Ti +
(D.13)
480 A
E
satisfies A(f) S for all t 2 T2(E).
Proof.
From (D.11) with d. = dr and (D.12),
(1) <
4 4 S2(Aa) 4sod, (1-Tf) + +2(())e -o (e-25)
8 s
Choosing so that the last term is < e/2 gives (D.13).
Corollary 2 (Power-law spectral tail). Suppose T(X) Ctand 9 as X with 3 = 1 (0,
Then
[minff, <
1/3
Crail
1/8
A In particular, when is small and s(Ti) I 1, the quadratic condition dominates:
T2(e) S
T
log
Ctail
480 (minfe, <2/s(T{()}
T2(e) S Ti +
1 --18
2/8 log
1
Ti + 480
log
480
D.2.1 TECHNICAL LEMMA
It will be convenient to approximate the discrete measure μ4 by continuous reference measure with density proportional to X 1-1/a near À = 0. This approximation is crucial in Phase II, since the long-time behavior of the kernel Ke depends only on the small-X (tail) mass of μas, which in turn reflects the power-law spectrum assumption. The following proposition shows that, with high probability, Pa has a similar tail behaviour to continuous counterpart.
Proposition 14 (Spectral/teache tail near 0 holds w.h.p.). Let 1 and define the spectrum i..dd NO0, 1), independent of (Mi), and set by A := i- Ha where Há, = EE= j_a Let wi 2-
Hd Pd: 8 Σ 2 8x.
Fix constants C, > 1 and E (0, 1), and define the tail scale Aa== = C. Ad C. d Haa and the geometric bins
Be (p*+1 Na, o² Aa], k= 0= where K = [ogg, (C+)l - 1. Then there exist constants C-, C_,C > 0 depending only on (a, p. C.) such that, for all sufficiently large d, with probability at least c -ad
x da z Pa(Ba) s C+ I à
C_
for all 0, I..... K.
(D.14)
Br
30

---

Published as a conference paper at ICLR 2026
Consequently, with the same probability bound, for every X E (0,Aa).
pa (a((0, XI) x X X'
(D.15)
and, more generally, for any nonnegative step test function (p(A) = Ex=D on 1n (A),
c- c x- p(A) & Jana ((A) s C+ /o- ==) d2.
Aa
(D.16)
Proof. We split the proof into several steps.
Step 1: Bin sizes are linear in d. Let J4 == (1 sisd: E B& Since == = / the
condition A S t is equivalent to 2 (tHa,a)=144 1/a, Thus the counting function
N(t) # (i: X {} d (tHa 11/ + 1.
Therefore,
[&l = N(p N(pht1Na) = (o Hd,a) 1/a (dHHaHa)))
Using AaHla, C.d a gives
[ul = p==p-(ttt11 (ktt))))/a d] -
Hence, for all d large enough,
dp dpa k/a < | S 02 c2dp
(k= K);
(D.17)
for constants CI2 02 0 depending only on (a, p2 C.). Thus |/1 p(dp=k/a) k/a), and K fixed constant (independent of d). Note: for the last bin Bx, the lower endpoint may fall below Ad, but by definition Iς only counts actual indices < d.
Step 2: Deterministic comparison of EiEle di
For i Is we have ph+1 Ad < X, S Ad.
Aa) [Al S
Σ As s (p* [
FE
On the other hand,
J - dd
(a) 1-1 (g)+ 1-2
Aa)
1-
1 - -ーー
Ad A p k(--))
(D.18)
Since a > 1, the exponent 1-1/a > O. Using (D.17) and (D.18), there exist constants D., D. > (depending only on a, p. C. and the bounded factor 11/1/a {(((a) 11a
-1/a, 1) such that
D_ A dd S Σ EASDO - ds
(k: = (..,,..,K,,
(D.19)
Bb
NEL:
Br
Step 3: Concentration for the teacher weights. Let St == Eier, (wf)² Since Se xir, and
[Tl 2 eid by (D.17), the standard x2 tail bound yields, for any E (0, 1),
P(Is- |T&|| <lTal) s exp( (1-1-114 ('(Tal) < 2c czd
Since K + 1 is fixed, union bound gives
P vk 0, o.....K K: (1 (1-8) (1-6) Ss S (1++))1Tl) 2 1 -ed
(D.20)
for some > 0 independent of d.
31

---

Published as a conference paper at ICLR 2026
Step 4: Comparing (a to the bin integral. On the event in (D.20),
8(1 -1 Σ de I Pa(Bx)
Σ KE
{x(wt)² - 8(1 + Σ A
=8
x
TET
Combining with (D.19) proves (D.14) with
C= 8(1 E) C= 8(1 +
Step 5: Tail. Summing (D.14) over (j 2 m) where is the unique index such that À (p Ads pm Ad], and using the geometric form (D.18), yields
pa( AJ) x Ad m(1-)) 2) x
(since pM x/Aa),
which is (D.15). The step-function bound (D.16) follows from linearity, as So.na yp adpa Ex as (B( and the same for the right-hand integrals.
The following lemma formally shows that after Ti, u(t) and s(t) cannot go above 1. In particular, it justifies the positivity of (t)).
Lemma 11 (Post-alignment barrier and correlation bound). Let Q diag(A.) 0 and let w satisfy (w" Qw") = 1. Consider the population gradient flow for anisotropic phase retrieval with Gaussian inputs, and define
s(t) == (w(t),Qu((t)
u(t) = (w(t), Qu").
Fix a time to (e.g. to = Ti) such that
0 < u((o) 1,
s(to) S 1.
Then:
1. s(t) S for all 2 to (forward invariance of {s S 13).
2. < v(t) S for all 2 to, and in fact u(!) for every finite t 2 to unless w((o) w
Work with the Q-inner product (x., v)q == (ar, Qv). Decompose into its Q-rrthogonal parts:
Proof.
w w0 + 2, (z, Qw") 0.
Then
8 (w, Qw) x² + ||=11.1 ||:18 = 5 u² 2 0.
(D.21)
Claim (pre-boundary: u 1). Since s(1)) S 1, Cauchy-Sehwarz ( gives
u((o) (wo(o)) Qw") < [[((b)]lQ || vo((o) < 1,
and by assumption u(to)) > 0; hence < u(tn) < 1. Moreover, as long as s(t) < 1, the same inequality implies u(t) S / st) < 1. Thus before any potential first time when s could reach the boundary indeed have u(t) <
Claim (uniform drift for s 2 1). Recall that
= 8(1 - 3s) s(2) + 16u u (2)
Using Amin (z, Q2) S (2) and (D.21), when 2 we have - 38 < -2, hence
à < 8(1 - 38) 8
< - 16 s (2) < 166mmn (2, Q2) 166Amin |1=1| = -16Amin (8 u²
(D.22)
Claim (no upward crossing at = 1). Let t > to be a first time with s(t)) and s(t) <
for 1 < 4 (if no such 4 exists, we are done). By the discussion in Claim 0, we then have w (17 <
V (((() < 1, hence u((1) < 1 by continuity. By definition of first hitting, there exists > 0 with
32

---

Published as a conference paper at ICLR 2026
s(t) 2 1 for all t E [t + (D.22) applies on (f1, E). Taking the right Dini derivative and using continuity of s u²
s(t h) s(t1)
Dt ((1)) == lim sup
S -16Amin lim (=(1)-u(1) = -16Amin (1 (1-u(t1)9) S
440
Thus the vector field is inward-pointing (nonpositive) at the boundary, which contradicts an upward crossing from s
> at Hence s(t) S for all 2 to-
Claim (u S 1 and strictness). For all i, Cauchy-Schwarz gives
so 0 < u(t) 1 for [u(()] < l)w(O)l || lle v x(1) 1
(positivity after T` follows from the Volterra positivity in Phase I).
Finally, unless (ty) = w², the real-analytic gradient flow reaches the minimizer only as t + 00, so u(t) < for all finite t to-
Next, we show u(t) and st) - 1. Lemma 12 (Convergence of s(t) and u(t)). As + oc,
s stt) + 1, u(t) + 1.
Proof. By Lemma 5, the trajectory w(t) is bounded, hence precompact in Rd. Moreover,
(tt) = - [[][(( S 0,
so L(w(t)) is nonincreasing and convergent. Phase Ila yields forward invariance the region {s }} there exists Ty 2 0 such that s(t) for all Ty.
By LaSalle's invariance principle, the w-limit set
= { w 36, + oo with w((n) - w
nonempty, compact, invariant, and contained in the largest invariant subset of (VL = ( ). By the critical-point characterization in Section (Proposition 2), every critical point with > of the form points satisfy 8 = and u = +1.
+w*, under the normalization Σ, \= 1. In particular, all such critical
Hence for any E Ω must have (s(w), u(w)) 1), (1)] Since u()) continu- ous, it follows that
s(t,,) + 1, u(t..) #1 for every sequence - Do with wtt..) forward-invariant (via the Volterra representation after T2), so u(t) > 0 for all t 2 Ty. Therefore, every limit point must satisfy u(w) +1. Thus (s(t),u(t) (1, 1)
Phase guarantees u(Ti) > 0, and positivity is
ac.
Finally, if s()) or u(t) did not converge, there would exist > O and sequence l, + oo such that either (s(tn,)
2 E or [u(t,) 2 E. By precompactness, we may extract subsequence w(tng) + w E 52 but then (s(tma). u(tmx)) + (1, a contradiction. Hence s(t) + 1 and utt)
E PHASE III: SCALING LAWS
We first show in Section E.1 that the MSE does not change significantly from initialization. In Section E.2, we then characterize its decay for > Tz.
E.I STABILITY OF THE MSE FOR < T2
Proposition 4. Let o Ei (wt) Under the assumptions of Theorem we have
MSE(T>) - o2 S
1/3
log
1/3
+
d
d
33

---

Published as a conference paper at ICLR 2026
Proof. Step 1: Variation of constants. For each coordinate i,
(1) 41:(1 - (1=((1)) wi(() 8Acu(()) @ :- / (1 (1-381 (1-3((T))de dr.
By Duhamel's formula, for all t 0,
wx(t) ct,,0tt) w0 + SAeuž / (0h1111))007))) u(T) dr.
(E.1)
Step 2: Phase clocks and the key split. Let Ty be the first time s(t)
Then θ increases on 3'
[0, Ti] and decreases on [T] oo). Define
(TT)) Fix t 2 Tj. Splitting the integral in (E.1) at Tí and changing variables to θ on each side, and using 0 we obtain the bound
max
A(t) == max Ot)). TT.
L' (r. 0(7)) di S c
4dr0max
1-e-4A-A())
(E.2)
X
X
where C' 0(8, so). This refines earlier estimates by accounting for the change of sign in O(t) (T) across phases.
Step 3: Frontier at T2 and inactive bound. At time T2, fix < (0, 1] and define the index sets
(T2) {i: 4A, A(T2) 2
(((T2) [d] (T2).
For E (T2), 4TA( S <. so 1 - 4A,(TT) < (TT) Combining (E.1) and (E.2), and after absorbing constants,
Iw (T>)l S ( (x)( + 8A/wof 11--0-1
+ +4A(Tz)) Σ cl(w((0)) + C²C |w
di
(E.3)
Step 4: Active correlation and energy. On IT(T2), x 2 </(4A(T2)). Applying the weighted Mauchy-Schward inequality E; |a.ga|| S 12/ 1/2 we get
|wi] lwss (72) <
(w:) 1/2
A (TT)(TT)))
1/2
<
4A(Ta)
Vdo? v((2).
(TT)(TT))
(Ta)
(T22(T2)
Hence the active correlation contribution satisfies
A(T2) çd
d (TI)(TT)
w (Tw((T2)
(E.4)
Similarly, for the active energy,
1
2x( s
Σ (TT;(TT)
A (T))
4A(T₂)
s(22) &
A(T₂)
(E.5)
d
(I
ςd
{d
(Ta)(Ta)
since s (T2) 1.
Step 5: Inactive correlation and energy. From (E.3) and Cauchy-Schwarz, and using | = do?
Σ (T(TT))) 1/2 < vd 20 (w00) do?) 1/2
Σ |ut|l (Ta)) (EI)(15)
||w0|1a
(EI;(Ts)
Therefore,
2
Σ | (en
S < 1/2
(E.6)
(TT)(Ta)
34

---

Published as a conference paper at ICLR 2026
Moreover,
26
Σ (EZE(AS)
wTb)???
< (w0) + x _ d
(E.7)
d
Step 6: Combination and optimization. Since
MSE(T2) - o² d Σ
wfwn (T2) d Σ (T2)
combining (E.4))(E.7) yields
MSE(T;) - of x
A(Ta)
A(Ta)
C+ 1/2
(E.8)
inactive
The right-hand side minimized (up to constants) by taking
A(Taa
A
d
which balances the active and inactive contributions. For this choice,
A(T2)
A(T2)
A(T2)
1/3
x
5r
x
ç"d
d
Therefore,
A(T2) d
1/3
MSE(T>)
x
+ d-1/2_
Finally, the Phase IIb analysis (via the Volterra-renewal argument) gives 1 - u(l) x A(1)-1/a, At t T2. u(T2) E, hence A(Tb) x = Substituting into the above bound gives
MSE())) of 2 (5-) 1/3
+ 1/2
Adding the Phase "plateau" contribution (log d/d) 1/3 yields the stated result.
E.2 EVOLUTION OF THE MSE FOR 2 T2
We first analyze the idcalized dynamics where u (Section E.2.1), and then control the approximation error in Section F.2.2.
E.2. IDEAL CASE WHERE = S
We begin by studying the evolution of the MSE under the idealization u 1.
Lemma 13. In the post-T2 idealization (s 1), the coordinate errors e,(t) w:(t) w evolve as
(T2 = eT e -4,T (T ().0).
Consequently,
d Σ 1=1
MSE + =
G (2))22
164,T = MSF Sa(T),
where the normalized spectral mixing curve is
(T22
d
Sa(T) = Σ T 164,T
T =
Σ
n 1.
E (T))2
35

---

Published as a conference paper at ICLR 2026
Proof. Setting in the gradient flow gives
ittt) 8Q(((t) w²)
Defining e())
w(l)) w² yields ε(I) -8Qe(1). Since Q diagonal with entries (At), each
coordinate satisfies ét = - (t), hence
(T2 r) ec(T2)e -A.T
Squaring and averaging proves the formula for the MSE. Factoring MSE(T;) defines the weights Ti and the claim follows.
Since Sa depends the (random) weights Ti, we introduce the proxy
Sx(T) = BaTi"a
3a = 16La:
which corresponds to uniform weights. The next result shows that this valid approximation.
Proposition 15. Assume that the teacher coordinates (wi yi are i.i.d. sub- Gaussian (in the eigen- busis of Q) and independent of the initialization. There exists constant C < co, independent of d. 0,
such that with high probability, eververy
every
[a((T) - s c(c* +
V loog
Bard
(E.9)
d
Proof. Fix 2 0 and write fi := e- e-16 e -ari Since (A:) is nonincreasing in i, the
sequence (fi) nondecreasing.
Let PE = Ek Ti. standard summation-by by-atts identity yields
d-1
Sa(7) (((r) =
Σ
Px
h 2 (sk+1 -Ka)-
Since Ex=l(fe41 ff fa - f1. obtain
k Ps
[B((7) Sa(7) s (Sa (fa-Si)
sup Isked
(E.10)
Noting that fa- - fi = e - e BaT gives the prefactor in (E.9).
Define
m. = (wt)²
Pt
||w11 1|
1=1
Then
k d
k
sup &
Ps
< sup
Ps P + sup
P
d
k
Write ;=
= Σi (T2)² = d dMS((T)), so = e2(2))2/Z. Using e; (Tb)² ((z))=(w;t} = 222
2w wi (T2) and Cauchy-Schwarz.
Σ (Tx) (wf)²1
Σ w. (T2)² 2/ wn (l
From (E.5) and (F.7) with Ç = 5* we have
d
Wi wa(TT (T₂)² S (C*)²,
hence
d |1w0112 (TT(22))le S S*
d
36

---

Published as a conference paper at ICLR 2026
so altogether E, (Tx) (w S dÇ* Moreover, Proposition 4 gives MSE(T²) o² + o(1),
hence Z with high probability. Therefore
es (T2))
(wf)² )2 | 2 S 5*
|| >²111
and thus sup, Pk PAl
- x² 111 5
Since ((w,)²); are i..dd sub-exponential with mean of Bernstein's 's inequality and a union bound over k = 1, I,.... yield
k (wt)²
sup IskSd
d j=1
(w))2
S vdlogd
with high probability. Dividing by E,(W3) x d gives
log d
sup Isked
Pk
with high probability.
Combining the previous bounds with (E.10) yields (E.9). Taking sup+20 and using fa - f1 S gives the result.
We now analyze the asymptotics of Sa(T).
Proposition 16 (Asymptotics of the spectral average). Let 1 and set wa (Bar) Then Sa(T) satisfies:
1. Early time (Bar < 1):
Sa(r) =1 BaT d
(Bar)² 20 of (BaT)
Σ
d
+
2d
1=1
Under trace normalization La = Ha, this simplifies to
Sa(T) 1 26 + +
2. Mesoscopic window
< xd < d):
_d Sa(T) = -1
d
3. Late time (xa Z d):
Sa(T) I expl -xp d a)
Proof. For E 1), e= + 2² /2 R3(2) with [Ra(2) S 2/66 3aT & 1, all Bari are small, and averaging the expansion over = 1,..., d yields the stated series.
(ii) Writing
1- Sa(T) = 1 d -- (1-e-(e0/09)
--
and defining (z( = (1//t) we have
10 g (1)) / (1-e-1=/0") dl x I(1 11-1 1
by the substitution y = (z/t)a, Since Iu is monotone decreasing in t,
d 1 (a(t) s Σ (1) s gz(1)) + +1, (t) dt.
37

---

Published as a conference paper at ICLR 2026
Thus
=10
Dividing by d proves the expansion for < Rd < d.
(iii) For i d, 1m 2 d hence e -BBTss e Averaging over i gives the bound.
Corollary 3. With high probability, uniformly for all 2 0,
(S + +v /log (1agg) (e-Berd (e- -e-ar))
MSE(T2+T) MSE(T2) Sa(r) = MSE(T2) (a(T))))((T)))) Sa(T) In parficular, in any regime where
Sa(T) (5" +
log d
(e Bard
(e-aard -e -BaT))
have
MSF(T2 + -SS-=MSST(T)))) (1+ (1+
Remark 7. Recall that for va == (Bar) and 88 == Bar = (sraf d)) we have in the late-time regime Sa 0C,
Sa(T) asd
On the other hand, the correction term in Corollary is order
(c* + logd ) o0
Hence the maltiplicative condition
Sa(T) > (5t +
log d
(e Bard -PaT)
reduces, in the regime sa >> 10
logd
8d
In particular, ( (log 1/3) (as in (E.8)), this condition holds as soon
d
1/3
da
d
1/3
Sa
log d
i.e.
T
Ba
log d
When 3a x d,, this corresponds to
d 1/3
equivalently
& d /=+1/3(logg)) d) 1/3 -1/3.
log d
E.2.2 HANDLING THE APPROXIMATION TERM
Let (t) 1 118(0) s(t) and (t) u(t). We will first show that the convergence approximation remains of order E after Tz-
Lemma 14. Assume Phase 1lb yields, time T2.
|8. (T2)) + (Tx(T2) S E.
(E.11)
Then there exists constant C >>0 0 and cu > 0 such that, if S ca, then for all 2 0.
18, (T2 7) 5,, (T, + r) < Ce.
(E.12)
Proof. Write 7 =t - Tz. After T2, the full flow satisfies
út) -QQw(t) (85, (t) 125, (t))QQu 12 5t()) Qwi(t.
This is linear time varying system. Its mild form (variation of constants) is
w(T++r) = sQr 2Qr (T2) L' -((((-))) (86., - -12 (T2+
Qw ds. (F.
38

---

Published as a conference paper at ICLR 2026
Set
B(T) := '/' ()((=s) (86.- (1oh-118.))(2ttt 12 Qw ds.
A(T) -QT (T2),
so that w(I2 + T) A(7) + B(T). We can interpret A(T) as the term giving the dynamics in the ideal case, and (T) as a perturbation term.
Step I. (Uniform semigroup bounds). Since Q > 0,
Lo H0 Q =
8 I,
Jo nc
Q1/2c 1 1 Q1/2. 8
Hence for any bounded h and any vector v,
11 e h(s)
1 1 |/ 10112,
(E.14)
8
1 (- 8Q1 (-8Q(r-)) h())
1 | /Q///u112.
(E.15)
Step 2. (Bounding B(r)). Let g(8) : (6. 120 (2 +5) On [7b, To T2 |g())) S 20 G(T) where
G(T)
sup (T6
+t))
ostsr
By (H..4)-(H.15),
[BTT]la S G(T),
[())B((T)l2 S G())
(E.16)
for some absolute constants C1, C2.
Step 3. (Comparing with the ideal trajectory). The "ideal" evolution keeps only the base drift:
ideal (T2 + r) n² + -QT (wr(T2) w²) = == + =BF=e(TT)))
so that
gideal (Te + r) = 1122222 (T2 + +T) 1|2,
w Ideal (T2 + +T) = -(wideal (T2 + T), Qw!)
Because =QT is a contraction.
1 1 s ideal (T2 + +)+ 16, S
|1 - u ideal (T2 + ++] S (Tx) SE.
Step 4. (Deviations of s and u). From w/ (72 + r) = A(T) + B(T).
+ s sT2 + 7) = = = و ideal (Te +) + 2(Q2/22 ideal ideal (T2 + ()) + [[Q"/B((T]13
uTT +T) = (A(r) + ((). Qw") = Ideal
(Th + + (B(r),Qw"),
Hence,
< ||
(B.17)
+T) < [B( 1IQ 1l2 s = c3 G(T),
(E.18)
using (E.16).
Step 5. Combining (E.17)-(E.18) and the definition of G(T) gives
G(r) < 2s + aG(()+++++))))))) + bG())? a == 20g + C32
b : 2.
Fix M 2 3 and set Eo := min ((M - (2a(M (M - 2)/(2)))) If I Eo. define 7* :=
supfr: : G(r) $ Me). Then on [0, 7*),
G(T) $ 2s aME + 6M < Me.
By continuity, 7" = c0, so G(() Ms for all 7. Setting C:= M gives
18. (T2 + T) + 8, (T2 + T) < Ce,, 2.
39

---

Published as a conference paper at ICLR 2026
Let -TT((E) and set
MSE (Tg +)) 1 d
MSE (T2 +T == 1 -
+ 2
Σ e (T2)² e 167.7
i=1
Define Ae(7) (Tu + +))= - eldeal (T2 + 7). Subtracting the full and ideal flows yields, for 7 T20. 0,
Se(T) = / 88((T-)) (T2+5. (c
ideal (T2+s)+ + (86 -1266) (T2+ (1,,)) Qwi ds.
(E.19)
Theorem 5. Assume (2.12) holds. There exist absolute constants A, B,C. C < 3o such that, for all
720 0,
MSE (T2 +T MSE (Te +)/ S AE (MSEideal (T2) MSE (Th (T2 +))
1 d d
B
Σ e1----B)77 (wz)² Ce Ce
(E.20)
where F(T):= suPosast | Vd. In parficular, if 8 S 6o is small enough so that Ca S 2
then the feedback term can absorbed to give
MSEfull (T2+ 7) MSEideal (I2 (I2+ x
A
(MSEideallTy (T2)
(T2) MSE (T2 + T)
Os
(H.21)
B
8² ā 211
( (w)))
(B.22)
Ce
and, e.g., f.g..orr S 1/3, -1-6 (1-Ca)- S 2 (after adjusting C Cf iffdeddd..
Proof. (i) Term with Qeideal, Using cideal (T2 + s) = =e -8Qs e(T2) and semigroup commutation,
l' SQ((-s) 8)( ideal
idiaal (T2 + s) ds = Te 8Q7 Qe(Tz).
Taking inner product with gideal (T2 + = e-Q e eT2)), dividing by d, and using xe" - < - with x = 16ATT,
c ideal (T2 +T), / --8Q(T=s) idiaal ideal
+ (T ds
1
(MSEideal(T)) (T₂) MSEideal (T2 +22+
8
Multiplying by 16,1 S yields the first term in (E.20) with A =
(ii) Teacher-forcing term. Let
T(T) = [' 8Q(+ ())( (86, 125.) (T2 + +) Qw* ds, g(s) = 88 +)) -126, (T2 + 8).
Then [g(s) < 20 and
T((T) Aw 1' g(s)e -Shi(r-o)) ds, S g(s) -8)((s-)) ds 5 20e e -A,T
8A,
Hence
25
(T)
(1 e 8hiT)2 (wf)?
d
which yields the second term in (E.20) with B - e- -8A < 8A,T recover the quadratic behavior
25
The bound is saturating: for small т, may
d Σ 1=1l
d |T(T)
< 400 ,
X (w
40

---

Published as a conference paper at ICLR 2026
whereas for large it saturates at O(E?) y (w
(iit) Self- self-inntractton term. Write
S(r) S(r) = [- 8Q( (-12,,77- ) Q Ae(s) ds.
For each coordinate
8) S((T) -122 //' dGe 11.-88111-5) (T2 (T2 Aei(s) ds.
Using |ö.1 < and the uniform L bound So tse du = Young's inequality gives
|[(())l2
3 sup
|
I
Vd
2
055ST
vd
Thus the self-interaction contributes a linear feedback term Cs F(r) in (E.20) with 3. Ab- sorbing this term to the left yields (E.21) whenever Ce < 1.
0
Corollary 4. Under (E.12) for S Eo small enough,
e² = Ci (MSEideal (T2) ( MMSD-Mda (T2 (T) + C2
( - (T2+T)
d
8
In particular, for each fixed the difference is O(e) relative the drop of the ideal MSE from T2 T2 T, plus a negligible (E=²/d) additive term.
F ADDITIONAL NUMERICAL EXPERIMENTS
F.1 APPROXIMATION (t) = (A, - (t)).
We empirically confirm the approximation used for the Phase III analysis. In Figure 4 compare the MSE of the population dynamic with the one predicted by the approximation (t) ze 8A, wi (f)) (i.e. we froze utt and s(t) at their convergence value) where T2 chosen such that u(22)
0.9.
100
10-1
MSE
= 1.1
1.5
2.0
10-2
3.0.0
Theory
1
104
105
106
Figure 4: Comparison between MSE obtained from the population dynamic, and the approximated one used to derive scaling law (th.) (log-log scale). Parameters: d 300, 7 = 10-3
41

---

Published as a conference paper at ICLR 2026
F.2 ONLINE SGD
In the main text, we focused on the gradient flow to highlight the phase decomposition. Here, we complement the analysis with simulations of online SGD, see Figure 5. We track the trajectories of the key order parameters when training with a small learning rate and without mini-batching. We used the same parameters in the previous section, but used empirical gradient and added noise E, ~ (O, 0.05).
Compared gradient flow, online SGD exhibits the same qualitative phases, but with additional noise. This suggests that our theoretical scaling laws are robust to stochastic perturbations intro- duced by SGD. Moreover, the magnitude of fluctuations increases with the spectral decay parameter Intuitively, when is large, only few top directions dominate the signal, so SGD updates concentrate heavily along these directions, amplifying variance and making the trajectory noisier.
42

---

Published as a conference paper at ICLR 2026
0.8
100
0.7
0.6
0.5
6 10
u(t), s(t)
u(t)
0.4
s(t)
MSE
0.3
MSE
4x 10
0.2
0.1
0.0
2 10
10
10
102
103
10a
10
10
TiT1
(a) 0.5
100
0.8
u(tt), s(t)
0.6
u(t)
-0
MSE
s(t)
0.4
MSE
4 10
0.2
3 10-1
0.0
-
109
10
102
10a
10a
10
10c
T
Tz
(b)
1.0
1.4 10
0.8
1.3x
1.2
ult), s(t)
0.6
u(t)
MSE
s(t)
1.1 10"
0.4
MSE
100
0.2
10
0.0
100
101
102
103
10a
10
10c
T:7
T2
(c)
1.5
Figure 5: Dynamies of online SGD for different
1000, // 10-3
43


Reviews:
[{"review_id": "CXi0iTppYD", "reviewer": "Reviewer_gLFC", "summary": "This paper considers the theory of scaling laws of phase retrieval, a classical nonlinear regression problem. The authors start by asking \"How does the input spectrum govern finite-time convergence in nonlinear regression, and can we predict the learning curve from the spectral decay?\".\n\nTo answer the question, they consider the gradient flow of optimizing the MSE loss, which can be formulated as infinite-dimensional, continuous dynamical systems. The dynamics is further analyzed through different phases in Section 4, including escape, approximate convergence, and spectral-tail learning. In Section 5 the authors provide a sketch of their proof, and then they wrap up the work with some discussions on limitations and future directions.", "questions": "1. Could the authors provide some discussions on the connections between the problem studied and real-world problems, such as how the reuslts would inspire new algorithms design for large-scale real-data problems.\n\n2. Are there any challenges of removing the big-O notations and asymptotics in theorems? For example theorem 1 requires d to be sufficiently large, how about the cases when d does not satisfy this?\n\n3. The analysis is mainly on the gradient flow. Would it be possible to analyze the discrete cases, i.e., gradient descent or even stochastic gradient descent algorithms, which might align better with real-world scenarios.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper gives a solid theoretical study of scaling laws of phase retrieval problems. \n\n2. The authors reveal that the problem can be viewed as infinite-dimensional dynamical systems, and can be analyzed by some math tools such as ODE, dynamical systems, etc.", "weaknesses": "1. It is unclear how the dynamics of gradient flow of the phase retrieval problems can inspire the design of new algorithms for solving real-world, large-scale problems. Although this paper mainly focuses on theory, it might be beneficial to include some discussions on the connections between this toy problem and real-world ones.\n\n2. Most of the results include the big-O notations, meaning that the conclusions can only hold when certain numbers are sufficiently large/small. This might indicate that the theorems and propositions are a bit limited.\n\n3. The mathematical tools and conclusions are standard in theory lliterature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBGxDIKsRU", "reviewer": "Reviewer_KHcf", "summary": "The paper uses phase retrieval as a simple setting to study non- linear dynamics, i.e., learning a single-neuron with quadratic activation in the presence of noise. This is a non-convex problem with two global minimizers, a local max at 0 and a saddle manifold. As a result, the landscape is nice in the sense of no spurious minima, but anisotropy (power-law decay eigenvalues of the covariance Q) makes convergence slow: in particular, the parameter trajectory escapes quickly from low alignment but then converges slowly because of small eigenvalues.", "questions": "- On the necessity of Gaussian data: would the same dynamics hold if the inputs were drawn from a non-Gaussian but rotationally symmetric distribution say? Which parts of the proof genuinely require Gaussianity?\n\n- Can one compute the scaling of the loss explicitly in the tail phase as a function of  $a$?\n\n- How does this multiphase trajectory analysis connect to other problems in machine learning. Do you think you can gain any insight say into training of e.g., deep linear networks from this work and the observed phases of training there?\n\n- Your work focuses on gradient flow under anisotropy which is what preconditioning approaches like natural gradient or Adam try to fix. Have you considered or thought about performing a similar analysis with methods like this to see if they can ameliorate the issues of slow escape from a saddle and slow convergence to a global minimum? In short, this setup seems like a nice testbed for studying these two problems, escaping saddles and slow convergence along flat directions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "I think the key originality this paper brings is the analysis under anisotropy. I also think on the whole that the paper is well written and clearly explained. In terms of significance I think the identification of slow convergence with the tail of the distribution of the data is nice, particularly around the emergence of this third phase not present when $a=0$. I was also think the fact that a larger $a$ implies faster decay of the loss in the first phase but slower convergence in the third phase is interesting and a nice characterization. Finally the fact that for isotropic data you can just track the magnitude and inner product of the weights versus the target weights, but for anisotropic data this no longer holds and you must track an infinite tower of weighted overlaps instead (proposition 3), I also liked. The analysis appears to involve a nice combination of techniques.", "weaknesses": "My only critique perhaps concerns the idealized setting, namely a quadratically activated neuron trained under full batch gradient flow on Gaussian data with power law decay. It is not clear to me how much of the narrative here applies elsewhere to other non-linear optimization problem settings of interest. As a result, I think the scope and general applicability of the results are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vsbUUFMEwJ", "reviewer": "Reviewer_aHGn", "summary": "This paper studies the learning dynamics of gradient descent (GD) in the context of phase retrieval with anisotropic gaussian inputs. The main technical approach is based on reducing the coupled dynamics of an infinite hierarchy of moments to a Volterra integral equation, and then obtaining solution bounds and asymptotics via Laplace-type analysis. \n\nThe main conceptual result, supported by theory and numerical experiments, is that learning roughly exhibits 3 phases: \n- Escape: The weights slowly escape the small gradient regime near \\$w=0\\$ and acquire nontrivial overlap with the ground truth weights. \n- Convergence: The relevant summary statistics converge toward their correct population values while the loss converges toward 0. The authors subdivide this phase into two qualitatively distinct subphases. \n- Spectral tail learning: The authors finally consider a second metric of learning, the MSE of the learned weights. At the beginning of phase 3, the MSE remains high despite low fitting loss. The MSE decreases through phase 3 at a rate that depends on the spectral decay of the data covariance; as one might expect, flatter spectra - i.e. smaller decay exponents - give rise to faster learning in this phase. \n\nAnother conceptual result that is discussed only very briefly is a tradeoff between fast initial learning and fast tail learning. Roughly speaking: low rank data drives fast initial alignment but leads to slow learning of small-eigenvalue directions. \n\nThe bulk of the paper is devoted to describing the analytical results and outlining the proof strategy, though the authors provide empirical support for their results through limited numerical experiments (2 main figures and several supplemental figure panels).", "questions": "**Major comments/questions**\n\n1. Clarify/amplify role of anisotropy\n    - The title and contributions section make reference to a tradeoff in escape and tail learning speeds. This tradeoff is not discussed further in the main text, and is not illustrated in any of the main figures. Given the prominence of this result, I feel it deserves more attention. \n    - It took a careful reading to understand what was being traded off - ie. “escape from mediocrity” and “convergence to low risk” (line 091). These terms ought to be defined clearly up front. This confusion was related to the fact that the paper studies two distinct measures of performance. See comment 2. \n    - In light of the results, I feel that the description of the anisotropic setting as a “completely different picture from the isotropic intuition” (line 087) is too strong. It seems more consistent with the results to say that anisotropy leaves the dynamics qualitatively unchanged while determining the precise scaling exponents and phase transition times. \n2. Up until Section 4.2, it is not obvious on first reading that the paper studies two distinct measures of performance - population loss and MSE. Several points of confusion could be avoided by making this clear early on. I think the MSE should also be defined alongside the population loss in Section 2 - Problem Setup, and that the text should state explicitly that the analysis studies both quantities, which become relevant in different phases of learning. \n3. Escape phase\n    - The concept of an escape phase in phase retrieval is apparently well known (eg. cited paper Ben Arous et al., 2025). I feel this is worth mentioning when describing the escape phase on line 256. \n    - I am somewhat confused by the initialization scheme. In particular, I don’t understand why the authors have $|w^\\star| = O(\\sqrt{d})$ but initialize the weights with $|w|=1$. This leads to issues later on: Remark 1 seems to suggest that the escape phase occurs because the initial gradient is small, in turn because $u(0)\\sim d^{-1/2}$. However, the third term of 3.1 has both $u$ and $w^\\star$, which has norm $d^{1/2}$. It’s not clear that the alignment term of the gradient is actually small relative to others. A very different explanation is given on line 256, which points out that since $w(0)\\approx 0$, the weights start out near the local maximum at $0$, and thus require long times to escape. Please clarify. (To my mind, the simplest initialization has both $w,w^\\star$ with norm $\\sqrt{d}$. Then $u(0)\\approx d^{-1/2}$ implies the alignment term in the gradient is small relative to the first two “norm terms”. Gradient descent quickly sends $s\\to 1/3$, and then one has a slow “alignment phase” wherein $u$ increases toward 1.)\n4. The authors should briefly describe the approximation scheme used in the phase analysis, perhaps around the statement of the ODEs in eqs 3.3-3.5. In particular, the analysis proceeds by assuming simple forms for $s(t),u(t)$, obtaining expressions for the simplified dynamics, and then bounding the approximation error. The initial approximation $\\Theta(t)\\approx t$ corresponds to $s(t)=0$ - ie. initially small weights, while the phase 3 approximation corresponds to $s(t)=u(t)=1$ - ie. perfect convergence of macroscopic overlaps. \n5. The authors may want to comment on the meaning, relevance, and broader applicability of the division of Phase II into two subphases. Is there any broader conclusion to be drawn, or is this merely a peculiarity of phase retrieval?\n\n\n**Minor**\n1. Line 083-084 uses uppercase $T$ for time while later equations use $t$. \n2. Line 323 typo (growth -> grows)\n3. Line 413 / eq. 5.4 - I don’t see a definition for $\\alpha$ anywhere in the main text.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper studies an important issue in learning theory. It considers a nontrivial, currently poorly understood, setting combining model nonlinearity and data anisotropy - both of which are expected to affect the detailed behavior of neural scaling laws. The technical analysis of the paper is interesting and rigorous, and is likely to be applicable in future work. The main theoretical conclusions of the paper are intuitive and appealing, and are supported by limited experimental verification/exploration. Finally, the paper is well written, and the derivations are relatively easy to follow.", "weaknesses": "**Robustness of conclusions and applicability to more general settings**. The paper obtains satisfying conclusions about the qualitative behavior of GD dynamics in phase retrieval. However, the neat 3-phase picture appears to depend somewhat on the loss structure of the specific problem under consideration. Currently, the paper does not appear to give any reason - even hand waving - to suspect that more complex models will exhibit escape, convergence, and/or tail learning phases resembling those of phase retrieval. The extent to which these specific results generalize is therefore unclear. This issue is especially concerning given that the results, while theoretically interesting, are likely to be of limited practical relevance in the specific context of phase retrieval, given the wealth of published fitting algorithms that significantly outperform (S)GD. \n\nRelatedly, I have a question/concern about the dependence of the results on the particular initialization scheme studied in the paper (see detailed questions). \n\n**Impact of anisotropy**. Data anisotropy is the main novel ingredient of the analysis. While this introduces novel and interesting technical challenges for the analysis, it appears to have a relatively simple effect on the learning curves. In particular, I don’t feel convinced that anisotropy “leads to a completely different picture from the isotropic intuition” as stated in the paper. The paper would benefit from more explicit discussion of the impact of anisotropy, especially relative to the escape/convergence tradeoff referenced in the title. (Right now, discussion of this result appears to be limited to two sentences in the Contributions section.) See detailed comments below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6BHEb9wJF", "reviewer": "Reviewer_8Pxo", "summary": "The authors rigorously study and characterize the gradient flow dynamics of the population loss for phase retrieval in a setting where the data are drawn from a normal distribution whose covariance matrix has eigenvalues following a power-law distribution. \nDue to the combined effects of nonlinearity and anisotropy across learning directions, they show that the dynamics are qualitatively different from cases where only one of these aspects is present.\nAnalytically, they find that the dynamics are governed by an infinite hierarchy of coupled ODEs for the order parameters. Using these equations, they are able to describe the entire learning trajectory and rigorously distinguish three distinct phases, in agreement with numerical experiments. In particular, they reveal that the dynamics are generally slower than in the isotropic case, and that the final phase is controlled by the smallest eigenvalues of the covariance matrix, corresponding to the hardest directions to learn. In this final regime,\nthey obtain an analytical scaling law describing how the mean-squared error decays with training time, showing that the decay exponent is determined by the power-law distribution of the covariance spectrum.", "questions": "While the works of Hestness, Kaplan, and Hoffmann are cited as motivation, those scaling laws relate error to number of data, number of parameters and discrete FLOPs. How do you see your dynamical scaling laws connecting conceptually or quantitatively to the empirical ones?\n\nIn ”Escaping mediocrity” (Arnaboldi, Krzakala, Loureiro, Stephan, 2023a), the authors analyze how online SGD benefits from over-parametrization (among other factors) to accelerate escape. Do you expect similar over-parametrization gains to influence the phase structure you observe here, for example, by modifying the escape from mediocrity stage or the later spectral-tail phase?\n\nDo you have insights on the qualitative differences there would be there would be if one studied the dynamics where the student norm (the Frobenius norm) is kept fixed to the same one of the teacher, so that the dynamics only aligns the vector w to w*?\n\nThere is a minor graphical bug in Figure 2 that causes the labels in the x axis to overlap.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The setting studied in this work allows for a complete and rigorous analysis of the gradient flow of the population loss in a case where both a nonlinearity in the regression problem and a power-law distribution of the data are present.\nTo my best knowledge, this is the first rigorous paper to do so. \nThe analytical description is very thorough and matches very well the numerical simulations, allowing one to build a general intuition about the interplay between these two elements.", "weaknesses": "The setting is intentionally specific to enable a rigorous analysis since the authors use both the generalization risk (whose landscape is devoid of spurious minima) and the gradient flow approximation, which does not directly capture discrete-time effects of an actual gradient descent. \nMoreover, while a scaling law is analytically found in the third regime, it is not directly linked to the classical scaling laws that connect generalization error with the number of data points (here assumed infinite in the population loss) and the number of parameters (here equal to the input dimension and also tending to infinity). \nThe technical sections are quite dense, making the paper more difficult to read in that part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)