Beyond Spectra: Eigenvector Overlaps in Loss Geometry
Authors: Gabriel Mel
Venue: ICLR 2026 Poster (2026)
Keywords: hessian
overlap
eigenvector
geometry
ridge regression
noise
free probability
algorithms
CIFAR
high dimensional statistics
generalization
covariate shift
double descent
multiple descent
random matrix theory
Primary area: learning theory
TL;DR: Loss geometry depends not only on train and test Hessian spectra but also on alignment of eigenspaces; we derive a universal fluctuation law, explain covariate shift and multiple descent, and develop scalable estimators for overlaps in large NNs.

Abstract: Local loss geometry in machine learning is inherently a two-operator concept. While a single loss is locally characterized by its Hessian spectrum, practical learning depends on both training and test losses, whose joint geometry is determined not only by their spectra but by the alignment of their eigenspaces. We establish general foundations for this two-loss geometry by deriving a universal local fluctuation law: the expected test-loss increment under small training perturbations is a trace combining train and test spectral data with a precise factor quantifying eigenvector overlap. We further prove a transfer law describing how overlaps transform under noise. As a solvable model, we apply these results to ridge regression under arbitrary covariate shift, where operator-valued free probability yields asymptotically exact overlap decompositions that identify overlaps as the natural quantities for specifying shift, and resolve multiple descent: error peaks are governed by eigenspace misalignment rather than Hessian ill-conditioning alone. We then validate the fluctuation law in multilayer perceptrons, develop scalable estimators for overlap functionals based on subspace iteration and kernel polynomial methods, and apply them to a ResNet-20 trained on CIFAR-10, showing that class imbalance reshapes train–test geometry through induced misalignment. Together, these results establish eigenvector overlaps as the fundamental missing ingredient in local loss geometry, providing both theoretical foundations and practical tools for analyzing generalization in modern neural networks.

Published as a conference paper at ICLR 2026
BEYOND SPECTRA: EIGENVECTOR OVERLAPS IN Loss GEOMETRY
Gabriel C. Mell.2
1 Centre de Recerca Matemâtica, Bellaterra (Barcelona), Spain
2 CRG (Barcelona Collaboratorium for Modelling and Predictive Biology), Dr. Aiguader 88, Barcelona 08003, Spain me ldee & gmail. com
ABSTRACT
Local loss geometry in machine learning is inherently two-operator concept. While a single loss is locally characterized by its Hessian spectrum, practical learning depends on both training and test losses, whose joint geometry is de- termined not only by their spectra but by the alignment of their eigenspaces. We establish general foundations for this two-loss geometry by deriving a universal lo- cal fluctuation law: the expected test-loss increment under small training perturba- tions is trace combining train and test spectral data with a precise factor quantify- ing eigenvector overlap. We further prove transfer law describing how overlaps transform under noise. As solvable model, we apply these results to ridge regres- sion under arbitrary covariate shift, where operator-valued free probability yields asymptotically exact overlap decompositions that identify overlaps as the natural quantities for specifying shift, and resolve multiple descent: error peaks are gov- erned by eigenspace misalignment rather than Hessian ill-conditioning alone. We then validate the fluctuation law in multilayer perceptrons, develop scalable esti- mators for overlap functionals based on subspace iteration and kernel polynomial methods, and apply them to a ResNet-20 trained on CIFAR- 10, showing that class imbalance reshapes train-test geometry through induced misalignment. Together, these results establish eigenvector overlaps as the fundamental missing ingredient in local loss geometry, providing both theoretical foundations and practical tools for analyzing generalization in modern neural networks.
INTRODUCTION
Modern learning algorithms are inherently local, and sources randomness (stochastic gradients, maite-sample variability, and distributional drift) are often small relative the underlying signal. A local quadratic approximation to the loss thus provides natural setting for analyzing learning. When the focus is single loss, local geometry is fully captured by the Hessian spectrum. Crucially, however, machine learning involves (at least) two losses train and test-and so local loss geometry involves two quadratic approximations. Their joint geometry is not captured by Hessian spectra alone; it requires critical additional ingredient: eigenvector alignment, or overlaps.
Despite the fundamental importance of eigenvector overlaps, most studies date have centered on Hessian eigenvalue distributions-ofter explicitly equating spectra with loss geometry. The litera- ture is extensive and examines Hessians from several complementary angles, including: (i) empirical measurement of cigenvalue distributions and their training time evolution, with links to optimization
stability (Sagun et al., 2017 Ghorbani et al.
Yao et al., 2019); (ii) random- random-matrix- wwt.---ttttitttteeee and mean field models (Pennington & Bahri 2017 Pennington & Worah 2018 2019 Liao & Ma-
2019
honey |021; Karakida et al.] 2019); (ii) class- and layer-structured spectral phenomena, such as identifiable outliers tied to data and architecture (Papyan 2020 Sankar et al. 202I); and (iv) Hes- sian/Fisher analyses relating sharpness (as measured by eigenvalue magnitude) to stability and gen- eralization (Keskar et al. 2017; Cohen et al. 2021 Yao et al., 2019). These studies give fundamen- tal insight into aspects of loss curvature, but ignore directional information that becomes relevant as compares two operators.

---

Published as a conference paper at ICLR 2026
The need to go beyond spectra is well understood in random matrix theory, where eigenvector con- sistency and overlaps with population directions are central objects in spiked models and correlated
ensembles (Johnstone] [00T,; Paul 2007| 1 2008: Benaych-Georgee & Nadakuditi] 2011 Bun et al. 2017] Landau et al. (023). There, eigenvalues alone do not determine statistical performance; rather, risk depends on how sample and population eigenvectors align. Related phenomena have also been observed in machine learning, where eigenspace overlap has been used to characterize shared Hessian structure across independently trained networks (Wu et al. 2022) and to predict compressed embedding performance (May et al. 2019). Here, we adapt this principle to learning- theoretic questions by analyzing train-test alignment. We show that overlap measures between the training fluctuations covariance (intimately related to the training Hessian; see Results) and the test Hessian yield a decomposition of generalization error into spectral and alignment components.
Applying this perspective to ridge regression resolves the puzzle of anisotropy -inducedd multiple de- scent (see Chen et al. (202T): [i & Wei Mel & Ganguli
(2nggll)(2021)))) (202T): Meng et al. (2023) for several distinct forms of multiple descent). Existing high-dimensional analyses connect interpolation peaks to eigenvalue distribution of the design matrix (Singh et al. 2022 Chen & Mei 2022). J. However, in
anisotropic settings where error exhibits multiple peaks despite monotonically decreasing minimum training eigenvalue, spectra alone do not explain generalization. Making overlaps explicit, we show that the appearance of multiple peaks is governed by the alignment between train and test Hessian eigenspaces. This corrects interpretations that implicitly attribute sample-wise multiple descent to Mcl spectrum ill-conditioning (Chen & Mei 2022 Mel & Ganguli [02T] Mel & Pennington| (2022), ) and
yields a simple geometric picture that may prove useful for understanding more complex models.
A second arena where eigenvector orientation is essential generalization under domain shift. In covariate shift settings, high-dimensional risk formulas in random feature models (Tripuraneni et al.) 202T) can be naturally expressed in terms of train/test covariance spectra and their overlaps. More broadly, many domain generalization methods encourage cross-domain invariance by aligning gra- dients, Fisher information, or Hessian statistics across domains (Rame et al., 2022 Hemati et al.) 2023 Le & (024), or by imposing structural constraints such as elliptic regularization (Hasan ct al. 2025). These approaches seek robustness across many possible unseen domains. Our perspec- live is complementary: rather than enforcing invariance, we derive explicit formulas for performance on a specific target domain. In particular, under covariate shift we show that, even with covariance spectra held fixed, varying overlap structure alone can increase or decrease test risk. Overlaps there- fore provide natural quantitative measure of the shift itself, predicting when a given domain change will help or hurt in a way that spectrum-only or aonain-agnostic analyses cannot.
Loss curvature plays an important role in both classical and modern analyses of generalization. Classical asymptotic corrections such as the Takeuchi Information Criterion (TIC) express the gen- cralization gap in terms of the local curvature of the population loss (see, eg., Thomas et al. (2020)) A second line of work uses curvature information at training time: sharpness-awarr and curvature- regularized methods- innluuding SAM (Foret et al. 2021) and its Fisher- and curvature- reguuarieee variants (Kim et al.) [022]] Wu et al., (024) bias optimization toward flatter regions of the training loss, motivated by the heuristic that such regions generalize better. Both perspectives are funda- mentally single-loss. By contrast, our framework is explicitly two-loss: we do not assume any relationship between train and test losses. Given a training loss and perturbation model, we char- acterize test coltormmancc-ttnttiaall on a distinct domain torouhhthroough the joint spectra of the two Hessians and their eigenvector overlap. The TIC emerges as a limiting single-loss case, but our theory reveals how alignment governs generalization beyond spectrum- criteria.
Translating overlap theory into practice at modern scale requires algorithms that go beyond spectral density estimation. A substantial literature has developed linear algebraic tools for im- plicit matrices, including volynomial/quadrature approaches and stochastic trace methods such as Hutchinson and Lanczos-basee, quadrature (Golub & Meurant 2009: [in et al., 20166] Ubaru et al.) 2017). These and related techniques have been adapted to deep learning estimate Hessian spec- tral densities and extremal cigenpairs efficiently (Adams et al. 2018 Papyan] 2019 Ghorbani et al.)
2019 Yao et al. 2019). Building on these, we develop novel estimators for overlap functionals between pairs of Hessians (train-test, population-sample and use them to study class imbalance effects on train test geometry.
2

---

Published as a conference paper at ICLR 2026
The resulting picture is that local geometry in machine learning is fundamentally bivariate: spectra determine the curvatures of train and test losses, while eigenvector overlaps determine how these curvatures interact to produce test error.
2 CONTRIBUTIONS
1. Two-loss theory of local geometry. We introduce a novel two-loss framework for lo- cal loss geometry that incorporates both spectra and overlaps 3.1 rectifying a widespread oversimplification that equates spectra with geometry.
2. General foundations. We derive and test a universal local fluctuation law showing how overlaps impact generalization 3.1.1 and general transfer law dictating how eigenvector overlaps are transformed by noise
3. Explicit formulas for nigh-dimensional ridge regression. Combining tools from random matrix theory with our overlap transfer law, we provide closed-form expressions for the overlap function between train and test Hessians anisotropic ridge regression 3.22
4. Unified explanation of covariate shift and multiple descent. We show that covariate shift is naturally quantified by eigenvector overlaps 3.2.I and that overlaps analytically resolve the puzzle of multiple descent
5. Empirical validation in neural networks, We confirm our theoretical predictions in mul- tilayer perceptrons, and use overlap machinery to show that the training Hessian acts filter shaping optimization 3.3
6. Scalable algorithms for Hessian overlaps. We develop novel, scalable numerical methods for estimating Hessian eigenvector overlaps in large-scale models, enabling practical use our theory in modern deep learning 3.4
7. Train-test misalignment under class imbalance. We show that class imbalance in CI- FAR induces misalignment between train and test Hessians, explaining the effects of class imbalance in terms of train-test loss geometry. 3.4
RESULTS
3.1 THEORETICAL FOUNDATIONS
Prior theoretical and empirical work frequently uses geometric descriptors the loss landscape- such "sharp" versus "flat" minima or valley structures-yet the relationship between these geo- metric notions and generalization remains imprecise. We begin by establishing general foundations for two-loss geometry to formalize this connection, and then derive fluctuation law that character- izes how perturbations to the training loss propagate to changes in the test loss.
Let I denote the d-dimensional parameter vector of model fws and let Cussin (w, (), Crest ( (w) be the swwice-differentiable train and test loss functions. The train loss Ltrain is parameterized by small variable representing general training perturbation. We remain agnostic about the source of the perturbation, which could be any combination of label/input noise, distributional drift, sampling effects, etc. Throughout, we write wi for the minimum the unperturbed loss Etrain (w, 0).
By analogy with the one-loss case, we refer to the pair of quadratic approximations obtained by second order expansion of Lirain; Liest around a point as the local Iwo-loss geometry. Concretely, define the perturbation gradient, and train and test Hessians follows:
Lwrain (Wo, e), Huain == d Ctrain (Wo, e), Hust = d V Crest(W) (1)
(note the scalings, chosen for convenience) and introduce the quadratic surrogate losses:
equad (w) = Lissin (wo, E) + f Aw +
2d AW Htrain
(2)
Lquad (w) Liest (wo) + 1 Stest Aw + 2d
Aw T Hrest Aw,
(3)
where 2test V Ctest (wu) is the normalized test gradient and Aw define the unperturbed test loss and test loss increment as follows:
wo- Finally,
Lo Lquad (wo);
AL
Lquad (wo Aw) Lquad (wo)-
(4)

---

Published as a conference paper at ICLR 2026
3.1.1 LOSS FLUCTUATIONS ARE GOVERNED BY EIGENVECTON OVERLAPS
Generically, the effect of perturbation is induce small gradient Z at the unperturbed mini- mum Wp. yielding a new minimum of Cquad a displacement Au. Directly minimizing ( gives the perturbation-induce displacement Aw gradient as the injected noise and to Aw as the (inverse-Flessian) filtered noise. Substituting the displacement into Lestt yields the following expression for the test loss increment.
- 2. We sometimes refer to the perturbation
AC = - - Pesst Hf
H train Heet 11 train 2.
(5)
Equation represents the simplest model capturing the interaction of nontrivial train and test ge- ometry in the context of noisy learning. The first-order effect is structurally simple-and, in sev- cral natural cases (c.g., label noise under MSE, analyzed below), vanishes exactly in expectation. The second order term, in contrast, involves interaction between train and test curvatures: letting Ctrain == E[AwAww " = E((H (Hrr = be the displacement covariance, its expectation 2d tr test C-raiu-- This simple trace expression suggests the importance of alignment between directions of large training displacement and directions of large test Hessian eigenvalue. One of the main theoretical contributions of this work is the following theorem making this intuition precise.
1
Theorem 1 (Overlap local fluctuation law). Let Prest Purain be the spectral measures of Heest Crrains and define {0(,,, A2) as the mean squared cosine angle between eigenvectors of Heests Ctrain at eigenvalues X1, A2. Assume E[Aw) Then
B|A //
A A2 O(A1, A2) Preat (dx1) Merain (d)2).
(6)
Equation (6) shows that neither train nor test spectra alone predict the expected generalization im- pact of noise: the decisive quantity is how graiing-inducee displacement directions route into test- sensitive directions via the overlap kernel O()1, A2). In particular, large test error arises when when high-variance displacement directions (large X2, corresponding to low-curvature train directions) substantially overlap high-curvature test directions (large A1).
Proof sketch. Letting (Atrstt .,(sst,, (A _uj urain) be the cigenvalues/cigenvectoss of Huest Ctrains
MrrlHeestCraann
A test xtrain [d (ujest u., (rainnn?]
1=
Writing the double sum as an integral over the spectral measures of Heest, Cerain yields (6). Sce Appendix B.2 for details.
While we do not treat stochastic optimization explicitly, in the same local quadratic regime, noisy gradient descent yields curvature- filtered steady-state covariance that, when substituted for Ctrains yields the same overlap fluctuation law (see Appendix B.2.2).
3.1.2 OVERLAP TRANSFER LAW
In many situations one must consider the overlaps between an operator and a noisy transforma- tion of another operator B, written B. For example, below in the context of ridge regression with anisotropic gaussian inputs, we consider the case that A, B correspond to the population test and train covariances, while / is the sample train covariance. More generally, A, B could represent the population test and train Hessians, and B the empirical train Hessian. In such cases, needs way of combining the population overlaps OA, with the noise, specified by O OB,6- We prove the following appealing transfer law in Appendix B.3
Theorem 2 (Free transfer law for overlap functions). Let B = F(B,X) be matrix rational expression. If X is free from A, B, then
O b,, = O,, B. (b, pn (db).
(8)
4

---

Published as a conference paper at ICLR 2026
(Freeness is notion of independence that is suited to large random matrices and holds asymptot- ically for a wide range of common random matrix models; see Appendix [B.)) Theorem 2 entails
(opendix(B.3)
simple overlap calculus that can be used to compute overlap functions of complex matrix models
from simpler ones. In Appendix C we use (8) to quickly derive expressions for train-test Hessian overlap functions in anisotropic ridge regression.
3.2 HESSIAN OVERLAPS GOVERN GENERALIZATION IN LINEAR REGRESSION
We now consider ridge regression, where the preceding theory is exact. Let training inputs æ E RRd have covariance Etrain == E Tx ]. and assume linear output with Gaussian label noise:
v(z) = J my x + &. E &~ *~,
We will also assume for convenience that w² ~ (),, so that the signal to noise ratio is fr Errain/ (tr Enrain + σ²) (ir denotes the dimension normalized trace). Given training set con- sisting of X E mRd (rows x: and labels E IR. ridge regression chooses w E Rd to minimize
Lirain(w) =
zे ||
X2 0.
2m
We write a == m/d for the sampling ratio. The (excess) test loss is measured with test inputs with (possibly different) covariance Euett:
2d (w w Etest (w We). With the scalings of 3.1 one has Herain Eerain + AI and Htest Etest, where Etrain == X X/m is the training set sample covariance. Note Etrain Crrain for large m//d
Liest (w) :- Bz,g (Ja" x === 00²²"
We now apply the local fluctuation formula (6). which the setting of ridge regression is exact. Our goal is not to re-derive known high-dimensional risk formulas, but to highlight their two-operator structure: test eтror decomposes into (i) train/test spectral scales and (ii) an explicit eigenspace- alignment kernel. This makes covariate shift and multiple descent analyzable as overlap phenom- ena. Letting the label noise supply the perturbation, the injected noise 2 == dV wtraii {) and (H
displacement covariance Ctrain == E [ train (H (((rrdnn))) [ are
2 = vd xTg,
Cerain = = Etrain "Erain(Strain + XT)-
(10)
where a := m/d :=m/d is the sampling ratio. The test loss increment AL obtained by substituting into the overlap formula (6). The training-side operators Strain Hirain = Etrain + Al and Curan commute and share eigenvectors, so for simplicity all formulas are written in terms of Eerain
x² E [AC] = Za ~
A2 1)2 O Ozioa Strato (A1, Prrrn (dA1) P (dA2);
(11)
where M.e. and M are the empirical spectral measures, and O Dxger Sran A2) is the eigenvector-overlap function (see (88). Since we will be interested primarily in the ridgeless limit A + 0. we will loosely refer to Etrain as the train Hessian. See Appendix for detailed derivations.
The fundamental conelusion from (II) that we will apply toward analyzing covariate shift and mul- tiple descent is that error is large when training perturbations induce large variance (small training eigenvalue X₂) in directions that align strongly (large O(),, )2)) with directions of large test loss sensitivity (large test eigenvalue A1).
In Appendix C using techniques from operator-valued free probability we derive asymptotically exact expressions for Lust. AL and the overlap function OHuain.HHes in proportional asymptotics where m. d oo with == m/d fixed. The main conceptual contribution of this work is that while the spectral densities of train/test operators set the relevant scales, it is their relative orientation- as quantified by the overlap function- determines how displacements translate into test loss. We illustrate these points in two settings: first, simple covariate shift experiment that provides geometric intuition and positions Or E.rai as the natural object quantifying shift; second, the puzzle of multiple descent (cf. Mell & Ganguli (2021)), where the overlap function allows a full analytical account. For clarity, in both settings we use the simplest possible model of anisotropic data: the "two-scale" covariance with spectral measure
a == p1 p2d.3.
(12)
5

---

Published as a conference paper at ICLR 2026
a
Parameter discplacements
Fixed Hessian spectra
Overlaps and loss
1.5
Hyah 2
Bw (shert)
Overlay
0.0
long/short
long/long
= n/4
Loss
Z
Zyst
e m/2
0.0
-1.5
0.0 Aw (long)
1.5
2-1
2
2
7t/
m/2
eigenvalue
0
Figure 1: Isospectral shifts isolate overlap effects in covariate shift. (a) Swo-dimensional slice of parameter space. Points: individual parameter displacements due to label noise. Lines show train (cyan) and test (red) Hessians with precisely controlled alignment 8. (b) Eigenspaces while Hessian spectra are fixed. Blue line shows d, m - oo theory. (c) For small 8, large dis- placements in learned parameters (aligned ellipses at top left, and purple overlap lines), and error is small (black line). For large 0, large displacement directions are aligned with sensitive directions of the test loss and test error is large (black line). Lines: theory; crosses: average from simulations. d, a, X, 102 10, 10-4 10-1/2.
rotated
aligned with the low-eigenvalue directions of the test Hessian
3.2.1 COVARIATE SHIFT INCREASES LOSS THROUGH TRAIN-TEST MISALIGNMENT
Equation 11 expresses A in terms of the overlap function O Ereat Eteain Relative to the population overlap Or this overlap is deformed by the finite sampling ratio of the training set (cf. transfer law of Theorem ( In Appendix C.3 we use the transfer law to state an explicit formula for O DEmcE. SErain" and then prove the following:
Theorem 3. As m, d + do with o fixed, the asymptotic test loss increment satisfies
d² dλ
/f
A2 Iznm, Drrai (21, MEre (d2z) MSreain (dA2),
E|CC
(13)
20 dd
(212 + X)2
where À is the effective regularization defined by the self-consistent equation:
r(z) (1-1 2 tr(z)
1. dis (1)
(14)
r(
Equation 133 parallels 11 but averages out all training randomness to express AL purely in terms of the population operators Etrain, Etsst Most importantly, (13) illustrates how O Enain aa the only quantity that can change under isospectral transformations to Etrain, Etest captures bona fide two-loss geometric effects that are invisible from either loss geometry in isolation.
To illustrate this point, we perform a simple experiment where both Etrain Etest have fixed two- level spectra (12) with scales s 2 = 2 20,2- and equal multiplicities. X = 10- and m/d 10 that Htrain Derain Eraain while Hust Etstt. Fig. [ shows the distribution of learned parameter displacements for different label noise realizations. As predicted, displacements have larger variance along long directions of Ctrain o² E /a, corresponding low-curvature directions of the train Hessian. At the same time, the test loss contours are determined by the test Hessian Eust- We construct a controlled perturbation in which Etett systematically rotated with respect to Etrain while all spectra are kept fixed (b), isolating the effect of overlaps. Fig. [c) demonstrates the consequence varying overlap. When the long directions of Strain align with the long directions of Etest (0 0), displacements occur in directions where the test error is relatively flat, yielding low excess test loss (Fig. ()) left column). In contrast, when the same train-long directions align with test-short directions (# π/ 2/2), the same magnitude of parameter displacement heavily penalized, and the test loss rises sharply (Fig. [[c), right column). This simple experiment illustrates the central role of eigenvector overlaps in the context of covariate shift.

---

Published as a conference paper at ICLR 2026
a
b
C
d
2-scale data
test
scales 4 scales
Spectrum and overlaps
0.4
Aoisi.Aal
1
0.2
p(ay)
10a
10 10-1
10-3 10-7
10a 10f
10-* =s]
1=s]
Crest. AC, Co
Crest. AC, Co
A2
Figure 2: Multiple descent in ridge regression. (a) Loss as function of the measurement den- sity a = =m/d for two-scale data. Note the peaks at critical values of a = 1/2, 1. Solid lines: theory: crosses: simulations with d 5000. Dashed, dotted, and dash-dotted lines show theory Creat AC, Co in the limit that the lower scale $2 + O, where bumps become true singularities. (b) Theory Lust A). Gold and blue lines indicate slices shown in panel (a). (c) 3 and 4 scale data which exhibit 3 and 4 peaks; legend same as (a). (d) Green histogram: empirical spectral density of the train Hessian X X/m function, O /2, giving overlap between a train eigenspace at eigenvalue 22 with the entire large-eigenvalue test space (ie. st). Note strong overlap for high train/test eigenspaces.
0.496; solid green line: theory. Purple line: overlap
3.2.2 MULTIPLE DESCENT IS EXPLAINED BY TRAIN-TEST OVERLAPS
Double descent well-established phenomenon in machine learning which test error exhibits a non-monotonic dependence on model size. More recently, several authors have described an ex- tension of this effect, termed multiple descent, which arises in settings where input data are highly anisotropic and the covariance spectrum contains multiple separated scales (see introduction). Fig. 2 b) illustrate multiple descent for two-scale data with $1 $2 1, 10-2, while panel shows how larger number of separated scales can create additional peaks in test error (see caption for details).
For two-level covariance, the overlap function determined by the solution to cubic polynomial that is easily solved numerically (Appendix C Fig. 2 d) shows the spectrum of the train Hessian (green histogram and theory line), and the overlap function (8], indicating overlap of a training eigenspace eigenvalue X2 with the large-eigenvalue (of) eigenspace of the test Hessian. Theoreti- cal and empirical overlaps are in excellent agreement (purple line and crosses).
The peaks of multiple descent are easily understood in terms of eigenvector overlaps. Fig. reports the error, training spectrum, and overlap map for the two-scale covariance model of Fig. The test- loss curve shows singularities at critical sampling densities the training spectrum undergoes phase transitions: at a = 1/2 an initially unimodal density splits into two bands centered near of and sÿ. and (Fig. Bb).. The corresponding overlap map O(),, A2) is approximately block-diagonal: modes near sf align predominantly with the of test subspace, and modes near si with the sy subspace (Fig. B(c)). Thus, the first eтror spike occurs when near-null training directions overlap the sharp test subspace, whereas the second arises when an even smaller training component overlaps the flat subspace but with variance large enough to dominate its small curvature. Fig. 3 provides a geometric schematic of the alignment of top and bottom eigenspaces of Hirain; Heest throughout this sequence. Until line 5, the minimum eigenvalue of Hirain always decreases as function of a-which, according to a spectrum-only analysis, should increase test error. Yet the error actually decreases between horizontal lines 3 and 4, precisely because the lowest train eigenspaces begin to overlap predominately with the low test eigenspace.
m/d /d (a). At the same densities
1 the lower să band develops a near-zero component
Summarizing, multiple descent arises from the interplay of (i) training components developing near- zero eigenvalues as a varies, and (ii) which test directions these overlap with-shar -sarp or flat, illus- trating the potentially extreme impact of (mis-)aligned train and test loss geometry.
7

---

Published as a conference paper at ICLR 2026
a
b
d
Htrain spectrum
Marain Heest overlaps
Hues
Hiraio
5
mid
4
training overlppe test space...
3
2
1
10a
10
8
10 53
1 = s?
8
10
5
1 = s?
AC
elgenvalue Àz
elgenvalue Ag
Figure 3: Multiple descent is explained by train-test Hessian overlaps. (a) AC due to label noise
(A, sī + 0 limits taken to illustrate true singularities; see Appendix [c5.1 for details). (b) Spectral density of Htrain as function of a (each density normalized to maximum of 1 and quantized). For high sampling density a, the training density acquires two components roughly centered on the true underlying data scales st s2. (c) The overlap function of Irain Hest as a function of a. For small a, all nonzero Htrain spaces overlap strongly with the large eigenspace of Heest- As a approaches the first critical value, with the small eigenspace of H1est. (d) Minimal model of train-test Hessian geometry. Cyan plane and ellipses represent the top and bottom eigenspaces of Htrain- Red ellipsoid represents level sets of test error. Error is controlled by both train variance magnitude and overlap onto test spaces.
spectral component appears, whose eigenspaces overlap almost entirely
3.3 LOCAL THEORY PREDICTS MLP GENERALIZATION AND LEARNING DYNAMICS
To test the quantitative predictions of the quadratic two-loss theory in controlled nonconvex setting, we trained small, constant width multilayer perceptrons (MLPs) to reproduce the responses of an MLP teacher network. Student networks were batch trained for a large number of iterations to ensure near convergence to the noiseless training loss minimum. Noise was then added to the training set and the network was trained further prather-beginnigg from the initial trained state to determine the effect of the noise on the initial local minimum. After training, the training loss increment was computed and compared to prediction of the local quadratic theory. Fig. a[abb) show the measured test loss increment against the local quadratic prediction for several orders of magnitude of input (a) and label (b) noise strength. All later panels refer the noise setting corresponding the red point in (b).
Fig. 4Tc) illustrates inverse Hessian filtering due to training dynamics. The gradient noise in- duced by the label noise has covariance E[22 T]. The purple scatter represents the overlap func- tion of E[zz J and Hirsin- Dot x, y position is given by Htrrinn E[zz eigenvalue and size is proportional to overlap. Note that the gradient noise and train Hessian are strongly aligned. After training, the parameter displacement covariance predicted by quadratic approximation is
Curain := E [ train [[HHrrinn) (H =) The overlap function of Htrain and the actual post-training covariance is plotted in red. Note how in accordance with quadratic predictions, variance is strongly inflated/attenuated along low/high eigendirections of Herain phenomenon we refer to as inverse Hessian filtering. The large displacements
not translate into large test error since the train and test Hessians are well aligned (Fig, (6), meaning displacements occur primarily along low test Hessian (oss-insensitive) directions.
To provide geometric intuition, loss landscape slices are shown in Fig. 4 for Lirain (w, 0), Ltrain(w (W, c). and Lrest A single 2D slice was chosen to contain the unperturbed minimum wo (white crosses), perturbed minimum (white stars), and parameters predicted by the local quadratic approximation (white "Y"s). Local geometry also predicts local gradient descent dynamics well (Appendix E. Fig. 7 Together, these results validate the predictions of the too-loss local theory.
8

---

Published as a conference paper at ICLR 2026
a
b
C
Input noise
Label noise
Eigenvector overlaps
10a
10 2
Moasured frec (Ac]
Other eigenvalue
2
10-1
10-1
Injected Filtered
10
10-3
10-3
10a
10
10-3
10a
10
10
10a
d
Predicted frac (ac]
Predicted [Ac]
Moon eigerie
Ctrain (W. 0)
Ltaain (W, E)
(west())
Figure 4: Validation of local fluctuation law in MLPs. Layer widths for both student and teacher were (5,5,5,1); nonlinearity: tanh; teacher network has gaussian weights with scale 4/v for each layer; loss: MSE with & parameter À = 1. (a.b) Predicted vs measured perturbation- induced &C/Co for increasing imput (a) or label (b) noise amplitude a. (c) Eigenvector overlap function between Htrain and the injected noise B[zs ] (purple), and post-learning filtered noise Cirain (red). Note how learning inflates/dampens variance along the low/high Heest eigenspaces. (d) 2-dimensional loss landscapes for 1 example simulation: noiseless Ltrain (left), perturbed Liraiu (middle), and Lrsst (right). Cross: noiseless training minimum; star: minimum of perturbed Lraii (ie. the new learned minimum); tri-star: parameters predicted by quadratic theory.
3.4 CALCULATION OF OVERLAP FUNCTIONS FOR LARGE SCALE NETWORKS
Applying our theory modern networks requires estimating the overlap function between the train- ing and test operators. These operators have dimension equal to the number of parameters -oteen millions to billions-so any approach that forms them explicitly is infeasible.
Here we give a brief overview of our approach, deferring details to Appendix F. We apply two sepa- rate algorithms, one for computing overlaps among outlier eigenspaces and another for the remain- ing "bulk" spaces. Outlier eigenvectors
straightforward to obtain using subspace iteration (Ap- pendix F22 cf. Papyan (2019); overlaps can then be computed directly. For the bulk eigenspaces, generalize well-known approach to spectral density estimation known the kernel polynomial method (KPM; Algorithm in Appendix (.33).
Given self-adjoint matrices A, B E Rd and arbitrary smoothing kernels G (z; o)) of width o, the smoothed total eigenvector overlap of A, B at eigenvalues X1, X2 can be written
d
(G GB..dll = d²
Σ 1,1=1
(AB.) -2200) (BAA))
(15)
where GA.A, = (A- Xy o)) and similarly for GB,>2- To obtain the normalized overlap function
treated above, one simply divides by the (G-smoothed) spectral densities of A, B at X1, Az-
Computing the trace in (15) prohibitively expensive for large A, B, and so we resort to Hutchinson trace estimation, which approximates tr [X] with the average of v Xx for several samples of v ~ NO 1). To ensure the trace is positive, instead of approximating (15), we use
tr (. GR,A,) ir A. GR,AA A E. |G// B.
G1/2 vll2.
(16)

---

Published as a conference paper at ICLR 2026
b
10 +
60
Hoac: eigenvalue
10 10
200
400
250
500
P
Hive eigenvalue
Floe eigenvalue
Figure 5: Overlap function for Hirain Hrests and class imbalanced Hessian Hiest for ResNet-20 trained on CIFAR-10. Cyan/red data: spectra of Haraiu Hrest- Dashed lines indicate outlier eigen- values. Purple seatters show overlap between each pair of eigenspaces/bulk spaces. Size and color reflect overlap magnitude. (a) Htrain test overlaps. Note strong alignment indicated by large over- laps along the diagonal. (b) Htrain Hiest overlaps. A large fraction of each Hessian's 's outlier energy lost in low-outlier and bulk spaces of the other, indicating poor alignment.
The KPM proceeds by taking the smoothing kernel G( o) to be gaussian of width a, and then approximates G1/2 A.A, GBAx using truncated Chebyshev series. (Kernel width and approximation degree K are chosen so that the truncated series sufficiently dampens the large-multiplictn near-O eigenspaces; see Appendix
1/2
F33 Thus (16) can be evaluated in terms of the vectors T; (B) T, (A) v, where T& is the ktth Chebyshev polynomial. These vectors in turn can be generated efficiently via Chebyshev recurrences using only matrix- products (see Appendix F for detailed treatment and application to synthetic data).
We simple controlled experiment demonstrate the scalability of our Hessian-overlap algo- rithms on a modern network and to illustrate how a common form of domain shift-class imbalance in the test set-produces clear change in two-loss geometry. We used a publicly available CIFAR- 10 pretrained ResNet-22 checkpoint from Chen 1 test accuracy: 92.6%). The train Hessian was estimated from 5000 examples and fixed throughout the experiment. Two-loss geometry was then compared between two scenarios: a class- balanced test Hessian estimated from 5000 randomly se- lected test images, and class- itass--mbllanneed Hessian from images with class labels 0, and 2. Spectra, estimated using subspace iteration and the Lanczos algorithm, are shown in Fig. 5)a) (train in cyan; test in red). Non-outlier eigenspaces were grouped into single bulk space for clarity. The strong alignment observed between the train and balanced test Hessians largely disappears when the test set is made unbalanced (purple scatters; bulk overlaps, omitted for space, exhibit similar pattern; Fig. 10 All ceessian-vector products were computed using standard PyTorch autograd on commodity hardware, with total runtime of a few hours. Runtimes are essentially linear in the model size and number of examples, underscoring the scalability of our method (see complexity analysis in F3
DISCUSSION
We show how, within two-loss geometric framework, overlaps occupy central role linking op- timization geometry, random matrix theory, and practical machine learning phenomena. We derive novel theoretical tools for computing overlaps, illustrate through several examples how spectra set curvatures, while eigenvector overlaps route variance into error- unif--ni covariate shift and multi- ple descent-and develop scalable estimators for overlap analysis in large models. A natural appli- cation of two-loss geometry is as diagnostic tool for explaining why some domain shifts are more harmful than others. Promising future directions include tracking Hessian overlaps through train- ing time, and alignment-awan optimization that attempts to improve generalization by encouraging strong eigenvector alignment between, eg., train and validation Hessians.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ryan P. Adams, Jeffrey Pennington, Matthew J. Johnson, Jamie Smith, Yaniv Ovadia, Brian Patton, and James Saunderson. Estimating the spectral density of large implicit matrices, 2018. URL
https://arxiv.ct wwwpaxiiiv..gg/ao... abs/ 110./88020000 03451
Florent Benaych-Georges and Raj Rao Nadakuditi. The cigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Advances in Mathematics, 227(1)4444-21,, 2011. ISSN 0001-8708. doi: /tt.:///woo.////oo/////ooo/o.//0
20000000..0..00.0000..00.....0.. URL https::/
sciencedirect com/ ott.a///c......t........oooaoaaa
John P Boyd. Chebyshev & Fourier Spectral Methods. Lecture Notes in Engineering, Springer,
Berlin, Germany, September 1989.
Joël Bun, Pran-Philippe Bouchaud, and Mare Potters. Cleaning large correlation matrices: Tools from random matrix theory. Physics Reports, doi http:///...........o..o./o//1/// p016/j.p httpysrepp201110.00000 10.005. URL URL httptttt////t
January 2017. ISSN 0370-1573. doi: 10.
10.005
Lin Chen and Song Mei. Spectral multiplicity entails sample-wise multiple descent, 2022. URL
https://openrevieww hww.:/ee//oorrr http:/ffrrrmm2dddg228888YYY
Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve, 2021. URL
Yaofo
Chen.
Pytorch
cifar models.
httpss//giihabb.cmm//eeeeeetttt
pytorch-cr ci far-maarreee model
Accessed: 2025-5-17.
Jeremy
Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021. URL http://///ppeeeeeeeeeeeeeeeeuuue net
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur, Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2021. URL
net forum ?id 6Tm1mpos
Behrooz Ghorbani. Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2232-2222222222 2241. PMLR, 09-15 Jun 2019. URL https: stttps://proceedings. wtt.:ssss..ppress//ee perres//9//ghorteee
9b. htm
Gene H Golub and Gerard Meurant. Matrices, moments and quadrature with applications. Princeton Series Applied Mathematics. Princeton University Press, Princeton, NJ, December 2009.
Ali Hasan, Haoming Yang. Yuting Ng. and Vahid Tarokh. Elliptic loss regularization. In
The Thirteenth International Conference on Learning Representations, 2025. URL https:
/oopeneeviww.co wwwp::etteeettttteteeemeem wowoooomiiiii.ooooooo.ooo 2oo
Sobhan Hemati, Guojun Zhang, Amir Estiri, and Xi Chen. Understanding hessian alignment for
domain generalization, 2023. URL httpps://axiv..og/////po http://bb/2300.1111
lain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis.
The Annals of Statistics, 29(2), April 2001. ISSN 0090-5364. doi: 00121////0/00092200555 doi ww.. /0. 1214 /at////////0002052223
URL TtRLhttp::///
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Secome International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1032- 1041. PMLR, 16- 18 Apr 2019. URL URL httphhtp/////ttt ngs ml htt.://eeeeeeooo......rree/e/ear
9a htm]

---

Published as a conference paper at ICLR 2026
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-bard batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017. htth::ss:// httttps://oppeeviieee /openrevie..
net d o Ygg
Minyoung Kim, Da Li. Shell Hu, and Timothy Hospedales. Fisher SAM: Information geom- etry and sharpness aware minimisation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu. and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11148-11161, PMLR, 17-23 Jul 2022. URL httpp::/prooeediggsss. www.:iinnnng.....ppp..... v1 62 / kim22f...mm html
Itamar D. Landau, Gabriel C. Mel, and Surya Ganguli. Singular vectors of sums rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model. Phys. doi: 10.1 10.1103/PhysRevE.. 108.054189 URL htttps://innd: aps. wwt.://gg//o////////////// 1103 11t.:...11111/hhhh.eeemm 108. 054129
Rev. E. 108:054129, Nov 2023.
Binh M Le and Simon S Woo. Gradient alignment for cross doos-- face anti-spoofing. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 188-199, 2024.
Yue Li and Yuting Wei. Minimum norm interpolatss Precise asymptotics and multiple descent, 2021.
Zhenyu Liao and Michael W. Mahoney. Hessian eigenspectra of more realistic nonlinear models. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, 2021. URL
RYNVO1xA8
Lin Lin, Yousef Saad, and Chao Yang. Approximating spectral densities of large matrices. SIAM Review, 58(1):34 65, 2016. doi: 10. 10/11333/39934283 URL /ttpp:://ee...r//I.....
130934283
Avner May, Jian Zhang, Tri Dao, and Christopher Ré. On the downstream performance com- pressed word embeddings. Advances in neural information processing systems, 12:11782-11793 2019. URL https::////// //ap:: httpp/pp..eaaat.ccccoaaaar.ooo wtw.:eawww...ccea........e.eaaaa
Gabriel Mel and Surya Ganguli. A theory of high dimensional regression with arbitrary correlations sample complexity, multiple descent curves and hierarchy of phase transitions. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7578-7587, PMLR, 18-24 Jul 2021. URL https: httpt:tttp//ooct /ttps///cccccein..... mlr. wwwpssssw39////
between input features and target functions:
html
Gabriel Mel and Jeffrey Pennington. Anisotropic random feature regression in high dimensions. In International Conference Learning Representations, 2022. htth:::::: htRL:tttts:::ppeeeeeett //tp:////////..e.... net/ http:/ormmmmii-ffaaaaaaaaaa2a.
Xuran Meng, Jianfeng Yao, and Yuan Cao. Multiple descent in the multiple random feature model, 2023.
09897
James A. Mingo and Roland Speicher. Free Probability and Random Matrices. Springer New York, 2017. ISBN 9781499969425.
doi:
URL http://dxx doi.
o 221110.10077771111-49999---333
Boaz Nadler. Finite sample approximation results for principal component analysis: A matrix per- turbation approach. The Annals of Statistics, 36(6), December 2008 ISSN 0090-5364. doi:
10.1214/08-aos618
doi. www.:do..og///// wtt.oogg/....../0...
1214/08 AOS618
Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, 2019. URL Utl.itttss//aariii.rr//b///////
12

---

Published as a conference paper at ICLR 2026
Vardan Papyan. Traces of class/cross-clas, structure pervade deep learning spectra. Journal of
Machine Learning Research, 21(52):1 2020. URLhttp:: jmtp:::///mmr..oo http://////m.ooooppppp//////////
20 933. html
Debashis Paul. Asymptotics of sample eigenstructure for a large dimensional spiked covariance
model. Statistica Sinica, 17(4):1117-1642, 2007. ISSN 10170405, 19968507. URL http:
www. jstorstor...o
st wttp:o///ttta///eeeeeeee/
Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 2806. PMLR, 06-11 Aug 2017. URL https: htttps://proceeddnggs htt.eeaeeddngg...........eessese mtt.press/v7c/
pennington1 7a html
Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of single- hidden-layer neural network. Neural Information Processing Systems, 2018. URL https:
//api aeop.sssemaniicccollaal
Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. Jour- nal of Statistical Mechanics: Theory and Experiment, 2019(12): December 2019. ISSN
doi: (0.1088/1742-54688bb3383 1742- 7742-5468/ab3bc-
1742-5468.
URL http: /ttp:://dx....... org 10..1/0..../8
Mare Potters and Jean-Philippe Bouchaud. A First Course in Random Matrix Theory: for Physi- cists, Engineers and Data Scientists. Cambridge University Press, November 2020. ISBN
9781108488082. doi: 01007/978110876899 URL. httt:://dx.xee /tg/10011117 9781108768900
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out- of-distribution generalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18347-18377. PMLR, 17-23 Jul 2022. URL cettps:/pproceedineg http:///wo..ioioo.ooo.erereraaaa html
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singu- larity and beyond, 2017. URLptts:://ppeneeeee http://eeet/ffr/rreeee forumm? 1id c 9gx
net
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to reg- ularization. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11):941--948 w35.: A.A0/article/vie/I11
May 2021. doi:
URL /tops:://ooss. aaai. org/ httg/indexxpphi php/
Sidak Pal Singh, Aurelien Lucchi. Thomas Hofmann, and Bernhard Schölkopf. Phenomenology of double descent in finite-width neural networks. In International Conference on Learning Repre- sentations, 2022. URL http::ttt/:// http:tttts://eeeeee///eeee
n forett/oorrmm1d1111: htt.:/ooommniaaiccnnnnnie
Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Picrre-Antoine Manzagol, Yoshua Ben- gio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on op- timization and generalization. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 3503-3513, PMLR, 26-28 Aug 2020. URL htttps:/pproceedi:: ptr.seeeediinss.m..rrr...sseeeee wwtresss//1088/hhaaa22222. html
ml
high Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate in high-dimensional random
feature regression, 2021. URL URL.Etpa::/arxvv.ccx///
08234
Shashanka Ubaru, Yousef Saad, and Abd-Krim Seghouane. Fast estimation of approximate matrix ranks using spectral densities. Neural Computation, 231111111,,229)1111111111131 05 2017. ISSN 0699- doi: 10. 16.1162/NECO 00951. URL htttps:///deet http:///t.oo.ooo....oo w NECO 00951
13

---

Published as a conference paper at ICLR 2026
Tao Wu, Tie Luo, and Donald C. Wunsch II. Cr-sam: Curvature regularized sharpness-aware minimization. Proceedings of the AAAI Conference on Artificial Intelligence. 38(6):6144 Mar. 2024. doi: wttp:///.......8..2......
URL httph:hts////oo aadi.
AAA//atticle/vlew//BB88
Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, and Rong Ge. Dissecting hessian: Understand- ing common structure of hessian in neural networks, 2022. URL
org/
2010. 04261
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. Pyhessian: Neural net-
works through the lens of the hessian. 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590, 2019. URL https: //ap:: semanticscho arr www.arr....ooeee..ee.oeeoeeooeoo Crg/CorppsID: corp
209376531
A STATEMENT ON LARGE LANGUAGE MODEL USE
Large language models were used to polish writing and in conjunction with other tools to discover relevant published work.
B
THEORETICAL FOUNDATIONS
B. EIGENVECTOR OVERLAP FUNCTION
To connect the finite-d decomposition manoom-matris and free- rre--probability tools, express eigenvector overlaps in kernelized trace form amenable to tree-probabilistic methods. Let X, Y be symmetric d d matrices with eigendecompositions
X
uuu
Y= Σ Uj0
1=1
Ø=1
For bounded functions fig.
o[f(X)((Y) 1
f(A))
[d (u]T,
(17)
d :=
If and g sharply peaked around Xı and A2, the sum concentrates overlaps between eigen- vectors with eigenvalues near (A1,A2).
A convenient choice is the Poisson kernel
K(x;p,,) (x 4) + 2²
with center a and width a > We define the overlap function
ir (X; 01) K(Y; 02)
O2)
lim 01.02
(18)
K(X; tr K(Y; A2, 02)
The denominator normalizes the total weight in the sum 17) to one, O(d1, X2) is the weighted average of the (scaled) squared overlaps d (u: 0z)² over eigenpairs near (A1, A2). Now [1) can be rewritten
// ( X2) dux (A1) dpy (A2).
ir [f (X)
(19)
In fact, another way to define the overlap function is to write px,y for the measure taking f.g + [$ (X) (Y)) and then defining O (A1,)2) to be the function making (9) hold, ie. O dax 0px
dux.y
14

---

Published as a conference paper at ICLR 2026
B.2 OVERLAP FLUCTUATION LAW
Here we prove the fluctuation law 6 Equation S
which uses the quadratic surrogate losses
train Ctest to compute the test loss increment, reads
AC d Fesst train Z ZZ= H train H H train"
H
(20)
Noting that Aw H train under the assumption that E[Aw]] = O, one clearly has
E[AC] tr H
Hursinl 2 test Curaiul-
(21)
All that's left is to show that the last trace can be expressed in the integral form 6 (Ag best utaut,, (Agraiu, utrain) be the cigenvaluucs/eigeneettos of Heest Ctrains
Letting
Heet Crrainl
x ) Xtrain [d (ut uj train 1
(22)
Defining the overlap measure
=
d
(d (u,
test train) { j Ayrain))
(23)
1=
j=1
equation (22) can be written
// \ de (dA1, dd2).
trest Cerain =
(24)
2d
is absolutely continuous with respect Mrest Merain. and we may define the Radon-Nikodya (A1, A2) so that
derivative 0()1,12)
dy
dftrcut Sittrrii
//
2d (rrain)
A1A20(A1: Prost (dA) Merain (d)z)..
(25)
On any atom (Mest, (rral)).
(((Atest, ( ((rain))
O X Xtrain)
(utest (rrii)) 2
(26)
Ptest (fxrest) H)ttrain (Atraimn)
as desired.
B.2.1 SURROGATE-FREE FORMULATION
For completeness, we derive the fluctuation law without the use of quadratic surrogate losses by making minor modification the train Hessian. As before, let Ltaain (w, e) and (tes(w) denote the train and test losses, assumed twice differentiable in w, and let wy be the unperturbed minimizer of Cerain(w (w. 0). For small perturbation e, write Aur w(E) wp for the exact displacement. By the fundamental theorem of calculus along the line segment wi + LAw.
L V Cerain + tAw, e) dt Aw.
Cersinl (W (wo
+ Aw,e) Ciraain CarCCariiiWWo,e e)
Define the effective train Hessian
L'' V Cirain + tAw e) dt.
Hell
The perturbed optimality condition u Cerain (wo + Aw;; O) therefore yields the exact displace- ment equation
Aw = (Heff ) 2,
where 2 = V Seraii (Wa.e) e). Thus, Aw is obtained by the same inverse- inverse-Hessian filtering law as in the quadratic case, with Herain replaced by Helf
15

---

Published as a conference paper at ICLR 2026
For the test-loss increment, apply ordinary Taylor expansion at Wy:
Leest WW + Aw) Crest (wo) + = Aw 1/2 Aw Aue + O O(I)Aw|)))
where Ztest = d V to test (wn) and Hrest = d V V. (wo) are evaluated at the unperturbed point and are independent of Substituting the displacement equation, as in the surrogate case one obtains:
AC = Sesst train
Hell
)
27
1 (H ( H (Hel ) (11e
Taking expectations over the perturbation, the quadratic term has the same form as in Theorem I Hee Ctrainl, where Curain = E (HHrm))) - (H and so after replacing Huaain with Helt
train: obtains formally identical overlap decomposition in (6) in terms of test-(effective) train cigenvector overlaps.
B.2.2 NOISY GRADIENT DESCENT
Consider gradient descent with small additive noise at each iteration, and take its continuum limit. In the local quadratic regime, such noisy gradient descent can be modeled by the linear stochastic differential equation (SDE)
dw, - -Huaiin wydt + /B/
where Σ is the noise covariance and B, is standard Brownian motion. Its stationary covariance Ctrain satisfies the Lyapunov equation
Htrain Cersia Ctrain
This covariance is setaur--iiiteeed-vviiiniee is suppressed along high-curvature directions and amplified along shallow ones mies-similarly to the perturbation-induced Crrain used in the main text. Substituting this covariance into the trace formula yields the same overlap fluctuation law and the same two-loss spectral-overley decomposition, with no modifications to the framework required.
B.3 FREE TRANSFER LAW
Here we prove the following free transfer law for overlap functions:
Theorem (Theorem Free transfer law). Let B combine B with source of noise X that is free from A, B. Then
O,,B(a,b) f O...(a,
b. b). as (db).
(27)
Note that despite its simple and appealing form, this relationship does not hold for general triples of matrices A, B, C-it suffices to check finite dimensional matrices with simple spectra. While the coefficients of C's eigenvectors in the eigenbases of A, B do follow change of basis law resembling formula (27). recall that the overlap function encodes the squared coefficients rather than the raw coefficients themselves. This relationship holds specifically because of the freeness relationship we've assumed.
To prove 27 start by noting that all of the spectral and overlap information for two matrices A, B contained in the measure on IR² corresponding to the functional
PA.B f.g (A) (B)).
For example, setting g -> 1 and (x) = gives access to all moments of A, and therefore to its spectrum, and similarly for B, while the overlap function precisely O du Bun Radon-Nikodym derivative the joint measure with respect to the marginals.
dua.R
the
For the remainder of this section, we will work in an abstract free probability space rather than with concrete matrices. Let (M, 7) be a W* probbabiiti space (in our application this corresponds to the space of d d matrices with 7 E tr). See Mingo & Speicher (2017) for details.
Let A, B, B, M be random variables and consider the problem of determining the overlap function O A. where B F (B, X), where X is source of noise that free from A, B. As noted
16

---

Published as a conference paper at ICLR 2026
above, all of the overlap information for the three possible pairs of variables A, B, B is contained in the joint measures
M A M.A.B. BB.
supported on (some subset of) IR?. We are free now to treat these measures as ordinary probability measures of two scalar random variables. Denote by (-) these scalar expectations. We now make use of the following two identities:
(f (x) g (U)) (Z.))~p = (f) (X)
(28)
and that for each in a suitably broad class of functions (eg. at minimum all Poisson kernels), there
another function Ly depending linearly on g encoding the "expectation of g [ = g (F (B,X))
over X, conditioned on B", ie. such that
[[(A. B) - - (g, (B)
(29)
for all bounded borel f. This Proposition | proved below using operator-valued free probability,
Combining these, have (f()) g (6) (a.)~=a..
If (A) (B)). Writing the right hand
side as scalar expectation and then conditioning on b.
(r g (0) (a,b)-nA. = (a) (a))))- Lg (b) bmpa
= (Q (b) b,g (b) (0))
where (b) = (f (a)) (a)) Applying 288 and again,
(f(0) (b)) (2.6)-a.. T [Q g [
(Q(6) (b) (b.0)-1O.
-(f(a)) (f (a(b)) badae bwuB
which shows that a, are independent conditioned on b:
PA.B = / HAI
Applying, for example, classical e-gaussian smoothing the measures of A, B, B, we can assume that px,Y < @ py for any two of the three. Thus we are free to form the Radon-Nikodyy derivative dpx Buy which corresponds the gaussian- smoothed overlap function Ox,y: (x. v)- Since Ox,y;e (2, v) px (dx) = pxlr=y (dz), we have that for any bounded measurable function
dux,y.
/ Φ (a.b) du A.
// < (a.b) ( MAIB= (da) (dda))BBBBl) MBlB=b
(bb) uBB (dlb)
/ʃ (a.b) / (O (a, (a, pa (da))
(Op. (b.6) 11 (ab) as (ab)
// (a.b) (OOBB (a, b) O ni (bb)) us (db) Ma (da) u db
so the last quantity is exactly the Radon-Nikodym density O A.
O Aff (a.b)= - / (a, (0.B))OO.... n. (b.6) μB (db).
Taking the smoothing to O, one obtains the transfer law for the overlap function. Note that may have to interpret the Ox,y as distributions (eg involving kernels) in case degenerate overlap between two of the matrices (eg. A = B).
17

---

Published as a conference paper at ICLR 2026
Proposition 1. Let A, B be free from X. and let B (B, be rational function of B, X. Then there is a linear operator L on functions such that for arbitrary bounded borel H,
B) = I (A,
(B)]
Proof. Let B be a x & linearization of B such that [B k, = y (F (B, Decompose B into its X-dependent and Xindeependent parts:
B XBx Bo.
Now form the linearization matrix
L (MM = (
Me -1 )
B
0
so that
- Me. B-1 -1 0
-B- 0
L-1 (M) =
In particular, LØ,1 (M) = (F (B,X)) and k+1,1 (M) = - (F (B,X)) So now
[OL(H(A.B) (0) =T [[ (A, (A) k+1,1
=T [ (A, g (F (B,X))]
As with B, decompose L into X X-dependent and X X-independent parts:
L (M) XLx Lo-
Assuming M is a function of A, B only, these two parts are operator-free (ie. with amalgamation over the space of (k: 1) (k 1) matrices with complex entries). By the additive subordination law,
(L(M) (0) IL. (-RXLx (gL(Ar) (0)))
From the linearization, 9L(M) (0) is block lower triangular, and XL only has nonzero components in the upper left block. This implies that RxIx (oL(M) (0)) is also confined to the upper left block, and that this entry is simply R-XBx (EB-1), where & == 6 id is the expectation functional of the operator-space. These imply
T TIl (A, B) (F (B, (9L(H(A,B)) (0) k+1,1
IgLo ( (-RXLx (IL(H(A,B)) (0)))
1+1,1
Bo - -R (EB-1)
0 -17
- -H (A, B ck
1-11
H (A, (Bo R- (EB
1)
The second factor be regarded simply function of B (Bo is the non-X part of B and EB-1: B - EB-1 (B, X) E M& (C). This proves that there is some operator L [g] as in the statement of the proposition. must obviously be linear in g, completing the proof.
C TWO-LOSS GEOMETRY IN ANISOTROPIC RIDGE REGRESSION
In this appendix we derive equation overlap functions that are used in figures 1-3. We consider ridge regression on multivariate gaussian input data with train and test covariances Etrain. Etest and with linear ground truth, y (z) va w. xr.
prove Theorem and obtain formulas for the loss and
1
We will assume for simplicity that w. N (0, Ia). A noisy training set is generated by sampling inputs as follows. The training set consists
3z N (0. Etrain),
y1 (xx) Eos ~ (0,
18

---

Published as a conference paper at ICLR 2026
for 1,..., We define the sampling density m/d. It will occasionally be convenient to state formulas in terms of a's reciprocal, := d/m.
Using the framing of section 3.1.1 will regard the noise as perturbing noiseless training objective. The train and test losses are formulated as follows:
m
Ciaain (w, =
yt
va w x.)²
+
|w|
m
Va w x)²
Leest (w) = Ex (v(x)
Note 1) we keep Lerain's dependence on the perturbation & explicit, and 2) the noise included in the test loss (and when it is, after averaging. it changes the loss only by an additive constant). Finally, let us write á (E) == argmin, Ltaain (w, {) for the learned weights, y (x) = v à a for the learned model, and Herain these scalings are chosen to keep spectra (1).
2 Ltrain and Hasst d 2 frest for the train and test Hessians;
We begin by deriving equation (1] by applying the fluctuation law (6 To do so, first compute 2, Htrainy Hrest, and Ctrain- Differentiating, we find
V Lurrrain (w, ( x X1)
vi vam
J Xu, +5)
(30)
Htrain d V2 Veraiin
(31)
Similarly,
Frest (w) = 1 (w W.) Etest (w W.)
2d
Hiest = d 2 Cteat Etest.
Next, =:= d.V. i. (wo, E) is the scaled train gradient evaluated at the unperturbed optimum wo- Since, by definition, Vtraaiin (Wo, 0) = substituting into (30) gives
z = d V Ltrain (wo, E) ( x X + A7) wo vax x
( (12x - +5)
(32)
= x
vd m
(33)
Finally,
Crrain
1E [[11 2) (H (H _ train
d EE [1l,,ain X² XTE XH [riin
og m H m
x X)H train
= =Qaa Eurain (Etrain + AI)-2
where Etrain == 1 1x X is the (uncentered) train covariance. Summarizing,
Vd TT m
(34)
Htran
Etrain + AI
(35)
Hiest Σtost
(36)
Curain = oāa - Etrain (Eerain + X1)-2.
(37)
Since [<]] 0, we have E[Aw]
- H1
Plugging directly into (6. we find
E /f A2 O Homms, Crran (A1 X2) P. (dA1) PC (d22).
(38)
Since Strain: Cerain commute, they share eigenvectors and we are free to replace O O Orent,Cruin with
HHest: Straio" Replacing the integral over Cerain 's spectrum with one over Etrain. we find
oá Za
N1
de Og S Ereai (2) p.o.s (dA1) M. (d)2),
E A|
(39)
(12 +22
19

---

Published as a conference paper at ICLR 2026
which is equation
High-dimensional ridge regression has been studied extensively, so rather than rederiving published formulas, as much as possible, we restrict attention to the novel focus of this article: overlap de- compositions. We treat the label noise as a fluctuation of the training objective, and derive exact asymptotic formulas for the needed train-test spectra and overlap functions that describing the effect of the fluctuation on test error.
All formulas obtainable from the general trace formula stated in the following two propositions, which we prove in Appendix D
Proposition 2. The equation
r r= 1=-111) {
ditEemat (t) (=)²
(40)
2 tr
has unique solution satisfying r E H for HF and satisfying O < for < 0. This
defines a holomorphic function r (2) on all of C/ that is obtainable for each 2 by fixed point iteration of the right hand side of (40) from arbitrary initial point ro satisfying to H for
2 E H and 0 < To < 1 for 2 < 0.
Proposition 3. Let
4 (z) ir (Erest Etrain) ) (zT - 1) ,
(41)
for bounded function f and complex scalar C|R20 As m, d with q d/ fixed.
(2) + fr (Erest Eirain) ) (zl - r Etrain)
where (z) is the solution of the self-consistent equation 44
Thus to perform the calculation we simply express all quantities in terms of traces of the form & (z), and then apply Propositions
C.I TRAIN-TEST HESSIAN OVERLAP FUNCTION
Since we will be interested primarily in the ridgeless limit À -> 0, and since the effect of nonzero is simply to shift the spectrum of Htrain == m _ X + AI, we will omit À in the computation of O Hrase Herain (A1,A2) and will write Harain A = 1 X X from now on unless explicitly stated otherwise. Thus, we interested in the overlap function of the matrices
Hiest Etest
Husin 11/2 m
ZT Z
The computation is simplified significantly by taking advantage of asymptotic freeness of 1 Z 2 we have asymptotically
from Draai, Leett. By Theorem
m
Ohrme Heraan (Ner Ars)
O Oxears Heraan (Ater Ace)
I OEien (Ncer A) D Hreoin (A, Avx) diss (A). (42)
In particular this shows, somewhat intuitively, that the overlap function of the train/test Hessians will itself depend on the overlap function of the population covariance matrices. Eq. 42) shows the dependence is quite simple: One simply composes the overlap kernels taking an Hirain >:1/2 1 2T 22 eigenspace
Eirain one, and taking a Etrain eigenspace to Etest one. The overlap of the train and test population covariances, Oz Exrmin is part of the input data of the problem and is therefore known. As for the second factor, Oz..in. Huriia this is simply the overlap function of the population and sample covariance matrices for an anisotropic gaussian sample. For- mulas for this quantity are known (see, eg. Pottters & Bouchaud (2020)). To keep the presentation self-contained we quickly derive an expression using operator-valuee valued free probability.
m
Following Appendix B. the train-test Hessian overlap function can be computed via
tr [[(Earuiii \1, 01) K K(Herain' 22, 02)] o-0 tr K (Etrain \1, 01)] tr| ((Htrann X2 02)]
lim Exvain Husin A21,
20

---

Published as a conference paper at ICLR 2026
where
p.: == - μ)² + o²
the Poisson kernel with center μ and width a. Moving the first denominator trace into the numer- ator and canceling factor of и from top and bottom, we find that computing O Heeat Htrain (A1, A2) requires the numerator and denominator traces
h (Eerain)
02 (A )21)² + 0
tr
02 (A
X21) + +
where h (Ferain) = K (Ptrainn 01) (TttKKEaainii, o1)
Using the definition of f (z) (41). and the resolvent form of the Poisson kernel,
K K(x; pt, - Im (# io x)
these become
Imt (A2 + i
-Imti (A2 i02).
Proposition implies
tn (z) + ir h (Etrain) (aI r (2) Eirait)
(43)
11 (z) (sI r (2) Eerain)
(44)
Im h (Etrain) (2T r (2) Etrain)
OSrannHerai (A1,
+ lim lim di +0
Im (=T r (2) Etrain)
where + is shorthand for limo- with z = x2 + io. Taking 0i + 0 sends h (Etrain) to delta function and collapses the trace in the numerator to the a eigenspace of Etrain.
Im (2)
O Oxrran Heroio (21,22)
x
Im (x) dpm (A)
Composing with Ox Stvain yields the overlap function Oz Osueat O Hoestt
C.2 OVERLAP DECOMPOSITION OF AC
Trace integrals are written terms of the spectra overlaps of the matrices involved. To explicitly determine the spectral density of Htrain, note that it can be written in terms of the trace in equation
PHrrntn (Aer) lim ir K "HHrrain Aue o)]. a
Using the same approach as above, we have the following for the z-Poisson-smoothed spectral density of Htrain
Pilreainoo (Atr) = Im 1 (Aer + i0)
1
2 (2) dus (A).
Collecting the results of the previous section and the fluctuation formula
0²
Aur
AC
Aue
O Oeerain Hrca (dtey Nar) (. (dDre) HH (dder), (45)
2a
(Aer X)
eigenspace overlap
train variance
where
Orren Herai (Pre Aud) / O Exrrin (rre A) O Devaiin Harain (), Ate) dpsemmm (2), (46)
21

---

Published as a conference paper at ICLR 2026
with
Im
O Eerain Herain (A1,
lim
g-r(s)A1
スーイ()パ dissaaiia (A)
This provides complete decomposition of the test loss fluctuation in terms of spectra and overlaps of the train and test Hessian.
C.3 PROOF COVARIATE SHIFT THEOREM
This subsection, together with the proofs of Propositions found in Appendix D proves Theorem
Formulas 45) and (46) show the effect of covariate shift in train/test sets decomposes naturally in terms of the overlap function Oxton,Eerni the two population covariances, (Note that there are two levels of overlap decomposition: the test loss increment is decomposed in terms of the train-test Hessian overlap function (45) which turn is decomposed in terms of the overlaps Evest: Errain-)
We equivalently start from explicit expressions for the fluctuation. Differentiating the loss and solving for the optimal weights directly, one has
Lrest (1 (E)) - of (tha (-1) Xt (-))) + 226 (-1)),
(47)
(equation (51) of Appendix C.4 Since AL = Liest Lo, and Lo is obtained by simply setting the perturbation strength oçę + immediately find
oe d Alia (-1).
AC
(48)
2x dA
Adopting the notation Ā = _ r[-xj Proposition 3 yields
Abi (-x) -At Mrest (A1 Errain
Ata (-)) + -Xtrr Etraiu (ir + Eerain dd
-2
Substituting into 488
2 AC
X'r x'r
Dreaiirraan (āt + Eirain)
}
-2
2a
Writing the last trace as an integral over the spectral measures of Eust, Etrains this becomes
N 20
f Xie (x+ Aar)
Aer
AC
EE Derain (Nues dax) dissoa (Mte) dyserain (Nur).
(49)
which completes the proof of Theorem Equation 49
parallels 45 but averages out the random
training inputs and label noise to express AL purely in terms of the known objects Y Errain Etest This expression shows that label noise leads to large increases in test loss when a direction of large train- ing variance (small eigenvalue Ate of Etrain) and a direction of large test curvature (large eigenvalue Are of Etest) experience significant overlap (large O Eerai (Auer )t))).
C.4 EXPLICIT FORMULAS TEST LOSS, FLUCTUATION
Here we derive explicit expressions for the full test loss and test loss fluctuation under general covariate shift. Since these formulas and generalizations of them are already published, this section is mostly for internal reference-especiieee especially for calculation of theoretical loss curves in Figs. and
Let X have rows n.J and & have components &.. Lirain can be written
Curain CtrainW,&.
2m J Xw, +
3 Xw|12 + 24 w w.
22

---

Published as a conference paper at ICLR 2026
Differentiating, we find
V Ltrain (w: {) Htrrin W
1 XT
XW, +5)
va m
H Htrain = d Ltrain
Ltrain
m X
Similarly,
Lrest (w) = 2d
(w w.) Evest (w - ww) ww,
Htest d 2 Liest Etest.
Solving 0 V Wiraaain yields
① H erain ( (T X) w. + (Htrain)
vd x Tgg.
&.
m
Substituting into Festt yields
Liest 40 20 x + X2T
Ctest (10 (E))
2 ir
X + 21)
Since (A + 2
d (A+ -1 ax
,we can write
1
X2 d
Lest (1) (E)) -
90t dx
tid (-1)
(50)
dd
1
goo (44 (-X) Mis (-X)) + 222 (-)))
(51)
2
Proposition implies
Ai (2) - EEest (2T - r (2) Etrain)
1/ (z) + -Tr Enest (zl - (2) Etrain) (1 - (=) Eerain)
which fully specifies Liest (1) (E)). The fluctuation easily gotten by setting of + and subtracting from Liest (10 (E)).
Reduction to published formulas
Letting Ā = r(-X) X) and substituting into 40 we obtain
A == ʃ
X
dussaamm (t),
(52)
X+t
which is eq. (8) of Mel & Ganguli (021) for the "effective regularization".
The fluctuation in 50 is
1 d AC qoo Ata (-2). dh
Since
A tr Errain
A I Etrain (Tr(->)²
Ataa (-))
r r--2)
- Errain (DI + Etrain) 7
get
Etrain Etrain
AC X'tr X'r
The authors define 1
dd dx*
80
1
Derain
27
AC -00 -i
Ps
Ät Derain
23

---

Published as a conference paper at ICLR 2026
which matches the fluctuation term of their formula up to constant factors differing in the loss defi- nitions. Next, the remaining term can be written
1 2
(-Xtuu Xtu (-1) + A d New (-2))
Lo = Liest - AC =
dd
Using 50 again,
MEtrain
Co
fr
XXTrr
train
Etrain
( Strain)
Comparing to 52). the first term is a (A -2 and differentiating gives
X
Exeain Etrain
tr
Substituting and simplifying yields
1
Lo
an
ad
2
Eerain 2
α tr
Seraii
Etrain
27
2
X'
a
ard
( Etrain
Once again using equation
52
{ x)
can be turned back into trace:
[ { Etrain
Eirain
Lo = X tr
Aer
2
AI + Etrain
2 Etrain
N'r
= 2
(AI + Etrain
which is equivalent to their second term.
C.5
k-LEVEL MODEL
At several points in the main text refer k-level input covariance,
k
pro..-
In this case the self-consistent equation for r 40 becomes
P1
1=1
which can written as p (r, 2) for some polynomial in r, z. Similarly, the overlap function simplifies to sum over the distinct eigenvalues of Etraini
Im 2-1(2)11
DStranniHrna (A1,
lim
x
E p,Im 2-r(8)8
24

---

Published as a conference paper at ICLR 2026
C.5.1 SEPARATED SCALES LIMIT
We now assume the scales are widely separated: 8;+1 S4. We will also work with the ridgeless formulas corresponding to À + 0 derived in Appendix C.6 For simplicity, assume $1 = 1. We will obtain leading order formulas for (a) as the ratio of successive scales is taken In Appendix C.6 h is defined and found to satisfy the following self-consistent equation (equation (55
1= + at dppErain (t)
The right hand side is a decreasing function of h and a decreasing function of a, we have that is decreasing function of a. Since the integral reduces terms with & < h do not contribute at leading order in $1+1/ 9s, we assume that h 2 1 and neglect all lower terms, giving
over the & eigenvalues, and since all
= p1 h a
so h = P1 a and AL
o a s0 (s, 1) duduse (s)
AC
a
the integral in the numerator can be written
| (s, 1) dusaca (8)
= - Erest Pr Premin=))
P1
where Ps: Proquimma is the projector onto Etrain 's a-eigenspace. In other words, the integral simply the normalized total overlap of Etest onto the strong training covariance space, and is equal to 1 for Etrain Eeett-
Now let us assume that h is near the scale sf. The self-consistent equation becomes
sf))
as
where Derai yields
(> 8?) is the total probability mass of all scales greater than s ie E}=1 Pj. Solving
Pi
h. which is consistent with the assumption that s7. Since h 2 we only get valid solution for >>> s)) Substituting back into the error expression yields
as
st o2 p:a² J (h-attz O (s, desirai (t) dpstem (s)
AC
(WS..an (441) a) (a MErrain
s?))
Evaluating the numerator generally requires a choice of Eeest 's behavior in the limit 8/41/s; - 0, but note that the denominator has zeros at a = /D will generically become infinite whenever is equal to the cumulative mass of some number of top scales. As a simple special case, letting Leest Derain this reduces to
(> 8?) M4 (> 1. and so the error
0² (a - Je
PiftSiuain 0 αx) (ar Meraain
AC
8?))
Since under this assumption,
Lo
5 hr Etrain (hl + (EErrai))
-11
1
P: when we take 8141/55 O, Lo only contributes at the highest scale,
as?
Myarat (> s?) < a S Pranain (> s(n))
2
a Ce-Menuin (30
Lo 2 x+ (pi a),
where a+ is the ReLU function.
25

---

Published as a conference paper at ICLR 2026
C.6 RIDGELESS LIMIT
Here we simplify our formula for the test error in the ridgeless limit. From (50)
we have
1
AC = -200 Aha (-))
2
dd
Co == Crsst AC = - 22 (-X). It will also be helpful to consult ti and r's explicit expressions matrix traces (equations 41 and
59
[11 - _
4a (z) = Ejest
(53)
(z) = q
(z - A) A
(54)
C.6.1 OVERSAMPLED REGIME
From 53) and since for m/d 1, the limiting spectrum of XTX X is bounded away from 0,
ha (-)) is analytic as À 0+. Thus in the oversampled regime
AC
g00 (0):
Lo +
From Propositions and 3
lia (z) - (0)
fr EEeett train
r (0) =
so that
AC, Lrest +
0e 2 a
[I
Etest
5-1 Irain
a-11
C.6.2 UNDERSAMPLED REGIME
Now assume a < 1. For + 0, ha and r's explicit expressions in (53) and (59) suggest tu (-X) (X-1) and r (->) = O (A). For convenience we will rewrite our formulas terms of (z)
-2). Substituting into the self-consistent equation for r 40) and simplifying gives
h (z) (gh (=) + +2) /
1 (2) + + durmaan (t).
Now differentiating and setting O, we find
1 = / qh +
t
dissiaain (1)
(55)
(
dpsreeai (4)
gl++3
N'
(56)
q gh++
dpseraii (t)
where we've suppressed the argument of h, We now write the eиror expressions in terms of these
d
AC
-006 Atia (-A)
dx d
(gh(2) + 1) [Sueat ((gh (A) + 2) E
g dd
Exestt Exeain
2 900
1 - q I (a dissimmn (t)
qiitte)
st O (s, t) djs (1) dessoa (8)
(gh+ee=
1 - q ( dprraan (t)
glite)
26

---

Published as a conference paper at ICLR 2026
while
1 2 (-x)
Lo
2
qh tr
Etest (qhl + Etrain)
29 qh S
s. qh +
00, 0 (s, dismrrnn (t) dytErmm (8).
Finally, the total loss is just Lu + AC
D CHARACTERIZATION OF y (z)
Here derive asymptotically exact expression for
tF (2) ir F (Erest: Etrain) (2I A) A
Let us abbreviate F =- F (Etrains itest). ). First,
1 - X X = Lrain
1/2
1 - - -TZ %
A
train?
m
m
where Z has standard normal entries, so that
tF (z) =r
Itrain Fe
1/2
21
Etrain m
1 TTZ Z
-1
(57)
Now define = Is - Es,2, where Es.2 matrix whose (5, 2) entry and has all other entries equal to 0, and let
1/2 train
FE
1/2 train
Erain
Σ
0
0
0
0
0
0
0
0
0
Q
0
0
0
1
Z
0
0
0
Vm
0
0
(E+ (E++)))++ has as its (5, block exactly the matrix in 57 and so F. (z) = 1 (B)),,1 where 97.+Q is the operator-valued Cauchy transform of Σ + Q
is straightforward verify that (B
By rotational invariance, Σ, Q are asymptotically operator free, meaning we can apply the operator- valued additive subordination relation (see, eg. Mingo & Speicher (2017) Chapter 10), which yields the self-consistent equation
g (B) = gx. (B R2 (gx+Q (B))).
The blocks of Q are standard normal matrices, and so its R-transform given by
RQ (M) E [QMQ),
where Es is the operator-valued expectation that takes normalized traces of all square blocks. Due to the large number of zeros in Q, only two entries of Ro (M) are nonzero:
[Ro (M)) M4.1
(Ro (M) = 9.s..-
27

---

Published as a conference paper at ICLR 2026
On the other hand, by definition g1 (M) = (M - 2) ")
Substituting back into the subordina-
tion relation and writing g for gE+Q (B), we find
0
0
0
=
1
Etrain
0
0
==
0
944
0
1
9953
0
-1
1
where simplify notation we have written E 2 FE 1//2 The entries of the right side straightforward to compute using elementary row operations. Performing just enough such opera- tions determine the (5, (4, 4), and (5, 3) entries, we obtain the closed system of equations
953 tr Eerain (=I 944 Itaai)))))
1
911
4953
951 = tr F (zI 944 Etrain) )
We can eliminate entirely, giving our trace
t (z) 951 = ir F -
([tll-rEtvain)) r-irain))
in terms of the scalar ;= 944 that satisfies
qtr Eirain (al r rEtrain) "I1""
A few remarks in order. First, note that can rewrite this trace integral the spectrum of Dtrain:
(1 11 dis (1) (0)².
(58)
It is helpful to compare (58) the explicit expressions for 914, 953 from the linearization before applying the subordination relation, which are
944 = + tr (2 1 A
(59)
= ir [=] - A)- Strain
(60)
Thus 944 (2) is analytic in everywhere outside the spectrum of A, and gs4 (H+) H and 0 944 (R<0) < 1 (the first inequality is gotten most easily by using 963 and g44
(953) )
In fact, these conditions along with the self-consistent equation (58) are enough to guarantee that
the solution unique, holomorphic, and coincides with Iı1 throughout all of C RR
Proposition 4. For E C/R20, there is unique solution (2) to (58) satisfying the conditions
r (IH=) C HF and ( < (R=0) < 1. (z) depends holomorphically on and can be obtained by
iteration of the right hand side of 588 from an arbitrary initial point in CRR200
Proof. Assume E _I Let f (r,=) =) be the map defined by the right hand side of 58
f (r,,)) = (1 (1 / br
dpssaai (1)
It is straightforward to check that f (1,z) 1H H+ Furthermore, point the boundary of H is a fixed point of f (., 2), since (R, C H+, and f (xc,2) 2) = 1. The Denjoy- Wolff theorem then guarantees that (:,2) has a unique fixed point in H and that this point is obtained by iteration of f (:, 2) from an arbitrary initial point in H Thus (58) together with the condition E H uniquely defines function r (2) for all HH
28

---

Published as a conference paper at ICLR 2026
Now fix 2g H Since (-, 20) H + H and f (.22g) is not Möbius transformation (it only can be if Strain is scalar matrix), the Schwarz lemma implies a 0r (r (zo) [2)] 1, which means
@ (f (r) -)) 0 Or (r, Ör
0,
and so the implicit function theorem implics there a holomorphic function solving 58 on some neighborhood of 2g that coincides with (=0) at 2p. Since (zo) E H this function must also stay in H+ in some (possibly smaller) neighborhood of 2p. and by uniqueness of solutions to (58). this implies that it coincides with r (z) on this neighborhood. Thus (z) IIII H+ is holomorphic each point of [H An identical argument proves the proposition for 2 E H
Now suppose < 0. Conjugating the right hand side of (58) by the map + 1/(1 -qx) gives self-consistent equation satisfied by 953:
t
y
dps (t)
(61)
1 1-
The condition 0 < r (2) < implies 9s3 < 0. Now letting h (y.2) be the right hand side of (61).
h (y, 2) y convex in y and satisfies h (0, 2) - 0 and h (-0c.) (-oc) oC, there is
unique solution to (61) with y < O, and thus a unique solution to (58) with O < r (z) < 1.
Since
& h (y, z) > 0, and differentiating / at the fixed point gives By
0 h (y, Oy
/
(1 qu)
= qu) dpsinnia (t)
8y
(1 (1- Q11 ql
=1 1 qU
+ z
dpeseenin (1)
(2 qy)) t)
< 1,
y (z) an attracting fixed point (:: 2). Since (.,z) a conjugate of f (., z), the unique solution
of (58) satisfying 0 < r (z) < 1 is an attracting fixed point of (() This implies that there is a neighborhood of that extends into the upper half plane whose iterates converge to r (z). But since < 0, (IHI+, =) H+, and so the Denjoy-Wan Wolfff theorem implies that all iterates of (:,2
initialized in H converge to the same point, which therefore must be r (z).
Finally,
d (h (v) - v) = M (v) < 0 implies that y (=) extends holomorphically to a solution of dy (TT) in an entire neighborhood of z. Since h (y (w), w) (w) = 0 for all w in this neighborhood, the solution point,
T dps (t)
(2)
By (h (1) - (z)) < 0.
By (h (z) -
A negative derivative implies that for sufficiently small neighborhood U of 2, (UNH C H
Mapping back to r (=) := 1/(1 - my(()) yields a holomorphic function satisfying (88 neigh- borhood U of = < O such that for (U n H-) C H By uniqueness of solutions in the upper half plane, must coincide with the function defined earlier on U Thus r extends holomorphically to the negative real axis.
The subordination relation implies 94a (2) = r (2) in a neighborhood of co, but both functions extend holomorphically all of CR implying they equal throughout. This completes the proof of Propositions and 3
E LOCAL GEOMETRY OF MLPs
E. GRADIENT DESCENT DYNAMICS
The initial gradient at Wy is == Lirsin (Wo. e). and the Hessian is H == 2trains so the local approximation for the training loss is
Cirain (w) = d² (w wo) + 2d (w - wwo) Hurain (w - wwp),
29

---

Published as a conference paper at ICLR 2026
train-test Hessian overlaps
eigenvalue
10
10
10"
10
aigenvalue
Figure 6: Eigenvector overlap function for one MLP simulation in the context of 3.3 A dot is plotted for every pair of train and test cigenvalues, with dot size and opacity representing squared overlap of the corresponding eigenvectors. Note the very strong train-test alignment indicated by the restriction of almost all overlap to the diagonal.
AE dynamics
simulation
0.99
theory
0.02
Measured fac (bc]
0.015
0.000
10
0.01
0.02
SGD steps
Predicted frec [AC]
Figure 7: Learning dynamics predicted by local two-loss geometry. Left panel: Test loss trajectories in response to different label noise realizations (individual trajectories shown thin green lines; average shown as thick green line). Noise amplitude corresponds to red dot in Fig. 4b). To reduce clutter, only theory average is shown (black line; see )62). Right panel: measured vs predicted relative test loss increment at two times, corresponding to the vertical blue and gold lines in the left
panel. Points corresponding the same trajectory connected by gray lines. Horizontal blue and gold lines show means.
where we've discarded additive constants. The gradient then
1 à (= Htrain We Herain w)
V Levain (w)
so gradient descent does
w + w 77 (2 Htrain Wy + H
(I H)rain)) n (z Hersin
30

---

Published as a conference paper at ICLR 2026
Thus,
(I
Nurain)) 2,
Awy W wo
Huaii
and so the test error satisfies
(ACrest), 7 Aw, Aw, Aew
(62)
1 (1 -HHuala) -
1
(1 nHuram)
(1 - n 2,
7
Hurain
2d
Herain
Hesst
Hurain
(63)
with == dV Crest (wo) and H1est == dV² Crest (wo).
To compute the GD trajectory for a large network, we precompute
grad (Cerain) (wo)
train Wip = hvp (Ltrains Wo). Up Htrain wo, and simply iterate
+ w (vo + hvp (Cirain w)):
where grad, hvp compute function gradients and Hessian-vector products.
F EFFICIENT CALCULATION OF EIGENVECTOR OVERLAPS FOR LARGE MATRICES
Here describe the Overlap-KEr -verlap-KPM algorithm, which estimates the unnormalized total eigenvec- tor overlaps for two matrices. That is, for self-adjoint A, B E Rdxd it estimates the measure
MABB
1 d
Σ JJ=1
(AAA)))
d (d
"
which accumulates all overlap of A₁ A-cigenspaces with all X2 B-eigenspaces. To obtain the nor- malized overlap function treated in the main text (eg. equation (18). one simply divides by the spectral densities of A, B Å1, A2. The problem of estimating spectral densities for large matrices has already received significant attention (see Papyan (2019) in machine learning context), so we assume that one can practically transform back and forth between normalized and unnormalized overlap functions.
F.1 RANK DEFLATION AND NORMALIZATION
As preprocessing step, we remove the outlier eigenspaces from each matrix obtained via subspace iteration (cf Fig. 5), and then normalize the spectra to the interval [-1,
F.2
RANK DEFLATION VIA SUBSPACE ITERATION
The overlap plots of Fig. 5 were generated via Subspace Iteration (SI). SI is a well-known method that generalizes power iteration, so we review only briefly here. See Papyan (2019) for an explicit implementation in machine learning context.
Let M be a self-adjoint operator with simple spectrum Ai,..., As. and take V e gdx with standard normal entries and then orthonormalize the columns. SI iterates
V + MV V VQ.
where Q is the Gram-Schmidt orthonormalizing upper triangular matrix of V. Informally, each application of M amplifies each ith eigenspace coefficient the columns V by A, which gener- ically leads exponentially greater weight on the leading eigenspaces. The orthonormalization Q
31

---

Published as a conference paper at ICLR 2026
prevents all eigenvectors from collapsing onto the same leading eigenvector. Since they forced to span an k-dimensional space, they must converge to the top eigenvectors M. Overlaps can then be calculated directly by computing pairwise dot products of columns of V.
After convergence, outlier eigenspaces are removed from the matrices by replacing each matrix vector product r M (v) with
v 4 Maes (w) - M (o) VVTo. c.
F.2.1 SPECTRUM NORMALIZATION
After removing the outlier cigenvalues, one may obtain bounds for the remaining spectrum via standard approaches (eg. the Lanczos algorithm: Pappan (2019)). Letting Amin: Amax denote the minimum and maximum eigenvalue (in practice, with a small amount of padding added), we then normalize the matrices to the interval [-1, I] by replacing + Mdes (1) with
(Amax + Amin
V + Maorm (v)
Maef (v)
V.
Amax
Auain
Amnin
F.3 OVERLAP-KPM
We now assume the previous preprocessing steps have been performed and in particular that A, B's spectra lie inside 1].
First note that for kernel function G one can write the kernel-smoothed overlaps exactly trace:
d
G - GB:) - [===a)==- = d²
Σ 4J=1
G(A. - A1: - A2:0) [a (UA, UB,)
(64)
Thus the goal will be compute such traces for each (λ1)2)) for some sufficiently small fixed kernel width a. Computing such traces directly is prohibitively expensive for very large matrices, and standard approach is to use Hutchinson trace estimation, random samples of, say, standard normal v, since
average v Mv over several
E, [ Me] tr [ME, [ve []
[M].
Informal experiments suggested better stability for estimation of PSD traces, replace the trace on the left side of (64) with
ir [ (AB
A1:o)
Now applying the Hutchinson trick, we sample probes U1 UI,..., and approximate
(B
TT (A- (BB G2/2 (A A1:0) Up
p (3 (A- -1:0) ux
To compute the summand, we generalize a standard approach known as the kernel polynomial method. Practically speaking, this entails approximating the kernel functions G1/2 (x: x) using Chebyshev polynomials T, (æ), which can be computed efficiently using To (z) = 1, (x) = I, and the recurrence
T, (x) = 1 - T (x), Letting o., 3 be the Chebyshev coefficients of the kernel functions,
j22 2.
G1/2 (x --;1))))== Σ
a,Ti (z)
1=0
11/2 (x )2:0) = Σ
3,7, (x),
1=0
32

---

Published as a conference paper at ICLR 2026
truncate to degree K and write
Cp =
G1/2 1 x2 G2/2 (A- )1;0) u/
(65)
K
Σ
a, (B) (A) Up
(66)
1,=00
Σ
0x 0, T: (A)T, (B)
(67)
k
(68)
Thus for P probes and order-K Chebyshev truncation, by appropriate choice of the coefficients 3. one can approximate a general function from the P (K + 1)* dot products
ManaA of (A) (B) (B) Ta (A) Ey..
This can be improved somewhat using the Chebyshev product identity
T3 (x) = (+) (1-))))))))
(69)
so that
( Mij.k. = (1/ Ti (4) (B) (A) 77 (B)
and all needed dot products can be obtained from the P(K + 1)² (2K + 2PK dot prod- ucts
M.j.k.p = vp (A) (B) Th (A) un..
0 OSiKS 0 S
1SPSP.
Algorithm efficiently generates all such probe moments with PK2 matrix vector products. Algorithm actually stores all 2PK vectors T, (B) Th (A) Un. but in practice, our implementation is significantly more memory efficient. We store all Tj (A) Up but as B's are added, one only needs to store the current and previous power of B. This amounts to ~ K vectors in memory at once.
Once the Ma are known, equation (68) is used to estimate the trace for each value of λ1, A2. yielding approximation the unnormalized overlap function of A, B.
Often in machine learning contexts, one or both of A, B has spectrum that is highly peaked around particular value. For the trace in (64) to accurately reflect the overlaps at λ1, A2. the kemnls- more precisely, their finite K Chebyshev series-must decay sufficiently quickly away from Ai, A2 to prevent the spectral spikes from overwhelming the overlap sum. Practically speaking, this can be diagnosed by 1) forming an estimate of A, B's spectral density, eg. using the Lanczos algorithm (see Papyan (2019) for implementation in ML context), 2) forming truncated Chebyshev series for the kernels, and 3) comparing kernel decay to spike height. Insufficient decay usually requires either decreased kernel width a, or increased Chebyshev degree K so that polynomial approximations accurately approximate the small tails needed to dampen the spectral spikes.
Algorithm evaluates O(PK²) matrix vector products. When these correspond to Hessian vector products for a model with d parameters evaluated on examples, this equates to a total runtime complexity of O(P33mdd) The number of probes and the Chebyshev degree K are usually small and can be taken to be fixed relative to m, d, runtime is essentially linear in the number of parameters and number of examples. Similarly, Algorithm| I only requires keeping O(K) matrix vector products in memory at once, for memory footprint of ((d)).
Overlap-Ken -PrMap--PPMM combines two standard components-Cheebssheev Chebyshev polynomial approximation of smooth spectral kernels and Hutchinson trace estimation and therefore its hyperparameter behav- ior is straightforward. The truncation order controls only the polynomial approximation error of the Gaussian kernel; because the kernel is analytic, this error decays exponentially fast in K (Boyd.
33

---

Published as a conference paper at ICLR 2026
1989). and in practice the estimate stabilizes rapidly once K exceeds modest threshold. The num- ber of probes P affects only the Monte-Carhe variance, which decreases the usual OLL/vP) rate. Empirically, we observe that the estimator stable over wide ranges of K, P (see tests on synthetic data in F44
Algorithm 1: Overlap-KPM for Eigenvector Overlaps
Input: A()), B(v) (normalized MVPs); degree K; probes P
Output: Probe moments Mijkk, for 0 S k < K; 0 2K; 15 p S.P.
for 1 to P do
sample probe U/- NoO,
D0,0,a + Upi U0,1, A(Ux)
for 2 to K do
L Vo,i. + 2A (Vo 1,1x)
10,3-2.1
for 1 = 0 to K do U (( for j = 2 to 2K do
Uj,K,N 2B 1
Ug 2,
for i to K do for k to K do for j = 0 to 2K do
M.j.d.. vu,i, Uj.k.
F.4 TESTS SYNTHETIIC DATA
Algorithm 1 with gaussian kernel is applied to synthetic data in Fig.
8
A,
E R
1000 1000
are
generated according to
A Wi,
B == W2 + A?
(70)
where W1, W2 are independent Wishart matrices with aspect ratio a = 5. The left panel shows the ground truth gaussian-smoothed overlap function of A, Note the nontrivial alignment due B's dependence on A. The right panel shows the approximation generated via Overlap-KPM. showing good qualitative match.
We performed informal experiments varying K, P to test the robustness of Algorithm | (Figure As expected, accuracy quickly improves and eventually saturates as the Chebyshev approximation order is increased. The variance the estimator constant of proportionality that depends on the input matrices. As figure shows, even for modest number probes (eg., P = 4), results can quite accurate.
function of P decays as O(1/vP) with
F.5 HESSIAN OVERLAPS OF RESNET-20
34

---

Published as a conference paper at ICLR 2026
Actual
Overlap-KPM
1.6
1.6
0.8
0.8
AA
Aa
Figure 8: Overlap- -verlap-KPM algorithm recovers overlaps on synthetic data. 2 Z1 1//m and 7,2 7 + A where Z1, Z-2 E Rm. are independent matrices with iid. standard normal entries. d 1000, a == m/d 5. Chebyshev degree: K 45; number of probes 4. Left panel shows actual eigenvector overlaps eigenvalues Aa λB, smoothed with gaussian kernel of width 1/16. Right panel shows approximation derived from the Overlap-KPM algorithm.
8
= 16
K 32
16
Figure 9: Varying K, P in overlap-KPM. Gaussian kernel with width of 1/32. Matrices A, B were generated as in 8
35

---

Published as a conference paper at ICLR 2026
Htest
H
test
8
Normalized
Atrain
overlap density
8
test > train
Atrain
sectional density
0
25
50
0
40
Atast
Nest
Figure 10: Overlap-KKP KPM algorithm for Hessian overlaps of CIFAR-10 10-trained ResNet-20. De- gree K 45 and P = 4 probes with Jackson smoothing applied to gaussian kernel Cheby- shev coefficients. Left column shows overlaps for Htrain Hast (ie. balanced test set), while right column shows overlaps for Htrain, Heest (imbalanced test set). Top row shows normalized overlap functions H. (Atrain Atest. and O HerminnHi.... (Derain- Nestt For ease of visualiza- tion, bottom row shows test, train sectional densities O Heran Mrast (Aerain Atest ) (Atrain) and O OHusin.Hi.... (Atrain: Mest) Mrrai Atrain) the average overlap of 1-D Hrest/ HHi eigenspaces full eigenspaces of Htrain In both rows, strong diagonal overlaps are visible in the left column that are reduced absent in the right column. Note also in the bottom row that the tail of the Aerain band extends significantly further for H' than for Heet indicating significant loss of high H eigenspace energy into the low-eigenvalue band of Htraiu-
36


Reviews:
[{"review_id": "USC4Pt6RzH", "reviewer": "Reviewer_Rqbg", "summary": "This paper analyzes the alignment between eigenvectors of train and test loss Hessians and its impact on generalization, extending prior works that focused mainly on eigenvalue spectra. A universal local fluctuation law is formulated to demonstrate the predictive role of eigenvector alignment for test loss. The theory is further applied to linear regression under covariate shift and multiple descent scenarios. Experiments extend the framework to neural networks, including MLPs and ResNet-20 trained on CIFAR-10, with a novel algorithm estimating eigensubspace alignment in the bulk of small eigenvalues.", "questions": "-\tPlease refer to the points raised in the weaknesses section.\n-\tThe developed theory resembles the Takeuchi information criterion (TIC). Could the authors discuss potential connections or differences? (see, e.g., Thomas et al., On the interplay between noise and curvature and its effect on optimization and generalization, AISTATS 2020). \n-\tHow can the analysis be adapted to account for stochastic optimization?\n-\tHow robust is the estimator proposed in Section 3.4? Some ablation studies on its hyperparameters would strengthen the argument.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper provides a novel and original analysis revealing the importance of eigenvector alignment between train and test Hessians, which is an interesting and underexplored aspect in the literature.\n- A new framework is introduced to evaluate the alignment between eigenvectors associated with the bulk of small eigenvalues.\n- The analysis in linear regression provides concrete intuition on covariate shift and multiple descent.", "weaknesses": "-  Although the eigenvector alignment analysis is theoretically interesting, it is unclear how this insight could translate into practical benefits for model training, as test data are not available during training.\n-  The experiments involving neural networks in Sections 3.3 and 3.4 lack clear motivation. Section 3.3 partially supports the theory in Section 3.1 but does not provide concrete insights into MLP generalization or learning dynamics. The purpose of the figures at the bottom of Figure 4 is also unclear and needs further explanation. Section 3.4 proposes an interesting scalable estimation method for bulk subspace alignment, yet the analysis does not seem to make it a central element of the argument, which leaves its importance underemphasized. Moreover, the section concludes by examining train–test Hessian misalignment under test-class imbalance, without providing additional insights into generalization.\n-  The paper’s organization could be improved for readability. Key equations (e.g., (13)) and concepts (e.g., smoothing kernels in (10)) are relegated to the appendix. Terms such as “error increment” and “bias” in Figure 2(a, c) should be explicitly defined in the main text.\n\n**Minor comments**\n\n- The caption of Figure 4 (referring to $H_{test}$) does not match the main text (referring to $H_{train}$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vnF18fDEJT", "reviewer": "Reviewer_G7EJ", "summary": "This paper studies how the relationship between the 2nd order shape of the train and test objectives affects generalization. They then use the general results to study several regression settings, including covariate shift. Finally, they extend the results to neural networks.", "questions": "Feel free to clarify the points above.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The results are interesting and provide an additional perspective from which to view generalization. I am not familiar enough with the details of the related literature to know if this is the first work to consider this, but I trust other reviewers will know more. The paper also contains a very large number of results.", "weaknesses": "As written, this paper reads like a physics paper. In order to be published in a computer science / ML venue, I think it needs a bit more exposition describing notation, exactly the approximations being made in Section 3, and what the operators in 3.1.2 represent in the ML setting. \n\n1. Is $\\Delta J_\\text{test} = J_\\text{test}(w_0 + \\Delta w) - J_\\text{test}(w_0)?$\n2. What is the source of the noise $\\epsilon$? Is it sampling noise, label noise, general?\n3. It would help to break up Section 3 into Theorem statements.\n4. It would help to have prose at the start of Section 3, 3.1 describing what those sections do.\n5. Consider separating out preliminaries for notation and results. The authors should be more explicit about things like what the noise is, what exactly $\\Delta J_\\text{test}$ is, etc.\n\nIt seems like a nice paper, but I would recommend either submitting it to a different venue or writing it more in the style of ML theory papers for publication at an ML conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RnWTDT63E5", "reviewer": "Reviewer_zuhF", "summary": "This paper studies the use of loss geometry for both train and test sets, specifically the eigenvector overlap between the two, rather than just their spectra, as measure of loss geometry and to predict generalization. It shows that this measure explains covariate shift and multiple descent through a unified lens via simulations on synthetic data. It also develops an efficient estimator to show how this measure can quantify how class imbalance can induce misalignment in train-test loss geometry.", "questions": "Can authors add details on compute and runtime for the CIFAR-10 results in Section 3.4? Is it possible to use the proposed method for larger models?\n\nSuggestions to improve writing/readability:\n\n1. The paper uses hyphens instead of em dashes in almost every occurrence, and in some cases, it uses en dashes instead of hyphens, please fix.\n2. The paper uses $J$ to denote the loss. I suggest using $\\mathcal{L}$, or simply $L$, which is more standard. $J$ could be misinterpreted as denoting the Jacobian.\n3. In line 157, the expectation terms are missing square brackets.\n4. In line 162, the order of $H_{\\text{test}}$ and $C_{\\text{train}}$ is swapped. \n5. In line 215, it should be ‘significant’.\n6. In Eq. (8), it should be $\\delta_{s_2}^2$.\n7. In Fig. 1, panel (b) is missing the y-axis label, please clarify.\n8. In most cases, the word ‘traces’ is used to refer to the solid lines in the plots. I suggest simply using ‘solid lines’. \n9. In most cases the subfigures are referred to as, e.g., Fig. 1a instead of Fig. 1(a), please fix.\n10. Fig. 2 caption states “Traces in panel a) correspond to gold and blue lines”. Please clarify.\n11. In line 323, the phrase “geometric cartoon” should be rephrased.\n12. In line 377, ‘2d’ should be ‘2D’.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper makes interesting contributions to use eigenvalue overlap between train and test Hessians, rather than just their spectra, as measure of loss geometry and generalization, and presents a scalable numerical estimator and empirical validations.", "weaknesses": "The paper is missing an expanded related work section with a detailed discussion on most closely related works. As someone who is not very familiar with the literature on random matrix theory, and eigenspace overlap estimators, it is hard to judge the novelty of this work relative to other works. It would be good to add a more detailed discussion comparing and contrasting the work with most closely related work.\n\nThe experimental methodology in Section 3.4 is somewhat strange. It is stated that “a CIFAR-10 trained ResNet-20 was obtained from Chen”, and then “5000 train and test examples were randomly selected to define train and test Hessians”. A more convincing experiment would be to train it from scratch using the selected train set samples. Additionally, why not use imbalanced train set with balanced test set to compare the effect of imbalance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzGlANJ57z", "reviewer": "Reviewer_GPhb", "summary": "The authors derive a universal local fluctuation law considering the train/test loss Hessian spectra and overlaps of their eigenspaces, and apply it to ridge regression, trying resolve the puzzle of multiple descent.\nThey also empirically validate their theory with MLPs.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper provides a new perspective on the importance of the eigenspace overlaps.\n- It explains the multiple descent phenomenon from this perspective.", "weaknesses": "- As (3) is very important equation in the paper, I'd like to know how to obtain the equation in detail. The explanation is very unclear to me.\n    - The approximations of (1) and (2) are not clear. In what sense, they are approximated? (e.g., $O(\\\\|w-w_0\\\\|^2), O(\\\\|\\epsilon\\\\|^2)$)\n    - What is the definition of \n        - $J_{\\text{train}}(w,\\epsilon)$ (Is it $J\\_{\\text{train}}(w+\\epsilon)$?)\n        - $H_{\\text{train}}$? If it depends on $w$ as written in the paper ($H_{\\text{train}}:=d\\nabla^2 J_{\\text{train}}(w,0)$), then (1) is not quadratic and $\\Delta w$ is not just $-H^{-1}_{\\text{train}}z$. \n        - $\\Delta J_{\\text{test}}$ (Is it $J_{\\text{test}}(w_0+\\Delta w,\\epsilon)-J_{\\text{test}}(w_0,\\epsilon)$? or without $\\epsilon$?)\n    - The approximations of (1) and (2), but equality in (3). How do we get the exact equality in (3) from (1) and (2)?\n- The caption of Fig 2a says \"$J_\\text{{\\color{red}train}},\\Delta J_{\\text{test}}$ and bias\" but in the panel it says \"$J_{\\text{test}},\\Delta J_{\\text{test}}$, Bias.\"\n- em dash? (L41, L222, L262, L348, L353, L361, ...)\n- errata? $\\mu_\\Sigma = p_1\\delta_{s_1^{\\color{red}2}}+p_2\\delta_{s_2^2}$\n- What is $\\tilde\\lambda{\\color{red}'}$ in (9)?\n- errata? (L265) Isn't it $s_1^2,s_2^2=2^0,2^{-4}$? The eigenvalues in Fig 1(b) should be $s_i^2$ as written in (8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mtTVO3o0GN", "reviewer": "Reviewer_tXsV", "summary": "The authors first formalize the local fluctuation at a local minimum of a model, and then the authors analyze the eigen space of the Hessian of the training loss landscape during these fluctuations. Finally, the authors relate the training Hessian to the Hessian eigen space of the test loss landscape to explain loss behavior during covariate shift and commonly studied optimization behavior, such as multiple descent.", "questions": "1. Perhaps the progression from eq 1 to eq 2 can be improved with a slight introduction on how eq 2 is achieved? Similarly, this can be improved through the development of eq 5-7 (or refer to the appendix). \n2. On line 156, the authors assumed $\\mathbb{E}z = 0$, is this assumption realistic during scenarios such as covariate shift? If the noise $\\epsilon$ is caused by covariate shift while the training data is centered, does this mean $\\mathbb{E}z \\neq 0$?\n3. How should one intuitively understand the paragraph from lines 214 to 217. Does this essentially mean that if the noisy part of the train landscape aligns with the more important part of the test landscape, we will see increased loss value during evaluation?\n4. In Proposition 1, can the authors explain conceptually what *\"X is free from A, B\"* means? Does it simply mean independence?\n5. Figure 1 a) seems confusing. What is the purpose of the red contour in Figure 1a)? How are they related to the cyan? Figure 1 b) and c) are cleaner and more understandable. \n6. Figure 3 b), does each color of the contour mean a specific ranking of eigenvalue?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The reviewer believes that the contribution of this paper is strong. Through analyzing the loss landscape between training and testing, the authors explain the behavior of a couple of important scenarios in modern deep learning, such as covariate shifts (commonly happen in applications), and multiple descent (commonly observed during optimization of over-parameterized models).\n2. The author not only provides analysis and empirical results with synthetic training/testing environments but also extends their findings to more complex datasets such as CIFAR. \n3. The work is novel.", "weaknesses": "Minor:\n1. The related work seems to slightly lag behind, for exammple, regularizers that encourage cross-domain invariance have recent advancements such as [1]. Maybe the related work can also touch on sharpness-aware optimization and other robust optimization techniques that focus on controlling the gradient and the Hessian. It will round out the related work nicely to tie it back to the more application side of machine learning. \n2. A suggestion on notation: in eq 6, a notation $q$ is used, but it was only defined until the page after. Then also a parameter $\\alpha = q^{-1}$ is used through out the paper. Perhaps the author can tidy up this notation and stick with $\\alpha$\n\nMajor:\n1. The presentation of the work can be improved. The reviewer sincerely hope that the authors can use the additional page during rebuttal to strengthen/clarify some of the theoretical points and figures in the manuscript. See following for details. \n\n\n[1]: Hasan, Ali, Haoming Yang, Yuting Ng, and Vahid Tarokh. \"Elliptic Loss Regularization.\" In The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)