Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting
Authors: Jiping Li
Rishi Sonthalia
Venue: ICLR 2026 Poster (2026)
Keywords: Random Matrix Theory
Spiked Model
Linear Regression
Generalization
Primary area: learning theory

Abstract: This paper analyzes the generalization error of minimum-norm interpolating solutions in linear regression using spiked covariance data models. The paper characterizes how varying spike strengths and target-spike alignments can affect risk, especially in overparameterized settings. The study presents an exact expression for the generalization error, leading to a comprehensive classification of benign, tempered, and catastrophic overfitting regimes based on spike strength, the aspect ratio $c=d/n$ (particularly as $c \to \infty$), and target alignment. Notably, in well-specified aligned problems, increasing spike strength can surprisingly induce catastrophic overfitting before achieving benign overfitting. The paper also reveals that target-spike alignment is not always advantageous, identifying specific, sometimes counterintuitive, conditions for its benefit or detriment. Alignment with the spike being detrimental is empirically demonstrated to persist in nonlinear models.

Published as a conference paper at ICLR 2026
RISK PHASE TRANSITIONS IN SPIKED REGRESSION: ALIGNMENT DRIVEN BENIGN AND CATASTROPHIC OVERFITTING
Jiping
Rishi Sonthalia
Department of Mathematics University of California, Los Angeles jipingl103248g ucla. edu
Department of Mathematics
Boston College
rishi wontha edu
ABSTRACT
This paper analyzes the generalization error of minimum-nomm interpolating so- lutions in linear regression using spiked covariance data models. The paper char- acterizes how varying spike strengths and target-spike alignments can affect risk, especially in overparameterized settings. The study presents exact expression for the generalization error, leading to a comprehensive classification of benign, tempered, and catastrophic overfitting regimes based spike strength, the pect ratio d/n (particularly as c - oo), and target alignment. Notably, in well-specified aligned problems, increasing spike strength can surprisingly induce catastrophic overfitting before achieving benign overfitting. The paper also reveals that target-spike alignment is not always advantageous, identifying specific, some- times counterintuitive, conditions for its benefit or detriment. Alignment with the spike being detrimental is empirically demonstrated to persist nonlinear models.
INTRODUCTION
Understanding the generalization error of overparameterized models is central challenge in modern
machine learning. Phenomena such as double descent (Belkin et al. 2019 Hastie al. 2022) and benign overfitting (Bartlett et al.] 2020 Mallinar et al. [022 siiler and Bartlett 2023 have
spurred research underscoring the critical role of the data spectral structure (Bartlttt et al. 2020
Dobriban and Wager 2018 Hastie al.) [2022 Kausik al. 2024 Mei et al. 2022 Sonthalia and
Nadakuditi 2023] Tsigler and Bartlett] 2023] Wang et al. 2024a). The spiked covariance model one commonly considered spectral structure Couillet and Liao 2022). In this model, the data matrix X = Z + A E dx comprising n data points in R is decomposed into a rank-one signal component ("spike") Z and isotropic noise component ("bulk"") A. Spiked covariance models emerge naturally in practice, for instance, in the features learned by neural networks during training Sonthalia al. (2025): Ba et 2022 2023 Damian al. (2022) ; Dandi al. (2024); Martin and Mahoney (202 Moniri et al. 2023 Wang et al. (2024b). While recent studies have examined benign overfitting in spiked models Ba al. 2023 Kausik al., 2024). they lack a systematic taxonomy spanning spike strength, spree alignment, model misspecification, and train-test covariate shift. This paper closes the gap for linear regression.
This work explores how general spike sizes and target alignments affect generalization error in least squares linear regression. We consider targets y generated by:
y az,3 a. +
Here, E Id represents the signal (spike) component, component, is observation noise, and B3, E Rd. The coefficients az and a model the target's dependence the spike and bulk components. Notably, if a # azz the targets are non-linear functions of æ z + a, introducing model mis-specification. We address two fundamental questions:
Rd corresponds the noise (bulk)
Q1: For a fixed aspect ratio c = d/ n, in the asympototically proportional regime, under what conditions does alignment of the target signal with the data spike improve impair generalization?

---

Published as a conference paper at ICLR 2026
Q2: In the lighndimensional limit where 0o, when do observe benign, tempered, or catastrophic overfitting regimes?
Contributions. We present precise characterization of the generalization performance of minimum- norm interpolating solutions in linear regression. Our exact risk decomposition pinpoints conditions for transitions between benign and catastrophic overfitting. This reveals nomena obscured by isotropic theories, clarifying how signal structure, data scaling, and overparame- terization shape generalization. Our primary contributions are as follows:
phe-
Precise Risk Characterization: We derive an exact generalization error decomposition (Theo- rem(5) into interpretable bias, variance, data noise, and alignment terms.
Comprehensive Categorization of Overfitting Regimes: We precisely classify benign, tempered, catastrophic overfitting regimes based on spike strength, overparameterizaaion (c d/nn, and target alignment (Table []. Surprisingly, for well-specified aligned problems, increasing spike strength can induce catastrophic overfitting before achieving benign overfitting. Misspecified problems show distinct transitions, often precluding benign overfitting.
Challenging conventional wisdom, we show spike align- ment always beneficial and depends on spike strength meeting critical thresholds (Table 2 For misspecified problems, beneficial alignment requires az / / in a specific, non-trivial range. Counterintuitively, very strong spike dependence (az/aa)) can render alignment detrimental.
Conditions for Beneficial Alignment:
Empirical validation confirms our theoretical phenomena, including surprising negative alignment impacts, persist in nonlinear models, underscoring broader relevance.
Empirical Validation: D
Benign Overfitting in Linear Regression. Significant research has explored benign overfitting in
linear regression (Bartlett et al. (2020) Cao et al. 2021] Chatterji and Long, 2021 Karhadkar et al.
2024
Koehler al. [021]] Liang and Rakhlin 2020 Mallinar al. [2022] Muthukumar et al. [2020
Shamir 2022 Tsigler and Bartleu] [023) Wu and Xu (020). Many studies assume a uniformly
bounded largest covariance eigenvalue or lack precise characterizations of its interplay with target
alignment and generalization. Our work allows this eigenvalue to grow, offering precise performance characterizations based on this growth and alignment. While Kausik al. (2024) considers spiked models, their focus is on noiseless, well-specified scenarios with specific spike scaling. Our analysis broader, encompassing observation noise, misspecification, and general spike scaling.
Many prior works (Karbadkar et al. 2024: Shamir 2022 Tsigler and Bartlett, 2023) on benign overfitting with low-rank signals plus isotropic noise require -raoogotttgy between signal and noise, sometimes imposing strong conditions like d = Q(n n).. We instead consider the proportional regime d/n -> e @(1), subsequently examining c <> oc. This setting is morally similar to allowing d = wn) and aligns with approaches like (Karhadkar al. (2024) which, for classification, shows misclassification probability can be upper bounded by Ce-d/n d/n vanishing as d/n + 0o.
Generalization Error with Spiked Covariance. While recovering spike properties (Sonthalia and Nadakuditi] 2023 Kausik et al. 20244] Nadakuditi 2014: Benaych- and Nadakuditi in spiked models (Kobak al.) 2020, Ba al. 2022 2023 Mousavi-Hossernee et al.) 2023) Moniri et al., 2023 Freeman 2025) are active research areas, existing analyses often characterize generalization implicitly (e.g.. fixed- -xint equations) or focus on specific spike strengths/alignmenn
2011. 2012) and analyzing generalization
In contrast, we provide explicit, generic formulae for gemeralization error, enabling precise categorization of overfitting regimes and conditions for beneficial spike aligument.
We further discuss connections to selected prior works, direct theoretical extensions to other settings,
and proof techniques in Appendix [
Notation The subscript on o, O, wr, Ω2 O will denote which quantity is being sent infinity.
OOr code is available at the anonymous GitHub repository: link
2

---

Published as a conference paper at ICLR 2026
Table 1: Asymptotic Generalization Regimes. This table summarizes conditions for when over- fitting is benign, tempered, or catastrophic in the limit where d/n - and subsequently c - co. The behavior depends on the spike scaling relative to the bulk, target alignment (3, relative spike direction u), and larget specifications A, az (train) and a üz (test). Here, 0 quantifies the scaled spike strength and r² the scaled bulk variance; the two primary scaling regimes are operator norm based (0² 2) and Frobenius norm based (0² = d. The wi, O, θ are all as we send + co.
Scaling
Benign
Tempered
Catastrophic
Well-Specified, No Covariate Shift: = ÔA = =z
â2 = a 0
0- =77²
7 = w G.
All other cases
2 7 We (1), 1 oa(c2)
8 dr²
Gx u
B, k u
Never
Misspecified, No Covariate Shift: a = a,, az = āz, a # az
0² =7²²
Never
All other cases
oc(c²) 27 (1), B. L
0 27
Never
Always
Never
Misspecified with Covariate Shift: a, + âA or az # āz
az + oz B, 1L u, 7 wc(l)
0 =
Never
All other cases
or ag à B, u.
(1 < 7 S oc(c2)
0² = =dr2
0g = âz ŌA.
All other cases
ag * âz and B, L
G. u
Spike Recovery: aı = 7 0, az = āz
(Appendix D
0 7T²
77² (1)
= 01(1)
77² = (c
0² dr
7² (1)
72 - =1(1
Never
PROBLEM SETTING
We study the generalization of minimum-norm interpolators in high-dimensional linear regression. Using a spiked covariance data model, we quantify how spike strength and alignment influence generalization and the emergence of benign, tempered, or catastrophic overfitting.
Data Model. We consider data matrix X Z + A E _ with signal component Z and isotropic noise component that satisfy the following assumptions. Specifically, we shall that the population feature covariance is Σ 8² uuu + I2I modeling rank-one perturbation of isotropic noise.
Assumption (Signal). Let u E Rd be a fixed unit vector representing the spike direction. Then
Z = 0uv
where 0 > 0 controls the spike strength, and the vector v E IR has i.i.d. standard normal entries.
Assumption 2 (Noise). The entries of A have zero mean and variance -2. The matrix A satisfies:
Its entries are uncorrelated and possess finite fourth moments.
Its distribution is invariant under left and right orthogonal transformations.
The empirical spectral distribution of 72d AAT converges to the Marchenko- Pasttr law as n, with d/n E (0, oo).
Spike Strength Normalizations. We consider two key scaling regimes for the spike strength relative to the bulk noise. These lead distinct generalization behaviors.
3

---

Published as a conference paper at ICLR 2026
Table 2: Conditions for Beneficial Spike Alignment at Finite Aspect Ratios (c d/n). This table outlines the specific regions where alignment of the target signal with the data's ' principal spike direction improves generalization. Conditions depend on the problem setting (well-specified vs. mis-specified)) the spike scaling regime (operator or frobenius norm based), the overparameterization level c = d/n, and the relative dependence of the targets y on the spike versus the bulk a az/ a
Setting
Alignment Beneficial Region
Well-Specified, Operator Norm Well-Specified, Frobenius Norm
>>
c e>
Misspecified, No Covariate Shift, Operator Norm
3c 3x2- (c"
Misspecified, No Covariate Shift, Frobenius Norm
2
A
1. Operator Norm Scaling (02 = =)r²); Here tunes the spike strength 0 relative to the noise comparable to
variance 72. When y = (1+ ve)2, the spectral norm of the signal component that of the noise component A. If y > (1 vc)², the spike emerges as an isolated eigenvalue beyond the bulk spectrum established by A, a phenomenon known as the Baik-Ber trous-Péché (BBP) transition (Baik et al. 2005). This scaling reflects spikes in learned neural network features
(Ba et al. 2022
Moniri et al. |l.] 2003).
2. Frobenius Norm Scaling (02
dr3): Here 8² dr matches expected signal and noise Frobenius norms [E]I) 7-] = [I|l ( and the spike has macroscopic proportion of the energy. Such strong signals can lead to improved sample complexity, potentially overcoming limitations observed in purely isotropic models (Ba et al. 2023 Mei et al. 2022).
Target Model. Given 24 the targets are obtained as follows:
= az B_ + a,a, T B, +
(2)
fixed constant) the true underlying parameter vector. The terms 2; and a are the i-th columns of Z and A respectively. The observation noise & are i.i.d. with E[ey] = O, Eie²] = - The coefficients az A E control the target's dependence on the signal and noise components. If az + a. the true data generating process for y differentially weights components of x,, causing model misspecification.
where B. Rd in uniformly distributed the subspace {B E
Generalization Risk. We study the minimum-norm interpolating ordinary least squares estimator: with
Bint X X'y. y.
= (2 + a
where X denotes the pseudoinverse. Given a new test data point (æ, 1), where = + and targets ăz 2 B. āΔă² B, ह with potentially with different coefficients āz a and model parameters Fe. the generalization risk defined as the expected squared prediction error:
R(BBnr) Ex,, (ū- 5) Ex.e.a.e [Ø T B,nt)
The expectation is over the training data (X, .) and the test data realization (æ, E). We shall denote the asymptotic excess risk in the proportional regime as follows:
Re n,d-+o0, d/n->c
lim R(Bmn)) 72
Remark 1 (Generalizing Prior Work). This problem formulation encompasses several existing models as special cases. For instance, isotropic regression settings studied in Hastie el al. (2022) are recovered by setting 0 = 0 (no spike) and az = 0. Spike recovery models, such as in Sonthalia and Nadakuditi (2023), correspond to specific choices like -2 = 1/d, +? = O, and a = O. Our generalized setup allows for a nuanced investigation of the interplay between signal structure, target alignment, and overparameterization.
Remark 2. Although stylized for theoretical simplicity, our setting naturally arises as a low-order Hermite approximation of nonlinear multi-index models, captured by (a.A, Ozz) prrmmettrrri....nnn
4

---

Published as a conference paper at ICLR 2026
Excess Errur vs Overpatameeericaaoon Ratio
Excess Crror vs Overrarrmmeteeizaiiio Retio
Aliane
23.0
stgred
20.0
37.5
Ex00G Eroor
25.0
32.-
10.0
73-
50-
Aggpet
And-a Agned
50-
20
dio
dio
(a) Operator norm scaling (02 2. Alignment ini- tially improves generalization, but have catastrophic risk as + oc. Anti-alignment yields tempered risk.
(b) Equal Frobenius norm scaling (0² dr Align- ment leads to benign overfitting, while anti-alignment results in tempered risk.
Figure Excess eπrог vs. overparameterization ratio e d/ in the well well-specified case. Each plot shows the risk for aligned and anti-aligned targets under different spike scaling regimes. The scatter plots are empirically obtained and the lines are theory.
This essentially makes our model tractable surrogate for a broad class of nonlinear targets, while still being simple enough to allow complete, closed-form risk analysis. Additionally, recent one-step Peature-learning analyses like Moniri et al. (2024) show that learned features features are polynomials of the inputs and the learned spikes. Motivated by this, we study arguably the simplest non-trivial polynomial target
O,A + az
Quantifying the Benefit of Alignment. A key aspect of our investigation is to determine when the alignment of the true parameter vector 33. with the data's s principal spike direction beneficial for generalization. We define alignment beneficial if the generalization risk R(Bine) (or R.),, is monotonically decreasing as a function of (3. 2²² E [0, 1]. Conversely, alignment is detrimental the risk is monotonically increasing function of (3_ u)?.
Characterizing Overfitting Regimes. Following Bartlett et al. (220)); Mallinar et al. 2022), we classify the asymptotic behavior of the excess risk, R₂ as We say the overfitting is benign if lim Re is zero, tempered this limit is positive and finite, catastrophic if this limit is infinite.
as benign tempered or catastrophic.
3 THEORETICAL RESULTS
Our core theoretical contribution is precise analytical formula for excess risk in the spiked covariance model. This result relies on Assumption 3 which encompasses both the operator norm scaling (62 ??? and Frobenius norm scaling (02 dr?) regimes. We develop general risk theorem by analyzing progressively complex scenarios. Specifically, our forthcoming theorems provide specific conditions for benign, tempered, catastrophic overfitting (as oo), and determine when, for finite c, alignment of (3, with spike u is beneficial or detrimental.
Assumption 3 (Scaling). As n, d with d/n - E (0, we assume that 0 and 72 satisfy ((7 0 < <Odr?) and ?2 = @(1).
3.1 WELL SPECIFIED PROBLEM
We begin by analyzing the well-specified case, where the target is a direct linear function of the observed covariates X + A. This scenario realized by setting:
az ôz ă = 0.
Consequently, yi a... T B. ex, and the model is properly specified.
5

---

Published as a conference paper at ICLR 2026
Theorem 1 (Well-Specified Risk). Given data (X, y) and (X. generated according to As- sumptions (Signal), 2 (Noise), Equation (Target Model), and Assumption 3 (Scaling). If the well-specification condition az a az â 0 holds, the asymptotic excess risk R2 is:
ifc<1 ifc>1
R₂
T-C Ce c-1 +aa [18.1] 1|. u)² 0-+2 20222-200- c-00"
(0=+7-0)=
where u is the unit vector defining the spike direction. Remark 3.
with 7 o(1) (a regime not allowed by Assumption but useful for sanity checks), the coefficient of (3_ u)² vanishes, the risk expression aligns with that of isotropic models, such in (Hssii al. 2022 Theorem 1).
Operator Norm Scaling (02 = yT?), In this regime, the excess risk for > becomes:
R. = [ (1-2) [118.1P++ 7c² - 22.0 -220-1 72 (7 c)2 (B) u)² + c-
(1-
The formula shows that alignment with the spike direction u beneficial and only if the coefficient of (0_ 2²² is negative, which occurs when y c(c
2). We consider different scalings for 7
Case I: 7 = (1()) (constant with respect to c). The condition for beneficial alignment, 7 > c(c 2). interacts intricately with the BBP phase transition condition, y + vc)². Let c, 4.212 be the unique solution to e(c- 2) = + Vc)² for c
For 1 < < c:: Here, e(c-2) < (1 + voy2. If clc (00-2)2277 < <><<<<<>>> (1 (1+ ve)2 alignment beneficial even though the BBP transition has not occurred (the spike is not resolved from the bulk).
For c.: Here, c(c 2) > (1 + V For alignment to be beneficial (7 > e(c 2)). the BBP transition must have occurred (as y e(c 2) = Y > (1 + vo) However, the BBP transition occurring is not sufficient for beneficial alignment. If (1 + ve)² <<<<><<> clc 2). the BBP transition occurs, yet alignment detrimental.
Regarding the type of overfitting
(while y remains constant):
lim R. 7² (1 (BT u)²).
Since this limit a positive constant, we consistently observe tempered overfitting when y = O,().. Case 2: 7 (1) (r grows with c). The behavior depends on the growth rate of y relative to The limit of the excess risk for B. f a oo is:
if (1) STS S()(cc")) if = @c² for const. & if = wc(e?)
lim R2 2?? c-000
||3 2 1) 1 - 108_-(T u)
Surprisingly, while - 0.(1) gives tempered overfitting, increasing spike strength (1) S 7 S (c(c") results in catastrophic overfitting, even though morally, this version of the problem has less noise, Additionally, we see that this catastrophic overfitting not present in the anti-aligned (3) u) case. More, aligned with intuition, we see that further increasing the size of the spike improves the generalization performance. Specifically, we get tempered overfitting = áx2 and benign overfitting if y = we(c²). ).ce), 3. and || 1.
For the ( 2² coefficient is (c-3) -34 Thus, for < c < 3, alignment beneficial and for <>> 3, alignment becomes detrimental. As + oo, if [3, u, the excess risk grows approximately as 2 [ 2² indicating catastrophic overfitting. In contrast, 3. 1. u, the excess risk grows like - 1/c))1 leading to tempered overfitting. This transition is illustrated in Figure Ia
Intuition: This result is special case of more general result Theorem 5 At high level, the complicated phase transitions are a direct consequence of competitions among the four terms seen in Theorem In particular among (1) Bias, Variinnce, (3) Target Alignment, where (1) and (2) are non-negative (harmful) and (3) is negative (beneficial). For the well specified case, alignment completely cancels the variance, but not the bias, which then drives catastrophic overfitting.

---

Published as a conference paper at ICLR 2026
Excess Error vs Overperennezericatioon Ratio
Excess Error vs. Overpssiatreeericetion Ratio
Alggndd fent sigeed
Aliane stgeed
C-2
10
Exccss Entr
Excess Error
101-
Ciddto
(a) Under operator norm scaling (0a cr with ag = 1, 0ut = 2, alignment initially improves gener- alization for small c, but becomes harmful beyond a critical point, lcading to catastrophic overfitting.
(b) Under Frobenius norm scaling (0 vdr) with 4 = and oz 1.1, alignment remains better than anti-alignment across all c, but henign overfitting is not achieved unless ag a,--
Figure 2:
Transition from beneficial to harmful alignment under mild misspecificctiin. The scatter plots are empirically obtained and the lines are theory.
Frobenius Norm Scaling (62 = dr The excess risk for e > simplifies to:
Res1 = 227 1 -1- (118.12-(133 u)) + T c-
We have few observations. First, if Ba u and ||3.|| the excess risk R. tends to c - 00 (benign overfitting). Second, if B. is not perfectly aligned with u. RR - a²-21 [ (3, (3,=))3)) u) 0
as c - oo (tempered overfitting). Finally, the coefficient of (,3, u)² the risk formula negative. Hence, in contrast with the operator norm regime, alignment is always beneficial in this regime for e > 1, and we visualize these behaviors Figure
Takeaways for the Well-Specified Case. As result, spike scaling profoundly impacts overfitting, especially with target alignment, which is not always beneficial. For aligned targets, increasing spike strength drive transitions from tempered catastrophic tempered - benign overfitting, while anti-alignment (B, 1 u) can mitigate catastrophic overfitting.
3.2 MISSPECIFIED CASE AND NO COVARIATE SHIFT
We next consider misspecified targets y with differing dependence on spike Z and noise A feature components. Specifically, we assume az * a but introduce no covariate shift between training and test distributions, i.c., âz az and â Q. This scenario models situations where intrinsic feature properties lead to differential correlations with the target, a common occurrence in practice. For notational convenience, define A. az aa with A az OAA
Theorem 2 (Misspecified). Let Z, Ž satisfy Assumption according Equation (2). If Assuuptiion holds with az āz a aA then
1
A, À satisfy Assumption 2 and y. y
1 I-c
2 c + 72
(3.
u2
A² 0²
02472
C < 1
R₂-
-e +
2 (1 2) + u) A2
844726 c-1
0²
04-22
2
<>
A key observation is that misspecification (az a.a) can itself induce double descent, even if 7: 0. This contrasts with the well-specified case where, if T? = O, double descent absent. However, in the misspecified case, we do not observe double descent if there is alignment B._ g.u 0.
7

---

Published as a conference paper at ICLR 2026
Equal Operator Norm Case.
For 8 = 7T?, the excess risk is
AR 2 AB. I-c m+1 T² (3. u)
Te I-C t c-1
c<
R =
222 + +13+91133111 - + c>l
For 1, the spike is detrimental. For 1, the behavior depends 02/aa. In partieular, if
1 az a 1 c
3c2 + 20y 2c
C
(2)
then we have that the coefficient in front of (3]_)²² is negative. Thus, when az/ lies between these thresholds, the spike helps, but the spike harmful outside this range. As - 00, i = (c(c)). the beneficial region shrinks and alignment increasingly harms generalization. On the other hand,
the spike is big enough (7 = we(c))) we have that the beneficial region limits to 0 S a2 S
Figures 3a and
3b plot the coefficient of (33. u)² for e = 2 and c 20 for ?
The upper bound on beneficial / is surprising, as stronger target dependence on the spike might be expected to always favor alignment. Additionally, the dependence on the level of overparame- terization also offers new insights. Consider the example of 7 = c, and az/aaa = Then when e < 2 or C (9 + /57)/2, we have that the ratio outside the beneficial region. Figure 2a shows that in the beneficial region, the aligned risk is lower than the anti-aligned risk. However, outside the beneficial region, the aligned risk becomes strictly larger than the anti-aligned counterpart.
Next, in terms of benign tempered vs. catastrophic overfitting, we have that
72 [700 u)2 + a 3.11]
B. y u, 7 = 0c(1)
B. L u, wc S S oo(c²)
0
lim R.
72 3. 1 (az (1 (1+ -2azas) (3T u)²]
Ba v 7 66²
c-000
113 (az 20zan) u)
B. Y.u, wo(c?)
1332111111
B. 1
For B. y (1)) S oc()²) we have catastrophic overfitting If (c))). overfitting is tempered, with benign overfitting precluded (Appendix Proposition 3) If y = wc
overfitting is
again tempered with benign requiring returning the well-specified case (a az).
Ridge Extension. As done in Li and Sonthalia (2024), is also possible to analyze the ridge regularized version by looking at the vesolvent away from zero. Since the general proof structure remains the same, we leave this as a future direction and empirically test this case. In particular, we follow the setting of Figure Ia (02 cT²) and vary the regularization strengths (A {0, 1, c² de)). In all cases the catastrophic band in our phase diagram persists, indicating that generally regularization does not remove alignment- driven catastrophic overfitting. The plot can be seen in Appendix [..2
Equal Frobenius Norm Case. For 8² = d the excess risk becomes:
R = a2.1..11 (1-- + (BT u)²
c-1 az =a)² 20A az
az 2 - The beneficial
For >>> 1, the beneficial region for the ratio az/aA is defined by:
region expands with c, making alignment increasingly beneficial in extreme overparameteriiztton (Figure Bc). Beneficial alignment can also be seen in Figure 2bb Here az/a = 1.1, which is in the beneficial region for 10/9. Finally, the overfitting is tempered unless a az.
3.3 MISSPECIFIED TARGET AND COVARIATE SHIFT
also have covariate shift between train and test. Specifically, az * āz or a * â. hence we have the spike/noise importance differ between train and test. For the equal operator norm case, show the following.
Lastly, in addition misspecifation,
8

---

Published as a conference paper at ICLR 2026
Codilideat 20
Contitxtent of UTT far cw
or seateeg)
(a) Operator scaling, 2. (b) Operator scaling, Large beneficial region.
(c) Frobenius scaling, 1000. The heneficial region persists at extreme overparameterization.
20. Smaller beneficial region
Figure 3: Phase boundaries for spike alignment impact. Coefficient of (BI u)²² as a function of az a indicating whether alignment improves harms generalization.
Theorem 3. Given data Z, Z that satisfy Assumption / A, Å that satisfy Assumption 2 and y.
according to Equation (). If Assumption holds, catastrophic overfitting occurs if âz = 0g. B. u, and wo(1) S 7 (e()). Additionally, if + az with 7 (1) and B, u we get catastrophic overfilling. Other scenarios yield tempered overfitting
Different covariate shifts pose varying challenges. In particular, if az j āz, (target's spike dependence shifts), then catastrophic overfitting becomes unavoidable for sufficiently large spikes. This contradicts the earlier theoretical intuition, increasing the spike size in this setting actually induces catastrophic overfitting instead of mitigating it.
Equal Frobenius Norm. In this case, we have the following theorem.
Theorem Let Z, satisfy Assumption 1 A, Á satisfy Assumption and y, ÿ according to Equation If A/Assumption 3 holds and az f âz then R. = for all 1. For az āz:
lim R. 72 [G0] u) (a 26 [1
If az A āz, catastrophic overfitting occurs. When 3_ and are parallel, we have that 3211,,,11((22 aa)². This is benign if and only if az u.. Notably, if training data misspecified (a * az) but test data is well-specified and matches the training spike dependence (az = âz ãA). benign overfitting becomes achievable.
3.4 GENERAL THEOREM
Prior results are special cases of our main theorem (Theorem 5). We present a high-level decomposition here.
Its full form is complex (Appendix
Theorem 5 (Generalization Risk). Suppose Assumption
Assumption and Assumption 3 hold.
R = E i. 2 BT
02
B
(-26 A8, AAT Bint)
Variance
Noise
Target Alignment
Bias. This is the squared error between the learned predictor Bent and the true parameter B, projected onto the spike direction In particular, the risk penalizes discrepancies only along the top eigen-direction of the population covariance Σ, reflecting the anistropic influence of the spike.
Variance. The variance is equivalent to 1/ 2l This mirrors classical isotropic regression results (Hastie et al. 2022 Bartlett et al. 2020)), but the norm 1/ || itself is dependent upon the interaction between signal and noise, the alignment between G. and u, and the scaling parameters.
Data Noise. The data noise term quantifies the contribution of the noise matrix A to the target outputs y; through A- Even in the absence of observation noise (72 = 0), target corruption via data noise create an irreducible error floor.

---

Published as a conference paper at ICLR 2026
Orgmmu teats
(a) 02 = 0.1, alignment helps.
(b) 0g = 1, mixed hehavior.
(c) 02 = 4, alignment hurts.
Figure 4: Alignment-plan phase transitions persist in deep networks. Generalization enror angle between spike direction u and ground-truth parameter B. when fitting data with 3-layer ReL.U networks. The effect of alignment switches oz increases, consistent with the phase transitions predicted by our theory. Experimental details are Appendix B
(a) az Ca. 1.
(b) az 1, Ca
(c) Ch
(d) az
Og
Figure 5: Generalization error vs. alignment for deep networks. Generalization error vs. the angle between spike direction and ground-truth parameter B. when fitting MNIST-derived data with ReLU network, for sweep over (az, (.a) E {1,4]2
Target Alignment. The alignment term measures the inner product between 3 and B, with respect the sample noise covariance. This cross-term captures how mismatch between Bint and B., especially when mediated by A, can amplify or dampen generalization error.
3.5 NONLINEAR MODELS
While our theoretical focus on linear regression, key phenomena like ag dependent non-monotonic alignment effects appear in nonlinear models well. We test this by training 3-layer ReLU networks to predict y (Equation (2) given X, where we vary the alignment angle between spike u and B, and record the generalization error. Figure 4 shows our results for three ag values. For az 0.1, increasing aligmment with the spike is detrimental. For ag = 1, alignment is beneficial, while for oz 10, alignment is detrimental again. This mirrors theoretical findings that there is region for beneficial alignment and a nuanced phase transition for different oz values.
To move beyond purely synthetic inputs, we perform experiment MNIST where we artificially inject a spiked direction into the input space. Please see Appendix I more details. We then train model vary (a az) 4 and the target-spike angle. We again observe similar pattern: the phase transitions we recover potentially generalize to nonlinear deep networks trained on real data, broadening the theoretical robustness our setting. Results can seen in Figure
CONCLUSION
This work provided precise analytical characterization of the generalization error for minimum normm norm interpolators in spiked covariance models. We decomposed the risk into interpretable components and comprehensively classified overfitting regimes based on spike strength, target alignment, and overparameterization. We reveal surprising phenomena, such as the potential for increasing spike strength induce catastrophic overfitting before benign overfitting in well-specified aligned problems, and that strong target-spike alignment is not universally beneficial, especially under model misspecifi- cation. These geggnment-dependee phase transitions, theoretically derived for linear models, were also empirically observed in nonlinear neural networks. suggesting broader relevance. Our results offer a more nuanced understanding generalization in the presence of data anisotropy. challenging conventional intuitions and providing detailed map of risk behaviors in overparameterized settings.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jimmy Ba, Murat A Erdogdu, Tajji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptoties of feature learning: How one gradient step improves the representation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: http::://penreview..rrt wwwp:rewwwwww..eee.o.oee
Td=akddwRG6FG
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, and Denny Wu. Learning in the presence of low-dimensional structure: A spiked random matrix perspective. In Thirty- Conference on Neural Information Processing Systems, 2023.
id=H1 TAOCHDWW
Zhidong Bai and Wang Zhou. Large sample covariance matrices without independence structures in columns. 2008.
Jinho Baik and Jack W Silverstein. Eigenvalues of large sample covariance matrices of spiked population models. Journal of multivariate analysis, 9766):1322-1408, 2006.
Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. 2005.
Peter I. Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(49):30033-30077 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical biri-nariance trade-off. Proceedings of the National Academy of Sciences, 116632): 2019.
Florent Senaych-Georger and Raj Rao Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Advances Mathematics, 227(1):494-522 2011.
Florent Benaych- Georges and Raj Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis, 111:120-13: 2012.
Yuan Cao, Quanquan Gu, and Mikhail Belkin. Risk bounds for ver-parameteriied maximum margin classification on sub-gaussian mixtures. Advances in Neural Information Processing Systems, 34: 8407-8418, 2021.
Niladri S. Chatterji and Philip M. Long. Finite-sample analysis of interpolating linear classifiers in the overparameterrized regime. Journal of Machine Learning Research, 22(119): 2021. URL
http::/ org. papers hvtp://v2222222
html
Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022. doi: 10.1017/978100912228
httt::://hhenyaee liao github
[o/bbok//]
Alex Damian, Jason D. Lee, and Mahdi Soltamolkotabi, Neural networks can learn representations with gradient descent, 2022. URL wtt.httss../ax....oorr///////// /ttp/////2226611....
Yatin Dandi, Florent Krzakala, Bruno Lourciro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step time. Jouraa of Machine Learning Research, 25(449):
1-65, 2024. URLpttt:: //tmt:://mmr.... jmir........... org htw.:/ooggpppaper///2.. 21////22115114..4 1543. html
Samet Demir and Zafer Dogan. Random features outperform linear models: Effect of strong labut-label correlation in spiked covariance data. arXiv preprint (xXiv:2409220250 2024.
Samet Demir and Zafer Dogan. Asymptotic analysis of two-layer neural networks after one gradient step under gaussian mixtures data with structure. arXiv preprint arXiv:2503.0085, 2025.
Edgar Dobriban and Stefan Wager. nigh-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1): 1):27-277 2018.
11

---

Published as a conference paper at ICLR 2026
Jake Freeman. Shrinkage to infinity: Reducing test error by inflating the minimum interpolator in linear models. arXiv preprint arXiv:2510.19266, 2025.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Kedar Karhadkar, Erin George, Michael Murray, Guido F Montufar, and Deanna Needell. Benign overfitting in leaky relu networks with moderate imput dimension, Advances in Neural Information Processing Systems, 37:36634-3663 2024.
Chinmaya Kausik, Kashvi Srivastava, and Rishi Sonthalia. Double descent and overfitting under noisy inputs and distribution shift for linear denoisers. Trunsactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httpthtttp//ppeennee otw. ottpe//www.eeoooooooooooeoooooa. d=R
Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization. Journal of Machine Learning Research, (1616)):1-16, 2020. URL htt.:tt: //tmm//mmmm/ I www..ogppgpg.//////.
19-844 4. h
Frederic Koehler, Lijia Zhou, Danica J Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting, Advances in Neural Information Processing Systems, 34:20657-20668, 2021.
Xinyue Li and Rishi Sonthalia. Least squares regression can exhibit under-paraameterized double descent. Advances in Neural Information Processing Systems, 2024.
Tengyuan Liang and Alexander Rakhlin. Just interpolate. The Annals of Statistics, 48(3):1329-1347, 2020.
Neil Mallinar, James B Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Pree- tum Nakkiran. Benign, tempered, or catastrophic: A taxonomy of overfitting. arXiv preprint arXiv:2207.06569, 2022.
Marchenko and Leonid Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of The Ussr-sbornik, 1:457-483, 1967.
Charles H. Martin and Michael W. Mahoney. Implicit seff-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021. URL
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis, 59:3-84, 2022.
Carl D. Meyer, Jr. Generalized Inversion of Modified Matrices. SIAM Journal on Applied Mathemat- ics, 1973.
Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. arXiv preprint arXiv:2310.07891, 2023.
Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks, 2024. URL /ttth:://aaxiiii orq/ htq/abs// abs 2310.00000 07891
Alireza Mousavi- Hosseiii, Denny Wu, Taiji Suzuki, and Murat A Erdogdu. Gradient-based feature learning under structured data. Advances in Neural Information Processing Systems, 36:7 1449- 71485, 2023.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83, 2020.
12

---

Published as a conference paper at ICLR 2026
Raj Rao Nadakuditi. Optshrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage. IEEE Transactions on Information Theory, 60(5): 3002-3018, 2014.
Ohad Shamir.
The implicit bias of benign overfitting. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 448 478. PMLR, 02-05 Jul 2022. URL http:::/prrceceinnggggg////////t
mlr. wrwss/ v whw.mmaaaaaaaaaa..aaa..m.m
Rishi Sonthalia and Raj Rao Nadakuditi. Training data size induced double descent for denoising feed- forward neural networks and the role training noise. Transactions on Machine Learning Research, 2023. ISSN 2835-8856
http:/eww.eeeeeeeeeeeeeeeee Forum F
dd= piM.w..VVVVV
Rishi Sonthalia, Michael Murray, and Guido' Montúfar. Low rank gradients and where to find them. In The Thirty-ninth Amnual Conference on Neural Information Processing Systems, 2025, URL
htttps://gpenreview net forum? 2id=de
Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24(123):1-76, 2023.
Yutong Wang. Rishi Sonthalia, and Wei Hu. Near-interpolator: Rapid norm growth and the trade-off between interpolation and generalization. In International Conference on Artificial Intelligence and Statistics, pages 4483 4491. PMLR, 2024a.
Zhichao Wang, Denny Wu, and Zhou Fan. Nonlinear spiked covariance matrices and signal propaga- tion in deep neural networks. arXiv preprint arXiv:2402.1011 10127, 2024b.
Yimin Wei. The weighted moore- penrose inverse of modified matrices. Appl. Math. Comput. 122 (1):1-13, June 2001. ISSN 0096-3003. doi: 1010116500066:0000000000000 7/doi .oriiorg/// 100 1ew.1111600066222 3003 (00)
URL https
Denny Wu and Ji Xu. On the optimal weighted & regularization in overparameterized linear regression. Advances in Neural Information Processing Systems, 13:10112-10111 2020.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Problem Setting
Theoretical Results
3.1
Well Specified Problem
3.2
Misspecified Case and no Covariate Shift
3.3
Misspecified Target and Covariate Shift
General Theorem
3.5
Nonlinear Models
10
4
Conclusion
10
Notation
16
More Non-Linear Experiments and Settings
16
B. MNIST Data
16
B.2 Ridge Figure
16
Prior Works Natural Extensions
17
New Techniques & Possible Theoretical Extensions
17
D Spike Recovery Case
18
Proof of Theorem
19
Step O: Rescaling
20
Step 1: Decompose Error
Step 2: Simplifying Terms
21
Bias
21
333.2
Variance
................................
21
Data Noise
................................
22
E.3.4
Target Alignment
22
................................
1.3.5
Helper Lemmas
22
E.4
Step 3:
Random Matrix Theory Estimates
27
E.4.1
Step 3(a): Showing that basic building blocks concentrate
28
E.4.2
Step 3(b): Bounding the Higher Moments
33
E.4.3
Step 3(c): Bounding 7 moments.
36
E.5
Step 4:
Bounding the Expectation of Products of Dependent Terms
40
E.5.1
Step 4: Bias
40
E.5.2
Step 4: Variance
41
14

---

Published as a conference paper at ICLR 2026
E.5.3
Step 4: Data Noise
42
E.5.4
Step 4: Target Alignment
42
E.5.5
Bias: Helper Lemmas
42
E.5.6
Variance: Helper Lemmas
46
E.5.7
Target Alignment: Helper Lemmas
54
E.6
Step Upscaling and Asymptotic Risk Formulas
58
Probability Lemmas
60
Proof of Specific Cases and Overfitting
70
Proof of Theorem
70
Proof of Thheorm
70
G.3
Proof of Theorem
70
Proof of Theorem
15

---

Published as a conference paper at ICLR 2026
A NOTATION
Symbol
Description Role
Typical scaling / range
First used
d, n2
Data dimension and sample size
d,n
+ oc with d/n
Sec. 2
fixed (0, xc)
Aspect ratio d/n Noise variance in ambient bulk A Spike (signal) variance
Sec. 2
Sec. 2
0²
0² 77 (operator-norm)
Sec. 2
l² dr (Frobcnius)
Spike-to-noise ratio = 0²/ /-² (effective out-
[0. oo): critical line (1+
Sec. 2
lier eigenvalue)
v2)²
02; 0.A
Coeffs. weighting spike vs. bulk in targels y
((1)
Eq. (2)
0z 0.A
Same coefficients test data (covariate shift)
9(1)
Sec. 3
B.
True parameter vector
1 =
Sec. 2
Spike direction in data covariance
|ulla
See. 2
A,
Bulk noise matrix, rank-one signal matrix
Ag ~ N/0,+=/d)) Z
Sec.
Ouv
Label noise and variance
N(,,
Sec. 2
Table 3: Glossary of recurrent parameters and symbols. All ((1) constants are independent of n, d.
Other Notations. We use lowercase lowercase bold and uppercase bold A letters to denote scalars, vectors, and matrices respectively. We use || |l2 to denote the Euclidean norm if the argument is vector and the operator norm if the argument is matrix. We use || || denote the Frobenius norm. When slicing one entry from a vector or matrix, we use both a; Aij and a,, Aij. where the latter intends emphasize the source of the scalar.
B MORE NON-LINEAR EXPERIMENTS AND SETTINGS
We used 500 data points in 750 dimensional space, with a hidden width of 1000, We used full batch gradient descent for 100 epochs with learning of le-4, Each data point is averaged 50 trials. Equal Frobenius norm scaling was used for the size of the spike.
B.1 MNIST DATA
For each trial, we construct spiked design matrix by combining rank-one signal with "bulk" component drawn from MNIST. Fix sample size and ambient dimension d (here d = 784). First, we sample a unit-norm ground- truth vector 3. E IR and, for each prescribed angle, construct unit vector u E I2 at that angle to B.. Independently sample unit vectors v, E IR" and set spike strengths 0, Vnr. The rank-one signal matrices are
Z fuv Rdxn
Z, = = Igdx
The bulk matrices are generated by sampling n MNIST images (train split for training, test split for testing), vectorizing each image into 7 and stacking these as columns to obtain
A Rdxx and A. Eddn
with raw pixel intensities in [0, 1J. The training features are X Z A, and responses are generated noisy linear measurement of weighted combination of spike and bulk:
y 3. (az Z + AA + +E,
Network architecture. epochs.
We use the same network structure as previously, but trained for 1000
B.2 RIDGE FIGURE
Figure 6 shows the risk that still manifests alignment-drive driven catastrophic overfitting with ridge regularization.
16

---

Published as a conference paper at ICLR 2026
Missalignd risk vs aspect ratio
10
(k-0)
masangnuc, ridge ()
1A
mibodigns, ridge
masaigges
Tostt MSE
1.0
0.-
62
d/n
Figure 6: Catastrophic Overfitting even in the presence of Ridge Regularizer.
C PRIOR WORKS & NATURAL EXTENSIONS
Li and Sonthalia 2024). This paper mainly shows that least-squares regression can exhibit double-descent peak in the mnder-parameterizel regime, by carefully analyzing the ridge-regularized resolvent bounded away from zero. They provide two examples where these spectral/alignment properties move the peak to 1 and argue that existing double-descent explanations, which focus on c 2 1, are incomplete. Our setting recovers theirs when the regularization μl = O, az a = (equal dependence), and learning denoiser (target is the noiseless data X). Our model significantly generalizes beyond the data model with interpolating parameters a az uncovering different insights about overfitting behaviors.
Demir and Dogan (2024) and Demir and Dogan 2025). Demir and Dogan 2025 analyzes
the Gaussian-min mixture/spiked data after gradient step. In contrast, provide closed-form generalization risk for the minimum-norm linear interpolator. In particular, this paper serves as the motivation for studying their setting, which the analytically tractable "base case" that isolates mhike-alignmen mechanisms. On the other hand, Demir and Dogan (2024) considers whether substituting one activation function for another results in the same generalization error; does not characterize the error itself, which is the goal of this paper.
In terms of proof techniques, our paper requires more sophisticated bounding approaches, as we argue in the subsequent sections.
C.I NEW TECHNIQUES POSSIBLE THEORRTTICAL EXTENSIONS
Our analysis combines ingredients that, the best of our knowledge, have not been put together in prior work on benign overfitting or spiked regression.
In particular, we develop a mixed spherical hypercontraactivity argument that controls products of random terms involving both bulk and spiked components. Instead of assuming strict Gaussianity, we only require zero mean, unit variance, and mild rotational invariance; hypercontractivity up to order 2k then yields concentration for multilinear forms of the type that appear in risk formula.
Two natural generalizations from our data model include (1) adding ridge regularization and (2) incorporating general low-rank data.
Ridge Regression. As discussed above for and Sonthalia (202)) how regularization affects the data resolvent can be tracked and similar Stieljties transform argument applics. In particular, check their Lemmas 13-15 for modified statements that can be generalized.
expansion for pseudo- inverse in Meyer 1973). This result from perturbation theory allows us to decouple spike and bulk and makes the subsequent analysis more tractable. A Woodbury-style formula for pseudo-inverse (extending Meyer (1973)) exists in WWil (200T), and the same proof structure follows. We can then bound contributions along each singular direction, which requires significantly more bookkeeping.
General Low-Rank Data. Our proof requires
Snterman-Morrison-een
17

---

Published as a conference paper at ICLR 2026
Due to the significant effort involved rigorously doing these calculations. We leave this future work. However, we believe that an extension to ridge with multiple spikes feasible.
D SPIKE RECOVERY CASE
We consider the special case where the goal is to recover the spike direction u. In this setting, the target depends only the spike component Z, with no contribution from the noise A:
ã = 0. ag = ãz > 0.
Thus, the target y is proportional to the signal plus possible observation noise e.
Equal Operator Norm In this regime, we have that the risk is
7a3=2 Resl (1 - (1- +017+111 (3 2)² 1 - Te
70(c2 + 7)0322 (c ()- c)2 (B u)²
Re>1
72
c c-
Here again, we see that when y = (c)), we have tempered overfitting and (z(1) r S oc(c²), we have catastrophic overfitting and for (c(c²) get tempered overfitting again.
Equal Frobenius Norm
In this regime, we have that
Re<1 = a2r² (B u)²
c- 1-c Res1
ca3 -2 c- (3T
u)²
1 -.
1
This generalizes the spike recovery setting studied in Sonthalii and Nadakuditi (2023). which assumed noiseless targets (7. = 0) and the equal Frobenius norm scaling. Our formula allows for observation noise and thus captures the more realistic case where the target y itself contains randomness not aligned with the spike. Here we see that we have tempered overfitting unless 7² o(1), which the case considered in SSonhallia and Nadakuditi (2023).
18

---

Published as a conference paper at ICLR 2026
E
PROOF OF THEOREM 5
Theorem (Generalization Risk), Suppose Assumption
Assumption
and Assumption 3 hold.
2 +
E
i. 2 Bin 2
22
3
A
+
B. A
Aā, AA Bint)
Rias
Variance
Dota Noise
Target Alignment
In particular, n, d do with d/n - CE (0, o0), we have the following expressions for each term.
Bias: For have that the bias term is
@2 (3TT))
(āz āz -z--az-t + (az - aa))
T² 0 + 72
+ +
c
c cd(02 + r2)
If e > 1, we that the bias term is
(B.. u)² āz az + (az a4 T2C
1| c-
02-2c
0 + 72 n(00 720)3
+
2 c
02 + 72c
aAA
(02 c2
Te
Variance: For c < 1, we have that the variance term is
0 + 0²
0
a 1 1|2 + 7 u)²
(az - a1)² aas2
2an an)
(02
722
82 + 72
+ 2 72
0
c
72
1
d(0
+2)
1-c
Fore have that the variance term
11
a
A
0
+ ++²(B.u))
Ø²
(az a.
0 + 72c d
(c (c- 1) 02 + T2
72
1
Ø²
T8 72
c
d(02 + 72c) c c-
Data Noise:
For all we have that
Target Alignment: For 1, we have that the aligmment term is
Ø²
( -2
aa) 0
(B. u) + taall3.12) u)
For >> 1, we have that the alignment term is
-2ā (o2- 0A)
Ø²
(BT u)² aal1,,111 (1
1 0²
02 72c
d 02 + 72c
Error terms: The largest error terms for all are:
o(1) + = o(1)
Remark:
We note that the above theorem very general and captures all of the theorems in the main text as special cases. It is worth noting that the theorem also incorporates different signal and bulk strengths for test data, namely for Ô and F.
The proof will be broken up into roughly steps
0. Rescale the problem To apply standard results rescale the problem. Section
+.
19

---

Published as a conference paper at ICLR 2026
1. Decompose the error into four terms. We shall refer to these terms the 1) bias, 2) variance, 3) data noise, and 4) target alignment. Section E.2
2. Simplify the expressions.
shall then use the result from Meyer 1973) to simplify the expression for each of the four terms. In particular, we shall express each term as the product dependent functions of the eigenvalues of X. Section
3. Random matrix theory estimate. We then standard results from random matrix theory such as Marchenko and Pastur (1967); Bai and Zhou (2008); Baik and Silverstein (2006) to obtain closed-form formula of the building blocks for the risk. Section E.4
4. Bound Products. We then show that products of our building blocks concentrate. Step (Section E.3 then collects the final terms.
5. Undo Scaling Step (Section E.6 gives us back the correct scaling.
Section has some generic probability lemmas that we need.
E. STEP O: RESCALING
In order to better align with existing results and use them accordingly, we change our scalings for now and switch back after our derivation. That is, we divide everything by vd. Hence, we shall use
0 uw = 0
|| vd
w
dd
w
as the spike. We shall let
1/2 = 0 |
w
and
d
Here, treat fixed unit norm vector and our spike is The noise after dividing by vd is
Z, uuur"
A, Vd
N
where are mean zero variance entries, Here the appendix, we shall use the letter p for т. Finally
let
X, = Z, A,
We can note that Bint, is still the solution to
|
B x,
where
CI, + A,) +
Vd
Vd
Vd
We define
- Vd = = N
(0,
T
d
Then when we want test, we shall look the rescaled error
(äz (āzZ
+ + BTe + A.)
Through Steps 4, we shall drop the subscript r.
20

---

Published as a conference paper at ICLR 2026
E.2 STEP 1: DECOMPOSE ERROR
Using the fact that has been zero entries and is independent of L. we see that we can decompose the error follows. Again here we consider ñ samples of test data and take the average (in expectation, this is the same as one test point).
I
3_ (a: 2 + aa
Binol 2 a)
1)1l3-
E
|0.3.2 Z- -B z
+ E
a Ā
BTns A
n
nt
2
(- AB AÄT Bint
3 2 3 kret
2
+
3
Br A
+
11
Variance
Target Alignment
We compute these four terms one by one in the following sections.
E.3 STEP 2: SIMPLIFYING TERMS
This section simplifies the four terms. We begin by recalling results from prior work. We state them here for completeness.
Theorem 6 (Theorems 3, 5 of Meyer 1973)) Define the following helper functions h T At k = Atu, t v
(I AtA), = + 170 A'u. = (I AAT ) 7/1 17 ||| + E
72 1f
|||| h
2 + c² and
n/|t|1
t n7
T =
AA_ h. k.
1/111 A'h h T nk,
n11/112
T h.
E
Then we have that
A k At
E Piq c<1
7 + A)*
A AAh
P2
c>1
The following subsections Bias F.3.1 Variance F.3.2 Data Noise F.3.3 and Target Alignment F.3.4
present the linear algebraic simplifications of the results. derive this results. We shall need some helper results that are presented in Section E.3.5
E.3.1
BIAS
Using Lemma we have that if c <
āz azz + E (az - (aa)
B + n E p.O
a.B. 7 BTT 2 =
7
7
and if 1
ā. 2 TT 2 BT - az) (azI (azI-aAAA!) Z-az
n|| B h
T
T
12 n
E P22
72
72
The bias equals the expected squared norm of this term (divided by n).
E.3.2
VARIANCE
Lemma
gives that
12202
Po3
E
BT kit
A
3 Z + A) Z3, + B, A(Z + A) + +A)TTA
d
27
a
+2
3. z + A) + A)TT AT B, + _ T( + A) + +A)Te
d
21

---

0'p00 02/02 Published as a conference paper at ICLR 2026
E.3.3 DATA NOISE
The data noise term is the simplest to understand. Preliminary caleulation gives us:
2 á BT
A
113.11
1
= n d
d
E.3.4 TARGET ALIGNMENT
To understand this term, we first note that is independent of everything else. Hence we replace AÁ T with its expectation A I.
2 p aBB. Bint d
A0 B_ Bint-
E 3. AAT Bint
d
Since has mean-zero entries that are independent of everything else. We that
Ee B. ((a28) Z (Z+ a. A(Z A)t) ]
Fe Bine]
(5)
=3 (a.B, Z(Z + A) aA A(Z A))) = =.33. (Z + A) Z B. + a (Z + +A)+TA ATB..
(6)
(7)
E.3.5 HELPER LEMMAS
Proposition 1 (Proposition 2 from Sonthalia and Nadakuditi (0033) ))
In the setting from Section
y uh + 1/112112 uk' At, c<1
( A)
uh + 1/1
72 us e>l
72
Lemma 1. Ift & 0 and A has full rank, we have:
i5 1T
più
c<
T Z + A)
Ti
e>
1772
Proof. After substitutions, Proposition
I
implics that for 1, T
A) Ž becomes:
E
A
7
A
E
Pi
[_||e|l?
k
A
- 11)) z
71
= rjj
AuuūT
+
TT
1)
AluōT
E n/|t/l2
A'u hu uT
by 2 = kuu
E
&
Since
Alu and hu
T Alu
5-1
we then have that
A'
+
11 TT K
Al ATuōT
5
nlltl1
T AA'u
hu σT
{
71
E
ñjE T
kō T
+
1| 12
TūT
t
+
E
17 1 |A
+ g2
-
711
En
ij T
Kō T
+
7|lk 12
T
6
+
1
P1
71 E
bT
E
71
"1
1/
5
ñ T
t
+
1k
PIū
PIūT
E
1771
"
i
+
P
pIù
17
m
mi
For > 1, we note that the calculation exactly the same. An example of such a calculation can be seen in the proof of Lemma
22

---

Published as a conference paper at ICLR 2026
Lemma 2. In the setting of Section 2 we have:
I 1 uh + Illle uk A
<< 1
A(Z + 4)
AA + " h s
o [1
hTh +fIln]
AAtuST 25 AAAuh uhu <> 1
Proof. For e < 1, Z, A are d x n with d < n. Since A is assumed to have full rank, Z + A has full rank with probability I, and hence
Z A) + A)* =1.
Thus, from Proposition
7 uh
1//10/12 uk" At.
A(Z A) = (Z + A) + A) Z + A)
71
For 1. since (Z + () Z A)T is no longer the identity matrix, directly expand using Theorem
5
17² s
(nlhil 8 + +1) :1))
A(Z + A) A
A + h T
Ah + +nk)
72
&
AA
T
5.
AA m
(milbal?
8 + 12)
72
&
Noting that TA AAI
AT
A
h
we have
A(Z + A) = AA
+
1
TT
1/
||
+
7AAiU
8 + h
{
72
E
AA +
! TTsT s
h S
TT
h h
n 1211
AA us
TE Y2
AAA AAtuh.
{
E72
72
72
We can combine the coefficients in front h T
to get
7
77th
1|122
7
12
+ 62)
17
|
1112
7
E
E72
872
72
The statement follows from here.
Lemma 3. I + and A has full rank, we have:
1 1-
&
BT 2 c<l
(. + A)
71
B Z
c>1
Proof. Using Proposition 1 for e and 2 iivo" T we have that
n
1211 uk
3. ( = BT
uh
A
Z
71
71
THE
uhuöö
T
1/|10|12 uk
A Aluuō
m
71
T
nh
uv T
17 1111 AuuōT
uk
AluöT)
m
Note & 1 TOoT.A A'u, AlukkAl
k k
|k||
The above equation becomes
n E(E - 1) 1/ 11 uō = (E(S-1)
2
7/1
71
7/
71
Using 71 1 | | + ?² to combine the coefficients, we have that
E(5 1)
n || ?
+ +
+
5
21
71
71
7/1
71
23

---

Published as a conference paper at ICLR 2026
This completes the proof for 1. Similarly, for 1, obtain
n5
1/111
A) 4)
B
uh
Us
Z
Y2
9/2
=//1111112
uhuū"
T
u.
uu
12
722
nh um
17
A A'wö
us
"0""
Y2
7/2
Note Ç - 1 = 1702
Atu,, s TyoTAtu.
= ||
The above equation becomes
713. (E(S
- (S(S-1)
uD =B. E(EE - 1)
17
|
z
7/2
72
7/2
7/2
Using 72 = =//||=/|2||||||
52 to combine the coefficients, we have that
E(5 1)
+
5² +
1180111111
-2+
5.
72
72
72
7/2
The target expression follows.
Lemma 4. E + and A has full rank, we have:
S. B. Z
c<1 AAIZ
B. A(Z + A)'2
n||l||
3 h u
7 +
LR.
e>1
Proof. We begin with e 1. Since A assumed have full rank, A has full column rank with probability 1, and hence
(Z + A)(Z + A)* = I.
It follows from Lemma that
A[Z + A) (. (Z A) + A)Ž 3. Z(Z A
-3 2 (1- - 3 5372. m
For c > 1, Z + A now has full row rank instead of full column rank. Hence, we do not have
ZZ () + A)t and need to directly expand using Theorem and its helper variables:
24

---

Published as a conference paper at ICLR 2026
B A(Z + 2) = B. A
A AAh's S
-292 T
72
kö + ?||$|12
5
113! A
Alh
p2d2
uv
72
i13. A
D20
P2
S
h
wi
72
n3 A
P20 T
n
1/ +
5
1
8T
P2
72
|//|| P
E²2
6
n3. A
P20
T
P2
7/2
En
n3. A
P20
5 P2
12 -
5
T
7/2
En
n3. A
P20
P28
p2
1712
TiE B, Ap2ū
1712
ÎHE B3
|n/1|=111
hT + n,Ak)
Tt
by plugging in the expression of P2
1/12
E
3. TT LB. AA'Z by ūkkō A "iuū"" T A 1. i|||$|1
Noting that B. hT is a scalar, we then introduce
= u and get that
im ||
/
G. h
B, h u 2
since iuu Z.
7/
72
Thus, the final expression is
8||8|| B,
E B.
h
u
Z
AATZ.
7/2
7/
Lemma (Bias Term). the setting of Section
2 we have that if c < 1,
E (az an)
1 E
àzB. 7 GTT 2 āz az
B.
p.ō
n m1
and if > 1,
".3. 2- -- 2 = B. (āz 1 az)
& (oAl- 2 a.
ž-a
n||8||
h.
h
1 ñ
E P
7/
12
1/
Proof. To simplify the bias term, first need the following expansion:
ā.3. 7 BTT =. 7 (3. (a. Z A, + R. 2 o 2( + A) a A(Z A) - TT A)2.
[Z T + A)'
From Lemmas we get simplified expressions for T (Z+AA) Ž. B, A(Z + +A) 2.3. A)t and plug them m. For c < 1, we get
āz 7 az 1 (1-5 5. B. as 71
i 5 E PTO
B.. 7
71
n
{ (az ann
áz az
BT. BT 2
T
?
71 m
25

---

Published as a conference paper at ICLR 2026
On the other hand, for have
āzB² % az 1
B
aa
/|
B h u
+
SBT. AAT
?2
72
7/2
11 7/2
$
0|
$
= B. (āz az)
(azl
2 2- 0x1
B. h. T u 4
T p28
7/2
7/2
72
Lemma (Squared Norms of pl and p2). Recall p
n/1
and p2 =
211
Aih
np..
1. P1 1|
52
71-
2. [p 12 |'||=||*
hATT Tth 2n"|| k.
Ath 1
Proof. For P1, we have
|||||3= (- 11 || 1 n/ke)
(-
-
|
||e112 + 15" 1/111s1
1/k
tk + 1A11
E
E
Using tk yields the first result, which we can further simplify as
|
1/211 g² 71
(2)
g2
For p2, similarly, we have
17 1/
1111=11
12212 =
hA iT - -nk""
A'h T -njk
|||1 g²
hAIT A + 2xy² 1/|2
k A 1111.112
Lemma (Squared Norms of qi and 92). Let qा
||| &
kT At h and qob 211h112
S h.
1//118112
20|14112
1. 1|=
k
TTA
kJ Ath +11112
E²
2. || 2--22-
|
Proof. Similar Lemma
6
directly expand the two terms:
2 (- 1/ k A
1n) (- (-mlte)l ATT k h h7)
17111111
27|16112
l|q
k
ATATT
& +
T
TTT + 1111122
E
E2
E
1| (-1))
[_]]|/bll?
n/ n/|h1||2
h_
+
|
since hs
{
{
2
|/|l
(17
1
12 ||||||| + +2)
g2
11/12
722
E2
26

---

Published as a conference paper at ICLR 2026
Lemma 8 (Preliminary Expansion of Variance). In the setting of Section 2 we have
2
7
E
B A
ER
3. 2 A) + A)T 2B.
B. A(Z () + A) A B.
P
2F 10 3. Z (Z A) A B,
72
ZZ ( + +A)TE
d
d
Proof. Since is independent the other terms, we replace ÁÄ with its expectation A I.
1 E The AA Bins
1 1 E
72 d TE [I] [
E
3 mn
ñ d
We now plug in the expression for Bint- Since e is zero-mean vector and independent from other random variables, terms with only one have zero expectation. A straightforward expansion gives:
72 B d (a. +E ET + AZ (B. (alaa++aaa +a +
After eliminating zero expectations as above, the expectation becomes:
72a2 B. Z(Z + () + A) ZB. + Pa B. A + A) + +A)T ATB. B.
E
Bint
=
E
d
d
d
27
a
B, Z + () + A)" A B.
72
(Z + A) IZ + +A)T
d
d
E.4 STEP 3: RANDOM MATRIX THEORY ESTIMATES
To do the estimates we recall the set up. In particular, we have that
Z nune" where 0 and
|||| 1,
vn
and the entries of
Aij = N (0. p²
Recall the following definition h
0 A1, k = Alu, t = 0 T - A'A), & = + tyo Alu.
(I AA u, ?1
r72
|| [1
+ E2, 72 |/|
1112 + and
|Ab1 t n/k
n||t112
P1
q.
TT
h.
E
1/ S
P2
Ath rikk
h
To show that each of the four terms, bias, variance, data noise, and target alignment concentrate in the limit, we do this in two steps.
(a) First, we compute the mean and variance for basic building blocks such ||h||2 and other variables. Section [.4.1
(b) Second, we provide bounds on the higher moments. Section E.4.2
Next, we prove bounds the moments of 74- Section R.4.3
27

---

Published as a conference paper at ICLR 2026
E.4.1 STEP 3(A): SHOWING THAT BASIC BUILDING BLOCKS CONCENTRATE
We begin by bounding the mean and variance.
Lemma 9 (Generalized version of Lemma 7 from Sonthalia and Nadakuditi (023)). Suppose
Asj have mean O and variance p² /d, the entries are uncorrelated, have funite fourth moment, the distribution is invariant under left and right orthogonal transformation and the empirical spectral distribution of f AA' converges to the Marchenko-Paate law. Additionally, if u and v are fixed unit norm vectors. Then we have that
e<l
and Var(11/b[2)= =O ((tnn) 73
1. [|]
e>l
to
(
p= c-l
1 c +o
1
and Var Var(]|||13) 0 (p.hn)
2. ||||+||]]
p² 1 c
p²
p+n
1
1
3.
||||>||]] and Var [| 0
1
4.
[[[[t|]]
= 1 and Var( O
n2
5
1
19=0(
$ 17
5.
E
and Var
17
17
max(n,, d) p²
(2
1
1
1 1/
6.
EE
+
+ max (n,, d) p²| -
+0
+
112
n
max(n,, d)p²
max(n, d2
and Var
112
max dd n)
Note that here max(d, n.). d, n are interchangeable in the variance big-Oh terms since they only differ by absolute constant We include the details for completion.
Proof. Items come from the original statement, which assumes unit variance. Here our variance parameter simply induces a multiplicative change. We now focus on item 6.
Let { = E/n = 1/n + v T A u. With = UVVT (SVD), A E R having i.dd p(O.pp/4 entries, and u. fixed unit vectors, we have ( = + E a, bea,, where min(d, n), V b U u are uniformly random on 5 and 5 respectively since U, V are random rotations.
"
Since A has zero-mean entries, only the non-cross terms remain in the expectation, and the fourth moment is
E(c*) = +
E (
E E(b,b, E|a;a,1 + Σ E
Elb,bybabilE Ela;a,ara]
17
2
R.j.k.
Furthermore, non-zero expectation terms require paired indices (since odd moments of the uniformly
random vector on the sphere equals 0). In particular, using exact spherical moments, we have B[a
x(n+2] E[aā] +. B[afa]]
(i + j). Ebb au+2
E[b = 1. E|b2b]] = 2(--2]
m2++2]
(i ( f):
1
1 d(d + 2)a(n 2)
E
+3 E
1
dn
d(d + 2) + 2)
0.
if
9 9Em
B[1/o|]
E[1/of]
1
d(d
+ 2) (n 2)
d(d + 2)n(n + 2)
2
dn
28

---

Published as a conference paper at ICLR 2026
Leading Order Scaling and Mean. Let N max(d, n), assume n, d with d/n - 1. Lemma from Sonthalia and Nadakuditi (2023) implies that if has unit variance entries, the moments of its inverse eigenvalue expressions of and are hence O(1). In our case, it will just scale with p instead:
E[ = 0(1/8"),
= and E[L/o - 0(1/p") etc.
In partieular, we also need the following exact expectation from the same lemma:
E
p² cl +
(8)
o?
p²
Since the above /, 13 have r = min(d, n) summands, this implies
- (2.r,)
_N (N=p²)=
r Is ===
N2 ) -0
(N/o)
1 = 0
=
NDt
Similarly, 12 has r(r - 1) R x2 summands, and
x²
I2 =
p+
=
N2
El5
+
1
+
since Ig dominates.
(9)
1
r7
max(d, n)pp
With a similar expansion for the second moment and taking spherical moments, we get that
En B[1/o
E[<3
+
Σ
E
Elb,bj]
172
dn
6,
min n)
1 1
by Equation g
1)
dn p² d
p
max(d, n) p² max n)p2
This gives the mean. Furthermore,
El= B[1/of]
E B[1/o7))
1 7
(
1 max((, m
(E 17
+0
(10)
1/2
dn
d2n²
Variance. Var(<?)= = E[5-1 - (1] From Equations the dominant term:
10 9
the overall scaling is determined by
)
5 2 =0
(
1
Var
max((d n)p²
17
Lemma 10 (General Terms). In the setting of Section 2 we have the following expectations:
1. For e < 1, BIT uk AtG.)
111-0 (3Ju)² + (+) and the variance is O(1(pAd)
2. For c < 1, T AT
(11-F + o (+)) and the variance is O(1((p&))).
3. For 1, E su T [.. = - =-]==11(3) (3,u², u)2 and the variance O(1/d).
(37
4. For 1, E AAtusT ATt) = -== [ u)) + o(1) and the variance is O(1/d).
5. For c 1, EB hT h,3.] 11..11
P-c-Ij + o and the variance is ((1(()=)))
6. For e > 1, EthAT TT A'h
fe-15
+
and the variance is ((1/(pd)))
29

---

Published as a conference paper at ICLR 2026
7. For c > 1, B||||| = + o ( and the variance is O(1/(pn))
Proof. For all these terms, we evaluate the expectation using the SVD A UEVT. with A VETUT, and important expectations from Lemma 5 of SSonthalla and Nadakuditi (2023) regarding the spectrum of A: suppose A has unit variance (general p² is multiplicative change), and let a, (A) denote the i-th singular value. We have
I-c (1)
c c<1
(1-c)a +o(1 c<1 (0-1)3 o() c>1
0. (A)
+
e>1"
(A)
e<1
[I-ofa +
e<l
p I-c
p
E
p
p
(A)
+
c>1
(A)
1c-133 +
1/2
c>l
(1I)
For the first term, we note that
B. uk T ATG. = -3. u)u ATT AiB.
= -3T (BTTuuu u)u TEIT TESTEUTT
ETUTB. 3.
= (BTu) u) i=1
u TU)(U (x): o7(A)
(3. u) Σ (uTT.))B ui) o?(A))
1
where u, denotes the i-th column of U. We further note that u 3. = u UU B.. Since permuting columns an orthogonal matrix does not break orthogonality and U is uniformly random, we have that the marginals u, are identical. Thus, we have that
T
1 Fu T un 3, ui) = = EF(u" 3. unl j (uTTp (.) since Elu.u = 21
follows from here that
E 3 uk ATB. (BI. u) Σ 1lu uu ui|l (A())
Σ ( c (1=+o(())
= (B, u)²
by Equation II
1=1
= 1- (3, u) +o
c
Since A is isotropic Gaussian, we have that U, V are uniformly random orthogonal matrices. Thus, u U and U B, uniformly random vectors on the spheres of radius || and ||,3_| respectively.
Hence, when we consider the squared terms to compute the variance, the term from the two uniform vectors will contribute ((1 (1/d²). Together with the singular value term (now squared to have O(1/p*)) and the summation, the variance is of order ((1/(=dd).
For the second term, we have that by Equation II
d Y(u
T A'A k= = u Tu ((AAT)))F= TUETUEEETTTER
(u (uTu,)) u,)² o7(A)'
Ejk A'AIT K] = Σ (u u u, 1
d
1
= Σ
d 1
à (1-c33 +0(1)) p² (1- (1- to (1)).
(7(A))
pt d
r=1
30

---

Published as a conference paper at ICLR 2026
where we again use E[(u² B((uu,;)² ||u|| = 1.
u;)²] 1/d since it is the entry of uniformly random vector of length
Similarly, the variance is O(1/(p²d) from the summation of d independent variances each of ((I/(o²()))
For the third term, we have that
n
B_ su B. 3, (I-AAA)ul(u' (onu((TB)).)).. = (3, -(3, u) E (3.] ui) uz)(u' u.).
Similarly, take the expectation (in particular, E[[ uz)(2² u;]] 1/d(0 u)) and have
1 - n Σ - (1-
(1- (B. u)
(B_ u)²
The variance for this term is O(1/d) from summation of d/c terms of (l1dd). For the fourth term, we plug in (I AAl)u and have
B, AA us 3. (3. 3 AAT (3. AA'u)²
From previous calculations, we have that
1 (3, u).
R|B AA u)
E
Σ (BT u.)(uT
Using Proposition and this result, we can then show
E[GT AA'u)?1 22 2/2
1 (3,
(1,T0)++0((1 +
follows that
Bi3 AA uss [.] 0-0
(B. u)² o(1)
The variance for this term is O(1/d), where the dominant term is summation of n d/c terms of O(1/d).
For the fifth term, we have
a B. 1ThTS h3, 2 Σ ().
(V (V. v))
(A()o
Since B.U.U U (and VT v) are uniformly random and independent of everything else, we only have the diagonal terms when we take the expectation. By Equation [I]
B. 1 d
c- ( + = | 12 c
1
E h h33
+o
n np
d c d
1=1
The variance for this term is 0(1/(p+&)) from ((d) terms of individual variances of ((1/(p
For the sixth term, by expansion and Equation II similar to above,
c p² (c 1)3 p4
to (
TAIT A A Σ B[( E[(VVTu)
u) E
= Σ E
(A))
(A))
iml
The variance is O (1)
For the final term, by expansion and Equation II
n
1n
1
1
[[]b]]]]] Σ E (n U) [U)
+0
+
((AA)
p2 -dc-1
p2 c
p3
The variance is (1/(6tn))-
31

---

Published as a conference paper at ICLR 2026
Lemma 11 (Zero Expectation). In the setting of Section we have the following expectations for
1. Ve, E|3, uh 0 and (ar uh,3.) O(1/(p2d)
2. Ifc E| A. uhB..] = 0 and Var AA A.w.Auuuha =1 O(1/(p2d)))
3. ffe> E1 sh and Var(B, shB.)) O(1/(Ad))
4. Ve, IE[K" A 1 = and Var(k' A = O(1/(p9d)
5. Ifc > EAAA [.] and Var hAA' (.) = ((1((p²d)
Proof. Similar Lemma 10 for all these terms, we evaluate the expectation using the SVD A UEV with A = VE
For the first term, we note that
min(n.d)
B. 1 (3. A = (3. VEIU B. =B.
Σ (VT).. B.); (A
Since A isotropic Gaussian, again we have that U, V are uniformly random orthogonal matri- ces. Thus, v V and UT B. are uniformly random vectors on spheres of radius ||w|| and [|3.|| respectively. In particular, they are independent and have mean zero, which implies
E B. uhB.] 0.
The variance will be d() as a summation of ((d) terms of O(L((22()).
For the second term, we note that
min(n, n.d)
min(n.))
B. AA'u-
Σ
(BTT),(UT))) u): and h3. (u VV B.); 1 oi(A)
Σ
Multiplying the two together yields
min(n.d)
3.. AA'uhB.
Σ (UT u ())(uT)), (U o,(A) 1
We note that T V is a uniformly random mean zero vector independent of everything else in the summation. Hence, the expectation is equal to zero, and similar to Lemma ??, the variance of this term is O(1(pp²d)) (a summation of (dd)) terms of O(1/(p²dt)),
For the third term, we have that
B. shB, = 3. (I- AA JuhB, AA))uAB, = -__ wh, - -T AA' AAluhb...
Then using the previous two parts, we get that each term has mean zero. Thus, we get the needed result. Using Lemma and the first two terms, the variance of this term is ((1/(p²d)
For the fourth term, we have that:
min(n.d)
I A = TUS =
Σ (u (U); (V v); (A)3
Similarly, using the independence of U, E, V and uniformly random entries, we get mean zero and variance (p
For the last term. we have that:
hAAL3.
Σ
(V (VTv), B.); (A
(min(
Using the independence of U, E, V and uniformly random entries, we get mean zero and variance (().
32

---

Published as a conference paper at ICLR 2026
E.4.2 STEP 3(B): BOUNDING THE HIGHER MOMENTS
To bound the higher moments, will the following Gaussian hypercontractivity lemma.
Lemma 12 (Gaussian Hypercontractivity Inequality). Let G ~ No 1) be standard Gaussian random variable. Let f - R be a degree k polynomial. Then, for any q 2 2, the by norm of f(G) is bounded by its L2 norm as follows:
L, 5a (
where the Lp norm of a random variable X is defined as || Il (PI))))))
Proof. Follows directly from (Mei et al. 2022 Lemma 20).
Lemma (Multivariate Gaussian Hypercontractivtty). Let G (GI...., (GI....,GM) NO, las) and let P: R R be polynomial of total degree Consider the Hermite expansion of P
P(x) = E
HaHa())
LEN"-IaaSS.
with coefficient random and independent of G. Then there exists a constant C' that is only dependent on M, r such that for any 2 2,
1/2
|P(G))L, SC 1)
Σ
Ikall2, at
|a|5r
Further, for all |oo] S r, we have that ||ca 122, 3 C9 (c Ii, then
|P(G)]L, S C(g - 1) Where the Ly norm is over all of the randomness. Furthermore,
Proof. Let H R + be the probabilist Hermite polynomial. Given a E NM define
M
H (z) = II
Hay (x2)
f=1
Then since P is degree r, then decompose
P(x) = Σ Ha()).
oEN Jalsr
Here |a| = E3a, ay. Since the Hermite polynomials are orthogonal, we can see that
[ H. (o) daa II az!
j=1
where YM is the density for an M-dimensional standard normal distribution.
[P(x))] = Ey [/ IP(x) (xx
Eg [cacal / (a()))) (x)
a Sr lalSr
Σ
||ca 12z a!
where a!
ayy
33

---

Published as a conference paper at ICLR 2026
Then using the ID Gaussian Hypercontractivity (Lemma 12 we that
[Ha())l
II Ha, (zy))la, j=1 M
II (q 1)/0//1| (x j=1
M
(q 1)lal/2
II
/ay!
==1
(q - 2
Thus, using the triangle inequality we get that
|((x))la, I Σ [a.oHa()))le, =
Σ
||<a Ik, (a(x))l
lalsr
alst
Thus
[P(()) S
Σ lalSr
Iea H. Ha(llle,, S 1 (
Σ
Icalle, Va!
lalsr
la|Sr
Then using Cauchy- Seuwhy-Sehwartz, we get that
1/2
1/2
Σ la|Sr
Ikallr.,
Σ
Ilaa lcalli,a!
Σ
la|sr
Finally, we note that
1/2
CM. ==
Σ1 1
JalSr
is some universal constant that only depends on M, r. Thus, we get that
1/2
[P(x) S CMMr (q - -
Σ
11aalli,at
la|sr
Using the assumption
Icaall, Collcal22
Then we get
[((x))le, S SCM, (q P(xx)l
Lemma 14 (Product Spherical Hypercontractivity). Lei 1,, la 0, let O Unif(S'), O₂ ~
U O3 Unif(S!s) be independent. and let H 1u+ x Rl2+1 glstl R be multi-homogeneous polynomial of total degree r. Then every q 2 2,
I -Crgl9-1)//2 0x)ll,,,
(H, Oz. Oz)
where the norms are with respect to the product measure. For homogeneows polynomials, the constant independent of the dimension.
Proof. H is multi-homogeneous of degrees 72, T3 with ri + + =r. G ~ N/(0, 41 G2 ~ (0, 11)+1), G3 ~ (0, 4ts+1) be independent with polar decompositions G. = R₁O,, where the R 's are independent of each other and of the O,'s. 's. Then
H(G1, G2, Gz) RT R2 R5 H(1,, O2, Os)
34

---

Published as a conference paper at ICLR 2026
so for any p O,
II E [R...
EE [II, G3)l
E 02, 02, 03)1
Then we have that
( [ O2 1a)l L.
[[ G2, G3)llL,
(12)
Apply Gaussian hypercontractivity (Lemma 12 to H(1,, G2, G3) (total degree r):
[H G2, G3)lla, I C(q - 1) [H G2, G3)lla
9 9 2.
Using Equation 12 with = and = yields
TE II (E 1)
[H 22 03)ll,, & C(q - 1)2/2
02,
For each since q 2 2 and R. > 0, monotonicity of Lpp norms implies (TE ()/r 2 (E R ///(m)) hence
E R 1R?.1y
1
1.
[ 11////
Thus the product is less than 1,
[H(0), O2 02)lla, I C 1)r/2 |H(01,02. As)
Lemma 15 (Product spherical hypercontractivity with random coefficients). Let 1 12,1 2 0 and ler θ ~ ) be independent. Let r E N and let R'++l R'a+1 R'a+1 + IR be multi- montti-homogeeneuu polynomial of total degree most Suppose the coefficients of P are random on an auxiliary probability space and are independent of (0,, O2, Q²). If the random coefficients satisfy ||ca]] Kallcallt, in the Hermite basis expansion, then for all 2 2:
[ S Cr.q (q - 1) ||
Proof. The proof is identical to that of Lemma 14 except we begin with the version of Gaussian hypercontractivity that handles random coefficients satisfying the stated assumption.
Recall
a a:= V:=VE b:=U TEUTUELL and Ug U B.
Then, since u, u are fixed, and U, V are independent Haar orthogonal matrices, we have that a, all uniformly random vectors their respective spheres. Additionally, using the assumption that B. is uniformly random such that B. u is constant. Uz is uniformly random on a sphere $d-2
Consider the following centered versions and polynomial representations.
1. Y₁ | E
[ 12] a EI ph)
2. Ys = ||A|| 2 E
[|| = b x 14) b
3. Y, : ||&||
[||4] 2T = a T - ((-- - 11)
4. Y, ||s 2
E
[| 2 b (II- (I-ΣEt) Me)
5. YE = E
E
= a T T ΣΦ
6. 7, B uk At3.
E [ uk At [.]] (BTT
E') 14,, (b_b))
35

---

Published as a conference paper at ICLR 2026
7. T2 = T A'A A T k E K A'A b] = b (E"TET) - -TT.))
=b=
- b
8. T3 = 3. su T B. [B su T [.] u) up - ΣΣ'}b )bb HTs (u up)
9. T4 = A B. - E B3 AA'us Br]
ug WΣ bb "1 (1
"r. (b (b)b) up)
10. T3 = h3 T hB3 E 13. h T...TAA, h = (up - -i, (a (a))(()uuu))
11. To = -(AA) T Ath T - E [ A'h =a (EΣΣETT² ) MToo)
12. 31 = S.. uhB, - B 33. wh = (8) u3 uuus
13. 82 B. AA AA'uhB, - E [3. AA'uBB. =u EΣ ba E' ug
14. 83 = B. shB. - -E 3. 3 sh = un(1 - -EE ba E'us
15. 5n = k A E kk Ahh
Hence we see that these are all homogeneous polynomials in uniformly random spherical variables. Thus, we can use Lemma [4] we get bounds on the higher moments. In particular, since the coefficients are only dependent on constants and Σ, we see that the coefficients are independent of a, b, u.g. Then using a change of basis we see that that coefficients of the decomposition are also random and independent of the input variables. Finally, since the spectrum converges to the Marchenko- we have that the coefficients have bounded moments. Hence the second assumption is satisfied.
E.4.3 STEP 3(C): BOUNDING 7s MOMENTS.
Lemma 16 (Moments of 74/1 74/n²), We have:
(i) For 7//12,
E
17
p²
c
n
+o
)
p2)-
Var
n1
1/2
=0(11))
0
p+n
1
(ii) For 72/1?
1
1
22
0
1
E
Var
0²
n
p&n
Proof. We decompose
7 = Ci m
i= 1,2, where Ci =||2|2 [|]], c ||| |A/
We begin by noting that ||e| depends only on V and is independent of U. Σ. ||s|| depends only on U and is independent of V, E. Additionally, ||| depends on U and E, hence is independent of V. Also ||/il depends on V and E and is independent of U, hence is independent of U.
Expectation Estimates:
Thus, have have that || and || are independent and || and || are independent. Thus, we see that
E[G]] = |[[| || E E[ k: 1
Using Lemma 9 again,
1
B || = 2|
p2 1 -
p2)
We plug them into the expectation and get:
E = (1 - c)
( -
c
+
0
1
=
c
+ 0
2
36

---

Published as a conference paper at ICLR 2026
Finally, we also have that from Lemma
9
62)
(1
1
E
Var
0
2
p
1p
Hence,
71
22
c
1 1
E
=
B
+
E
p2
+
A similar argument applies for 72/11² using the corresponding results for || ||]]|?
Variance Estimates:
Again using independence, we have that
Var([]e[|||<4]]}> Var (ar((1|13)) (|(:11))
+ Varr))))) + V[ar(1))
1
1 c²
1
0
O
+1
c)²0
ptn)
O
n
p^n
pt
(1 c)2
m2
ptn
We then use Lemma 34 to compute the variance of the sum:
5²
5²
Var
(1
Var((1) +
Var
172
177
1
1 pptn
+vo
p
1
p
This proof is similar to the other case.
Lemma 17 (Moments of (2a/12)²). We have, as n, - 5o with d/n +c+1 1, (i) For n//²²
1
2
1 pz
(
11
1
E
+
+
0
Var
= 0
1p2
112
p n
(ii) For 72/1/²
E
7/
1 2
o (1))
Var
(22) 2
0
1
0²
172
0z n
Proof. Write, for E [1,2].
2t = C. +
:=
||t|| |k]
Eo = ||s|| |/]]
1/2
1
Means. Using Lemma 16 and the fact that for any random variable
E [Y
E| + Var(Y)
we get the means.
Variances. Using
y²= + (EE (Y -V-E + (Y E
37

---

Published as a conference paper at ICLR 2026
Thus, using Lemma 34 we have that
2
(VA(B(X)) Var(X,)
Var(Y3)
V VVa
(Var((Y-B11)))
B
By spherical hypercontractivityy for d polynomials,
7
2
2 Var
?
E
E
17
hence
(()
25
2
2
17
1i 17.
2
Var
E
I
E
Var
172
17.
Using E
26
(1) and Var
Opp-~1111 gives
21 Var
0
17d
pmn
as claimed.
Lemma 18 (Finite Negative Moments of 7s). Fix p > O. There exists an ((p) such that for all n, d 2 (p), we have that for <
2 p2
E [
2p
[ pl
<
M
and for >> 1, we have that
2p 172p
E 20
2PE
o
E
S
M
where 01 is the largest singular value of A, T = || d-n m) Bela 2 2
Beta (znd, 4)). and ||s||
Proof. Recall our SVD A = UEVT and that
?1
1/ ||||| |
g²
and
72 = 0/|10||211122 =
Then we have that
d
b
1
||
611
Similarly,
a
1
1
>
|a|1
o
01
of
Thus, we see that
1
n1 2
1| of
and
2 n7 8 1 of
|| depends only on V and is independent of U, U. || depends only on U and is independent of V. V.E. og depends only on E and is independent of U. V. Therefore, on independent of T := ||t|| and of S : ||s||2.
Thus, we get that
o
and
P < 72
#
27
|p
2p
2
||
2p
Then taking the expectation and using the independence, we get that
E
1
E
E
2p
and
E
1
E
2p
P
2
38

---

Published as a conference paper at ICLR 2026
For < (where d < n). the right null space of A (dimension n d) is uniformly random (n - d)-dinnssia subspace of R".. The squared norm ||t|| represents the squared length of the projection of the fixed unit vector E IR" onto this random subspace. The distribution of such squared projection norm is Beta ( (²2" 2), as can be represented as the ratio of two independent
chi-sque thi-squared random variables: Σ D==1 G Di=1 G.,, where G,~N((0,1) 1) IID, which follows the desired Beta distribution. Similarly for e > 1.
Since the eigenvalue distribution converges to the compactly supported distribution. We can see that for sufficiently large n. d, we have that there exists an M 2 1 such that 01 I pM almost surely.
For Y Beta(a, 9) and p < a,
I(a - p) p(a + 3
E
BY
p)
Moreover, using Stirling on the r ratio,
a1 B1
P
P tn
a1
=
1 c
(<<
and B|
0/2 + B2
c
p
n,00000
(c >
1).
022
Thus, there is an M such that
1
P
2p
1
p
2p
E
S
MP
and
E
M
1
Lemma 19 (Moments of 172 /7/7)) We have:
(i) For ///
n²
p²1²²
1
112))
0 (
E
+o
Var
71
c + p2
02)
71
(ii) For xp 1/2
n12
1212
112 = O (1) 0²) Var
+
112 + 0²
Proof. By Lemmas
32 and 16
the expectation of T?? /71 can be computed by:
p²m²²
71
Bln/np1
+
p²d
r 0²
o
By Lemmas
33 and
16 the variance of 1? /vi can be computed by:
n
71 172
Var
Var
+ o
Var
1
1
+
(nfc+ p2)=
!
by the scalings of and p.
The proof is similar for the other term.
Lemma 20 (Moments of x7* //2)). We have: (i) For n²² 177
p²n²
= 0 (
+ +0(1,
Var
(n + po.z
39

---

Published as a conference paper at ICLR 2026
(ii) For n² 1/2
p'nn p²)²
I
+
+1),,,
Var
0
12
(n²²
Proof. The expectation of n1 ///// can be computed by Lemma 19 By definition have that
17'
1
112
p² to (02) +0 ())
E
E
+ Var
n2
n/c
The variance follows Lemma and Lemma
7
Var
0
2
71
since the mean O(1).
The proof is similar for the other term.
Lemma 21. Suppose E R whose entries have mean O, variance Tox and follow our noise assumptions. Then for any indepedent random matrix Q E TRax,, we have
EeQQ [E [T==] - [T(()))
Proof. We have that
n
Qe
6xE e
We take the expectation of this sum. By the independence assumption and assumption Bleys,] when * j. we then have
Es.Q e [
Σ E 2 =
Σ Qui (Tr()T(()
E.5 STEP 4: BOUNDING THE EXPECTATION OF PRODUCTS OF DEPENDENT TERMS
we decomposed the error into four terms Bias, Variance, Data Noise and Target alignment. In Section E.3 we wrote each of these terms as the sum and product of various "elementary building blocks". Section E.4 we should that these elementary building blocks concentrate. this section, since have tight concentration (i.e., the higher moment bounds). We and Lemma 33] which shows that the expectation the product can be approximated by the product of the expectations. In this section, we do that calculation for our different terms.
In Section E.2
Lemma 36
E.5.
STEP 4: BIAS
We begin with the bias term. Recall that for e < 1, the expected bias by Lemma
5 is equal to
& (az (az-
772 52 (az-aa)) aa) (B. u)² +
E Bias] = E àz az + ?1
17 o
where the cross term equals due to e having mean zero entries. These remaining expectations are given by Lemmas 22 informally via:
Lemma
X Lemma 23
17
For can plug in the value to get that the expected first term is given by
0,7 u)² ( (āz-az) az) + rpct x (az -a)) + +0(1) + ?
p²
40

---

Published as a conference paper at ICLR 2026
and the second is given by
To 1/² 1f-l + o + +0 (
2 1
n²
Adding them, we then have the desired result:
([izz ( az)
p²
13
rpc+ +P (az -aa) (3,T + 2
to 71
+0
1 + p2
For c > 1, we instead have the following expanson:
& (azI- - 2 aλ n||i|1
E 72
B. (āz - az)
u 7 + 3.
P2
72
t
t
The bias equals the expectation of the norm of this vector. Taking the Frobenius norm, we have the six terms. Among the cross-terms, (t1, (3) and (t2, t3) have zero mean since 23 contains & whose entries have mean O. We now look at the other terms
E 43 1
E
P25
E
P2
1?
by Lemma 21
7722
T7
The expectation given by Lemma 23 Subsequently. Lemmas
22
24 25 give E|| t1 [1 E ||| [2]
(t, ts} respectively. Informally, we can compute the bias via:
||00 (āz az) + & (azl aAAAt) 2 aA
n 1
i 5
E Bias]
E
B. h T u T 7 +
72
72
72
[|3] + |1[1|22|1 E|
2T
t3)
Lemma
22 Te Lemma
23
Lemma 24
2 x Lemma 25
Similar to c 1, adding them together and dividing by 7, we get
1 + (az- - -22) a
p² (az
||..12 /c- (c-1
t/² p²
72
17² + p²
(,Tu)² (āz - az)
d (c
(17² + p2 22
c (17² + p2)3
+ +0 11
+
E.5.2
STEP 4: VARIANCE
Recall that for the variance, we have the following expression (Section E.3.2
7 2
723 B A(Z + A)
E
3
E
3 Z(Z
A) ZB. +
[Z +
ATB. A
2i
3. z A) A) A
(Z A) (Z TT
d
d
In particular that the expectation will be the weighted sum of the expressions from Lemmas 26 27
28
29
Informally.
(a Lemma 26 2azas x Lemma 28
a x Lemma 27
29 Lemmal
This yields that for < 1, after simplification, the variance is
(az (az=)a)² (1²² + p²²) d² (1Pc 22)2 1-c
n/cc 7 1
2 11 2 u3)
+ 2a aa)
d
d
plpcct (oce+p)111 12)1-1 )
17 2
+ +0(1) +0
c-cp
41

---

Published as a conference paper at ICLR 2026
For similarly simplify it to:
||
a
a
1/
u)² c
1/2
az
(0a-0a) oa)²
d
c
1 p2
c- + p2
+
d
p²1 + p²) c- p²))c )
c
+
t(()) 0
1
p² c
n
E.5.3 STEP 4: DATA NOISE
Recall that for the data noise, we have the following expression
0²20² 11..11
Noting that 113.11== ((1), we see that this term has no more randomness and we do not need to estimate anything.
E.5.4
STEP 4: TARGET ALIGNMENT
Recall from Section E.3.4 that the alignment is given by
E [,
A)T Z Br OA AT]]
d
From Lemma 30 have that
# (() c<1
E
B.
+
A)
2
B.].
p" 8²tzf"c
g- 13. +
+0
e>1
and from Lemma 31 have that
|13.112
1-1 (3. u)2 (3.Tu²² + (0)
1²c
c<l
E [3] [2 + A)
IT A a.]
[10.112 111.3,11 +2
d + (((T + +0)
e>1
Thus for e < 1, the entire interaction term now becomes
2ā 1p² (aaIl3.11 + (az aa) aa) (B,
17c
+ =011)
d
2+1ee
For >> instead have
Aā
aAA
0A
n² 112 p²
azz 113.1
a
n
(B_ u) - o(1))
d
1
p
E.5.5 BIAS: HELPER LEMMAS
Lemma 22, In the same setting as Section
2 we have that for e < 1,
E ( (ăz (ax-asst -az-azz az +
E (az - aa) (aal)aa)
(32(3.u0) u)²
71
p²
2
= 13, (āz - az)+ + 1pcc + p² (az - aA)
+o(1) + +
For >
C S (azl - -aaA-)) 2 $1]
E
B. (āz - az)
12
= -0T (āz az) +
p²
oz - 04 aa)) 2
+ + +0 (2)
112 + p
42

---

Published as a conference paper at ICLR 2026
Proof. For < 1, we first expand the square and get:
àz (áz-az+ + E azz (a)) = (āz-az)
1 2 r72 gं
ng
(az aaz=
(az-aa)(áze (az-aaláz-aa)
17
By Lemmas 9 and 20 then that, using the square root of the covariance bound the difference between the expectation of the product and the product of the expectation.
rz
52
E
= E
+
Var
Var
7
n72
+ o(1)
(npc+ p2)2
1/2
+
o
p2n
+
+0
/'1p
1
(1)
+o
+0
(np p2)2
n/2
9
17
n² ?1
E
E
E
+
V
Var
Var
71
1
1212²
(1)
1
+
+o(1)
0
p2
17
1
p'n
0
1/cc
Combining these terms together, we have that
E âz - az + E (az (az-ash
2
n²(B,. 2)²
71
772
p41p
1
= (3. u)²
ip ip(āz - -x,)++
+0
+
(az aaa²
172
(1pc+
p2)z
172
27p
12n
(1)+0
+o
+
(az
-aa(āāz - ozz)
"1 (1pc+p2
= (BT
(āz (āzz az) + n²c + p2
p²
(az - -as)
+ 0
1
+0
1/2
mmm
We now consider c 1. Recalling that 2 vuu
we let 0i āz az and expand:
E (azl I - AA ?
BT
(āz - Iz)
72
_azl- (azl-a-AAl)] T
f (uzl - (AAAA)
22 (āx-az)1 E
- (āz - -az)
+
B.
?2
72
(āz - az)
$
az)l TuT (āz - (āz-az)(+
S (azI-aaAA!)) Ba
72
72
E 3.
-,, 7/2
((azI (oz)
20
(azIaanAAi)un B..
72
Not that for the second and third terms, we have that E, only depend on the singular values of A and the rest only depend on the singular vectors. Hence, these terms are independent.
First note that when d > n, the number of singular values equals n, which is less than the dimension d. As a result,
AA - - TVE'U U
I Inxn Onx(d-n)
UT.
0 0(d-n)xn
0 ((d-n)((d-n)
Then we have that
1 c 11192.12 B 3.] = D> E [(B.U) = n 1113.112=
E
(13)
1=1
43

---

Published as a conference paper at ICLR 2026
since B. U is uniformly random vector of length || in R after the rotation U.
For the middle term, by Proposition 2 and the above Equation
we have
E |3. (azI a a AAAA ((azI - -.AAA
= [B. u)² a [3_ AA'uu [ + a [13] AA {A
2 = az (az- =a)² (3, (3T + o1).
Similarly, for the last term, we have
[ (azI axAAiun [.] = oz -4) (3_ u) + o(1).
Thus putting these expectations together, we get
E (B. u²²
2
5² (az (az- -0a=) 201 & (az (az-"a) -
(az=-a1]]
E (B. u)² aa & (az )
7/2
72
72
Similar to the
1 case, we take the expectation for terms involving
& and %
get:
7 13,8, u)² ((āz-oz)) az) + + p² (az (2x
p² -a) 2
+
mpm
we have that Lemma 23 (Expectations involving and p2). In the setting of Section 2
1. For c = d/n <
2
p
||
c
+ +o(1)
(1)))
+0
(
e
qp
+
p²
p2n
2. For c == d/n >
E
2
P2
c c
172
(17² + 0²)2
n²c + p²
+ o(1) +0
(
p²n
1
Proof. First, Lemma us that
17² 121
pp
11
7
7/1
Then recall from Lemma 9 that
1
1
Var([|&]13) = =0 (1()
E
k
|
and
02
1
p2
p'n
and Lemma tells us
p²r
n²²
E
0
and
Var
0
n
1 +
p
7
Again Section E.4.2
tells us that the assumption of Lemma 37 is satisfied and that
E
P1
2
E
1
E
7
E
[[
Var
1j2
Var (1|A)12)
71
7
71
ie +
p²n²
1
c
+
))
+0
+
(0(1)
(1)
p
17
o(1)
0
cetect
p²
p2n
44

---

Similarly, we can do the same thing for the other term. For the middle term we note that from Lemma Published as a conference paper at ICLR 2026
Using Lemma 6 for P2.
5²
1
P2 |p21
==
A'hT
2 A +1)
1/
7/
To begin, we start estimating
n
hATT AihT
E
Using our Spherical Hypercontractivity, have that |s| and h.AtT Ath satisfy the assumptions for Lemma 36
Then using Lemmas 9 and I0
we we first have that
[||+||]] = and
Var (1|x) =
E [[AATAhh"]]
A/n 0² (c 1)3
o (2) and
Var (g. h T uu B.) =0 (
=((2a)
Thus, using Lemma 37 we have that
2 TEEEE KATT Ath +0 max ( =
E [||s||[hatt A ] = (E [1]s|1]]]
p
1 -1-22)
c
)
1
+ 0
p (c - 1)3
e
p c--1 +
and using Lemma 36 since all the means are ((1), we have that
Var [||| hhA TT A'hT) O (max (Var ([|=|]). Var (hAt Ath = (
"² does not satisfy the higher moment bound, Then Lemma 20 gives mean and variance of n" Since 7/2 and cannot be directly included in the product, we can include it via the classical bound:
17 ||
hAt TT A'h
T
E
n
[1
HATT
+ A
Var (|| Tth ( Var
1/4
72
72
(14)
D'n
(1o= to (0(2)) +0 (
1 p
+ p2)2 + o(11)
(15)
(17
n²
o11)
(16)
(1) 2ン)²
E K A'h
=0 and Var (k TAT'"" = o p
and Lemma tells us
=1- 1 and
Var (1|s) = =0 =
E ||
and
E
5 (5)= = ( p2n
E
and
Var
17
1
Thus using Lemma 37 we have that
T
E
2
8
Aih T
= 0 10+0
45

---

Published as a conference paper at ICLR 2026
Thus using the standard covariance bound for the expectation of product versus product of expectation, we have that
7 1 k
1
E
2
A'h
=0+ +
Var
2
0
0 0
=
72
72
n2
For the last term, we have that, using Lemma 37
2
1
1
)
E
1|2
+
1
1
1
+o
+
c-1
c
1p
and from Lemma 36
Var
[kll
= 0
1/
phnn
Then using the standard bound, we have that
17 2
|
17
E
52
||:]
2
Var
1pt
0
1
72
2
112
7/2
p²nn
p²1
1
1
1
+0 (ph))
(17² + p2)2 +o(1)
to
+0
+
ptn)
p12p²
c
1
nP:p²
1 p2pp
1
1
+
+
C (17² + p²)²
17pp2
p2n
Finally, putting all three terms together we get
E
7/
52
P2
12
c (n2 + p2)2
n
o(1)
c 1 (11
1 p2
+ p2)²
0
p2112)
1
+0
+
1 ppn
1 n2c+ p²
+ + +0
c 1 (1p² + p2)²
p
From the above proofs, we make an important observation that the individual terms from Lemmas 10 11 16 all have means O(1) and variances (1/n). Hence, by Lemma 36 we can bound the variance of a product of terms by (1/n),, given that these terms satisfy the Temma assumptions. Essentially, only y/² /77, and 1y² 1/²/// // fail the assumption on higher moment bound, so we deal with them via the classical bound after carrying out the product. This simplification ensures proper concentration and will be used at times in the following proofs without reference.
E.5.6 VARIANCE: HELPER LEMMAS
Lemma 24. In the setting of Section
we have that for e >> 1:
E
||
2
B
h
T
u Z
ita
||3.11 c
c-
n
+ +0 (1)
72
d
(12 + 22)2
Proof. Since 2 iuu T we have that
0 /||=|| 2
7
|/|1
B h,T, = 2 172 s/|
3. hhhh T u 2
a
B h T h
72
=
72
172
7/
Similar to last lemma, using Lemmas 37,
20
we get
17 s||
3_ h. h,3
= E
17
E
[]
2 E [IT. 113.]
+ Var
11 0 1
7/
(1-
113.112
1 ) +0 +
+ +0(1)
+ o
(02 + 12133
d
pccc 1)
(2d
|1
(c-1)
1 p²
1
O
d
217 + p2yz
m
46

---

Published as a conference paper at ICLR 2026
Hence, it directly follows from here that
n|
172
|||
E
a
3,
Br
h
T u
T
2
0
E
Br h h,3.
72
1p2
75
a2 a
113.11
(c- (c-1) 1
+ +0 (17² + p2)²
n²²p²
11
Lemma 25. In the setting of Section
we have for 1:
(āz az) I (azI aaaAa))))) 22 uhB, =0 =0 (*))
E
3
72
72
Proof. Using Z juū we can expand this into three terms. We can take expectations in a similar way via Lemmas
TO
Let c₁ äz az. Each term contains zero expectation:
E
c
n||s|
3_ uh3,
772
E
1j
E
[[|
E
[
uh +
Var
n²²
O
7/2
72
72
1/2
!
0 (
C1
Var
72
72
||11
E
17 az
B. uhB.
IE
17
E
IE
[||
IE
[3T ut
+
Var
o
7/2
112
77 72
77
72
1/a
Var
0
172
?2
71
1/ a
/| A. AAA A.A.t/BB..
7/2
=
azii²
1/2
11'
E
5
[||
2
G
AA LAiunB.]
+
Var
1/
!
az
11a
1
=0 (
Var
0
172
72
Thus the cross term concentrates around zero rate of On/nn).
Lemma 26. In the same setting as Section
2
we have that
22 2²
1-c (3 2 + o1) +
e<l
E B. Z(Z + (Z zp.
"retp"
c - (BT u)
+ 2 + +0
p
e>1
Proof. We start with c < and expand this term using Proposition !
3 Z( 4) z0, 2 h 2g²
11) 17 [|]
T A (3) 2² 277x 1
(3.
k. Ath 18]
7
We then start plugging in the expectations of these three terms and the "cumulative" variance of the sum according to Lemma 37
22 7
(B) u)
(BTTu)²t 2
17
E
E
Var
O
1
171
77
71
p'n""
p ))
1
c²
(
= =BTTu)
(nPc+ p2)3 + +0(1))
+
+
+
0 n
1
1
(3T (3Tu²² 2²² + +
0
TTpc 2
n
47

---

Published as a conference paper at ICLR 2026
1/1 7
0 (1) mm)
E
(k
AAAA A u)²
= =(3 2
E [1[t11]] T [ AlAiT]]
+
Var
{
= uB. u)²
(1+c+ pz? + ((11) (1 c)²
p&yp²
/
6²
+
pr )) + 0 -0
pt
(1
c)3
npt
B, u)² + o(1) +0
(nfc+ p²)2 1 -
and
1 ATh (B. u) (B_ u)²
2 E
(
(
E
A' |
[1+|
0
12
n
Now we have the expectations and errors for the three terms. Combining them yields the Lemma statement.
For > 1, we recall that h.s 0, and Proposition 1implies
TT z3, = '72 1
(3, u)² + 1/
2n 217/11221 u)²
3. Z(z + AZ + T)
(B, (3,u)²+
B. uhsu B.
72
7/
7
1/ + +, 1//11111111111111
(B. u)
2/
11 12
uB. u)²
72
272 |/ (3. u)
7/2
Hence, we can take expectation:
|| 22 (BT u)² + +0
B13 z(z + AZ + +)+ zB. =
L22
77
1/2 e
(3 1)² 011) + +0
172 + p2
c
Lemma 27. In the same setting as Section 2 have that,
|| + n + +
(r"cpp I-c (BT (BT1)²² 20
(3T u)² -1)=+ c<1
E BT A(Z + A) + AB.
||3.1
"
18.11 (0 (3. u)²
+ + (
c>1
c(c-1)
Proof. We use similar expansions that follow from Lemma
1/ (BT
A + A) + A)+T AG, 11,3.112
TK ATTTAAATTNN A u)²
77
7?
2 21//
20/211112
2n BT whB
u T
BTuuk ukk AtB.
77
m
Lemma 26 gives the expectation of the first four terms:
22 c²
( 11
|| + (Tpc+ p2)a (B3. u)
We have done the following expectations in Equations 18
E 3 uh.3.
17 |||
1²cc
o(1) +0 +
E
0
3. uk AJB.
71
1/c + p
Combining these results yields the lemma statement.
48

---

Published as a conference paper at ICLR 2026
For core> 1, with hs 0, s'A'AA'' AA 0, hA4t h, have the following expansion by Lemma 2
A A)) A)T AG. 3 AAAA B. + 77 =
" h.
3. h.3.
7/2
1/
17 ||
"/11/b|125²2 B__ AA" AA'um T AAT3.
B, AAA'u T AA'B.
75
72
21 11$11
h.h. 2n
h1.
B. AAiuh..
7/2
7/2
217
||
11
3. AAtuhB. + 2n |//
3. AA AAtuhB..
72
7/
We can combine the coefficients
n ||||||
|||||||
2
17
11 + +2)
21/
1/1
12
72
72
12
722
17 11
1
12
82)
1/
121111
1/
72
7/2
2
72
12
7/2
Then we have that:
B. A(Z + A) A)tT AB,
1/
3_ hB, + 11111111
2n
= 3. AA 3.
B._ AA A AA
B. AA A''uhB,
72
Recall from Equation 13 that
B AAÍ 3.] 13.11%/c. We then proceed similarly with the other
expectations using Lemmas
9
TO
19
1 s1
7n²
172
1
3 h hB.
E
[||=||]]
[ h
hB.]
+
Var
72
L22
72
71
1212
(1- 2
|13.112
))
(1
+
1
+o
+0
172 02
02
p²
dp2
1.111
1
d
1/2 + p
n1
12
1112
n12
72 -
12²
E
(BT AAtu)²
= =E
E |1
E (B.T AAtu) + Var
7/2
72
p21
(1))
c
+o (0(2))
(3T (1 + ())))
+o
1 + p2
x² c-
c2
2 (3. u)²
! n
tl( +0
2/2 /² de 1)
by B. AAT A.tuh,
1
E
172
1
E
= E
E
E
[B.
Attu33..] +
Var
O
72
12
17
722
1
=0+
(17)
We combine these results to produce the lemma statement.
Lemma 28. In the same setting as Section 2 have that
(2 02) 2
7²c
(BT u)² + o(1) +0
c<l
E B z(z + +A) + A)T A3.] =
p T-c "'
- (BT u)2 o(1)
+0 (1)))
c>1
+0
49

---

Published as a conference paper at ICLR 2026
Proof. For 1, expand it using Proposition Lemma expectations have been evaluated in the proofs of Lemmas 26 27
Note that all of the relevant
m1 B.
+ ./11e112
20111101/E
z + A) + T) A,.
uhB,
B. uk ATB.
(3,
hAT
7
79
17 |||l of
?/11A|1 u)
A
(3. u)²
P
The expectation of the last three terms is given by Lemma 26 Equations T8 19 respectively. We can plug them in and compute the expectation:
The first expectations come from
17 (17 p²) c²
n/cc
B.. z + A) + A)T AB.] =
(B. (8. +00)++((1)) (0(1)++0
(++* + p²)² 1-c
1/c p²
For c > 1, again with hs = 0 and s T "TA= = 0, B. Z(Z + A) + AB, becomes:
AA
3T 7 uh AAA
π sh Q2
|
172 111
T h u T AA' B. AA'
+
TTh
su
7/
Q2
72
+ 3 72111/]
n sh
|>||
17 h1l?
T/ h T u AA 3.
u.
AA i
h h
su
AA'
7/2
7/2
7/2
n
[| uh
?
AhA
wu
AAA
B.
7/2
72
72
1211 |
77 |||
uh
wu
AAÍ
Ga
7/2
72
THE AAA
1|s1 uu AA' B.
wu
AAA
7/2
7/
TE hAA'3.
17 12111
=33, u)
AA' B
since 72 = - + E²
7/2
72
We need to evaluate two following expectations. Similar to c < 1,
n
hhAtt, 3. = =0 (
E
n²
11² o 1
(
E
AAI3.
= E
E [112111
E [3. AAtu] +1 Var
(T2)
72
72
p2m2
1
1 c
(
1
{ (B. u)) + +
+
+o
1/2 + 2
c-
n
(B] v)
1
+ to(1)
+0
2 + p²
Finally, we have that:
1 1
+ +0(1) + 0 (
E [3 () A)"T A3.]
(3 u)
172 + p²
c
Lemma 29. In the same setting as Section
2 have that,
cd p-o
0
1
c<l
E 6 T (Z + A)'(Z + (A
p
pz
p
2
d
m
e>1
pm
50

---

Published as a conference paper at ICLR 2026
Proof. For c < 1, we first expand this term using Theorem 6
A + T A - S [pia)
A )
TT At 5 Spid e
< + AZ) (AT))++TTee = ==
71
-ETATA &+ 21 ST A AATTte kte
25 A'qipTE
2 E k
2² A'qipe e + z pid qiple
&k
Note that Lemma 21 and the fact that t = 0 imply that the second term has zero expectation:
2m
2 tAtA k =
Ee
A
TatTtte kts
E
Simiarly, we will later use:
Eg T A te) = -AAATT E. [ e| = - Tr((t TrTt(k 0.
Note that these equalities exact without taking the expectation over other sources of randomness besides e.
We now expand the other terms by one and compute their expectations along the way. We start by eliminating zero expectations and taking expectations w.r.t. e using Lemma 21
1
2
q
AAAAT k
T
=E
A
ATATTA
25
25 T At
/n|l||?
E
At 4:Pi
E
E
A k hT
nikk
71
E
2n/ ||
[1112
T A'AA
AAIT kts
T ATAA
ATT kk
mE
2ny ||
2n E
A'h" ts
"Ahh K
E
71
71
2m
||t|
2nLT2 k Ath"
E
k
ATA
k
71
71
2n
2m T k
1/|
17111 t + ni )
E
t
TT
A
qqpp
E
At
k k+hh T
71
2m |+||" ||
2n7a |k]1
E
(K TT
TK TTTAth" J)= TET TTa
E1
71
2ny |1el1
2n²
(KT
ATAAAITT TT
(K A () TT
71E
71
2n ||||1*
||
2 11411 k
k
ATATT
TT
k
Ath
21
715
51

---

Published as a conference paper at ICLR 2026
By the squared norms in Lemmas and Lemma 21
E
E
Piqi q:p)
=
11:11
19
T²
1/111/11
n
KA KATAT k 2nliel1
E k A'A + -110112)
V1
7
5²
E²
2011e11 KJ A + +1N11P)
(111))
KAIA
711
n"
||
2ny 1 |
T ?IRRPPIN
-2
KA
k+
k A'h
7152
7/
We combine like terms and simplify the coefficients. For the term
T ALAIT:
|0
20y
201211112
11111112
2
11112118112
2
1 E2)
-2n 11E1?
+
7152 ?
7
5²
71
n E2
2
1
= =r2021141r
+
7/1
E2
n 5
25
=r:'??1011
71
2
77 ||
71
For the term k A'
2n |||||112111111
|t|1
2n
-2
2n
T1E
715
71.
7/1
Combining these terms together, we have:
E | T (Z + A) = E E AAATTE E
1211011-22
TE
A
2MÊTZ 71
K
AA +
|1
22
71
71
Similarly, using Lemmas
21
we have the following:
[
T 2 cd
d p²
E
= T
Te nE
to
by Equation
(1-
q 1
k A
ATAAAAT
fT
k
= E
17
||||
E R A ATA
Var
1
72
p2pp
111)
(1 - c) (1
)1
( 12
+
to
n 2²
02
0² (1 c1-c33 c)3
112
à
(BT (B,u)² + o(1) +0
1
p²
p²
c)2
E
Th
K
T
At
h
E
E
E
1s
A
Var
172
o
1
0
1
711
71
17
n
1:
172
122
1
E
IE [[]6]]]
21
E
[[]
+
Var
o
71
7
11
p²17²
1
1
c²
_
1
(
c
)
0 (1)
+o
+o
+
iic 0
1-c
1/
(1)
+0
1/c+ + p² 20 - 0
m
52

---

Published as a conference paper at ICLR 2026
After simple algebra, the result follows from here.
For > 1, we can expand similarly using Theorem 6
Z + A)) E=E T At + ! Ath ST Sp29
Att + " shAtT
S
72
72
29 T
2 E A q
E A & +
A SsA
72
72 1
A' A T 42D. + E²
21
p2d q.p.e
72
72
expand the other terms one by one, marking those with zero expectations:
17 s
A
hAtT TT E
=
E
1/
hAL
A'h
E
2E
2
17 [7]||[11
E
T A
42p
=E
E
A'
8 8+ h
hAtt + +nnk'
72
722
2
E
Alh
hAlT +
72
E
2ny ||
2n5
E
TAh hAtT
A
72
72
2ny 1|
hATT A'h 2nÊTE k A
E
7/2
7/2
21
27
17 s|/
Ahh 8
92p E
E
A
T
8 + hT)
hAT AT + n E
72
É
E
2m T
A
1
hAiT
+ +nk
72
E
E
2n 21/
21/
|||1?11A111
E
TAth hATT
E
TAt k
7252
726
2nyd
||/||=|//||2 37
|
|
12
E
hA IT
A
k
AAh
E2
72
Using the squared norms from Lemmas 6
c2
P24- qop. e
= E
P2
122
|11112
1/
10
+111111 }
E
hAIT Ath T
Ath +
72
E2
C2
1/'1
|||
2r 1
)
E
72
hATT
Ath
k A
728
72
Similarly, we combine the coefficients: For the term HAIT IAITA! AT
|||||
17
217
||||||||
17 8|
17² 1
||/12
2
1
T
C2
52
7/
22
T2
52
72
+
22
__2 62
2
1
7/62
7/2
72
6
25
12
7252
725
72
|||
72
53

---

Published as a conference paper at ICLR 2026
For the term k' A'th"
2n |
|r/||s|/|||h|
2nT
2m
-
72C
725
Combining these terms together, we have:
1 2
2nÉTÉ
7
[ T (Z + ()+ = E E AAAATE
hA
T
A
A T
+
72
92
Similarly, replicating the proof with the counterparts, have the following:
E A'A
T
Te
+
1²(c
12
E
17 | Ahh T
hAiT IT
c²
+ 0 0 +0
(
r7² + p
p c 1)3
T k Ath
1
E
= =0
1112
112 + p2 p - 1)2 + +o +0 )
After simple algebra, the result follows.
E.5.7 TARGET ALIGNMENT: HELPER LEMMAS
Lemma 30. In the same setting as Section
we have that
n²c
(3. u) o(1) + ( <<1
E B. A)
z 3.]
2 p
2 (3. 2) + +0 e>1
Proof. For e < 1, from Proposition T
we get that
T1C
1/t11 A
B (z A)+T z B.
B
h
T B. +
ATT ku B..
7
To begin, we start estimating
E
3. u B.
Using our Spherical Hypercontractivitty, we have that e and 3__ h u.." B. satisfy the assumptions for Lemma 36 Then using Lemma
9 we have that
E and
Var L ) = (Ft
p²d
and Lemma 11 we have that
Var (B.nT T u TT.) o (
[ h u [.]]=
=0
and
pedd
Thus, using Lemma 37 we have that
E 3 u B. = o
E
p²d
and using Lemma 36 since all the means are O(1), we have that
Var 5 3. h 3.) = (max Var [
Var (B T u B.) B.)) =o
(0
54

---

Published as a conference paper at ICLR 2026
Then Lemma 19 gives mean and variance nf Since x does not satisfy the higher moment bound, and cannot be directly included in the product, we can include it via the classical bound:
7,' 74
E
m
3.
h
u
3.
=
E
1 71
E 3_ h u B. + V
Var 5 B. h YoTu" B. Var
1f²
= 0
71
71
(18)
For the second term, begin with
IB
[|]]33] A ku 3.]
Lemma 9
tells us that
B||||| =1 1 and Var (1(el3) = O 0
and Lemma tells us
1
and Var (B. ATT ku kuT B.) = =0 -
E [3] A ku B.] p² (B. u)² +o
Thus using Lemmas| 37 and Lemma 36
we get that
- and
Var (1 ATT ku (.)) = 0
E [| AtTAa ku [.] (3] 2²
to
+0
0²
Recalling the mean and variance for
from
19 we have that
9 1 3J ATT ku B.
=
1
E
||
12
AtT ku 3.] + +10
Var
772
0²1
(8.) u)² p²
+o
+
0
)
+0 1
1fc
p²
0²
17
n2c
1
= (B, u)2
+ o(1) +0
(19)
rpc
Combining these two terms yields the first result.
Similarly, for > 1, Proposition gives the expansion:
B. A) z B. =33 For the first term, we begin with
T uh.
1/12 us )
B.
nh B. h. 4 B.
1/11
G_ su B.
72
72
7/2
7/2
E
3 h. u Ba
Recalling form Lemma 11 we see that
and Var (B. TT. T.) = =0 -0(1t)
E
[ h u B. =
0
Thus again using Lemma 36 and Lemma 37 we see that
E B h u To. 0+0
and Var ST u TB. 0.)
E
pd
pdd
Next using the standard covariance bound on the expectation of the product. We see that
E
n
B h
u
T B. = =
1
+ +0 = -0
71
Pd
55

---

Published as a conference paper at ICLR 2026
For the second term, begin with
E
G! suB.].
Recall from Lemma 9 we have that
||| [|]
1
1
Var([If]]3) =0 pan and
(ptnr)
p² c
0²
and from Lemma 10
E su..]
-11-)
(G. u)²
and
Var su.3.) 0 (12
Thus using Lemma 36 and Lemma 37
we get that
(3__ u)
(
and Var (|1h|/3, suB.) (1)
E [|] suB.]
+ o
p²
0²
Recalling the mean and variance for n² from Lemma 19 and using the classical covariance bound for expectation of the product, we get that
2 11 3_ su T B.
1p
(
0 _
n12
E
=
su' 3.] VeTT
+
Var
72
72
72
p²n²
(3.T
u)²
))) +0 (
1
+o
+0
+
1/2 + 02
12
1/?
2 02 (B. u)² + o(1) +0
Then adding the two together, we get the result for e > as well.
Lemma 31. In the same setling as Section 2 we have that, for e <
p² + n2c (3_T222 u) + (1) (1²)
n7²c
+ + )
E E + T)tT ATB. |13.11 112
73
for c 1
1/2
113.
1 (3T +o(1) +0 +0
[3+ AT = 11.111
1/2 + p²
c
Proof. For < 1, using the expectation from Lemma 30
get
E 1 1) B.] E |.. (I Z(Z + A)7) B.]
n²c
! m
=113.112 p² 1²c
(B.T u)² +o + 0
For c 1, using Lemma 2 we get
3. (Z + A) A.
B, = =3, AA
n hTt
1/
h T h
7/11111
AA 1 us
mE AA un
B..
72
72
72
72
We then compute the expectation of each term above. To begin, we have that
E | 113 by Equation B
Next, we recall from Lemma II that
EB hT s TT.] = 0 and Var(3 TTB.) = 0 (1
pd
56

---

Published as a conference paper at ICLR 2026
and from Lemma that
5
Var (5
=0
E
7
+
p²
and
17
(pmn)
p2n
Thus, using Lemmas
36
and Lemma 37 have that
E
S
3
3. =
and
Var SB.nT
p²n
p
Then recalling the mean and variance of 1/2/22 from 19 using the standard covariance bound on the difference between the product of the expectation and the expectation of the product, we get that
T 3. 8 B.
( and
1 3. AA AAuun,
O 1
n
7/2
n2
Furthermore, for the next three terms, recall from Lemma 10 that
E|B, h h[]]] ||3.111 2
1
+
and
Var (B. h h.B.) =0
p²(c-1)
p2d
p²
and
E [ AAAus uss" [.]
c-1 c2
(3) +o(()(1))))))))))))))))))))))+ and
Var (g) AAtusT (..)s 0 =.)= 0B,
AA'us" To.
and from Lemma
Var (B. AA ulAB) = (
E|B AA AA
and
p2dd2
Then recalling from Lemma we have that
Var(I|s| (
E 1 =
and
Then using Lemma
36 and Lemma 37
we have that for third term
p²d )
+0 (1)
Var (1s||² h.3.) = 0 ((
||||=|123,11 h h,3.] p2d
11
12
and
for the fourth term
(}))
1
))
c-1 c (B. + +0(1) +
(1(2))
1
E| h AAtus" 1
0² c-c-1 +o
02)
p²d
(.. (BTu)² u)2
1
+ o(( +0
pee
p²d
with variance
us 1) =0 (0)
Var [111111 AAtus
p²d
For the first term, we have that
E 5 AA uh, = 0
and Var (5 ($3. AA AAtnn..)
=0 -00 p2d
E
pdd
17
Adding the last three terms and using Lemma
34 twice, get that
E
B.
[1s]|2hT h + AAAsTAAt
5 AAAuh) B.
p2d 11..112
13,uu²
p²c + 0 o(1) 00
17
With variance
Var BT [1s|p h TAAtus
SAAiur) B.
o
(
"
Then recalling the mean and variance of 17² \/22 from Lemma the expectation of products, we get that
19 and using the covariance bound for
n
||3.112
1 (sT u)² t0(1)+
E
3.
|| h + T AAl uus"
+ AA uh
B.
122
"7
172
+
d
Adding all five terms, we get that
E B. [Z A)TT ATB.] = |8.1
113.112
1
(3TTu)])
+ + +0
112 + 1²
d
c
57

---

Published as a conference paper at ICLR 2026
E.6 STEP 5: UPSCALING AND ASYMPTOTIC RISK FORMULAS
In the previous step we derived downscaled expressions for the four constituent terms of the risk: Bias, Variance, Data Noise, and Target Alignment. We stop our abuse notation and are explicit again about douwnscaled vs. upscaled.
Bias (downscaled). For e < 1, the bias term is
((2)= i ((022-02) az) + r²c+ p² (az aaa=
p
- 112 + p2) )
c
1
+ to
+ (0)
For c > 1, the bias term is
112 1 (1 - 22) ))
p²
+ t 2 ||3.11
nppp2
(B. u)² (āz -22)
6-1
d
(172 + p2)2
- 1 p²
+
+
(1ي² + 12)²
Variance (downscaled). For e < 1, the variance term is
p²
d a 2 +B. u)² (az ((02 aaa) (aaz) 2 (1 + p²)
²²
qpc
+ 22al(az-aa)
(12c 22)2
c2
d
n
+ 1 cp2
p2( 1 1-
For > 1, the variance term is
p²
a3
a²
172
np²
2
d
n
|13.11
+ (B_ u)²
(BTu)²
az 2a
+
Tggr
d
1 + p2)
c 1
n2 + p²
p2 c
p²
p²)
c
Data noise (downscaled). The data noise term is
a 0² 1..
Target alignment (downscaled). For 1, the alignment term is
2ā 2p [Iall8.112 (az - aA) (3.u)² u)2 120
1/e
d
For > 1, the alignment term is
A0² aA ||8.
11² d 272 + x
1 (az - 3A
17² (3, u)² w)²)
d
17 2
These formulas are expressed in terms of the concentrated building blocks, but still at the "microscopic" scale in which is ((Vi), p (1)), and 72, O(1/d).
In this section we return to the macroscopie, or upscaled, version of the problem. Specifically, we multiply each term by d and reparametrize according to
0
+2 =
2 2
while keeping p. ő fixed. This normalization ensures that the effective spike strength isotropic noise level p. and label noise Te,r are all of order one. this scaling, the risk times larger than in the downscaled representation, and the resulting formulas cleanly separate the contributions of the four terms.
The terms change follows
58

---

Published as a conference paper at ICLR 2026
Front factors (after multiplying by d).
j
02
xd
p²:
xd ā'p².
digr + re.
(20)
d
d
Denominator identities.
8² ccp
cf - 02 p²,
x²
(21)
Frequently used ratios and their upscaled forms.
P²
0²
(22)
7 C p
8 + p2
1
8
(23)
IPc C + p2
A + p
17
6
(24)
272 02
2 + cp
Ø²
cp
(25)
172 p2
82 + cpp
112 12
02 p²
(17 + 22)3
(02 + c2222 e
(26)
(n2 + p²)
c²
(02 + cp²) 1 (02 + p2p 1-c
(27)
(n+c+ 2²)²
1-c
Noise terms with aspect-ratio factors.
After multiplying by d and substituting T? d
d
112
22
c
for 1- cp²
p 02) (
1
02 1-c
2
p2)
(28)
2
C
1
f
(29)
p²
2T1 + p) c-11
ppc-1)
p² c-c-
( + cpp) c-
Alignment-specific identities.
n²c 0² +
Ø2
1/2 112 + p2
Ø²
(30)
T/c
p² + 02
02 + cp²
We now state the explicit upscaled limits for each component. As before, we present results separately in the underparametrized regime < 1) and the overparametrized regime (c > 1). Each term has little o(1) error term.
Bias.
For the bias contribution is
@ (((ā2-az) az) + 02 x² laz aa)] aa)) (BT [0T)) + +2
p²
1 d c 02 (202111) )
For > the bias is
g² (8Tu)² (āz (az) + p² ( az 22) ))
+ +
||3.11
(c-1)
g2
c p²
2
8 + p²
+ 2
d
02 + p2
d
0²
+ p2)
59

---

Published as a conference paper at ICLR 2026
Variance.
For 1, the variance contribution is
x² 11 u) ( (lo2 (az-aa) aa)² ( + cp²)
0²
+ 2aa - aA)
(02 + p²)²
1 1-c
+2+ + p2
1 c
0²
c
+Te
1
1 -
d
+
1 1-
For
L, the variance is
x²
13.11
a
a² d
0 + cp2
8²
+ (8.
u)²
c- 02 + cp²
c
8²
Q2 34 2
c
+ Tc
0² c - 1
1 1
d 2
(02 + cp2)
8²
c
c
Data Noise.
The data noise term is independent of c:
a² p² 3. 1
Target Alignment.
For c < 1, the target alignment contribution is
82
_ā aa [aa13..12 + (az - (A)
p² + 02
For > 1, the alignment term is
-2 aAA 13.
12
0A d
0 + cp2 |1..12
0²
(az - 24
0²
(3, u)2
0 + cp
Lastly, replacing P. P with i, and using d/ n + e yield the detailed expressions in Theorem up to simple algebra (rearranging terms and simplifying the fractions).
F PROBABILITY LEMMAS
Proposition 2. If u, v E R are fixed unit norm vector and A E Rddn is a Gaussian matrix with i.idd ((0, 1) entries. If n, then we have that
(u (uT (n) 22) (1 (u (TT))))(t b)) n))
n T AA'o)\= d(d +2)+ 2)
c (uT u o(1),
d-1
Var (u (ATa)))) = (2)
Proof. Let p= AAA. This is the orthogonal projection matrix onto the column space of A, denoted C(A) Gange(AA)... The subspace C(A) is an n--dimnnsional subspace of Rd Because the entries Ag are i..dd 1), the distribution of the random subspace C(A) is isotropic (or rotationally invariant). Consequently, the distribution of the random projection matrix P is also rotationally invariant. That is, for any fixed d x d orthogonal matrix Q, the distribution of PQ T is the same as the distribution of P.
We are interested in B[(u T Pv) Let 0 be the angle between u and v, such that cos = u To (since they are unit vectors). Due to the rotational invariance of the distribution of P. we can choose an orthonormal basis without loss of generality. Let Q be an orthogonal matrix such that u' Qu e1 (1,0.....0) .0) and e' Qu lies in the span of e and e2. Specifically, o' cos( + sin(0)e2. Let P = -QPQT T p' has the same distribution as P. Then,
u T Pu = (Q T u²) P(Q v') = (QPQ Jo' (u') p'v' Substituting u' = and v' = cos(O)e + sin(0)
u T P e P' (cos(())e + (fn)))
cos( T P'e,) (e P'ea)
cos( P/ sin(
60

---

Published as a conference paper at ICLR 2026
where Pij the elements of P'. Since P' has the same distribution P, drop the prime for calculating expectations involving the elements. Let u. T Pu. We then need EE] [
E[x*] = [[((ss((0) (0) P11 + (1)
E[ (0) P0) + (0) + 2 cos sin P1 P12]
cos (0) sin (0) + (0s sin (in(
Calculation Moments. We need to compute E(P)) and P1]11
Consider reflection matrix R that maps to -22 and leaves other basis vectors unchanged (i.e., R diag(1, 1,,,, -1,1,...,1)) Since the distribution of isotropic, invariant under reflection. Let P = RPR RPR. P has the same distribution P. The components related:
Ph (RPR) Ru P Pıi
and
P (RPR)12 R11 = (1)P12(-1) = -P12.
Therefore,
E E P1] B E[Pn P12) = P1j
This implies 21(P11P12] 0, so P 112 = O.
The diagonal element Pn1 e Pe, Pei l2 represents the squared norm of the projection of the fixed unit vector e₁ onto the random nn-dimensional subspace C(A)). This variable follows a Beta distribution:
d-n n 2 2
Pi ~ Beta
The mean and variance of a Beta (ax 3) distribution are n/2 and 3 (d n2/2, so a + 3 = d/2.
and
respectively. Here,
a+3
(a+3)
B[Pr]] d/2 d
n2/2
n
Next
Var(Pn)) (n/2)((d-n)/2
n(d n1/4
ne(d - 18 4d + 2)
2n(d n) dd + 2)
(d/2) +1) (d) + 2)/2)
Now we find E [[P]] using E[P?)] = Var(Pn) + (E[Pn))?::
2n(d - n)
11 2
E
(2d + 2) +
(
2n(d n) + (2 2)
d=dd +
2nd - 2n - n²d + 2n
d - 2)
2nd + ndd
()
n(n + 2)
d(d +
We use the property that P is a projection matrix, so p P. The trace is Tr(P) n. Also Tr(P²) Tr((P) n. We can write Tr(P²) = TTPPPPT since P is symmetric.
d ΣΣ Σ (Pb))
Tr(P2)
61

---

Published as a conference paper at ICLR 2026
Taking the expectation:
T(Tr(P2)) E
P2
E
P
By rotational symmetry, E[PA] is the same for all i, and [ sy)
is the same for all + j.
d
Σ
E P
=
There are d diagonal terms and d(d - 1) off- diagonal terms.
dE[Pi] d(d - 1) [[P
Substitute the value for E[Pi] (assuming d > 1):
n(nn + d(d + 2)
d
+ d(d E =
n(n + +2)
+ d(d - 1) B =
d + 2
n(n+ + 2)
n(d 2) n(n + 2) nd
2n n²²
22
n(d m)
d(d - 1) F[ = n
d+2
d+2 2
d+2
+ 2
n(d - n)
B d(d - 1)(d+2)
Substitute the moments back into the expression for [[X²]:
E E[x²] cos² (0) + sin (0) + 2 cos sin(t) -0
Using cos( (8) v. co) (u e)². and sin sin)
cos² (8) 1 (u w) n(d - n)
nnnn + 2) ((d+2)
B[(u AAT2) (u v)²
+ (1 - (u (uTz2)} d(d-d)) - 1)(d 22)
(1 (u (uuT))1)0 )) -n)
n
(u (u)))(n+2)
d(d 2)
d-1
(u v)
7
Calculation of Variance. Recall that reflection diag(1, 1,,,, implies P RPR. (equal in distribution) and thus IE[P of P12 vanishes. Therefore, we have the following expansion:
0, and in general any mixed moment with an odd power
E|x*] = cos* 4 E PA + cos 0 sin" (EP Pf1 + sin # [E
(31)
We start with E(P Since Pn Beta(a, B) with a n. B za We need the higher moments for the Beta distribution: for m 2 1,
a (m) (a + (m)
n (m)
E[PM
7
x ((m) x(x 1) .x 1).
(m)
In particular, we have the following third and fourth moments:
(3) 1
(
E[PH (3)
c + (2)
E
+
(3)
c3
(4)
We now move on to E.P From idempotency, (P²) = Pu gives the row identity Pu1
PPJ.
Ed_ Ph. Multiplying by Ph and taking expectations, we have that
E[P = E(P + Σ E[P? Pi = E.Pl + (d - 1()
62

---

Published as a conference paper at ICLR 2026
E[Pi [[
(2)(3)
(2)
1
(4) = =0
B[P] [1
d-1
d
($)
(2)
d
c
We still need to evaluate or upper bound B[P From P11 = Ek=1 P2 we have Σk=2 Pik =
P11 Pn. By
d
d
2
P
Σ
p2 1k
= (P11-P11)
=
k=2
k=2
Taking expectations, we get:
(d (d- ()- 1111 B
21[P1] 1[P
- {
1 d-1
(1 2
1
- 0 (1)
E/P - d-1 02
d2
We now plug these expectation bounds into Equation 31
E E = cos 1 A (4)
1
6 cos ssnn 0 + 0 (1) sin
+
0
(4)
d
1 σ u 0)² +0
d
Recall from the prior proof that:
E[x²] = cos² n(n + 2) d(d + 2)
+ sin2 8 n(d - n)
d(d - () 2) ā ( u) + (
Finally, we have that the variance is of order:
Var(x*)=B[Xx> = -[x+ -XX"))()1 =0())
Lemma 32. Let a jf 0 be constant and suppose that ( = a + o(f(m)) as n - oc. Then,
= + off())).
a
Proof. Write ( = a Tn with rn (n)) Then
1
a 4
Fa
a 1+ En
Using the expansion
O(w3)
as u + 0,
1 u
with rn/a, we obtain
1
1
In + ((rn/a))))
1 Fa + +(r2)
a
a a²
Since r. = (m)) and f(n) 0, we have 22 = o (nf).. Therefore
1 df(n)))
a
which is the desired expansion.
63

---

Published as a conference paper at ICLR 2026
Lemma 33 (Variance of a reciprocal). Let X be random variable satisfying
E[X] = a> 0 and Var(X) = = o(1),
and assume that X is bounded away from with high probability. That is, there exists C' E (0,a)
such that
Pr[x C7 =1-((1
If there exists an M such that
E
x
S
M
and [xx (xxx) 0
Then
1 X
1
Var
= at
Var(X) + t (Var(x)),
so in particular, Var(1/X) = d(1)
Proof. Let == a. Then
E Y = 0,
E [Y²] =
EE [
By Taylor's theorem with Lagrange remainder for f(x) = 1/ I, there exists = -(X)== E (0. 1) such
that
1 ---- Y
y2
+2
Z
2 0.
X
a
Q²
(a 0Y)
Write A =
Y
Z. Then
1
Var
E
A
(B])))
X
We will show
o²
E
[A>
a1 + o(02)
and (E [E]))) = =(o0)
Let G = (X 2 C} and B = (X < CJ. Since C < and E [Y²] = o², Chebyshev gives the
1x>
quantitative bound
E [
σ²
Pr Pr 2 - c
(a C)2 = 0(o²) = o(1)
(a C)2
Second moment E | We split over and B.
On G. Since + OY 0X + (1 - 0)a 2 C, we have
Y2
y1 C
| <
2 <
C
Therefore
{-/
Y
2
1
E
1a
R
2
1
E Y 7 1cl
[E[Zic]]
a
a
We bound each term as follows.
F
F [ = 0
T6
and, using 1G I and Lyapunov/moonotonictyy of L norms,
(1 -10)) o
E || 2 lc] -
Moreover,
E - (v21n] E[y21a] (B(Y")) 1/2 Pr 0(03) Pr [B] =o
64

---

Published as a conference paper at ICLR 2026
Hence
Y
2
o²
IE
+2)
16
+
o(o
a
a
On B. Using the algebraic identity
[ 1 x--1²=
y2
X
x2x2'
Cauchy-Schward and Hölder (with exponents 2, 2) give
E
A
1
1 a2
E
Y2
1B
1 a²
IE Y
1/2
(E X
1B))
1/2
1
[E|x
(B x
10 1/4 Pr Pr[BI
X2
a²
Under the lemma's 's assumption
[X $] 1 M, we get
E
A 1 B ( Pr [r[ 1/4
= o
Combining the G and B parts,
a²
E
a
+ o(
a
Mean correction (E [A]) Since EE [Y] = 0, we have
E [
lc]
+ E 7
18)
On G, Z S so
E 111c]
lc
5
0².
C
C
On B, The inequality
y2
X²
Z
0Y)
y3
holds on set because on this set as X < a, meaning the point a 0Y lies between X and a, so + OY > X. Thus, using Cauchy-Schwarz and Hölder,
Y2
E E 1 B
S E
X3 B
S (E [Y
1/2
E x
BB)) 1/2
s0(02) (R
1/4 o(02).
Thus E |||||
(o²)
and therefore
= -(02)
Putting the two steps together,
0² a + σ
Var
x
E -
(B[A))
Var(X) + o(Var(X))
Lemma 34 (Variance of sum). Let A and B be any random variables with finite variances
V(A) = Var(A) and (BB) = Var(B). Then,
Var + B) = (V((A+/V(()))
Proof. Recall that
Var(A + + B) = Var(A) + Var(B) + Cov (A, B).
By the Cauchy- Seachh--Shwwaziiiie inequality, have
CCov (A, B)I S (V(A)((B)
Thus,
Var(A ( VV = -(V(A)+++
65

---

Published as a conference paper at ICLR 2026
Lemma 35 (Variance of one product). Let A, B be real random variables with means [[]] b = E B and finite variances. Assume
E (A a)²|
- KA Var(A)
E BB b)¹} - KB Var(B)²
Then, with C4 = (KaKB)
VVar(AB) I |a| Var(B) + [6] Var(A) + C. VVr(()) Var(B).
Moreover, Var(A), Var (3) + 0. Var(AB) = (ar))r(B) (b²Var(A)) + o(Var(A) + Var(B)).
directly follows that if all the means are ((1).
Var(AB) 0 (Var(B)) + O (Var(A)))
Var(ABC) O (Var(C)) + O ((a((B())))))))) + 0 (Var(A)) and so on by induction.
Proof. Write
AB ab a B bb
Using Var(U +V) Var(U)+ (ar)))+Va(V))22CCC C V) and [Cov(U, V)I S Var(U, get
Var(AB) = Var aB bb AB)
$ ( |aly Var B) + 1b1y Var A) + V Var AB
2
Since Var(A) Var(A) and Var(B)
Var(B), it remains to bound Var (AB). By
Cauchy-Schwand (Hölder with p = q 2),
Var AB
< E A2 8
(B (B[ 1/2 (R (R[B*T) 1/2
Since assume fourth-moment control E [
S KA Var(A)² and E [B*] s K Var(B)?, then
Var AB
(KAKB)'4
(Vr(())Var(B)).
Hence
Var(AB) & (lalv + MM (ar(A) + C4 (ar((A)aar(B)) 2
C::= (KaKB)
For the moreover part, using the exact variance expansion,
Var(AB) = (VVa(B))+ (A) Cov Bar (AB) +2 Cov AB) +2b Cov (A, AB)
we bound the three remainder terms using Cauchy-Schwarz and the fourth-moment control:
Var AB
S E
<
A4
1/2
[B4]) 1/2
s <C Var(A) Var(B),
Cov
BB
v
Var B3
V
Var
AB
< C. Var(B) V Var(A),
Cov [, (B)
/ Var A
Var
AB < C4 Var(A) / Var(B).
As Var(A), Var(B) (B) 0, each of these is of o(Var(A) + Var(B)).
66

---

Published as a conference paper at ICLR 2026
For the covariance term, Bauchy-Schwarr and the inequality 2ue S cu? + e (for any 0) with := / Var(B) |b| / //Vr(A)) give
2aab Cov(A, B - 2/abl Var(A) Var(B) & E e Var(B) + +=
Therefore,
Var(AB) < (1 + a Var( (1+ (( o(Var(AA) Var(B)).
Choosing, e.g., E yields
Var(AB) ((a 0 ((2 o(Var(A) Var(B))
which proves the moreover statement.
Lemma 36 (Variance of general product). Let m 2 2 and let X..... Xm be real random variables with nonzero means μ4; = E[X;] # and variances f((n) == Var (X,) - n oc. Assume that for some integer M 2 m (it is enough to take M = m))
E X;
14l 2M
= O(Var(X:)))
foreach each = I,..., m.
(32)
Then
Var
II X3
= of m
fl(m)) 2) = o
max fi(n)) 15asm
Proof. Write ;;
= X; μu so that E[A]] and | = o;. By assumption Equation 32 with M 2 and monotonicity of Lp norms,
| = (v(()))
for every 15 & S m, ..... m.
Expand the product multilinearly:
2
II
P4
Σ
II jEB=
My
II
A
[+
\ieS
Taking L2 norms and using the triangle inequality,
II
X
II JES-
|ngl
t=1
OFSC(m)
iES
L2
For fixed nonempty S with |S||
k. apply Hölder with exponents all equal to 2k:
|Alll =o
(I
II A
< II
VI
iES
L
iES
1ES
where we used م
O(VT.) for
Let G Vl((). Summing subsets S shows
m
m
m
II
II
14
<
(1(++)))) < A(
La
where 3 m on n)). Hence
Cj and is a constant depending only on m. {mc), and the moment constants (not
2 O(
171
-0(2 F((n))
Var II
X
<
p4
L2
Since is fixed,
(Σ Ci 2 < m² max; ft. giving the claimed bound.
67

---

Published as a conference paper at ICLR 2026
Corollary 1 (Higher moments of the centered product). Fix p 2 1. Under the hypotheses of Lemma 36 then
m
III:
X,
EE X
Cpm
E
(II JES
B[X,]]
II IVv. o(1), iES
(ASC(m)
and hence X
X, x o
Lemma 37 (Expectation of Product vs. Product of Expectations). Fix à: 2 2. Let Xi,..., X be random variables. Assume:
1. Uniformly bounded means: sup,, [[[x M < oc.
2. Vanishing variances: Var(X,) fn) with f((n)
n oe for each i.
3. Moment control up order k: For each i and every E (2.....k).
E I B[X,]["] < Cp Var(X,)
with constants Cr-
Then for finite k, we have:
EX; o (( fi(rr) 1)-
max fa(m)) ISisk
E
X;
1=1
1=1
Proof. Set A; = X, E[X,), so EA, = O, Var(X,) = Var(A.) = [n)), and by assumption
| 1,, = (EI //8 s ///o fi(n)1/2
= 2....,
Using the multilinearity of expectation,
&
(
II II
[
Σ SC(M]
II NiES
jES
Thus,
X,
II
E[X
Σ
II II
E[X,].
O*SCIKI
LIES
jÉS
Then taking the expectation and noting that Is E[X,] is a constant, we get
E
X;
EXX = Σ
E 4
E[X,].
(ASC
LES
JES
E[Ad] Hence every singleton term vanishes exactly, and the sum begins at |S| = 2. From the bounded means assumption,
If (2) then E [[Les A,
II E[X,] jEs
S MM 1-1
VS C [k).
Fix a nonempty subset S with |S| m 2 2. By generalized Hölder with all exponents equal to m
(so Eies m
1).
E
I A
<
A 11, Em.. S II
(cL/m f((n)1/2)
Cm II fin)..
LES
FES
EE
VES
Therefore, for every with [S| m 2
E
A
E.X,
<
M
k-m Cm
(m))
LEE
SES
68

---

Published as a conference paper at ICLR 2026
Let f((n) 0. Denote by
Em (.n......., (1) = Σ
II c
SCIR |S|=m
YES
the m-th elementary symmetric polynomial. Summing the bound from, we get
E
X
II
EX,
<
M mm em(ei..... Ck).
1=1
x=1
m=
Let M, == maxggmsk M fe-m Cm- Since cm 2 0 for G 2 0,
k
M km Mm-mCm cm S M, Σ
Cm :k.
m=
m=2
Recall the identity
II 2=1
(1+ +1++)) c.) = Σ
em (c) =1+
1+
cm cm Ck).
m=0
m=1
so that Σ
2== cm
i== G) - Σ Cj. Hence
I M. [I (1+ -
E
X
EX2
Σor
ci
1=1
1=1
Let E = Σ C4
- as - co. Since log(1 + u) S u for u 20,
[I = exp Σ log(1 + ci) S exp(E3).
Thus, the difference is most ç E [0, E). so =
- By Taylor's theorem, e= 1 111 1 2 for some 2 < 6 (since & and of S {²² Therefore,
k
M. = = O(E3),
X,
E.X,
I 2
Li=1
as E and e= + Since E = (E Vf(()) we get the result.
Lemma 38 (Moment preservation under monomial ++ Hermite change of basis). Fix M N and
degree E N. Let
M: {x7 TEN", sl) ::= (Ha CENN (a)
with (x(x) -II H
Ha, (xz) the probabilists' Hermite basis. For any (random) coefficients (athier define the random polynomial P(x) Ehis. ag x", 2, Then there is deterministic,
invertible matrix T T(M,r) such that the Hermite coefficients {callalsr P(x) Ejater Ca Ha(x) satisfy
To.
Consequently, for any p 2
||calle, S
[Taz Gaallllyll,, for all a,
hist
so if each a, E Lp then each ca E Lp. Moreover, since is invertible, the converse also holds: if each ca Lp then each a., Lp
Proof. In one dimension, each monomial admits a finite Hermite expansion 2: _|m/2] fm (x)) with deterministic coefficients tmji in several dimensions, take tensor products obtain x² = Emals Tan, H. (z). Ordering multi-indices by total degree yields block upper-triangular, deterministic, invertible matrix T T(M, r). Linearity gives Ta. p 2 Minkowski's inequality yields ||ca l|.,, E, Ty,,, < Σ, [Tayl [aall r,,, finiteness of all ||al|ı, implies finiteness of all ||ca]le,, Invertibility gives the converse using T-lc and the same argument with T-1
69

---

Published as a conference paper at ICLR 2026
G PROOF OF SPECIFIC CASES AND OVERHITIIGG
G.1 PROOF OF THEOREM
Proof. We set az aA = âz a a. 0 0. in the above Theorem 5 and note that greatly simplifies each term. Algebra shows that for c <
8²
Variance = a 21,,11, + ? c c
Ø²
Bias T8 d(A + +
d(A2 + +2)
Data Noise = 2²7 a² 13.1, Target Alignment =
-2..31113111
While for > first send d, n and many terms become asymptotically 0. In the end, we get that:
Bias = (²8 u)² 1 (1-
T²C
Data Noise
==²>"13111
8² +
+²c
1
11 1
1
Variance = Variance=
a² (B, u)²
+ T
02
T
c
1
((1-)
0
Target Alignment 20 20
(B.
u)²
+ ||3.
Ø2 + 72c
Adding these terms together, see with simple algebra that many terms cancel can be combined, establishing the stated formula.
G.2 PROOF OF THEOREM 2
Proof. We set az âz 0 ã.A. 0- = 0, and send d, n - 0o in Theorem A. = az 2s and A = a0 0AA- Then some algehra shows that for c < 1,
Recall that
72
2
Bias 0² (3,T.u)233
Data Noise = 1
@ 72
82
Target Alignment = 2222=21133
AT ( u) A1
0 + +
1 0+ + 72
0
Variance 13.71 + T 2
+ 73, u)
A + 20 A
(02 72)3
@
For c >> 1, we have that
72C
Bias (3.
Data Noise 03372113.112
1 + +2c
Target Alignment -2a33 ? 113.112
0²
A ( (B.Tu)2 1)2 A.
1 + T2
Variance = a²²"" ||..12
02
+ (B. 4)² 1- c 0
c
-
72
We proceed by adding these terms together and the results follow from algebra.
G.3 PROOF OF THEOREM 3
Proof. We set Ø = 0 and 7 = r in Theorem 5 and have the regime of equal operator norm 0 = 2.72 Since we are interested in the limit + co, we only consider the overparameterized case c >>> 1. We first take the limit d, n + oo and have that:
Bias = 73T u)² (vilāz-az)+
-0z) az)+ (az
cV7
Data Noise = 21,3,1113311
7 + c
Target Alignment = -2
((02-
8A
7
(3, 4) + o.A 1|3. 1
+
70

---

Published as a conference paper at ICLR 2026
Variance = r²a3 113 + (3. (c- (c-1) (az 0A 0A)
c ?
2
+ T - --1).
The rest follows from simple calculus: if ăz + az. = wc(1). and 3. u * 0, the bias will diverge and other terms are controlled, yielding catastrophic. If az az w.(1) > S oe(c²), and 3__ u o O, a similar thing happens. In other cases, all of these terms are controlled and become finite values the limit lime Re- -e giving us tempered overfitting.
a 11,-1113111
β
x² [10 (3. u)² +0311.111
B / u,7 = Oc(1)
0g + 0z 1 u,7 = w(1)
lim Re
0
ag = uz B. 1 u, (1) S & o(c²)
72
[W+1)= 02 - 26 2σ (0. u)² + +
az á B, 1 u,? dc?
7² (a 2ā
2ā 2)² u3) (..11
az u B, + u, w((?)
G.4 PROOF OF THEOREM 4
Proof. We start with the first part and assume that az # āz. Similarly, we have that = 0 and T in Theorem 5 To achieve equal Frobenius norm, we set 02 dr2 and send d, n - oc so several terms would vanish.
In particular, for e < 1, we have that
72 2 Bias (3( 62 az az + (az - aa) g2 72 = (8, (Vllaz- + (az - aA)
Vd
d d
is clear that this term becomes oo since the term inside the parentheses scales with d. Note that the variance and data noise non-negative, and target alignment is controlled. We have that R. 00 for c E (0, 1).
For the same logic follows, and we also note that:
(Vllaz + (az ) ) -
a1 vdc
2
Bias 03 u)² (áz-az az + az
as
0 + 720
d d
which scales with d with other terms controlled. Hence, R. = oo for e e
Now assume that az ãz. Since we are interested in - co, we only consider >>>>> 1. First, from algebra and taking the limit for d, n, we have that:
ad cVd 2 Bias = (BTTuu² (a2-
+0,
Data Noise = 13,72113.113
c
d
c+c
Target Alignment = 22ā (11z-2) aAA
1
(B. u) a.A
Variance = =2a03 113.112 + 72 u)²
c
(az - aA 2
+ is
c
c-
We now take and many terms vanish in this limit, yielding:
lim R. 2B u |B.11 = =² (13. (0)Tu)(a32 -2 (1,02)+113.111 113.1|22]
6-000
71

---

Published as a conference paper at ICLR 2026
Proposition 3 Mlon-existence of canceling scale parameter). Let az > 0 be fixed scalars, let u. B3, E R be fixed vectors, and set
113.11
b b= (3. [0,a].
For every positive real number define
f(6) 0² a + (u²(1++2)) 2)) - 2azas)
Then
f(6) > for all Φ >
Consequently the equation f()) = 0 has no solution with @ E (0, 00).
Proof. If (i.e. G. is orthogonal to u) we have ((0) a a 0, positive & can cancel
the expression. Hence assume 0.
Writing r b/a (0, 1] we obtain
f(6) aooa + azz 20 r+ + a2r
Since r $ 1,
a + azzzaz 2 2 a az(az 20 (as - (aa= 20. 0.
Thus the square bracket in (+) is the sum of a non-negative term and strictly positive term.
72


Reviews:
[{"review_id": "wKa76VFDk4", "reviewer": "Reviewer_bT6s", "summary": "This paper studied the asymptotic generalization error of the minimum-norm interpolation solution of linear regression when the data has a rank-one signal in the population covariance. Under the proportional limit, the authors derived a systematic description of the generalization error with respect to the aspect ratio, the signal strength, the alignment between target direction and data signal, model misspecification, and train–test covariate shift. This detailed analysis gives us a full picture of benign, tempered, and catastrophic overfitting regimes. For instance, when the target is aligned with the data signal, increasing signal strength can have a transition from\ntempered overfitting to catastrophic overfitting, then to tempered overfitting, and then to benign overfitting,", "questions": "1. What is $(\\tilde{X},\\tilde{y})$ in Theorem 1? Do you only consider one test data point $(\\tilde{x},\\tilde{y})$? Please clarify the notion $\\tilde{Z},\\tilde{A}$ in all theorems. For instance, in Theorem 5, are you considering $\\tilde{Z},\\tilde{A}$ as vectors $\\tilde{z},\\tilde{a}$?\n\n2. Do you assume $\\alpha_A\\neq \\tilde{\\alpha}_A$ in Theorems 3 and 4? Why is there no $\\alpha_A$ in the limit of $\\mathcal{R}_c$ in Theorem 4? It would be better to clearly state the equal operator norm and equal Frobenius norm conditions in Theorems 3 and 4, respectively.\n\n3. Can you compare the test errors of training 3-layer ReLU networks in section 3.5 and linear regression on the same training and test datasets? Are the test losses for 3-layer ReLU networks always smaller than linear regression model?\n\n4. Typo: line 1592: (Lemma 12\n\n5. Where do you use Lemmas 13 and 15? Can you explain the application of the Gaussian hypercontractivity in the proofs?\n\n6. Why do you need to assume $v$ in (1) has i.i.d. standard normal entries? In Assumption 2, do you need to assume independent entries of $A$, or is it uncorrelated enough? There should be a detailed discussion on the three conditions of Assumption 2. Can we use the first two conditions to imply the last condition of the Marchenko–Pastur law?\n\n7. In (2), can you consider $\\beta_*$ as a fixed vector? Why do we need to assume that it is uniformly distributed in some subspace?\n\n8. In this paper, you focus on the overfitting regime. How about the underfitting case when $c\\to 0$?\n\n9. Is there a heuristic explanation of Figure 2a? Why could there be a region where the aligned risk is lower than the anti-aligned risk, but the aligned risk becomes strictly larger outside this region?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written, and the statements are clear to me. This paper provides a comprehensive understanding of how varying signal strengths and target-input alignments impact risk in overparameterized settings. The transition between benign and catastrophic overfitting is an important and interesting topic for generalization theory and the ML community. It also includes extensive simulations for both linear and nonlinear networks.", "weaknesses": "The main limitation of this paper is the model assumption. This paper only studies a linear model, which may not generalize well to nonlinear models for the transition between benign and catastrophic overfitting. Besides, it only considers the minimum-norm interpolation solution. It would be more beneficial to generalize these results to the ridge regression case, where altering the regularization parameter may yield more diverse outcomes. Additionally, the regimes of benign, tempered, and catastrophic overfitting are highly intricate from a theoretical perspective. It would be beneficial to provide some heuristic explanations of why these phenomena occur differently in various cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5R2hqSjUE", "reviewer": "Reviewer_uSkU", "summary": "This paper studies the generalization error of linear regression using spiked covariance data models, characterizing the impacts of spike strength, target-spike alignment, target model misspecification, and train-test covariate shift. The theoretical output of the paper is an exact expression for the generalization error in the considered setting. This allows the authors to identify the regimes where spike strength and target-spike alignment lead to improved generalization.", "questions": "1. What is the positioning of this paper in comparison to the papers I mentioned in weaknesses 1 and 2?\n\n2. What is the motivation of the specific data model (in line 45)? If it is to introduce a non-linear function of $\\mathbf{x}$, why don't you use $\\sigma(\\boldsymbol{\\beta}^T \\mathbf{x})$ for some nonlinear $\\sigma: R \\to R$?\n\n3. Is it possible to connect the data model to the one I just mentioned?\n\n4. If the target is a non-linear function of the input (as stated in lines 49-50), how can we expect the linear regression to perform well? What is the point of studying this case?\n\n5. What happens if you apply ridge regression (with a regularization constant that increases with $c$ (ratio of dimension to samples)) in Figure 1a, for example?\n\n6. Could the authors explain the practical relevance of their setting (specifically, their data model of input and target)? Also, is it possible to identify real-world datasets/scenarios that approximately satisfy the assumptions and experiment with them?\n\n7. Is it possible to extend the assumption to multiple spikes?\n\n8. Is the setting (together with the assumptions) of this work enough to characterize the generalization error for neural networks trained with one gradient step (Dandi et al. 2024; Moniri et al. 2023; and [3] above)? If so, it would be a strength?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Rigorous asymptotic analysis of the generalization error of linear regression in a setting involving multiple interesting variables (i.e., spike strength, target-spike alignment, target model misspecification, and train-test covariate shift).\n\n2. The writing of the paper (including figures and theoretical results) is of good quality. Therefore, it is easy to follow.\n\n3. The studied setting is somewhat interesting, and it can be useful for understanding some effects of data distribution on the generalization error.\n\n4. The paper includes experimental extension (results) for multilayer neural networks, which is beneficial, although the experimental results are also limited to the synthetic data setting considered in the paper.", "weaknesses": "1. The paper lacks highly relevant literature and contextualization relative to the prior work. Specifically, there are at least three papers (about generalization errors with spiked covariance) that are quite related to this work but not mentioned in the paper:\n- [1] studied \"generalization for least squares regression with spiked covariances\", which is also the fundamental topic of this work. Also, the settings are quite similar for [1] and this paper.\n- [2,3] analyzed two-layer neural networks with spiked covariance, which leads to spiked covariance for the features learned by the model (as discussed in the introduction of this work as well), so these works are also relevant. Specifically, [2] focused on the effect of spike magnitude and spike-target alignment, which is also the focus of this paper. \nTherefore, without proper contextualization relative to these prior works, this paper is weak in terms of positioning and novelty.\n\n2. The fundamental theoretical challenge relative to the prior work should be discussed. Specifically, considering the closeness of this work to related setups in [1], Hastie et al. (2022), Sonthalia & Nadakuditi (2023), and [4], the difference in the proof/derivation techniques should be explicit. Since this is currently missing, the results in this work can be considered trivial generalizations/extensions of the other results. \n\n3. The motivation for this specific data model (in line 45) is unclear. While being the main object of study and the source of most of the claimed results, the authors do not explicitly motivate the data model $\\mathbf{y} = \\alpha_Z \\boldsymbol{\\beta}^T \\mathbf{z} + \\alpha_A \\boldsymbol{\\beta}^T \\mathbf{a} + \\boldsymbol{\\epsilon}$, other than saying it is a non-linear function of $\\mathbf{x} = \\mathbf{z} + \\mathbf{a}$ for different $\\alpha_Z, \\alpha_A$ cases (introducing mis-specification).\n\n4. Considering that the target is a non-linear function of the input (a.k.a. mis-specificed case in the paper), studying the performance of a linear model becomes irrelevant (not that interesting). What is the point of studying this case?\n\n5. The benefit of spike-target alignment on the generalization is known in the literature [2]. On the other hand, most of the cases where the alignment hurts the generalization in this paper can be attributed to the lack of proper regularization. For example, in most of the figures where the alignment is harmful, the authors set the spike magnitude $\\theta = O(\\sqrt{c})$ with respect to $c = d/n$ (ratio of dimension to samples) and let $c \\to \\infty$. In this case, the model is highly overparameterized, and the norm of the input is large (due to the spike magnitude), but there is no explicit regularization beyond the implicit nature of the min-norm solution.\n\n6. Spiked covariance with a single spiked direction (Assumption 1) is considered, limiting the practical relevance of the found explicit expression for the generalization error. Also, the practical relevance of the considered target $\\mathbf{y}$ model seems unclear.\n\n\n**Related work that are not mentioned in the paper:**\n\n*[1] Li, Jiping, and Rishi Sonthalia. \"Generalization for least squares regression with simple spiked covariances.\" arXiv preprint arXiv:2410.13991 (2024).*\n\n*[2] Demir, Samet, and Zafer Dogan. \"Random features outperform linear models: Effect of strong input-label correlation in spiked covariance data.\" arXiv preprint arXiv:2409.20250 (2024).*\n\n*[3] Demir, Samet, and Zafer Dogan. \"Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure.\" International Conference on Learning Representations (2025).*\n\n*[4] Li, Xinyue and Rishi Sonthalia. \"Least squares regression can exhibit under-parameterized double\ndescent.\" Advances in Neural Information Processing Systems (2024).*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qaPOguKwwQ", "reviewer": "Reviewer_KtDe", "summary": "The authors study a min-L2-norm regression task.\nThe data x is a Gaussian along a direction u plus an isotropic bulk, while the labels y are a linear model of the projection of x on direction u plus a linear model of the bulk part.\nThe predictor is a linear model learned with min-L2-norm regression.\n\nThey study:\nGeneralization: MSE evaluated on a new data point (including noise, and possibly with different coefficients in the target function - to model covariate shift). They consider the excess risk to remove the noise plateau.\nAlignment: how beneficial it is to have the task weights aligned to the hidden direction u for the generalization? \nOverfitting: when c -> infinity (n << d, few samples, perfect fitting of dataset is possible), does the generalization go to zero (benign), to a constant (tempered) to infinity (catastrophic)? \n\nThey consider both the scaling in which the spike dominates, as well as the one in which the spike is of the same order of the noise (spectrally).\n\nTheir main result is Theorem 5, providing a full risk decomposition for the setting under consideration. \nIn Section 3 they study generalization, alignment and overfitting in the case of, respectively:\n- Section 3.1: well specified problem, in which the target function is linear in x (and not linear in the spike and bulk contributions of x)\n- Section 3.2: missspecification but no covariate shift: the target function is not linear in x, but testing is done with the same target function as the training data\n- Section 3.3: missspecification and covariate shift: the target is non-linear in x as in the previous case, and testing is done with a structurally identical target, but with spike-bulk contributions altered.\n\nTable 1 classifies all the regimes studied by the authors.", "questions": "In your work you consider min-norm interpolation, i.e. the limit lambda -> 0^+ of ridge regression with regularization strength lambda. How hard would it be to generalize your results to finite lambda? Would that make sense in the context of studying overfitting? \n\nTypos:\n- line 45-46: z and a should be defined clearly given X in line 34. Are they the columns of Z, A? This gets defined only in line 180\n- line 45-46: what is the dimensionality of y? It seems a scalar, but is boldfaced. Same for epsilon in eq 2.\n- line 103: \"we shall ...\" seems to miss a verb\n- Figure captions sometimes lack the values of parameters (such as d and n etc) at which the curves/experiments are plotted", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors provide a complete classification of overfitting in linear regression, studying in particular how alignment between data structure and task helps or hinders generalisation, in a toy but fully controllable learning setting. This creates a nice benchmark to understand the phenomenon in more complicated settings (non-linear estimators for e.g.).", "weaknesses": "It is unclear to me how specific the results are to the data model. The authors do not discuss which elements may generalize to more complicated data, and which are surely specific.\n\nThe presentation is a bit heavy in Section 3. The authors present the classification of behaviors, but provide little intuition on why the behaviors observed make sense, or why it challenges common beliefs. This results in a bit of a dry description of the phenomenology, from which is difficult to gather a more general take home message (which is more nicely summarised in lines 60-70).\nMaybe this is not possible, but I invite the authors to see whether this element can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iSeOXGJR5g", "reviewer": "Reviewer_ywoE", "summary": "This paper analyzes the performance of min-norm interpolators under a spiked covariance assumption for the data in high-dimensional linear regression. The authors consider the asymptotic proportional regime where both $n$ (number of samples) and $d$ (input dimension) diverge to infinity with a fixed aspect ratio $c = d/n$, and specifically examine scenarios where the spike in the covariance is correlated with the teacher vector. \nAfter deriving the limiting value of the excess risk, the authors classify different conditions leading to benign, tempered, or catastrophic overfitting in the overparameterized regime. The main theoretical finding is that increasing spike correlation can, counterintuitively, induce catastrophic overfitting. \nThe authors provide limited empirical evidence suggesting that analogous phenomena appear in 3-layer ReLU networks under similar data conditions.", "questions": "Most of my questions for the authors are connected to what was previously written in the Strengths and Weaknesses sections.\n\n1. Have the authors tried to verify the theoretical predictions with other network architectures beyond 3-layer ReLU networks? For example, does the same behavior on synthetic data appear for Convolutional networks or Residual networks?\n   After establishing the phenomenon on synthetic data, do similar alignment-dependent phase transitions appear on real-world datasets for the same models?\n\n2. Can the authors provide guidance on how practitioners can determine whether their data exhibits spiked covariance structure? What are the practical tools or diagnostics that would indicate when this theory applies? Has this been shown on some specific datasets for example? This would greatly enhance the practical relevance of the work.\n\n3. The paper shows one experiment with nonlinear networks but provides no theoretical justification for why the linear analysis should transfer. Can the authors provide:\n   - Theoretical analysis (even informal or pointing to relevant references) of why these phenomena persist in nonlinear models?\n   - More extensive empirical validation showing the boundaries of when the theory does and does not apply?\n\n4. Given that the $(\\alpha_Z, \\alpha_A)$ parameterization represents a specific form of misspecification, can the authors comment on what types of real-world problems might naturally exhibit this structure? Are there examples from machine learning practice where targets are known to have this differential dependence on principal and bulk components?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents several technical strengths:\n\n- The paper shows a high degree of mathematical rigor: The theoretical analysis appears sound, with the novel results properly recovering the known results from Hastie et al. 2020 as special cases. The generalization to spiked covariance models beyond isotropic settings is a natural extension of existing literature.\n\n- The systematic classification of overfitting behaviors (benign, tempered, catastrophic) across different parameter regimes provides a detailed characterization of the problem space. Table 1 is especially effective in summarizing the different cases.\n\n- The paper uncovers counterintuitive yet interesting behaviors, particularly that alignment with the spike is not universally beneficial and that increasing spike strength can worsen generalization before improving it.", "weaknesses": "While I recognize the substantial theoretical effort invested in this work, I believe the paper in its current form has some significant presentation and validation issues that limit its readability and impact, especially for the broader machine learning community not only interested in theory.\n\nMy main concerns are the following:\n\n1. The paper lacks in accessibility. The paper prioritizes mathematical formalism over intuitive explanations, with limited discussion of *why* these behaviors emerge or *when* practitioners should expect them. \nFor a venue like ICLR that serves a diverse ML community, the paper should consider a more accessible presentation alongside mathematical rigor. \n   \n   *Specific recommendation*: Each major theorem should be followed by an intuitive explanation section that uses simple examples or visualizations to explain the underlying mechanisms. For instance, why does increasing spike strength lead to catastrophic overfitting before benign overfitting? What is the intuitive mechanism?\n\n2. The data model (first display on page 1) represents a significant simplification - a rank-one spike plus isotropic noise. While mathematically tractable, it is unclear whether this captures realistic data structures encountered in modern machine learning. Real data show both hierarchical structure and other types of correlation with the ground truth rule. Additionally, real data have been shown to follow power law covariance structure.\n   \n   Moreover, even from a purely theoretical perspective, the way non-linearity and model misspecification are captured through the two coefficients $\\alpha_Z$ and $\\alpha_A$ (which differentially weight the spike and bulk components) is quite restrictive. This specific parametric form of non-linearity—where the target depends linearly on spike and bulk components but with different coefficients—represents only a narrow class of possible misspecifications. \n   \n   *Specific recommendation*: (a) Include a discussion about the relationship between the spiked covariance model and real-world data, with empirical evidence showing when real data approximately satisfies these assumptions. (b) Discuss the limitations of the $(\\alpha_Z, \\alpha_A)$ parameterization for capturing general forms of model misspecification, and acknowledge what types of practical scenarios this does and does not cover. (c) Consider extending the analysis to more realistic spectral structures or more general misspecification models, even if only as future work.\n\n3. The experimental section requires substantial expansion and can be greatly improved to better justify the studied model:\n   - Only one experimental setup is shown (3-layer ReLU networks, Figure 4)\n   - No experimental details are provided in the main paper, forcing readers to search Appendix B for minimal additional information\n   - The connection between the linear theory and nonlinear experiments is not rigorously established\n   - No experiments on real datasets are provided\n   \n   *Specific recommendation*: Add experiments on: (a) deeper networks with different architectures, (b) different activation functions, (c) real-world datasets where spike structure can be validated, (d) ablation studies showing which theoretical predictions transfer to practice\n\n4. Even considering Appendix B, the experimental details are insufficient for reproduction. Critical information missing includes: exact network initialization, optimization hyperparameters beyond learning rate and epochs, how the spike direction and alignment are constructed, how results are aggregated across trials, and the random seeds used.\n   \n   *Specific recommendation*: Provide complete experimental details in a dedicated section or supplementary code repository, following ICLR reproducibility guidelines.\n\n5. The term \"phase transition\" in the title and throughout suggests sharp, discontinuous changes in behavior. \nHowever, Figure 3 and the mathematical expressions suggest smooth, continuous transitions from positive to negative coefficients except for the case of $c=1$ which has been extensively discussed in the literature.\nThis is more accurately described as a trade-off or crossover phenomenon rather than a phase transition in the statistical physics sense.\n   \n   *Specific recommendation*: Either provide evidence of sharp transitions (e.g., derivatives of risk with respect to parameters showing discontinuities) or adjust the terminology throughout the paper to more accurately reflect the continuous nature of these transitions.\n\nAdditional minor concerns:\n\n6. The proof sketch in Section 3.4 that explains the risk decomposition is deferred to near the end of the paper. This interpretable decomposition should appear much earlier to help readers understand the subsequent results. \n\n7. The paper would benefit from a section explicitly discussing what practitioners should take away from this work. When should they worry about alignment? How can they diagnose whether their setting corresponds to the catastrophic regime? What are actionable recommendations that one gains by understanding this model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)