Closed-form $\ell_r$ norm scaling with data for overparameterized linear regression and diagonal linear networks under $\ell_p$ bias
Authors: Shuofeng Zhang
Ard A. Louis
Venue: ICLR 2026 Poster (2026)
Keywords: Norm
scaling law
deep linear nerual network
linear regression
Primary area: learning theory
TL;DR: Closed-form scaling laws characterize how every $\ell_r$ norm ($r\in[1,p]$) of minimum-$\ell_p$ interpolators scales with sample size, in linear regression and diagonal linear networks.

Abstract: For overparameterized linear regression with isotropic Gaussian design and minimum-$\ell_p$ interpolator $p\in(1,2]$, we give a unified, high-probability characterization for the scaling of the family of parameter norms $ \\{ \lVert \widehat{w_p}  \rVert_r \\}_{r \in [1,p]} $ with sample size.

We solve this basic,  but unresolved question through a simple dual-ray analysis, which reveals a competition between a signal 
*spike* and a *bulk* of null coordinates in $X^\top Y$,  yielding closed-form predictions for (i) a data-dependent transition 
$n_\star$ (the "elbow"), and (ii) a universal threshold $r_\star=2(p-1)$ that separates $\lVert \widehat{w_p}  \rVert_r$'s which plateau from those that continue to grow with an explicit exponent. 

This unified solution resolves the scaling of *all* $\ell_r$ norms within the family $r\in [1,p]$ under $\ell_p$-biased interpolation, 
and explains in one picture which norms saturate and which increase as $n$ grows. 

We then study diagonal linear networks (DLNs) trained by gradient descent. 
By calibrating the initialization scale $\alpha$ to an effective $p_{\mathrm{eff}}(\alpha)$ via the DLN separable potential, 
we show empirically that DLNs inherit the same elbow/threshold laws, 
providing a predictive bridge between explicit and implicit bias. 

Given that many generalization proxies depend on $\lVert \widehat {w_p} \rVert_r$, 
our results suggest that their predictive power will depend sensitively on which $l_r$ norm is used.

Published as a conference paper at ICLR 2026
CLOSED-FORM ly NORM SCALING WITH DATA FOR OVERPARAMETERIZEI LINEAR REGRESSION AND DI- AGONAL LINEAR NETWORKS UNDER l, BIAS
Shuofeng Zhang
Ard Louis
Rudolf Peierls Centre for Theoretical Physies
Rudolf Peierls Centre for Theoretical Physics
University of Oxford
University of Oxford
shuofeng, zhangepphysics cx ac. uk
ard. ard.louus@physiccs.oo ac. uk
ABSTRACT
For overparameterized linear regression with isotropic Gaussian design and minimum-(,, interpolator with p E (1, we give unified, high-probability characterization for the scaling of the family of parameter norms { p1ppl (ee(t with sample size. We solve this basic, but unresolved question through simple dual-ray analysis, which reveals a competition between signal spike and bulk of null coordinates in Y, yiclding flosed-forma predictions for (i) a dete-depnndent transition n, (the "clbow"", and (ii) universal threshold separates ||,,] 's which plateau from those that continue to grow with an explicit exponent. 'his unified solution resolves the scaling of all & norms within the family r E [1,2] under fy-biased interpolation, and explains one picture which norms saturate and which increase n grows. We then study diagonal linear networks (DLNs) trained by gradient descent. By calibrating the initialization scale a an effective Pef( (a) via the DLN separable potential, we show empirically that DLNs inherit the same elbow/threshold laws, providing a predictive bridge between explicit and implicit bias. Given that many generalization proxies depend on ||0p||r, our results suggest that their predictive power will be highly sensitive to which E, norm is used.
2(p 1) that
INTRODUCTION
Many modern generalization measures for machine learning tasks are anchored on the parameter norm instead of parameter count (Neyshabur et al. 2015cc Yoshida & Miyato| 2017 Miyato et al.) 2018, Cisse et al. a 2017). Yet, most analyses overparameterizzer regression still treat "the norm" monolithically tyypllt defaulting f2. If one is going be used, and how does that choice interact with the inductive bias that selects the interpolator (e.g., minimum-l,,? This question has been comparatively less studied. We address this question first in simpler but core setting- linear regression and then connect the picture to diagonal linear networks (DLNs). Our experiments reveal that sweeping (r, p) produces non-trivial behavior: even for the same interpolating predictor, some e, norms plateau while others keep growing with distinct slopes; in mixed cases, the elbow's location shifts with p, and which r's plateau depends on the setting.
parameter norm, which ł, should
In linear regression it is well understood that the value of shapes the inductive bias (sparser as p41. 1, denser as 2)), making the r-p interaction concrete. Beyond explicit Ep penalties, first-order optimization can implicitly select a geometry: in overparameterized linear regression, gradient methods recover the minimum-lz interpolant; in separable classification, gradient descent converges to max-margin solution; and in diagonal/deep linear parameterizations. the separable potentials governing the dynamics interpolate between sparse- and dense-leaning behaviors depending on
initialization and parameterization (Tibshirani] 1996, Frank & Friedman, 1993 Hoerl & Kennard
1970, Chen et al. 2001] Zou & Hastie (2005) Hastic et al. 2015/2/222222
[2055]
et al, 2018))/unasee
al. 2018 Ti & Telgarsky 2019h) Chizat et al. 2019
Woodworth et al. 2020 This variety of

---

Published as a conference paper at ICLR 2026
explicit/impliitt pathways for p-like biases motivates unified treatment of the famity {1| and explains why different l, proxies can exhibit qualitatively different n-dependence under fixed training pipeline.
Concretely, we study the minimum-l, interpolator in high-dimensional linear regression with isotropic Gaussian design (d > n. p E (1,2), and characterize- iin closed form and with high probability how the entire family (11@) [re(I.n scales with n. A one-dimensional dual-ray analysis exposes competition between a signal spike and a high- nigh-dimensional bulk in X Y. yielding: (i) data- dependent transition size 74 (an elbow in n),, and (ii) a universal threshold r+ = 2(p 1) that separates norms that ultimately plateau (r > r.) from those that continue to grow with explicit exponents (r S r.). We also extend the picture to DLNs trained by gradient descent: calibrating the initialization scale via the network's s separable potential that gives an effective exponent Per(a), and with this calibration the observed é,-vs-n curves inherit the same elbow/threshold structure as explicit minimum-f, interpolation. When the inductive bias is unknown priori-e.g e.g., the operative of the training pipeline is unclear-our results imply that choosing the "right" r for norm-based generalization measures be delicate, since different (r,p) pairs can produce opposite scaling behaviors (plateau vs. growth) as n increases.
Our contributions:
Strong sensitivity of the parameter norm as a function of the pair (r. We find strong qualitative effect for the scaling of the parameter norm with data: for fixed p. certain E, norms plateau while others grow with different slopes; varying p moves the elbow and reassigns which r's plateau.
Closed-form scaling laws for parameter norms. We derive the first unified closed-form scaling laws for this problem. For p E (1,2] and all [I, we identify the universal threshold r. 2pp - 1), give an explicit expression for the transition size n., and provide plateau levels and growth exponents in both spike- and nulk-dominated regimes via compact dual -aa argument.
3. Extension of our theory to DLNs. We map the DLN initialization scale to geometry: + Pet((ar. Using this map, we transfer the theory to DLNs and verify the predicted elbow/threshold behavior of the parameter norm empirically.
Implications for practice. Because many norm-based generalization measures and diagnostics depend on || our results imply that practitioners using norm-based bounds eexiiss-eppecially in more complex models such as DNNs-should be cautious: conclusions can be highly sensitive to the choice of r, and the sensitivity depends on the underlying E, bias that selects the interpolator.
2 RELATED WORK
The focus of this paper is a basic question: for overparameterizee linear regression and related diagonal linear networks (DLNs), how do the parameter norms (r scale with sample size when the interpolator is selected by an Ep bias? The links generalization are therefore indirect: norm quantities often appear as proxies modern generalization measures (Neyshabur et al.) 20155) Bartlett et al. 2017 Dziugaite & Roy 2017), so understanding their n-scaling is informative, but do not develop new generalization bounds here. Relatedly, recent analyses derive explicit norm upper bounds as intermediate steps toward generalization-otl
via Gaussian min-max techniques-For
interpolators and max-margin procedures (Kochler et al. 2021 Donhauser et al. 2022).
The e, family of linear-regression interpolators. A large body of work characterizes how explicit L penalties shape linear estimators: ridge/Tikhonov (f2) (Hoerl & Kennard 1970), lasso (l))
(Tibshirani] T996. Efron et al., 2004] Knight & Ful 20000; Zoo! 2006), elastic net (mixtures of &
and (2) (Zou& & Hastie 2005), and the bridge family (Ep for 0 < - 2) (Frank & Friedman, 1993):
basis pursuit gives the sparse interpolating extreme under equality constraints (Chen et al. 2001
Candès & Tao 2007 Donoho 2006 Bickel et al. 2009). High-dimensional convex-geometrie
analyses explain when these programs select structured solutions and how their solutions move with the data geometry (Chandrasekaran et al. 2012 Amelunxen et al., 2014 Bühlmann & van de Geer, 2011] Wainwright] (2019), and recent developments give precise characterizations for ridgeless (minimum-(z) interpolation and its risk (Hastie et al. 2022 ?). Our contribution complements this
2

---

Published as a conference paper at ICLR 2026
literature by treating the entire norm family (1lugll)rr((tp)) for minimum-4, interpolators (with p E (1)2)) and deriving closed-form, high-probability scaling laws in n across r. In this sense we move from "which p shapes which estimator?" "given p. how do all by diagnostics behave as data grow?"
Overparamettrization in regression and deep networks. The deep-learning stimulated ve-cxamination of overparameterized regression, revealing phenomena such as double descent (Belkin
et al.
2019 Nakkiran et al. 2021 Zhang al. 2017 Nakkiran et al. 2020a Adlam & Pennington and benign overfitting for minimum-noo norm interpolators (Bartlett et al.] [0200] Hastie et al.
2020
2022
Muthukumar et 2020). These results show that linear regression can capture qualitative
behaviors seen in deep learning models and that the selected interpolator's norm matters for risk. Our work leverages this bridge as motivation only: by explaining. in closed form, which e, norms plateau and which grow (and at what rates) under an lp bias, we clarify what one should expect from norm-based proxies commonly used in deep-net analyses. Because practical pipelines for deep models rarely specify the effective p. our finding that ||ppp 1p depends sensitively on the pair (r,p) suggests caution when interpreting norm-based generalization diagnostics.
Explicit/mplicit regularization and DLNS. Beyond explicit penalties, optimization can select solutions with implicit geometry (Soudry et al. 22018: Lyu & Li, 2020; Gunasekar et al. 2018 2017). In overparameterized linear regression, gradient methods recover the minimum-é2 interpolant; factorized or deep-linear parameterizations, the training dynamics induce separable potentials that interpolate between sparse- and dense-leaning behaviors depending on initialization and parame- terization (Saxe al. 2014b; Gunasekar et al. 2018 Ji & Telgarsky) 2019b] Chizat et al. 2019 Woodworth et al. (020). We build on this perspective for DLNs: by calibrating the initialization scale to effective Poff, we show empirically that DLNs inherit the same elbow/threshold laws for [|| [ [++)+
as explicit minimum-4, Lp interpolation.
Proof techniques. Our analysis borrows standard high- dighenimmnsional tools used throughout the modern regression literature Gaussian concentration, blockwise (signal-vs-bulk) decompositions, and dual certificates in convex programs (Vershynin 2018: Tropp 2015) and combines them with a one-dimensional "dual-ray" reduction tailored the Ep penalty. Two closely related works derive norm upper bounds as an intermediate step toward generalization, using the Gaussian Min-Max Theorem (GMT) and its convex analogue (CGMT): Kochler et al. (202I); I); Donhauser et al. (022)). Their CCMT/CGMT-base proof's are conceptually similar in spirit; by contrast, our argument proceeds from first principles via a simple dual-ray balance and yields closed-fomm n-scaling laws without invoking GMT/CGMT (see also Gordon 1985); Thrampoulidis al. (2014) for the GMT and CGMT statements).
3 FAMILY OF NORM MEASURES OF MINIMUM f,-NORM INTERPOLATOR IN LINEAR MODELS
We now formalize the object introduced in the overview: for E (1,2) in overparameterized linear regression, we study the family is to characterize how these norms scale with sample size n. Our results identify (i) data-dependent elbow n, and (ii) a universal threshold r. = 2(p 1) that separates plateauing from growing e,'s
where üp is the minimum-e, interpolator. Our goal
Data and settings. i..d. (0, Ia), and
We consider overparameterizet linear models with X E Rund d > n, rows
y= -xw + E {~~((0,021.)
The minimum-fp interpolator is
Tp
E
arg min wERd
||l|| s.t. X
p E (1
Let [0 denote the support size and 73
||0 112 + 0². In contrast to interesting recent
work by Donhauser et al. (2022). our theory is not restricted to the w' = e, limit of extreme sparse regression.
3

---

Published as a conference paper at ICLR 2026
3.1 MAIN THEOREM
Theorem 3.1 (l, scaling of minimum-fp interpolators). Fix E (1,2), set = p-I. r [). Assume
_
and take
d + (1) and
liminf d R-+00
Kbbukk 0.
n
n
Let w have support with |S| = 8, and let
up E arg min
||lB S.t. Xw=Y
wERd
Write Wg = ||0 11 and me == El 2 for Z ~ NO 1). Define the ray scale t, via
| |X
19-1
w.h.p.. (1)
YIR
n W + (d s)
+ my 13 n 1/2
OTT (sn + + s
spike
bulk
Then, w.h.p. (see Remark A.7
|0pll x max 19-1 9-1 ||0" 1w=' ||""l (1g-1))) (d s))//r (1. vvVT)
max 1/r, 11-1) (t. TaVr)) 1-1
(2)
Introduce the transition scale
T - (bbull Wg
(3)
In the two extremes, we obtain: Spike-dominated (n >> n.):
1+8
8
n
21 17,
T 2(p 1),
W
1pplr x
(4)
[w"
I1 9-1
<> 2(p 1).
W
(g-1)r 1)r
Bulk-dominated (n & n.):
|1 = max
Kbblk Ts n 1 Kullk 72-9 || [1 (4-1)r n 8-1
1-1
Kbulk To maxt 1/r, ()/2 11/2
Since d x Kbulkn, the last term equals depending only on (p, Kbulk, r). Remark 3.2 (Dual viewpoint). The constrained problem min,. || | s.t. Xw Y has unconstrained dual max ATy +1|x Allg, with KKT conditions Xw = Y and X X V (wr). Restricting to the ray A = tY yields t.- = ||Y||/ Yll.. The "spike" vs. "bulk" terminology refers to which part of
Iu d=s s max 1)/2) Vn. All x hide absolute constants
controls t..
Proof sketch. The behavior of the minimum-f, interpolator can be read through simple dual lens: rather than track the optimizer directly, we examine dual certificate that both fits the labels and respects norm budget after passing through the design; pushing the dual along the label direction (a one-dimensional "ray") reveals single diagnostic scale where the budget tightens, and this scale is controlled by two competing sources in the correlations X Y: "spike" part (true signal coordinates) that coherently accumulates with n, and a "bulk" part (many null coordinates) that aggregates small, mostly noisy effects. Balancing these two contributions defines depandeeenddnt transition sample size n,: for n & n, the bulk dominates, the solution's mass effectively spread many coordinates, and the family (1, grows with in the way bulk formulas predict
4

---

Published as a conference paper at ICLR 2026
(including the usual cross-r ordering and an ttyp- trend visible in the plots); for > n, the spike dominates, mass concentrates on the support, and clean threshold determined by p at = 2(p 1)-sppits the outcomes: l, plateaus for above the threshold and grows with gentler, explicit slope for below it. Standard concentration for Gaussian designs justifies the spike/bulk decomposition and the stability of the ray scale, and the KKT linkage between the dual certificate and the primal coordinates turns these ingredients into the unified bound, the expression for n.,, and the regime descriptions stated in the theorem. Full details are deferred Appendix A
bhreshold-deeermineed
COROLLARIES FOR CANONICAL TARGETS
To make the unified laws in Theorem 3.1 concrete, specialize them two canonical targets: (i a single spike w* = ét, and (ii) a flat s-sparse vector with equal magnitude a on its support. Substituting the problem-specific scales W || I1g and 73 -I|w=11 o² into the elbow formula equation and the spike-/bulk- dommnntk-ddmmneeeddddd expressions equation equation 5 yields closed- form, high-probability predictions for | and the transition size n, We record these specializations
below as Corollaries 4.1 and 4.2 and use them as reference overlays in our experiments.
4.1
SINGLE SPIKE
Corollary 4.1 (Single spike). Under Theorem 3.1 with w = 61 and T² = 1 0², for any r E [L.p]:
Bulk-dominated (n < n.):
10 z r(d - 1) n//
r g+1 n 11
ifr S 2(p 1),
Spike-dominated (n >> n.): 110llr x
ifr 2(p 1).
Interpretation. Here W,=1 and n. x (Nibulke r 9) 32/6022) from equation l, curves plateau at level x 7² once n > n.: for r S 2(p-1) they continue to grow with slope
For r > 2(p-1) the
4.2 FLAT SUPPORT
Corollary 4.2 (Flat support). Under Theorem j E S (|syl 1). for any r [L. p). wh..p.
3.1 and a flat w* on with |S| and wz s for
(sa² 02) 1+1
n-
r 2(p 1), 2(5-1)
s|a|2
Spike-dominated (n Cn.): ||ūpllr
s -1-1 sa² + o2
2(p 1) < S
la|
Bulk-dominated (n S cn.): || max { Kuulk Ta n 2, Koulk T 2-9 84/2 (a| n -1
Kbulk To 8 max/1//r. 1)/2] n -122
Interpretation. Here W,=s|a|² and T?=sa²+ol so equation
gives nr
(AbulkkT"////// (ssl which grows with (the elbow shifts to larger ri). In the spike-dominated plateau branch (r > 2(p-1)) the level scales as s -1 (sa²+o²)/ which is typically of the same order as the single-spike plateau for moderate s.
Comparison across targets. The threshold r = 2(p 1) and the n-exponents in both regimes
are unchanged between Corollaries 4.1 and 4.2 The differences lie in the scales: (i) the tran- sition size moves from n, x (Kbulk 4/W (g- (q-2) with W,=1, -²=1+o² (single spike) to n. x ( H// (/W)2//(-3)) 22((=2) with W W,=s|a|7, + +o? (flat), which scales roughly linearly in (cf. equation 3). Hence the elbow for regime change shifts to larger n when we move from é to a flat wn with 500 (ii) In the spike-dominated plateau branch (r 2(p 1)), the level changes from
5

---

Published as a conference paper at ICLR 2026
p=11
p=15
d=0.1.p-1.9
Genandication wrer 1.1.
thery (bok
Genenaaation waar
theory tbult
Generatation
C
thesyy (32) dem))
theory up
theory
Generaitation error
Genera wenaraltaation are
3.04
21
1.0
4 namn
J narm
80
2000
4000
1020 2000 3303 4502 (Sampless)
3000
2000
4000
n (samples)
[Samples)
(a) p 1.1 (spars)ty-leannn)
(b) p = 1.5
(c) p = 1.9 (dense-lcaning)
Figure Single spike w" = c1; explicit minimum-l, interpolation. Ordering across r and the presence/absence elbows follow Corollary4:: 4.. the bulk panels rise like n/2 and the spike-side panels plateau for 2p-1),, consistent with equation Atttiooio/qquuoooo
x (single spike) to - (sa2==")/+////) (flat) [ef. equation 4 and Corollary 4.2]: for moderate this produces comparable numerical magnitudes, which is why the vertical ranges in our figures are similar. The regime labels (bulk vs. spike) and their slopes/plateaus therefore provide the informative contrast.
4.3 LINEAR REGRESSION WITH EXPLICIT MINIMUM- BIAS
Here the inductive bias is explicit: for chosen p. the interpolator is the minimum-f, element among all w with Xw = Y. Sweeping p slides the solution from a more sparse-leaning geometry as p & toward more dense-leaning geometry 2, revealing how the objective itself shapes the family [11rpplr)r.
Experimental protocol. We set d 50, 000, 0.1, sweep p E {1.1, 1.9), and vary Each plot overlays test MSE (left axis) and representative l, curves (right axis). For flat w" experiments, kept ||w" ||2 i.e. - Additional noise sweeps reported in Appendix C
What the figures show and why. In Fig. | (single spike), the left/middle/right panels follow the corollary's regime rules. In the left panel, for > 2(p-1) the curves flatten after the transition, while for smaller they retain the predicted growth; thin reference overlays (where present) trace these slopes. The middle panel exhibits a clear elbow near the predicted n.,, which is close to 1.4 x 103 under our experimental parameters; beyond it, the r > 2(p-1) curves plateau in line with equation 4 while the others keep their slope. The right panel stays bulk-dominated across the range, with the & traces growing approximately as n 1/2 and ordered across the bulk formula prescribes.
In Fig. 2 (flat with s=50), the same slope/plateau rules apply, but the transition scale is larger: the elbow for p=1.5 appears at a later n (or just off off-range), consistent with n, increasing roughly linearly with in equation Across panels, the absolute &, values are numerically similar to Fig. []] this matches the flat-support plateau level in Corollary 4.2 which for moderate is close to the single-spike level. The informative distinction thus where the curves switch from bulk growth to spike plateaus and the persistence of the 1/2 slope regimes that remain nolk-dominated.
Experiments with larger sparsity. We repeat the explicit minimum-f, runs at larger supports, e (500, 5000), with the same ||l* 1|2=11 and noise level (a 0.1); see Appendix Figs. S13 $14 The qualitative picture from ==50 reappears but shifts to larger consistent with the transition size n, in growinn with s. For small p (p=1.1), the prolonged dalk-dominated window makes the double-descent pattern visible-genenne generalization error first increases and then drops (most clearly at
--50000whiil while the blue f curve keeps rising along the bulk guide across the plotted range (Belkin) al. 2019 Nakkiran et al. 2020b Hastie al.) 2022). For larger (p-1.5, 1.9), the curves remain monotonically decreasing; the minimized lp traces drift only mildly upward (no flattening within the range), reflecting the rounder geometry that avoids early over-reliance noisy bulk directions. In all panels, the dashed overlays track the bulk/spike trends and the expected r-ordering of the f, diagnostics, matching the regime structure highlighted in the theory.

---

Published as a conference paper at ICLR 2026
s=01.==11.==00
0.1,
1,5. p.
d= 0.1, 199.650
Genaneication wrer 1.3. norm
therry (bc
Gencralizatin
Generotation theory
thesry (32) dam))
Genvraitation ereer
Generaliiation emor
1.
1.00
mOmm
0.8
10
Gemeralization
0.95
J narm
65
0.4
1020 2030 3030 4000 5202
2050 (sammllss)
4600
2050
4000
(aammplss
(samples)
(a) p = 1.1 (sparsity-lcanin)
(b) p = 1.5
(c) p = 1.9 (dense-lcaning)
Figure 2: Flat wt (s = 50); explicit minimum- interpolation. The scaling rules mirror the flat-support corollary: bulk growth persists until larger transition scale, while spike-side values plateau; absolute levels are comparable to the single-spike case, as predicted by the plateau formulas.
4.4 DIAGONAL LINEAR NETWORK WITH IMPLICIT BIAS
Diagonal linear networks (DLNs) deep linear models whose weight matrices are diagonal so that the effective predictor is the coordinatewise product of layer cottmntters-rovidde a tractable testbed for understanding optimization-indeee induucd geometry and implicit bias overparameterizee systems.
They connect classical analyses of linear nets and factorized parameterizations (Saxe et al.) 2014a
Ji & Telgarsky 2019a Arora et al.) 2019 Gunasekar et al.) 2017) with recent perspectives on how initialization and parameterization interpolate between "rich" and "kernel" behaviors (Chizat et al.
2019 Woodworth et al. 2020). A particularly useful feature- fornalized for DLNs via a separable gradient-flow potential is that the scale of the initialization, denoted a, continuously tunes the implicit bias: small a yields a sparse-leaning geometry (an llike penalty up to logarithmic factors), while large a approaches an flgtype geometry; see the potential Qa. and its limits (Theorem 1 in Woodworth et al. (2020)) and related characterizations in Gunasekar et al. (2017): Arora et al. (2019).
Calibrating a via an effective p. To compare DLN runs with our explicit minimum-fp experiments, convert into an effective by data-frec calibration. Following the separable potential view, we evaluate Qa on k-sparse, unit-62 probes and fit the log-log slope of its k-dependence; matching that slope to the exact A 11-p/2 law of || [| yields a monotone map a + Pef((a) with limits Petr (a) + as o - 0 and Petr (a) - as a - xo. This calibration is independent of (n, ar) and lets us select a range comparable to p {1.1, 1.5, 1.9). A full derivation and
values that span sparse-to-dens visualization of a + Pett (a: are provided in Appendix
Finite learning rate. With single-spike target (w* e1, sparsity s=1) and small initialization (a = 0.00102, so Pan Rv 1.10), we find that the learning rate Ir can materially change the f-vv--n scaling once label noise is present. When a=0, the f11 curve rapidly plateaus and is essentially insensitive to Ir (see Appendix for for more details). In contrast, for a {0.1, 0.5) increasing Ir produces a steadily rising fT.1 and shifts the elbow to larger nı; at the highest noise the effect is strongest-Ir=10- yields monotone growth across our range, whereas 1r=10-3 exhibits transient rise followed by relaxation toward a plateau, indicating a rightward- moving elbow. We observe qualitatively similar trends for larger sparsity (s-50). A natural explanation is that finite step size together with noisy gradients turns (stochastic) gradient descent into a noisy dynamical system with an effective temperature that scales with Ir and the noise level. The resulting diffusion broadens the stationary distribution and biases the predictor toward rounder (less sparse) increasing Po mass leaks into bulk coordinates, delaying spike dominance and inflating &, before the eventual plateau (Mandt et al. 2017 Smith et al. 2018 Yaida 2018 Jastrzebski et al. 2017).
Experimental protocol. We set 50, 000, a 0.1, sweep a E {0.00102, 0.0664, 0.229) (which according to our to calibration s= f1.1, 1.5, 1.9)), and vary n. Each plot overlays test MSE (left axis) and representative f, curves (right axis). For flat w* again a Js
Additional noise
sweeps are reported in Appendix C
Because has been empirically calibrated to (a), the DLN panels minror the scaling behavior seen with explicit minimum-l,, for w" = e; (Fig. smaller a (smaller p.#) enters the spike-dominatee regime earlier that, for > 2(p-1). 1), the f, curves flatten after the transition; larger a remains
7

---

Published as a conference paper at ICLR 2026
o-0.1, 0.0.00122, + Gonora encr
(At/s<1.10) norlnorm
0.1, 0.0664, 1r-0,00L, ==1 (fl))=1.5)
0.1,
In-0.001,
1(d)-1190)
General zalii erer
++ (.35 norm
Gonorstiacion enlrr
415 nom
1111 norn
I_1_Inorm
+ L13 nom
Generallaatoon entor
Generaitzetion arror
2.00
Gantrallaatian
0.75
1 harm
85
9.55
1_
0:50
6:25
2000
4000
1000
2053
6650
5000
2050 (sampless
[ssamples)
[samples)
(a) 0.00102, 0.1,
(b) 0.0664, Ir 0.001, 8=1
(c) 0.229, 0.001,
Figure 3: Single spike =' C1; diagonal linear network (DLN), After calibrating a to Petl (a). the regime structure matches the explicit p case: smaller a exhibits earlier spike dominance and plateaus for > 2(p-1); larger a remains bulk-dominated with n 1/2lliike growth.
0.1. a- 0,0010, 1r.0.1. 5 186gl=1.10
orm a.1.0...0..mm 0.0654 IreO 001, sm 5o (Rol=1.50)
0.1, a- 0.229, r-0.001, 50 (Atu)=1.90)
lost
Green
( torm
Ceneralaen inneratzztionnoo emer
114
1.11 nom
1.11
Genyeallatton wror
Generaliaatio arror
Generaliztton
0.35
firm
1.00
Horm
0.5
0.00
0.55
1503 2010 3000 4000
5500
2000
6030
2300
4003
(Isamples)
(sampics)
([Samplls)
(a) a 0.00102, 0.1, 50
(b) a 0.0664,, 0.001,
5050
(c)
0.229, 0.001, =0
Figure 4: Flat w* (s 50); diagonal linear network (DLN). The same scaling rules hold, but the elbow appears at larger n- n-in line with the flat-support transition scale-while absolute l, magnitudes
remain comparable to the single-spike case.
bulk-dominated longer and the traces grow with the characteristic 1/2 trend. For the flat target with s=50 (Fig. 4 the same rules apply but the elbow shifts to larger n, consistent with the s-dependent transition scale in the flat-support corollary. The absolute magnitudes of ||,||, are similar across the two targets, as predicted by the plateau formulas, so the informative contrast again lies the location of the elbow and the presence/absence of plateaus
bulk growth. We do not overlay theory on the DLN plots: our guarantees are stated in terms the explicit parameter p. and deriving a closed-form a-indexed analogue (especially under finite learning rates) is outside the scope of this work; the r Pet calibration serves precisely to make the scaling correspondence visible. In Appendix F we discuss how can we extend our main theorem to DLNs with explicit a.
CONCLUSION AND DISCUSSION
We provided the first unified, closed-form characterization of how the entire family of norms || scales with sample size in overparameterized linear regression under minimum- lp interpolation (p (1, 2)). A one-dimensional dual-ray argument exposes competition between signal spike and a bulk of null coordinates in T and yields, with high probability: (i) data-dependent elbow n, at which bulk and spike balance [Eq. 8 and (ii) a universal threshold
r. = 2(p 1),
which separates 6,'s 's that ultimately plateau (r > r.) from those that continue to grow with an explicit exponent (r < r.) in the spike-dominatee regime (Theorem 3.1). The formulas give plateau levels and slopes in both bulk- and spike- ppike-dominated regimes, and specialize cleanly for canonical targets (single spike and flat support). Empirically, diagonal linear networks (DLNs) trained by gradient descent inherit the same elbow/threshold laws once the initialization scale a is calibrated to an effective Pott (a) via the separable potential. Together, these results show that which b, one tracks matters: for a fixed Ep bias, different é, 's can exhibit qualitatively different n-laws.
Intuition behind the regime transition. The dual-ray lens reduces the interpolation geometry to single scale controlled by || TYll, (q p/ (p 1)). The bulk contributes x (d (d-5) 5) my 72n9/2
8

---

Published as a conference paper at ICLR 2026
while the spike contributes x n. g: and their balance sets the elbow n,, Above the elbow, the KKT map raises correlations to the (g 1) power, the sign of 1 -リ- 2(z-1j dictates whether the bulk- term decays (plateau) or grows (slope). This is the origin of the sharp threshold r. = 2(p - 1). Geometrically, smaller (sparser inductive bias) lowers r*,, so more e, 's plateau once the spike dominates; as p f 2, r, approaches 2 and spike-side plateaus recede, consistent with the special role of p = 2 where there is n-driven transition in the proportional limit.
Implications for generalization proxies. Many diagnostics and bounds in modern learning scale with a parameter norm (or a reparametterization-awvae surrogate). Our results indicate that the predictive power of such proxies is norm-choic sensitive. For a given Ep bias, & above r. stabilize (after n.) and can serve data growth through explicit exponents. In practice, the pair (nx, r-) acts as norm-scaling signature. Reporting only one norm- form-nften/ f2-risks conflating bulk spike effects and can obscure regime changes that are visible in the l, family.
coometry-alignee capacity proxies, while e, ' below r. continue to reflect
From explicit to implicit bias. By calibrating DLN initialization via simple slope-matching map + Peff (a), the empirical DLN curves line up with the explicit minimum-f, predictions under f- Pett (a). This provides a quantitative bridge between explicit and implicit bias: initialization steers the effective geometry, and the r.) structure is inherited. Finite learning rates in the presence of label noise act like an effective temperature, increasing Pett and shifting elbows rightward consistent with recent views of SGD as noisy dynamical system.
Relation to double descent and benign overfitting, The bulk-side growth (x n in prominent terms) and its eventual handoff to spike control rationalize when increasing n first harms and then helps: early fits draw from many noisy bulk directions (large norms and higher variance), while beyond n, the spike dominates and the relevant l, plateau. Our explicit exponents and thresholds sharpen this picture and make precise which &, will display which trend at given (p, n).
Scope and limitations. Our guarantees assume isotropic Gaussian design, p E (1, squared loss, and exact interpolation. At = 2 the proportional regime admits no n-driven elbow. The DLN extension uses data-free calibration to Patr(a) rather than a fully rigorous, learning-rate-awar theory. Finally, classification losses and non-linear features (beyond DLNs) are outside our formal scope.
Actionable guidance. (i) When using norm-based capacity control, choose the norm with the geometry: if training is bi-bia (explicitly or implicitly), track & with r > 2(p-1) to obtain a stable, post-elbow proxy; use 2(p-1) when one wants a readout that continues to reflect data growth. (ii) Empirically estimate (n., r.) by fitting the predicted slopes to a small e, grid; this gives compact fingerprint of model-data geometry and practical meter for bulk vs. spike dominance.
Future directions:
Beyond isotropy and Gaussianity. Extend the dual-ray analysis to anisotropicesub--aauiiin designs (via whitening) and to heavy-tailed covariates; characterize how n, and possibly r. deform with the spectrum and tails of X.
From DLNs to nonlinear nets. Replace the power link by depth-dependent implicit links in deep (nonlinear) architectures (e.g., path-norm or neural langend/ieature-llamnng regimes) and test whether an r.ytype threshold persists.
Algorithmic knobs as geometry. Develop a theory of Paff that accounts for step size, batch size, momentum, and label noise (Langevin/SGD limits), turning these knobs into quantitative geometric parameters with predictions for r.,.
Classification and margins. Generalize the scaling laws to separable classification with cross- entropy/hinge losses, relating r, to margin exponents and the growth/saturation of norm families along max-margin flows.
Tighter, r-aware bounds. Build generalization bounds that track the family { {|[θ]]+ and explicitly incorporate the elbow/threshold structure, connecting to PAC-Bayes and nargin-based analyses.
Practical diagnostics. On modern deep models, measure several f,-style surrogates (e.g., path norms) across data scale to estimate (m., rx) and evaluate which norms are reliable predictors of test error across regimes.

---

Published as a conference paper at ICLR 2026
Overall, our results advocate replacing the monolithic notion of "the norm" by a family view. The elbow n, and the threshold r. provide simple, interpretable invariants that tie together explicit/mpliiit bias, data growth, and norm-based generalization measures, and they offer compact vocabulary for in high dimensions.
describing- and ultimately
Reproducibility Statement All experiments in this work are fully reproducible with mini- mum hardware requirements. The anonymized code can be found at htttss://gtthut com/ shuofengzhang/min pocdebbaee
LLM usage typo/grammar slips.
General-purposs LLMs are used this paper to aid or polish writing, e.g., checking
REFERENCES
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-graines bias- variance decomposition. Advances in neural information processing systems, 33:11022-11032
2020.
Dennis Amelunxen, Miroslav Lotz, Michael B. McCoy, and Joel A. Tropp. Living the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, 3(3):224-294, 2014. URL https::/toott
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
URL https: wttpaaiivvvvo..... org/ abs 1905. 13655
Peter L.
Bartlett Dylan J. Foster, and Matus Telgarsky. Spectrally-normalizedd margin bounds for neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017. URL
https://arxix: /btgg//bb/715555 08498
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression, Proceeding of the National Academy of Sciences, 117(49):30063-30077 2020. URL
https://doi
org/ 1073 /paas........... 1907378117
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical variane trade-off. Proceedings of the National Academy of Sciences, 116(32)115449-15855 2019, URL
doi 1tt.:/..............poo....11110
1071
Peter J. Bickel, Ya'acov Ritov, and Alexandre B. Tsybakov. Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics, 3705):170- 2009. URt.https::/doi. org/
10.1214/08 AOS620
Peter Bühlmann and Sara van de Geer. Statistics for High-Dimensional Data: Meth- ods, Theory and Applications. Springer, 2011. URL
378-3-642-2012-11
Emmanuel J. Candès and Terence Tao. The dantzig selector: Statistical estimation when is much larger than The Annals of Statistics, 35(6):2313-235) 2007.
Venkat Chandrasekaran, Benjamin Recht, Pablo A. Parrilo, and Alan S. Willsky. The convex geometry of linear inverse problems. Foundations of Computational Mathematics, 12(6):805 849, 2012. URL |=tpp:///oo..oorr.....0////12110
Scott Shaobing Chen, David L. Donoho, and Michael A. Saunders. Atomic decomposition by basis pursuit. SIAM Review, 43(1): 2001. URL https://doo..rr/200.111
ESD0311445003990
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pp. 2933-2943 2019. URL BtR.sttps:: ptttps://arxxe
002.0b/996110009676 07956 arXiv: 112.09181..07956.
10

---

Published as a conference paper at ICLR 2026
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. Proceedings of the 34th International Conference Machine Learning (ICML), volume 70 of Proceedings of Machine Learning wwwpspesssssom Research. PMLR, 2017. URL. https: http:////occeeeeaaae ot sseellll1 html
Konstantin Donhauser, Nicolo Ruggeri, Stefan Stojanovic, and Fanny Yang. Fast rates for noisy interpolation require rethinking the effect of inductive bias. In International Conference on Ma- chine Learning, pp. 5397-5428. PMLR, 2022. URL http:tss//ppooeeeggg....-p.ppppp
David L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289-133)
2006. Uttptttts://dbi 100 http:/...1119//11..... 1w19/1119TTT.. 2006. 871582
Gintare Karolina Dziugaite and Daniel Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In arXiv preprint arXiv:1703.1 1008, 2017.
Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. Least angle regres- sion. The Annals of Statistics, 32(2):07-499, 2004. URL
(79053604000000000
IE Frank and JH Friedman. A statistical view of some chemometriis regression tools. Technometrics, 35(2):109-135, 1993. URL /ttp::///b...rr/1...11 01w0:110..1000000000
ott.://...............00..00001. 1933 1tt.i.999.....88 155.10485533
Yehoram Gordon. Some inequalities for gaussian processes and applications. Israel Journal of Mathematics, 5064))265-289, 1985. doi: 10. 10.1007/BRO22644726
Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Sre- bro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems (NIPS), volume 30, 2017. URL RtLLitttss//axxiv /ttp/aaivvv...///////.e//a.aa.lt org/ abs 092800
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The
Lasso and Generalizations. CRC Press, 2015. URUhtttss:///oi.rr//// http://c...11/1111111000
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-
dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Biased estimation for nonorthogonal 00w.:004007.66.....1110 Inttps://dol http//oi.oogg/1..1 1080 000.0100000000000000000000000000 000000000100000 1970. 10488634
problems. Technometrics, 12(1):55-61 67, 1970. doi:
URL
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd, arXiv preprint arXiv:1711.0462. 2017. /O0/T//RR/ttttee/aaaaar.oo//////
04623
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In Interna- tional Conference on Learning Representations (ICLR), 2019a. URL /tttps::/
net feet/fos Eorum? wow.:mmidddlglliilaatl..
arXiv: (.ckiv:1811.00 8810.2203..
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations (ICLR), 2019b. URL httpsstts///axx
org/abs 1810. 02032
Keith Knight and Wenjiang Fu. Asymptotics for lasso-type estimators. laso The Annals of Statistics, 28 (5):1356-1378, 2000. URL URL.ttpss///aa..cr//1222222
Frederic Koehler, Lijia Zhou, Danica J. Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2021. URL ///rriiv... org/abs abs 22006 09276 arXiv:2106.09276.
11

---

Published as a conference paper at ICLR 2026
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations (ICLR), 2020. URL org/ horg//bb//ae 1906. 05890 arXiv:1906.05890.
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18( 134):1-35. 2017. URL https
www. jmlr wwwpg//ppppssaee http://ppprs/suuueeeeeeee
214 pdf
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for
generative adversarial networks. In International Conference Learning Representations (ICLR), 2018. Utttttts://aaxxit org. wttpo///aaa//////// 05957
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever, Deep double descent: Where bigger models and more data hurt. arXiv preprint 1rXiv:1912.02292, 2020a.
URL https: org/ ab9/1 02292
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv: wwXiv:1912.02299 2020b. URL htttss://pperreee available.
ew. net. formee www..e/oooonmmdd.naaaa (ir-ddd-BB1bbyy ICLR 2020 version
Prectum Nakkiran, Gal Kaplun. Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data still hurt. Journal of Statistical Mechanics: Theory and Experiment, (12):12400) 2021. Utthttps::/ //to://ooi..o www.ol..org/.... 1088/
1742-5468/ 000.55488ccc22/
Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro. Path-sgd: Path-normalized optimiza- tion in deep neural networks. Advances in Neural Information Processing Systems, volume 28,
2015a. URLthttss://aat http://///aaaa..ooo///////////// 02617
org
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceedings of the 28th Annual Conference on Learning Theory (COLT), 2015b.
URL
ml r http://eeeee..........reeee////e
html
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceedings of the 28th Conference Learning Theory (COLT), volume 40 of Proceedings of Machine Learning Research, pp. 1-26. PMLR, 2015c. URL https:// proceedings 1 wwteees/v40/eeyeeee
html
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representa- tions (ICLR), 2014a.
arwiv org/ www./o//////bab/.///o.oe
6120 arXiv:1312.6123
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. In International Conference Learning Representations (ICLR), 2014b. URL htttp:: http:://aaxvvv//. org/ abs 6120
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations (ICLR), 2018. URL http:/ttps///ppeeevvieeee
net Eorumoidddiiir.o Eoormmmmmmm--11111Xx
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Reseurch, 19(70):1-57, 2018. URLhttps: h/wp:::///wwww.// jml miml org wttppppprssa......o/./...e.e. 1 ume 1111-118/11111111111111111111111 Fpdf
Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. The gaussian min-max theorem in the presence of convexity. arXiv preprint arXiv: 408.448777 2014.
12

---

Published as a conference paper at ICLR 2026
Robert Tibshirani. Regression shrinkage and selection via the lasso. the Royal Statistical So- ctery: Series B, 58(1):267-28) 1996. [tt.atttse//aa...rr//.......1.11
1996.
Joel A. Tropp. An introduction matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1-230, 2015.
Roman Vershynin. High-Dimensional Probability: An Introduction with
Appph:cciicoiiiiii.p.ppppccctoot Applications in Data Science.
/wwp://wwwww/... wtt.://aww.iii.....eoooooo.oeeee /bokks/ Cambridge University Press, 2018. http://ppww/.w.....creee.e..eee.
Martin J. Wainwright. High- Sigh-DDimensional Statisssi A Non-Asymptoric Viewpoint. Cambridge University Press, 2019. URL
Statistics:
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Proceedings of the 33rd Conference on Learning Theory (COLT), volume 125 of Proceedings of Machine Learning Research, pp. 3635-3673 PMLR, 2020. URL https::/pprceeeedgggggg////////
mlr. wttppp/ee//////.oooooorooooo..oa
html
Sho Yaida. dissiipiinnniiiiiiiiiiioooooooo relations for stochastic gradient descent. arXiv preprint arXiv:1810.00004, 2018.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint 1705. 2017. URL httptttt//aaiii.orrrr./.ee////
1705. 10941
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Repre- sentations (ICI.R), 2017. Utt. htttas///eeeeeeeeeeeeeesse//rrss
net www//fooouuu /
Hui Zou. The adaptive lasso and its oracle properties. Jourme Joumaa of the American Staristical Association 101(476): 1114776):111114444 14299 2006.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B, 67(2):301-320, 2005. URL httttttts///oo.orrttt
1111/j. 11w.14677762 9868.
A MINIMUM-I, INTERPOLATOR WITH S-SPARSE GROUND TRUTH
For completeness, we first introduce again the mathematical settings and restate our main theorem. We study p E (1. set q -ーr E [2, oo), and let E [1,y). Dimensions n, d E N with d 2 n. All X Radd have i.i.d. N 1) entries; columns are X.,-- Noise & ~ (0, I²I,,, independent of X.
The signal w* E _Rd is s-sparse with support S c. [d], |S| = s; we write w for its nonzeros. The response is Y = Xw" + &. The min-C, interpolator
Tp arg min [| 1p Xw Y}
(p ensures uniqueness)
our object of interest. Shorthands:
= |
Wy: = ||="11
Σ W3]
jeS
Remark A.1 (Standing assumptions and probability shorthand). We work in the proportional regime d E (1),
Kbulk lim inf d- (0, 220000
n
n
so d - s = (n) and s = Onn) (we do not require s S n). Unless stated otherwise, all hidden constants depend only (p, Kbulk) (and on when relevant), and "w.h.p." means probability at least 1 Ce en 2d- 7 When we simplify remainders using s & n (c.g., /sn + s ~4 /sn). the corresponding s > n form is always available and does not affect any conclusions in Theorem A.2
13

---

Published as a conference paper at ICLR 2026
On proportionality. The assumption d/nn - k is only for cleanliness of exposition and to keep constants tidy; it is not essential to the argument. All places where enters (c.g., the bulk l embedding and the uniform column- control) can be under the weaker-and and often more realistie-conditions
d
inf n-
Kbulk > 0, log d = o(n),
5 5=
In particular, our proofs and conclusions (same exponents in n, the threshold r, = 2(p 1). and the high-probabilie events) remain valid even in "larger" aspect-ratio regimes (including d/n oo) as long as log d = o(n) and the bulk density is bounded below. Under these weaker assumptions the hidden constants uniform in (n, d, 8) and depend only (p, r, Khulk) (and on fixed upper bound for s/n if desired), so no changes to the proof' are needed.
A.1 MAIN THEOREM
Theorem A.2 (Theorem restated). Fix E (1,2), y - p r [1, and suppose liminf(e
s)/n Kbulk > 0 while d/n + E (1, o0). Then, w.h.p..
@pplr max t n2 9
q-1 (0-1)r (d 8)2/1 (sV 9-1
max 1/r, (a- 11/2} (t, (s+T)) 9-1
spike main (8)
bulk (89)
remainder
(6)
where the ray scale t, satisfies
19-1 |
T2n n,W, + (d - 8) mg T8 na/2 + + slte/2))
w.h.p. (7)
with m, = E 21' and Z NNO,,) 1). Define the dual-transition scale
n,
(Khull W
(8)
Then, w.h.p., the following asymptotic simplifications hold:
Dual spike spike-dominated n >> nh
T 9++
n
21 17,
r r - 1),
W
Wplle
(9)
T
||
19-1)) l)rt
r> 2(p 1).
W
Dual bulk-dominated < Th.-
[| max {
bulk To bulk Tg 2-9
||
|| 11/-11) n
-1 Nuulk max{ 1/r, (]-1) 11/2
(10)
(Equivalently, using d- -=s Kbulk7, the third term be written L. max(1/r, (4-1)/2) vn.)
Remark A.3 (When the third term absorbed). If < 2(p 1) and s S C - 3) for an absolute constant C, then the third term in equation 5] dominated by the first term (their ratio is S (x/(d- (//(ds))2// s))) In that case, equation
5 reduces to the two-term maximum
|| = max { bulk T 1
bbulk - ||ll |1 2-1 }
For > 2(p - 1), no uniform absorption holds in general; the third term can dominate when 1(g--111) small relative т.,
Remark A.4 (Boundary p = 2). At = 2 (so = 2) the exponent in equation g diverges. In the proportional-d regime (d/n x) there is no n-driven transition: the relative sizes of the spike and bulk are constant-level. In the finite-d regime (below) a concrete n n-threshold does exist because
(d s) does not scale with
14

---

Published as a conference paper at ICLR 2026
A.2 KEY LEMMAS AND PROOF OUTLINE
Roadmap. We prove Theorem 3.1 by (i) reducing the min-(, interpolator to a dual maximization and restricting the dual to the one-dimensional ray À = (Y, (ii) decomposing || || yll into a spike term (j E 5) and bulk term (j & 5). and (iii) converting back to the primal via the KKT map, which raises correlations the power (q 1) and produces the three-term maximum in equation 6 The elbow at r = ((p1)) comes from the sign of 1/r - 1/(2(p-1)), i.e., exactly whether the bulk-type contribution grows plateaus in the spike-dominated regime. We work single high-prooabillig event & (defined below) on which all concentration facts hold simultaneously.
Global event.
Let & be the intersection of the column-norm, spectral, and bulk events from and A.11 Then P(E) 2 Ce 2d- All bounds below hold on E.
Lemmas A.5
A.8
A.2.1
DUAL PROBLEM AND KKT
We briefly review Lagrangian duality for convex programs with equality constraints and then apply to the minimum-&y interpolator.
Primal problem and feasibility. We consider
min f(w) subject to wERd
X with f(w) ww
where p E (1, 2]. Since E Ruxd has full row rank n a.s. (for 2 n with i.idd N/ 1) entries), the affine constraint set {w Xw = Y is nonempty for every Y E R" The objective f is proper, closed, and strictly convex for p > 1 (indeed uniformly convex). Therefore, the primal minimizer üp exists and is unique. Introduce Lagrange multiplier IR" for the equality constraint, and form the Lagrangian
[(w, A) f(w) + (A, Y - -xw). The dual function is obtained by minimizing the Lagrangian over wt
{s(w) (x X, w) +Y,)) -
g(\)
inf wEB'
where f* is the convex conjugate of f:
(z(2
sup WERd
{(2, w) f(w)}.
Since f(w) Ei=1 |w;lB/p i is separable, its conjugate is f*(=) E 121% (1/1)ll=115- where q = (p 1) the Hölder conjugate of p. Indeed, for each coordinate
sup {a - 1tp/p)
1E3
achieved at sgn(2))=/0-1 with optimal value |2/9/q. Therefore the dual function
1 q
g(A) (Y,
| All
Dual problem and strong duality. The dual problem is maxXER= g\)), i.e.
max D(A), X
D( || A|l
This is concave maximization problem smooth concave objective with no constraints). Strong duality holds in our setting by standard convex duality: the primal is convex, the constraint is affine, and feasibility holds (Slater's condition for equalities reduces to existence of feasible point). Hence
min MEC X
f(w) = max D(A). XE
For convex programs with equality constraints, the Karush-Kiu Kuhn-Tuckerr (KKT) conditions necessary and sufficient for optimality under strong duality. They read:
(primal feasibility)
Xw Y, (stationarity) 0 0f(w) XTX.
15

---

Published as a conference paper at ICLR 2026
Because p f, is differentiable R with gradient
V (ww) = |w| p-- C w (w) & |w|p-1,
so the subdifferential collapses to the singleton {W) and stationarity is
Vf(w) X
At any primal-dual optimum (Wps X-) therefore have
X m Y, = = 10p p-2 の Gp.
(11)
The conjugate f* is differentiable with V fr (z) = ==
=z sgn(2) O |2|0-11 and the gradients
are mutual inverses: Vf* (Vf) - Applying V both sides of X X (ū,,)) gives the coordinatewise KKT map:
pp = (V
= sgn ((x (xTx))) 9-1
(12)
Equivalently, Vp ( and x= vf(pp))
At optimality, Fenchel-Young gives ((ūp) (x = (0pp X xT)). Using Xúp Y and the expressions for / and I* yields the identities
x
1 (Y
(13)
These will be used repeatedly pass between the primal and dual scales.
The affine set {w Xw = Y} is a translate of kcr(X), and minimizing ||l||p over it finds the point where scaled lp ball first touches this affine subspace. The outer normal to the e, ball the touching point is V normal lies in the row space of X. In coordinates, equation 12 shows that each coefficient of wp (g 1)-power of the correlation between the corresponding feature column X and the dual multiplier X"
kp.p Ip-2 @ Wp and the KKT condition X V f ((r,)) says that this
Specialization at p = 2. W - Vf())= = w and (2) = Then equation reads X X = W2 and X X = Y, which implies XXT X* = Y and hence X = (XX - Therefore YY
6 X (XX -
the minimum- (Moore-Penne Penrose) interpolator. For p * 2 the same structure persists but the map 2 h z Vz) sgn(2)]22(>>> nonlinear, which is exactly what introduces the (q - 1) --porr the subsequent spike/bulk analysis.
Why duality helps here. The dual objective
D(A) 1 2|I
separates the data dependence (linear in Y) from the feature geometry through || A|lg In our Gaussian design, the coordinates of X T split naturally into the spikes (indices in S) and the (d (5=s) s) bulk, for which we have precise & concentration (Lemmas A.8 and A.1Ip. Because D is homogeneous in a simple way along the ray X = tY.
(tt) = +| |
we will use the ray scale t, (the maximizer of D(tY)) as a canonical scale for A*; Lemma A. shows ||>*|| 112 x t. ||||| and provides blockwise controls on X T The KKT map equation 12 then
converts ||
|| T
(- 1)r into ||,,, via ||| 100-000111 || b which is the backbone of the
unified bound equation
A.2.2 CONCENTRATION FOR Y AND XTY
Let m ' for 7 10.
16

---

Published as a conference paper at ICLR 2026
Lemma A.5 (norm of Y). With Y Xw" and T3 11w0 12 + 0², have
|| = -2 n (1 + o())
w.h.p.
More quantitatively, for every t >
Pr | - r.2l 2 2n. + 2T²2) S e
Proof. For each row i E [m], (Xw*); = Σラ= WyX,, is M(,, 1 (12) since the Xis are i.i.d. (0, 1) and independent in j; the independent. The noise & (N,0 is independent of X, hence
|Y
Y ~ NOO, (n).
X..
T
The standard Laurent- inequality for x² variables (see e.g. Ann. Statist. 2000) yields, for all
t > 0,
Pr Pr(IV - 1,n 2 2T Vnt + 2T2t) S se Pr Pr(T.n 1 2 2T v nt S se
Taking en gives Y r o(1) with probability at least - -- on
Lemma A.6 (bulk coordinates of XT Y). Conditional on Y. for each j $ S,
(X,,, N(O, |
and the variables {(X,s, V)sss are i.i.d. given Y. Consequently, with my == TE for 7 ~ NO,,,), 1).
Σ (X,,, (d s) mn 18 (1 + o()) (d s) n//2 w.h.p.
Quantitatively, for any fixed 9 2 and any (0, 1).
(X j. 1)1a
Pr
mg
Y
< 2 eexp(-co(d min(u?, (u)
|
JES
Proof. Fix & S. The vector X,, ~ (0, In.) is independent of (X. k)Es and &. hence independent of Xw" xw= &. which depends only on the columns indexed by S and on &. Conditional on Y. by rotational invariance,
() - || Zz. Z₁ NN0 1),
and independence across j & S follows from the independence of the columns {X.s)ees-
Let Wj: := 12 my. Then W, are i.i.d. mean-zero and sub-exponential with [[yle < C. (a standard fact for polynomial functions of a standard Gaussian, see, e.g., Vershynin's High- Dimensional Probability). Bernstein's inequality for sub-exponential variables gives, for any u >
Pr
d-sI W j&S
> u |1) S 2exp(-cc(dd s) minfu? u))
d
Multiplying back by || proves the conditional concentration display. Since (d -s) s) x n by assumption, taking + 0 slowly (e.g. = (logn)/(C s)) yields
X Y = (d - 5) s)m Y (1 + o(1))
with probability least (ddss > -e -n (unconditionallly) Finally, Lemma A.5 implies ||Y||] x r/n1/2 w.h.p., completing the proof.
17

---

Published as a conference paper at ICLR 2026
Lemma A.7 (Signal block with integrated uniform column-nome control), Let X E Raxd have i.i.d. (( 1) entries, S C [d] with |S| = S, and Y == Xw" + { where & ~ ~~N(0,, I2)) is independent of X. Write + = I| 2 + + and W, := Eees lw31 for 2 2.
(i) Uniform column-norm concentration (over all d columns). There exists universal E (0, 1) such that, for every u > 0.
|x
Pr
max 15add
> - S 2d exp( - en min(u²,u),
(14)
n
In particular, for any fixed 7 O,
7) log E (0, 1] for a large, and Pr
Un
max 1Sd
> Un
S 2d
CR
n
(ii) Spike decomposition, explicit definition of Sj, and q-moment bound. For each j E S, define
E = (X:s) kEST(())
X + E).
(15)
Then
(X wj || + Gz.
(16)
Moreover, for each fixed j E S,
E[(; [
Var(Sy (X,,s) (T² - (wj)²) ||
(17)
and, conditional on X..j
G~~ ~~100
||X 1B3)
(18)
(We do not assume or use independence between the collection {Site the proof below controls their aggregate via operator-norm bounds.) Consequently, with probability at least - 2d -e--ee-vveee -eyns
Σ jES
(X r) =
+ :
+
(19)
where the o(1) (as - oc) and the hidden constants depend only on (hence on p). The mixed term Bjesla,19-1100 absorbed by Young's 's inequality into the n leading term and the Ejes|by] remainder, with a harmless change in constants.
Proof. Part (i): For a fixed: j. -z = | _ x² By Laurent-Massart, for all x 2 0,
Pr(Z, 2 2 nz + 2x) Se Pr(n- 2, 2 2/nT) S Se-
standard choice of x (see derivation below) yields the Bernstein-tye type bound
Z
Pr
< 2exp( en min(u?, u))
(Vu > 0),
(20)
for some universal E (0, 1). Summing over j = 1,..., d gives equation 14 For the explicit choice
Mn (1 log d/(en) < 1 (for n large).
2d expep(-2o exp((ccnu)) 2d expep(-2o (- + 7) log d) 2d
u.n 3 get Pr(Z, 2 un) S 2n
(Derivation of the Bernstein form): If E (0, 1], choose and unn to get Pr(n Z, un) < " If u 2 1, choose coun (c.g. co 1/16) so that 2//zz + 2x un, hence Pr(Z,
wn) I e coun Combine and absorb constants into
Part (ii): The decomposition equation 16 is immediate from
Y = wj X
w + E.
kES
18

---

Published as a conference paper at ICLR 2026
and independdence/rrtatioonal invariance: conditional on X (X X.,x) NO ( for k + J and ( ~ o00, (12), all independent. Let 0j == w [ l1 and by = Sj so that
(X.,, ag bj. We show:
Σ JES
-a11== (1 and
Σ1yl S 8T n8/2
jES
with the stated probability. Conditioned on the event from (i) with a = Un = o(1)
max 15iSd
Una
and by mean-value bound, || = n2 + (unn)) uniformly in j. Hence
1aa
Σ JES
|u X 112g n
w3l (1 - O(.,)) n (1++0000))
JES
JES
For any index set T C [d]. we write X [T for the submatrix formed by the columns {X... : E T}. When convenient we abbreviate X Xy. For vector E Fd Wy denotes its restriction T, and T the complement of T in [d]. Let G == X5 Xs and D == diag(] jé) Then
b = (by)ses (G D) w/ + X.C. We bound ||bl| and then pass to 4g Recall b = (G D) + Xg where G = =3 Xs R and D = diag(1) (1)))))
Bound on |I - D)w3ll2- We have
- D) I ||G - Dilop 110 S (IG- - nT + D nTs(lop) ||
(21)
Singular-value bound for G - nJs. Let 8max (Xs) and 8min (Xs) denote the largest and smallest singular values of Xs. By the standard Gaussian singular-value concentration (see Vershynin, High--Dmensional Probability, Thm. 4.6.1), for any 2 0,
P Smax (Xs) S /n + vs + smin (Xs) 2 vn - -stt) 1 1-226 2e 18/2
(22)
Conditioned on this event,
|[G mllllop max { Smax (Xs)² 2 n, smin (2
< (Vr+ + +) - n V n (vn v t)2
+ + 2 + 2t( / v8) +
(23)
Choosing (= V8 in equation
22 equutiionn 23 yields, with probability at least - 2e -//2
||G nTallop S s + 2 2/5(V 58) + < 4/ns + 4s.
(24)
Diagonal bound for D - nlg. By the single-column deviation bound equation 20 for any u > 0 and any S
X - > - n
Pr
2xxp( en min(u², ,,))
Union-bounding this over the s indices j E S and taking
us = s
(25)
we obtain
13
Ce
3 3 n,
IP max jES
us <
Ce de
(26)
$ n.
19

---

Published as a conference paper at ICLR 2026
hence, on this event,
D nTllopp max ||
n| S nug vns.
(27)
Combining equation 21 equation 24 and equation 27 we arrive at
|I(G - D)w v (4y's + 4s + vns) |l"l22 S (5yns + 4s) 1
(28)
with probability at least 8/2
s/2 Ce-é'vns,
Now we bound |X E Conditionally on Xs, the vector X56 is Gaussian with covariance
= Var X55 Xs) o² G.
Write the eigenvalues of G as fl, ,i...., M1, 2 Then
X Σ X 2
x== of Zi iid N( 1).
The weighted x² tail of Lausent-Massart (2000, Lemma 1) states that for all 0.
P
2
A; +
2 2
x + 2maxx 2s) x Xs e Σ x
(29)
t=1
Since = O'rr(), Σ, X = o²tr(G2) s o's ||G| and max; X; o² inserting
these into equation 29 and choosing gives, with conditional probability 2 1 1
< < 0² (1r(C) + 4s ||G||op)
(30)
We now bound tr(G) and |G'llop on the events already used in Step A. First, by equation equation 26
tr(G)
Σ
1 s sn + us) sn +8ns.
(31)
JES
Second, from equation 22 with Vs.
|G||op Smax (Xs) < (VT V8+ n + 4yns 4s.
(32)
Plugging equation
31 equation
32 into equation 30 and taking square roots, obtain
|| S σ V sn + syns + 4s (n - 4 + 4s)
$ 0 Vsn Vovns + 2//n 48)
&Co(Von + s),
(33)
where in the last step used Vsyns 33/ < { (VST
l2 and bg bounds for b. Combining equation 28 and equation 33
lbbl < [I(G D)w 1l2 + || < c( ((vns) I ll + +o
(34)
In particular, when < n the terms are dominated by /ns and
|bble & OT vsn
(since 7" = ||w" 112 1 +
(35)
(Refined q-moment bound via decoupling). Introduce i.i.d. "ghost" columns (Xis independent of (X, E) and set
Uz),
Uj X. W$y6) +
20

---

Published as a conference paper at ICLR 2026
By standard decoupling inequality for Gaussian chaos of order two (de la Peña and Giné, Decoupling: From Dependence to Independence, 1999, Thm. 3.5.3), there exists Cg < oo (depending only on q) such that for all t > 0,
IP
14 >1 S Cq P
Σ //1011/0 1/Co) JES
JES
Conditional on {uy}. ,. the variables [yi are independent centered Gaussians with variances
||u 12. On the singular-valle value and column- norm events used above (cf. equation 22 with t v and equation [4], uniformly in j.
|uy 2 - slop ||w" 112 + IkEl2 < C(n + 4yns 1s) || Co S C (n s).
Hence, conditionally {uy}. each 15/1 l is sub-exponential with -orm & Cr3 (n + s)9/2 Bernstein's 's inequality then yields
ΣSS S C Cr s n/2 + s2t//2) jES
IC$1
with conditional probability at least Ce
Unconditioning and applying decoupling gives, with probability at least 2d Ce
Σ jES
1b, Σ = Σ 141
EIGI S CT² (r2 + s14//2)
(36)
JES
In particular, if S n this simplifies to Ejes 1b,1 I CS n//2.
For the cross term, for 2 and any a, we have the elementary inequality
||a + 6/8 |a|* s (aa + 16/9),
(37)
for constant Cg depending only q. Summing equation 37 over E 5 with aj = wz || and by = Sj. and applying Hölder,
Σ JES
la + 6
|a,18|
S Cg Σ
|a,1 1b,
C. Σ 16,19
jES
JES
P-1 (ay1")
I CC
16,19)
C Σ
1b119.
(38)
jES
jES
jES
Set
A=
1a,1%,
B: Σ
1b,19.
JES
JES
Apply Young's inequality with conjugate exponents
q-1 and q: for any >
A = B < e +
(4-1)
9-1 A
1 1) B.
B
1 +
(39)
With A n' 0 (u.)) and the bound B < C (s + 1///2) from equation 36 choosing a fixed E (0, (e.g. == } absorbs the mixed term into the leading A and the B-remainder (with harmless change of constants). Consequently,
Σ jES
ll, + 6,1" |a,15| ( ss :
which yields equation
19
When s S a the remainder simplifies to Oss T, n//2).
Combining Lemmas A.6
yields the decomposition
||X T 1l n Wg (1+o(1)) + (d-s) my 72 9/2 +
w.h.p. (40)
21

---

Published as a conference paper at ICLR 2026
A.2.3 BULK EMBBDDDN AND GAUSSIAN l RELATIONS.
Lemma A.8 (uniform l control on the bulk operator). Let q E (2) and assume Kbulk lim n d-a that, with probability least Ce simultaneously for all RR..
There exist constants < cq S SCq < oc, depending only on (q,Kbul)), such
cy (d (d- 11AI2
(dd-s)
(41)
jES
42 (Here we absorb the Gaussian absolute moment mg Z into the constants Cg. Cg: in equation
we keep my explicit.) Moreover, for every E [1.g]. there exist constants C C. < 00, depending only on (bulkb,)), such that, w.h.p., uniformly in X E IR".
1/t (d 5) 1/t 1|Al12 s 1I(K.
S C. (d 5)=s mi 1/t ||ll22
(42)
where my = E 21 for Z ~ 1).
Proof. Fix X E R", and if X + 0 write a == A/1 E g"- By homogeneity,
Σ
Σ
(X.,s, u)
(43)
jES
jęs
and similarly for any g].
(X 1²) 1/8
[I(I (X. = =|1l|2
(44)
j&S
Thus it suffices to prove the bounds for unit
Let == 5 and m = [ d 8. Fix u - and E [1, Define
(w) [X EET.
Since the columns {X.s are iid NNO, and independent of u, the random variables are i..d.
Definition A.9 (Orlicz uu norm and sub- Weibull class). For v E 2 and a real random variable Z, the Orlicz norm
| K" s =
1 = inf [ 0: Eexp
If |1 < 00, we say sub- Weihull of order v. Special cases: v (sub-Gaussian) and v Sub--xxoneentia). Two basic properties
P(|Z > x) exxp - -c(x///Z|le.)) (Vz 2 0),
(45)
7 - E pb < 2 ||
(46)
Definition A.10 (Gaussian absolute moment). For t > 0, let 2 ~ NNO, 1) and define
m, = E|Z|' 2t/2 ( ++1
2
/ π
(u)) in a/, (with explicit mgf computation). Since (X. u) N 1), write Classification of y (u) ~ (O, 1) and set W = 1 For any K 0,
W 2/t
2/t
1212
K
=
K
K2/L
Let
1
0 = K2fE
22

---

Published as a conference paper at ICLR 2026
Then
Eexp (W/p((W))))) 2/t E exp z²).
Compute this expectation explicitly: using the standard normal density c0(2) (2x) =212
El E
n=²
(02)
dz 22 / e=² E
==21/2 12/2d dz
3
1
(
e
-0) dz 2m
for <
(47)
2t
1 -0
v1 20
1/2 Equivalently, since Z ~ x². the mgf of xi is - 20)-1/2 (1-20)-11
for 0 < 1/2, which matches equation 47
now choose so that 1/2 and the expectation uniformly bounded by a constant S 2. Take
1
1
K₂ = (41)
(12 1).
(48)
Kt
2/t
4t
4
Then, by equation
47
E Exxp Eexp(( (W//K))
1
V2 < 2,
(49)
K.
21
where we used I [1, q] (hence 2 1). By the definition of the Orlicz norm,
|| | Ilaze I Kr = (4t)
(50)
Centering preserves the class up to factor 2 (by equation 46 hence
21" my Il I 2K, 2(41)*/2
(51)
Finally, define
v(l) = min (1, 2/1].
(52)
Since 2/1 1 for / and 2/1 1 for 1 2, combining equation 51 with equation 52 yields the uniform (in u) classification
( m4 Ilearo I K2 with Kj = 2(4
(53)
This bound is uniform in u because (X.,, u) $ 1) for every fixed u E Sn-1
Empirical-mean mean concentration fixed u. From equation 53 and independence across j T, Bernstein cormstein-type inequality for sums i. sub- (ub-Weibull(y) variables (e.g. Theorem 3.1 in Kuchibhotla- Chakrabortty, 2018) yields, for any E > O.
P
(w) mi)
> < 2oxp -cxee m min
(e) 6² K2 (k:
)
(54)
JET
Taking s = 8m, with d E (0, 1), and absorbing the fixed ratio m./ K2 (which depends only on f) into the constant, we obtain
(
1 Σ ((((()))))))))))))) mi > :dm)) S 2exp a m min(5" gu10)).
(55)
JET
where c > 0 depends only on (hence only on p). In the sub- sub-Exponential range t E [1,2], v(t) = and equation 55 simplifies
P
( m, > >8m²) s 2exp( - m miin(8,88))
(56)
JET
Finally, note that
Ey(1 (u) m
(57)
by Definition A.10 completing Step 1.
23

---

Published as a conference paper at ICLR 2026
Now construct net the sphere and uniform bound that net. Let E E (0, be fixed absolute constant (to be chosen below). There exists an E-net Ng C g"-1 with
[ < (1+ 2 E
S C.
(58)
Applying equation 55 with 6, E (0, 1/4] small absolute constant depending only on and boundinn over yields
N m
Σ JET
1 (v) m
bc m
S 2 exp( ct m min(6,,81) 5.3)
2 exp nlog C. - dm)
(59)
Because 2 Kbulk?i and Khulk the right-hand side of equation 59 is Ce-on, Therefore, with probability at least Ce simultaneously for all u E Ns.
> 0, by taking ó, fixed (e.g. d, = 1/4) and fixed (e.g. = 1/8),
(1 1-5mmm
(X.,, S (1 + 6x) m.-
(60)
JET
We are ready extend from the net the whole sphere. Fix arbitrary u E g and pick N with 2| 2112 I E. For any a, b E R and any t 2 1, the elementary inequalities
la + b1t <
2 (Ia|* + (6|²),
lal' < 2 (la + 61' + (6(²)
(61)
hold. Applying equation 61 with a (X u) and (Xar, v). we get
(X j. u) < 2 (X و v) 1r + j u) 1*) > 1' (X i1 u) - (X u u)
(62)
(X و u)
(63)
Average equation 62 and equation 63 over j E T and divide by m to obtain
(X io u) 1' s 2
1 m
(X 3o 2)) + 1 Σ
(X o)) -0))
(64)
m
JET
jET
jET
(X.s, u) 1r 21-2 1
Σ
(X.g: v) 1-- Σ
(X.j; u - 0) -
(65)
m
m
m
jET
JET
jET
For any w E R",
1'=
=|[w||
Σ (xx,5,
@ =
w
(if w # 0).
(66)
m
JET
JET
Define the functional and its extremal values
1 Σ (Xij))))
A(u)
S: sup A(v), -
1: inf A(u). ag8
m
jeT
By equation 66 and u112 Ss.
(X u u- v)) 10= 1|u m
1x SES
(X.J,u-o)
m
JET
JET
where we used the definition of in the last inequality. On the event equation 60 (from Step 2). A(v) E [[1 - 5,)m,, (1 + m.] for every N.. Plugging these into equation 64 equuation 65
yields
A(u) <2 2 (A(v) + e's). A(u) 2 2'-(A(v) - -E'S.
24

---

Published as a conference paper at ICLR 2026
Taking the supremum over g"-! in the upper bound:
2'-1
S S ((1 + b m, + E
S
(1 + 8t) m.
1 2t
Taking the infimum over gn-1 in the lower bound:
2 2 21
5t - -5'S.
Choose fixed & - 1 and ε s 5. then
27-1e² (28)
(1/4)
v 2
S 8'
80 1 - 2 2 7/8 and thus
2 (1 + 8e)m, S C me:
S S 7/8
for a constant C < oo depending only on t. Substituting this bound for S back into the inequality
for I gives
1 21 6.)m - c.C.m 2 ma, for some a > 0 (depending only on t). Therefore, with probability at least 1 - Ce
m <
( u)
C2 me
simultaneously for all u E gm-1,
(67)
JET
Multiplying equation
67 by m d - s and using equation 43 with t yields
Cq (d 5) ||II S
Σ
s C, (d - 5) ||Al
J4S
which is equation 41 Likewise, combining equation 67 with equation 44 gives
d (d-s) m, 1/t ||al22 I [[(( Diesl S // (d (d-s) 5) m. 1/1 1|ll,,
which is equation 42
A.2.4 SPIKE l CONTROL FOR X
Lemma A.11 (spike control for X YY. Fix any & [1. and y 0. With probability at least
1 1- -Ce
(I(X Y)I Jes): n m|ll"lle ll (1 Ouu)) + C mmxx 2
(68)
where := (1++)) +1 d/ccn) [ = o(1) and (x)+ = maxfx; 0}. In particular, if S then the error simplifies to
(I((X))))))
|| (1+ O(un)) + C vn vn max// 1/2)
(69)
All constants may depend on (hence on p) bui not on (n, d,
Proof. For each j E S.
(X Y' = w3 |1
Si,
G== (X.S) Σ
w + +5).
(70)
kES
Conditional on X.j=
BiG₁ X = 0,
Var(G = ( (x- 1
(71)
and C X..j ~ N(0 (T:- -
by independence and rotational invariance. Define by = وo (az)ies, b: (by)jes-
aj = 1
25

---

Published as a conference paper at ICLR 2026
By the uniform column-none norm bound equation 14
with Un o(1), have
||X,,
max SSDSd
1
U,z
with probability at least 1 2dd
(72)
On this event,
((aas))sssle
= Σ
1/t
Σ JES
Juzl' (1 (unn)))) 2/8
jES
= n 11 lle (1 + ((un)).
(73)
Let Xs be the submatrix with columns {iss and set
G G==
D diag [1x
From equation 70, in vector form
b= D) + X55.
(74)
We bound the two terms separately.
(i) Control of (G - D)w3. By the triangle inequality and operator norm submultiplicatiiity, ((G-D)w33
| ||w"|22 I (I|G nde lop I|D mndallo)) || w|2"
(75)
Gaussian singular-value concentration (Vershynin, HDP, Thm. 4.6.1) gives, for any 2 0,
P max (Xs) S vn + V Smin (Xs) 2 vn V8 -t)) - 2e 1//2
(76)
On this event,
|G - nTsllpp = max 8max (Xs)² - n, n - smin (Xs)²
I (Vn + v8 1)² - V n (vn V 1)2
s nss + 2t( vn + vz) + 1.
(77)
Taking v yields, with probability - 2e 5/2
||G nI llop S 4
ns + 4s.
(78)
By the S-only column-norm event equation 26 (with us s/
|| nTellop max
||| x n| nus v Vns.
jES
Combining this with equation 75 and equation 78
yields
|I la < C (vns + +) ||22 with probability least 1 1- 2
Ce- -a/ns (79)
(ii) Control of XJ Conditionally on Xs. one has X ~ NO0, oGG) Writing [ for the eigenvalues of G and X; ;= o² Laurent-Massart weighted x² tail (2000, Lemma 1) yields, for
all x 2 0,
P
Z > Σ d +2
Σ X3 x + 2(max At)
Xs
S =
(80)
Using = Σ, X s Ass |[l|ep and max; d = O2l|ol and taking x gives,
with conditional probability > c
|X < d² (T((G) + 4s [[G|lop).
(81)
On the event equation 76 with = f and equation 72
tr(G) X 11 < sn(1 + un) sn o(sn), ||G|lop (Xx) 4/ns 4s.
JES
(82)
26

---

Published as a conference paper at ICLR 2026
Plugging equation 82 into equation 81
and taking square roots.
| x Co
with prob. 21- =1-2e- -
(83)
Combining equation 79 equation 83 and equation
74
||bl[ C (v sn + s)
with prob. 1 -2d C0
(84)
For E [1, 2}. the norm monotonicity R' gives
|ible, & 8 1//t-1/2 11b122-
(85)
For t 2 2, | [1b|2-- Hence, for all t E [1
||bll, I s (1/1-11/2+ lbbl2 I CT V 1/2) + 1+(1/t-1/2)+))
(86)
where used equation equation 86 reduces to
84 In particular, if S then 1+(1/1-1/2)+ S max and
lible. S CT,, vn
(87)
Finally, by the triangle inequality,
[1(, + biDsesllee < [[(ls]])ees la 1(1b)))sss
(88)
[1(la + byl)seslle
2 [(las))ses le.. (((6))))esl))
(89)
and combining with equation 73 and equation 86 (or equation 87 when s S n) yields equation 68
(and equation 69
A.2.5 RAY CONTROLS: MINIMAL COMPARISON AND BLOCKWISE BOUNDS
For the ray À tY we have the one-dimensional dual objective
1 q
D(t) = (Y,+Y) || (tY) +||1
|| |
(90)
Since D* -q 1) || Y 1 < 0 for all t 0, D is strictly concave [0, oo) and admits unique maximizer given by the first-order condition D'(I.) 0:
| ETTYI
1--
(91)
At this maximizer,
(tt.) = + |X Y1 -(1 ) t X T (1-1) |X (LYY)
12
1
(92)
Lemma A.12 (Ray controls). Let E (1,2), q= - E [2, oo), and define 1, by equation 9 With
probability at least - Ce-a(d c(d-s) Ce -evns (constants depend only on (q, Kbulk)), the following
hold simultaneously.
(One-sided value comparison).
D(X)) 2 D(t.)
and
| 2 || [tTY()))
(93)
(Dual-norm scale). There exist 0 C1 x depending only on (g, Khullk) such that
a4418bb s ||*||22 S C1 1|
(94)
(Bulk block level t [1,4). For each E [1,q] there exist 0 < a S C. oc (depending only (t, Kbuulk)) )) such that
oe (d (d=s)(/mm) 2/t 411[1112 5 ("),, sesl, S c.2lt (d - 8) ") t.]lX|12s (95)
27

---

Published as a conference paper at ICLR 2026
where m. E| for Z ~ 10.
(Spike block: two-sided t-level perturbation). For every E [1.
[(<X),; jesll = (1 M) jesl. l. + C24 12 (0/t-111111 (Vi+ + V8), (96)
for a constant C2 = C2(q. Kbulk). In particular, if S n then
L. (.11(1Xx.5 X)D + C3 To V max
(97)
In the last display we used | = T1 + o(1)) from Lemma A.5
Proof. We work the intersection of the high-probability events supplied by Lemma A.8 (both equation and equation (2).. Lemma A.6 and the singular-value hound equation 22 this intersection
has probability least 1 Ce-on
(One-sided value comparison equation 93
By optimality of X and the definition of f.,
D(X*) 2 ((4))
Using the Fenchel- -oung identity at the optimum (see equation 13) and equation 9
D(**) (1
4) | 11
D(L.) (1 (1- + ( || (1.)))
hence equation 93
(Dual-norm scale equation 94 Lower bound. From D(>*) 2 (t..) and equation 92
D(*)) Z (1- 1--))..11113
Since & || we get
|| 2 (1 (1- L. ||
Upper bound. Let
S(A) Σ (X AX1,
B(A)= = Σ ((
JES
SE8
From equation 1
D(X*) = (1(2)) + B(x))) (B)))))+(B1)))
By Lemma A.8 (left inequality in equation 41)
B(²*) 0g (d - 8) ||
Combining with D(X*) < ||V|||||*|| lla gives
(1 - 1 (1-))c((d
cd 8) | < ||
(98)
Next, Lemma A.6 vields
Σ (X = (d - 5) my | 18 (1 + +ol)),
348
so | 2 cdd s) |Y 18. From equation 91
(t,1)1ll2))= 0-2 || g+1
1 |
<
||x Y 118
c (d s)
Comparing with equation 98 gives |1 S C (1+||1||2)011 and hence ||||||2 S C1 4, |Yll2.
28

---

Published as a conference paper at ICLR 2026
(Bulk block equation
x 95
Apply Lemma A.8 at level t (two-sided inequality equation 42) with
(d mm | v (IKX, \))) $E8 l. I (d-s) ||^
Substitute ||^112 x | from equation 94
(Spike block equation 96 equation 97 Set h = X L.Y. Then
S = t, X. s.
For any t 2 1, the triangle inequality gives
((\D,s - t. (1(,),, MiDses) + | le.
and the analogous lower bound with a minus sign. By norm monotonicity in R and operator norm submultiplicatiitty.
| g S s(1/e-1/2)+ 1ITTs/112 S s(1/1-1/21+ smax (X s) |//|22
From equation 22 with t = Vs, mmax (X.,s) & C(vn+ + v8) w.h.p.. and from equation 94
||hll2 ||A- t.Yll 1
|| 12 || S (C1 + 1) ||
Putting these together yields equation 96 If S Lemma A.5 gives | T.
+ o(1)) and
(1/C-11/2) (Vn + v - 2/n s 2//,1/
which implies equation 97
A.3
PROOF OF THEOREM
With these lemmas in place, we are ready to prove Theorem
A.2
Proof of Theorem A.2
We work on the intersection of the high-probability events provided by Lem- A.11 and A.12 this event has probability at least -Cee --Ce-adda) Ce -e-evne consistent with Remark A. All constants implicit in x depend only on (q, Kbblk)-
mas A.5
A.6
A.7
A.8
2d
Along the ray À = tY, the one-dimensional dual objective
D(I) = 1
strictly concave with unique maximizer given by the first-order condition (see equation 91
1--1
(99)
By Lemma A.5
Y rī o(1)), and by the decomposition equation 40
X
Y n W (1 + o(1) + (d s) mg T n / (1 + o(1) + Oss n2/2)
Substituting into equation 99 yields
T2
w.h.p.
(100)
n&W, + ((d - 5) m, + (s)) n/
By strong duality and Fenchel- Tenchel-Young (see equation 13
sup D(A) (1 (1- 1) ||x ||XT>>11 X111 (1 (1- 1 9
(101)
X
Evaluating D on the ray at t, and using D(X*) 2 D(t.) gives
29
||
1 | 2 (XT(.Y))
(102)
29

---

Published as a conference paper at ICLR 2026
Moreover, by Cauchy-Schwarzz Schwarz and equation 94
|| X* 1 = (x) & | 1/2 2 t. 1111 = + || 1
Combining with equation [02] we obtain the two-sided scale
|| = || z 17 | Y 11
Using the coordinatewise KKT map equation 12
up Vf sgn(X
Hence, for any r E [1. pl.
1 = 1 (q-1)r-
(103)
Split the (q 1)rmm over the spike block S and the bulk block S and note that ||lll ||us|ll ||use ll implies ||alle x max(Iluslle |lus-lle:
d x max { [I(KX X Aes Dh (K AMDsss ( }.
(104)
(We used max (a, b) S (at + + v 21/² max (a, b) for t 2 1.)
Set t 11r S q. By the spike-ray perturbation from Lemma A.12 (see equation 97 when
$ S n)
[I(K jeslle (1(((x (XI) sesll,. C Ts T max (1/t,
(105)
(If > n, use the general form equation 96; the conclusion below is unchanged up to constants since ((1/)-1/22++ which is captured by the final "spike remainder" term.) By Lemma 1. at level 1,
(Vn+ /z) s(1/t-1/2)+ S 2/n 8 /max(1/2.17
[ Y)1) selle = n || lle (1 + o(1) = C' To vn (1/1,1
(106)
Combining equation 105 equation 106 and using (a + b)0-1 < 20-2 2 (a-- + b9-1) for a, 2 0, we
obtain the following uniform two-sided bounds (recall ( (q 1)r S q):
-1--1 s c to t n 9-
| 11 + (t+ vn) 8-1 max
(107)
9-1 2 4. 12 - I. TT max maxx 9-1
(108)
Applying the mean-value inequality to the map + 29-1
[ + g) 1 1| s C (2- + y--1),
with æ= t.nll="ll lle and y = Ct, Te / max 1/2), we obtain
(I((..,;A))) sesllle 9-1 = 1. n 9-1 ||0 - (1+0(1) + (4 max1 (4-1)/2, 1)/4) 112-11
(109)
Recalling (g- and [w=lle x ||w*| ()-1) l)r- we obtain the spike contribution stated in equation
(For completeness: specializing equation 90 = q together with Lemma A. at t yields the same rute and remainder exponent as in equation 709
By Lemma A.12 (bulk control equation
95) together with equation 94
[I((X.,. X ()
( 1)r x (d - 111011111111110 ()r) t, ||
Raising to the (q - 1)-th power and using || ||||2 x T. /n (Lemma A.5
[I( X") | ijr (d - ()=ss)are 2/ (n)) r-1
(110)
Plug equation 109 and equation 110 into equation 104 This yields
lrr x max { 19-1 n ||
8-1
9-1
(d-s)2/r (4T)) 9-1 maxx11/n.( (1. vv) 9-1
()-1)r
30

---

Published as a conference paper at ICLR 2026
which is exactly the three-tern term unified bound in equation 6 When 2(p 1) and (d (d- s) 2 the third term is absorbed by the bulk term, recovering the two-term maximum.
In the proportional regime (d s) equation 440 to define
Kbulk balance the two leading terms in || Yllg (cf.
T2 (i) Dual spike-dominated regime > nA. Then || Yl1 = n n&W, and equation 100 gives
n W = (d - 8) Tg n0/2
na/2 x Kbulk W
n x
Kibulk
T
W
which matches equation g
T,2
+
19-1
n (9-1))
(111)
naW,
W
Consequently
Tgt+1
(d-s) 5)111 (44 Tv/T)
9-1
W 13
(112a)
rg+1 max (g1/(2)1)/2) n 20 17.
max (1/1)//11)1)) (4x T2 9-1
(112b)
W
In particular, when r S 2(p - 1) the two "bulk-type" terms are of the same order (and are dominated by the spike main when 2(p 1)); this recovers equation
(ii) Dual bulk-dominated regime < nu. Then ||X
1
(d S)T2n
and
-220 1-
6 x (d - s)r2n9/2
(113)
(d - 8)
Therefore
(d (d- 5)=s) 3/1 (t. Ts rv//n) 9-1
4-1 bulk
Ts
1 -
(114a)
8 maxt (o- Tv)T)) 9-1 Khilk Ts max{ 1/r, (m-1)/2) n 112
(1 14b)
Taking the maximum together with the spike main term gives equation 10 whenever the third term is absorbed; otherwise the third term with exponent max(1/ (/r, (g 1)/2] 1/2 may dominate.
This completes the proof of equation 6 (three-term form), the energy scale equation 102 hence the proof of Theorem A.2
A.4 TWO CONCRETE COROLLARIES: SINGLE SPIKE AND FLAT SUPPORT
We keep p E (1,2), p E oo), r E [1, pl. and Kbulk lim inf s)/n > Recall the p-1 unified bound from Theorem A.2 We will repeatedly use the identity
|ppllr x max 1 1--1 9-
9-1 w | (d - -)2//r (1. T
(115)
maxf 1/r, (0- 12-1 (t. To vn) 9-1).
(116)
together with
X |
T. -
n. x Kbull Wg
Wg
Σ 1l3f9
- o
jeS
(117)
31

---

Published as a conference paper at ICLR 2026
Case (i): single spike (s 1).
Let the support be { and write [wj/
0. Then
1 a²,
| 1( a, T3 a² + 0
(118)
The transition scale simplifies to
(a² )0/2 222
n, x
Kbulk
(119)
a9
In equation
115
the spike remainder is dominated by the bulk term since
spike remainder bulk
-d (d-
for large d.
(120)
Dual spike-dominated (n >> n.). Using the phase form equation 9 we obtain
(a² + o 1
r 2(p 1), 26
|1pllr x
(121)
a² + o²
r> 2(p - 1).
Dual bulk-dominated (n & nt).. Using equation 10
Trppll x
max bulk (a² + o2 - bulk (a² + a a n an-1n x-13.
(122)
(The third term in equation 10 equals bulk J.n 11221/2
1/2 and is dominated by the first term for large n.)
Case (ii): flat signal its support. Then
Assume w/ as for all j S with |sy| and |S]
||w* v la).
Wg lal,,
| ((g- sto la),
of = + (123)
The transition scale grows linearly in s:
The x
Khull
(sa² + 02/0/22
$1//0
-²2
- Kulkk 8
0² 9-2 (1+80²
1
sa.
(124)
Dual ppike-dominane (n >> n.). From equation
9
(sa² 02) 2
n
10-1)
r 2(p 1),
|a/
(125)
8 - sa
2 + 0
r> 2(p - 1).
aa
In the noiseless case (a = 0),
>> 2(p - 1): @ 2/r la],
S 2(p 1): ||0
--d |a|
1)-1)
(126)
Dual domk-ndmm (n & n.J. From equation 10
1-1 bubk (sa² 22/1/2 n 1. 2 Kbalk (sa² + 02)²=² 8//r|a/ 9- 2-1
@ max
(127)
Kiulkc (sa² + 02)1/2 (00 n -1/2]
(128)
When 5 2(p -1 and S (d s), the third term is absorbed by the first (Remark) A.3
32

---

Published as a conference paper at ICLR 2026
B FROM INITIALIZATION SCALE TO AN EFFECTIVE by: A SLOPE- VIEW
visualizes the mapping Peff (a) use throughout. The construction is data-free (independent of n and σ) and relies only on the gradient-flow potential that characterizes the two-layer DLN implicit bias. Pseudocode can be found in Algorithm
Figure S1
We start from the separable potential
3 du 2 14+ + arcsinh
(3)
9
(129)
i=1
((=) l aresinh
"1 (5)
(130)
At the coordinate level, letting &o (t) = a² q(t/a²) gives
t
va (t) arcsinh
(131)
2a
1
(132)
a² /²v/4 4+ (t/a²)²
4a + 1
Asymptotics for q control the limiting geometry (all logs are natural):
((2) x² z²
+ 0(*),
- 0,
(133)
192
1
(2) = z(log - 1) +
+ 00.
(134)
2
Hence Qa behaves like Eg as a--000 oo and like an t1-tyee penalty (up to a log) as a a
To turn this into quantitative a - mapping, we evaluate Qa on the k-sparse, unit- probes
30h) E R., For this family,
8((A) 10/)- || 1, #li 1 * 0) = k.
(135)
(x(3())) = a² k
(136)
a2 2
while ép (calibrated via [| has the exact scaling
1 =k
k 11
(137)
Vi
We now fit a log-log slope to the k-dependence of Qa and match exponents. Fix a > 0, choose logarithmic grid KC (1,2,.... d) (c.g., up to 104), and solve
log (3(3)) c(a) s(a) log
k E K.
(138)
Comparing with equation
137
which grows 1/2) yields
(a)
s(a)
(an = s(1))
(139)
2
The limits in equation
133 equation
134
imply
a 00:
Qa()()))
+
s(a) +0, Pea(a) +
(140)
4az
a5k
1 a 0 (0(3())) = VT (log( aavk
- + 2a²k - akkk + OaSk2RR).
1)
(141)
s(a) + 1
Pet (a) 1.
Thus (a) increases smoothly and monotonically from 1 to 2 as grows, exactly as depicted in Figure S1 The inverse problem problim-choosing a for a target p" E [1,2] is the scalar root Pef (a) = p which we solve by bisection using the monotonicity in (Algorithm)
(142)
33

---

Published as a conference paper at ICLR 2026
Algorithm 1 Slope-matching map + (a)
Require: Log-grid A of a values; log-grid K C C fl....,d, df of l values Ensure: {(a: Pett
(a) ) : a A}
1: for all a E A do
2: Initialize lists
D X flog ki. Y = flog Qa (3(0))
3: for all & E K. do
4:
2k + (/(2////
5:
Compute g(2x) using the closed form in equation 130 if [ex] is small, use the series
(<))
= / /4/1 + 2550 +... for stability
6:
Qx t a² k ((x))
7:
Append log k to X; append log Qx to Y
8:
end for
9:
Fit 2 c(a) + s(a) X by least squares
(etr(a) (1-s(a))
> by equation 139
11: end for
return {(a,p.n (a)) a A}
Algorithm 2 Inverse map p" + a* by bisection in log
Require: Target p" E [1, grid K;; bracket < O'uin < Omnxx with Peft (Amin tolerance E>0 Ensure: a" with (Pea((a)) -
S p" S Peff (Qimas);;
S
1: Umin + log Ommnn Umax +- log Olmax
2: while Umax Umin > 8 do
3: Umid 6- {Umii + Umax),
Okmid f- e 2
4: Compute Peff (amid) via Algorithm restricted to this single
5: if Pott (Amia) < p" then
6: Umin Ulmid
7: else
8:
Umax f Umid end if
9:
end while
11:
return a +- e MMein t/2aaa/22
C ADDITIONAL NOISE SWEEPS: O E {0, 0.5)
replicate the experiments of 4.3 and $4.4 two additional noise levels, 0 and 0.5, keeping everything clse fixed (same {I.1, 1.5, 1.9} for explicit minimum-(, Lp runs; same a E {0.00102, 0.0664, 0.229] } for DLNs with the same a + Petf calibration in Appendix B same seeds and learning rates indicated in the panel captions). Each plot overlays test MSE (left axis) and representative l, curves (right axis).
Experimental protocol.
What the figures show and why. In Fig. S2 Fig. S9 the slopes and regime rules from Theorem 3.1 and Corollaries 4. 4.2 are unchanged across σ; noise only rescales т, and thereby shifts the transition size n, x (Kbulk / /W, 22/,))2((222 [equation 8 and the spike-side plateau levels [equation 9]. Thus, compared to o 0=0.1 in the main text: (i) at 0=0 elbows appear earlier and plateaus (for 2(p-1))
occur sooner and at lower levels; (ii) at 000.5 elbows are delayed and spike-side plateaus are higher.
Bulk-dominated panels retain the 1/2 growth and the r-ordering equation 10
D FINITE LEARNING RATE EFFECTS
We consider the single-spike case w* = 61 and a small shape parameter a = 0.00102 (so the calibrated Per (a) = 1.10). We vary the learning rate Ir {10-1 1 10 10 33 and the label-noise level a E {0, 0.1, 0.5). All panels plot generalization error (left axis) and f norm (right axis) versus sample size
34

---

Published as a conference paper at ICLR 2026
Slope matching mäp "4-00 Qu
Elfoctive exponent Par
1.8
1.6
pou1.30
1.4
0:0012
1.2
1.0
10
10-3
10
10-1
10'
Initialization scale
Figure $1: Slope-matching map Peff (a) (Algorithm of Qa () against the exact &o p-p/2 scaling of ||g(K] annotated; their corresponding a are solved by Algorithm
1 obtained by fitting the k-sparse scaling Target points (p E {1.1, 1.5, 1.9}) are
p-11
p.15
0-0.0,
Generaliationn encrr 2 noms
theory (spiee dem.)
zeseralaciio ener
theory (ulk-doml theory Cpp dom
4111 nom
theory
Ceneraliratin entor
Caneer wwwerlltaiionn arae
2.15 nom
3.0
L norm
85
2000
4000
0.A
1020 2000 3303 4305 3000
0.0
2070 (samples)
4007
(samples)
(Samples)
(a) 1.1 (sparsity-leaning)
(b) = 1.5
(c) 1.9 (dense-leaning)
Figure S2: Single spike w* = G1; explicit minimum-f, interpolation (a = 0). Farlier elbows
and lower spike-side plateaus than at 0-0.1; bulk-side traces keep the 2/2 slope, consistent with equation 9 equation 10
Observed effect. With clean labels (a 0), the 4111 norm is essentially flat across n and insensitive to Ir (Fig. (TO), consistent with a low-pef (sparse) implicit bias at small a. When label noise is present (a (0.1,0.5)) increasing the learning rate makes ÉT11 increase with n (Figs. $11 S12); the transition point (the "elbow") beyond which the norm would plateau shifts to larger Ir grows. Within the accessible sample sizes this rightward shift makes the curve look bulk-dominated and rising if the effective exponent Peff were larger.
Why this happens. Finite step size together with label/gradient noise injects additional stochas- ticity into the discrete dynamics. A useful approximation views (stochastic) gradient descent Langevin-type process with an effective temperature controlled by the learning rate and the noise level; this broadens the stationary distribution and leads to wider, less sparse solutions (Mandt et al., 2017] Smith et al.) 2018 Yaida 2018) Jastrzebski et al.) 2017). For a single-spike target, that diffusion leaks mass into ofg-signal coordinates during early training, nudging the geometry away from "C, -like" toward higher-p regime and delaying when the spike dominates-
the elbow shifts right. With clean labels, the gradient remains aligned with the spike and the small-step implicit bias toward path/diagonal-nont solutions is recovered (Neyshabur et al. 2015a; Gunnaeekar et al. (2018). The same qualitative phenomenon also appears for the denser case 5=50 with smaller magnitude.
E LARGER SPARSITY FOR EXPLICIT min [ |wllp LINEAR REGRESSION
We revisit the explicit min |w]| experiments at larger sparsities s E (500, 5000} for p E (1.1,155 1.9) under the same Gaussian design and noise 0.1 in the main text. Each
35

---

Published as a conference paper at ICLR 2026
p=00.p=11,
$0
d- 0.O, s- 50
d= 0.0, 1.9.650
Genandication wrer 1.1. nors
therry (bc
Gencralizatin
Generoiation theory
thesry dam))
Generiitation ereor
Gemeraliiation emor
1.0.
20
100
8.8
numm
J narm
0.00
0 1020 2030 3030 4000 5202
1900 2600 1005 6030
5000
0.85
2050
4000
(5ampllss
(sampless
(samples)
(a) p = 1.1 (sparsity-lannnn)
(b) p = 1.5
(c) p = 1.9 (dense-lcaning)
Figure S3: Flat w (s = 50); explicit minimum-f, interpolation (a = 0). Same slope/plateau rules
as Corollary 4.2 with a reduced transition scale and lower absolute E, levels compared to 0=0.1.
o- 0.0, a- 0.00102. (1-0.001, x-1 (8te)=1.100
+ Genoratzetion erce 1,3
0.0664,
(66))=1.50)
o-
L.11nomm
0.220, In-0.001, s (kk)=1.90) Cenerstizatin enor
++ L335 nom
-+ 1.1 nom
Goneraitaton oreer
2.00
Generakzzatoon ermor
J horm
0.05
0.50
8.00
2020
4000
1000 2303 3000 4030 [samples)
5000
2000
4050
n. (samples)
(sampless
(a) a = 0.00102, Ir 0.001, 8=1 (b) a = 0.0664, 1== 0.001, 8=1
(c) a = 0.229, 0.001, 8=1
Figure S4: Single spike w = e1; DLN (o = 0). With a calibrated to Pet (a), the regime structure mirrors the explicit p case: smaller Par exhibits carlier spike dominance and plateaus for >> 2(p-1); larger Peft stays bulk-dominated longer.
panel reports generalization error (left axis) and several eo-nssms of the same interpolating w (right axis); gray dashed curves are the bulk/spike theory overlays used earlier.
Comparison to s=50. Across all three p values, the larger-s experiments reprise the main text regime structure at larger sample sizes. For p == 1, lengthening the bulk-dominated segment makes the initial increase in generalization error clearly visible (especially at s=5000), after which the curve turns downward as aligoment improves. For E {1.5, 1.9), the same right-shift occurs yet the curves remain monotone; the rounder objectives keep the estimator from over-relying on noisy directions early on. In every panel, the blue f111 curve remains useful "regime meter": rapid growth signals bulk influence, and gradual approach toward the spike guide signals improving alignment-even though none of the f, curves truly flatten within our plotted range.
Small p (here p=1.1). Relative to the s=50 panels in the main text, both larger-s slices preserve the same two-phase story but the handoff happens later in n. At s=500 (Fig. S13p), generalization error is flat-to-slightly higher at small while |/w]] rises rapidly; as grows, generalization error begins to fall and the blue curve bends toward (but, in our range, does not meet) the spike overlay. At ==5000 (Fig. [S14)). the shape is unmistakable: generalization error first increases to a visible peak at intermediate n and then drops. The f1.1 curve keeps climbing throughout the displayed range, tracking the bulk-dominated guide before gradually approaching the spike prediction (without flattening). This "up-then-down" with more samples matches the double-descent picture for interpolating estimators-early fits lean on high-variance bulk directions; only later does the solution align with signal-wel documented in linear and deep settings (Belkin et al. 2019 Nakkiran et al. 2020b Hastie al. 2022
Larger p (here p=1.5 and 1=9. Compared to s-50, the curves again shift rightward in n, but the qualitative picture is unchanged: generalization error decreases monotonically over the whole range for both sparsities (Figs. S13p-c and S14p-c). The minimized fp-norms (red for p=1.5, green for p=1.9) drift only slightly upward rather than plateauing, while the auxiliary f1.1 diagnostic continues its steady growth along the bulk guide. The absence of an initial increase in generalization error is consistent with the rounder geometry of larger-p balls: the interpolating solution spreads
36

---

Published as a conference paper at ICLR 2026
dh 0.0. 0,0064, 0r-0,001, 5 50 (8o)-1.50)
T-0,001,
(8(a)=1.90)
Genera lacion erar
Conrratgetinn arter
1.11 nom
Caneraitaatton aror
1.00
Ourerallationn
moor
1.5 norm
2.85
nume
2000
4026
2300
4003
(samples)
(tmppccs)
(ssamplles
(a) a 0.00102, Ir 0.001, (b) a = 0.0664, = 0.001 s=50 (c) a 0.229, Ir 0.001, 8550 S
50
Figure S5: Flat w* (s 50); DLN (a 0). The elbow shifts with support size in the flat-support corollary; plateaus for r > 2(p-1) occur earlier and at lower levels than at a=0.1, while bulk-side 1/2 growth persists where predicted.
0.5.5 111
O= (.5.p=a. p:15
0.5.5, p
Genemaicattin
(oul-ddmm) (apice dom.)
Generaa on
(bulk-dom))
Gencralzation amer
C11 norm
1.1.
7.1 narm
theary te-dem.
2_1.1 nom
theory (buitdom.)
Commriiiiation error
Generallattion erral
7.1. nam
1.25
4.25
nomm
5 2 horm
5.0
Generaliee
0.75
2.15
0.50
1.10
2000
[Saamples)
[samples)
[samples)
(a) p 1.1 (Sparsity-leannng)
(b)
1.5
(c) p 1.9 (dense-lcaning)
Figure S6: Single spike w* e1; explicit minimum- interpolation (a 0.5). Larger
increases both n, and plateau heights relative 0=0.1. Bulk-dominated panels retain the n trend; r > 2(p-1) traces flatten only after the later transition, in line with equation equuaioon 10
weight more evenly and avoids the brittle, variance-heavy fits that create the small-p bump, echoing analyses of benign overfiting/ridgeless least squares and
shrinkage of descent
cones (Bartlett et al., 2020 Hastie et al. 2022 Chandrasekaran al. 2012 Amelunxen et al. 2014).
F EXTENDING THE L,-SCALING THEOREM TO DIAGONAL LINEAR NETWORKS
This section blueprint for porting our main f,-scaling theorem from the minimum-f,, interpolator to predictors selected by training diagonal linear networks (DLNs) with arbitrary depth. The goal reuse the entire spike+bulk argument with minimal surgery by swapping in the right implicit regularizer and the right one-dimensional balance. The guidance below covers both the two-layer case and the general depth- D case, aligning with the characterization of implicit bias in DLNs proved by Woodworth et al. (220))
In our min Lp analysis, the predictor among all interpolators is selected by separable power potential, and the proof runs through dual "link" that maps the ray variable back to primal coordinates. DI.Ns fit exactly the same template:
For two layers, the implicit regularizer is the hypentropy-type separable potential, and the link is the corresponding odd, strictly increasing map (Woodworth al., Thm. 1). Non-uniform initialization simply reweights coordinates multiplicatively throughout.
For depth D 2 3, the implicit regularizer is again separable but with a depth-dependent link; Woodworth al. (Thm. 3) identify the unique depth-L) 1) link and its inverse. Practically, you can treat it as "the D-link" playing the role occupied by the power map in min Ep and by the hypentropy link D
No other structural change is needed: once the link fixed, every step of our proof goes through with the same spike/bulk decomposition and the same ray reduction.
37

---

Published as a conference paper at ICLR 2026
sm
0.5, 1.5, 5 50
d= 0.5, 199.5-50
Generalitatin
Goncralizatio
Generoitation
theory
Generniization enor
Ganeralltation emor
10
norm
12-
Generalization
J narm
2.
1.15
1030 2003 1900 4200 5000 (samples)
1900 2600 3305 6030 5000
2000 [Samples)
4080
(sampless
(a) p = 1.1 (sparsity-lcaning
(b) p = 1.5
(c) p = 1.9 (dense-lcaning)
Figure S7: Flat w" (s = 50); explicit minimum-lp interpolation (o = 0.5). The same slope/plateau
rules apply, but both the clbow and plateau heights shift upward with via TN and equation
a- 0.00102, In-0.1, s-1 (8te)=1.10) + Genera emrrr -t- (111 norm
0664,, D00,
0.229, 0=00,001, (8k))1.90)
Geseralzaiinn mmmm
Cameratzation entr
1.35 nom
2.11 noem
+ L19 mm
Generalizaiin error
Generalization eroo ento
2.25
26
E norm
f nomm
5.0
202
0.7
2.15
1000 2000 3030 4000
5000
2:000
400n
2000
4000
(samples)
Esaapless
(samples)
(a) 0.00102, 0.1, ==1
(b) < 0.0664, 0.001, s=1
(c) a 0.229, 0.001, s=1
Figure S8: Single spike wt e1; DLN 0.5). After calibrating a Poff. bulk growth persists to larger n (larger n2,), and spike-side plateaus for r > 2(p-1) emerge later and at higher levels.
As in the min e proof, restrict the dual variable to the ray spanned by the labels and determine a single scale t from strictly monotone one-dimensional balance. Conceptually:
In the kernel-like window (small arguments of the link on both spike and bulk), the link linearizes and the entire analysis collapses to the = 2 case verbatim. This is the "lazy" regime.
In the rich-like window (arguments large on the bulk and/or dominant spike), the nonlin- earity of the link controls the transition. For two layers, the balance yields Lambert-W controlled scale: for 3, the depth- link gives faster, polynomial-in iimmiallii-niitiaizzatoo tran- sition. You do not need closed form just the monotonicity and the small/large-arguml argument asymptotics.
Bulk block. Replace the power moment used in the min Ep bulk bound by the depth-appropriate scalar functional that averages the link across standard Gaussian coordinate. Operationally:
Define a bulk scalar by applying the DLN link at the ray scale to single Gaussian coordinate and taking its &, moment (to the 1/r). This plays the exact role of 1/t in the min lp proof.
Use the same Gaussian embedding for the bulk design to lift this scalar to the full bulk contribution. In the kernel-like window you recover the p = 2 scaling exactly; in the rich- like window you get the accelerated depth- growth predicted by the link's large-argument behavior.
Keep track of the global scaling coming from the link's overall prefactor (this carries the initialization scale); it multiplies both bulk and
terms.
Spike block. On the spike coordinates, keep the original two-part structure:
Spike-main: apply the link to the mean shift determined by the signal; a single coordinate dominates the -ie-dimensionll balance, the selected predictor saturates the spike scale and becomes essentially independent of the initialization (up to lower-order logarithmic or depth-dependent corrections).
38

---

Published as a conference paper at ICLR 2026
(n-00,,,_m011..e0m 16tel=1.101
g- 0.5. 0,0064 00-0,
5 50 (8o)-1.50)
(8(a)=1.90)
Gener Lacion erer
Gentratattonn
horr
Generalitaten
1.225
2:35
1,200
0.3
2.30
1000 1000 3503 4000 5020 (ssmples)
2205
4000
2300
4003
n @Samplest
110smmples
(a) & 0.00102, Ir = 0.1, 50
(b) a = 0.0664, 0.001 ==50
(c) a 0.229, 0.001, 5050
Figure S9: Flat w² (s 50); DLN (a 0.5). The o-driven increase in т, shifts n, to larger n:
otherwise the bulk vs. spike regime behavior matches the theory and the explicit p experiments.
o- 0.0. 0.0.0102 1n-0.1, s-1 (At//=110)
Genera fastinnne norl
(flal=1.10) (.0
d- 0.0, 0.00102, 1=00.001, 5=1 (éta)=1.10) Genundiattencnee
C1.ino
Gemerattal Coneraltaation enerr
nom
Conneralizetion
05
0.5
2000
4000
2002
4003
002
n (saamlles)
(samples)
Isamplest
(a) = 0. a 0.00102, Ir 0.1, (b) σ = 0, a = 0.00102, 0.01, s=1
(c) a = 0. a 0.00102, =
s=1
0.001, s=1
Figure S10: w" ej (sparsity 8=1), clean labels. f..1 rapidly plateaus and is insensitive to learning rate, consistent with low-Pett implicit bias small
Spike-remainder control the residual Gaussian fluctuation by the same operator-norm and concentration events as in the min l, proof; its l, size is the bulk scalar (at the ray scale) times maax 1/2), again multiplied by the link's global prefactor.
When spikes are meek relative to the bulk (no dominant coordinate), the spike block lincarizes and you are back in the laws.
Unified bound. After these replacements, the final display has the identical three-term structure:
DLN predictor's e, size maximum of (spike-main, bulk, spike-remainder),
with each term obtained from the min Ep counterpart by: (i) replacing the power link with the DLN link; (ii) inserting the link's 's' global prefactor; and (iii) using the DLN bulk scalar in place of the power moment. In the kernel-like window this reproduces the p 2 version exactly; in the rich-like window you get either bulk-controlled growth (Lambert-W for two layers; depth-accelerated for D2 3) or spike saturation.
Depth and initialization intricacy.
Depth D 2 The depth- link odd, strictly increasing, and has a simple linearization at the origin and an explicit rational form away from it (Woodworth et al., Thm. 3). This yields the same kernel-like reduction and sharper rich-like transition than at D 2. You never need its closed form form-only its monotonicity and asymptotics.
Non-uniform initialization. The per-coordinate shape the initialization simply reweights the separable potential and carries multiplicatively through the link. Every bound inherits these weights purely multiplicative way (Woodworth al., Thm. 1).
Limits. Large initialization recovers the minimum-é, norm predictor; vanishing initialization recovers the minimum-&1 predictor (with the usual caveats how small "small" must be). These are the DLN analogues of the kernel and rich limits and hold for all depths covered above.
39

---

Published as a conference paper at ICLR 2026
0.1. a.1..am...0.. am0.00102 Imm0.1, sm1 (81a)=1.10)
-+ Gonora restiiniiiiinnn ener -t- norinors
0.00000
(810)=1.10)
om pm1.1.05 2m 0.00102, 1-0.001,.-11 (6fa)=1.10)
- Genemi sabon -I- 23.3
Geterra Gentrralraato eroor
ener
95
2000
4000
2:00
4030
0.00
2502
4032
n (samples)
Isamplesl
n Isamplest
(a) 0 0.1, a = 0.00102, Ir
(b) a 0.1, a 0.00102, (c) 0.1, a 0.00102, 0.01, s=1
0.1,
0.001, 8=1
Figure S11: w² = e₁ (sparsity s=1), moderate noise. Larger learning rates produce a steadily rising l11 and shift the elbow to larger decreasing ir suppresses the rise and restores near- near-plateau.
Geneee lsetion wrrrr +
0.00102, 1.,0.1, s=1 (8.1)=1.10)
2111 norm
0.5, 0.000102, a-0.02,511 (die)=1100)
Generalzation 133
ou 0.5, au 0.00162, 001,0001, s=1 (6te)=1.88 .1)
-+- Genent aetioltttiiioo emo -+ (10 nom
Gemeralization enror
Ganeralia Ganeraltation ertor
Genera tasten
1.0
1.0
0.5
1000 2006 3050 4000
5000
1500 2000 3350 4000
5050
2000
4002
(samples]
(samples)
Isamplesi
(a) 0.5, 0.00102, Ir
(b) 0.5, 0.00102, lr
(c) 0.5, a
0.00102, Ir
0.1,
8=1
0.01, s=1
0.001, s=1
Figure S12: w" e1 (sparsity s=1), heavy noise. The leamiig-rate-induued inerease in f.1 is strongest at high noise: 1r=0.1 (and to a lesser extent 0.01) yields monotone growth with n, whereas 1r=0.001 shows transient bump and then relaxes toward plateau evidence that the elbow shifts right under larger Ir.
A handy dictionary for porting the proof. To translate any display lemma from the min &p analysis to DLNs, we can make the following substitutions:
replace the power map by the depth-appropriate link (hypentropy at two layers; the depth-D link from Woodworth et al. otherwise), including its global prefactor.
Power link - DLN link:
2. Ray scale + DIN balance: keep the same one-dimensi strictly monotone balance along the label ray; solve it numerically or via asymptotics (linear in the kernel-like window; Lambert-W at two layers and power-law at depth 2 the rich-like window).
3. Bulk scalar: replace the power moment by the &, moment of the DLN link applied to a single Gaussian coordinate at the ray scale; lift via the Gaussian embedding exactly as before.
4. Spike block: reuse the deterministiccpla--Gassss decomposition, the operator-norm and concentration events, and the same l, geometry, only the link and its global prefactor change.
With the substitutions above, the E,-scaling analysis for the minimum-f, interpolator transfers directly to DLNs of any depth. The proof structure, the spike/bulk decomposition, and the final three-term form remain identical; only the link and its scalar balance change. Two layers inherit Lambert-W bulk scale; deeper networks transition faster with initialization due to their depth-D link. In the kernel-like window, everything collapses to the p bounds almost word-for-word.
40

---

Published as a conference paper at ICLR 2026
d= s=
Genennticetion moce 1.1.1 nom
thecey (Du bul-dom))
Goncratzation
thecry (apice dom.)
norm
theory (bulk-dom)
rom
theory (bulk-dom))
Cetrrnitaaiionnoom ereer
J nOm
Generaliation
nOTT
Gentraliation
1 form
8.8
0.00
2050
4000
2000
4000
(samples)
(samples)
[Iaamples)
(a) p=1.1, 85500
(b) p=1.5, 8=500
(c) p=1.9
8=500
Figure S13: Large sparsity, ==500. Black- generaaenelliizationn emor; colored--,, norms of the same
interpolator (blue: &1.1 red: l.s. green: {19); gray dashed-bulk/spiie overlays.
(m p 1 5000
5000 L1.5
dencralizaion erer
thesry (bu k-dom)
C1.
1.3.1 noms
thesry (aslno-dom))
nom
theory (bulk-oom)
thcory (bulisdom))
Caneralipatoon area
1,3.5
1.075
norm
Generaltaation
0 L norm
1.005
2000
4000
2.99
2003
4000
2000
4000
n (samples)
(sampless)
[1samples)
(a) p= 1.1,
S 5000
(b) p-1.5 =55000
(c) 1.9, S 50000
Figure $14: Even larger sparsity, $55000. Same conventions as Fig. S13 Increasing s shifts the bulk crossover to larger n.
41


Reviews:
[{"review_id": "o5UvG2Txdt", "reviewer": "Reviewer_cAzm", "summary": "This paper presents a unified theoretical and empirical study of how the parameter norms of overparameterized linear regression models scale with data under an ℓp-biased interpolation. Specifically, for minimum-ℓp interpolators with $p \\in (1,2]$, the authors derive closed-form, high-probability laws describing how the family of norms $\\{\\lVert \\widehat{w}_p \\rVert_r\\}$ for $r\\in[1,p]$ changes with sample size $n$. A central insight is the identification of a data-dependent transition point n^\\star (the “elbow”) separating bulk- and spike-dominated regimes, and a universal threshold $r^* = 2(p-1)$ that determines which norms plateau and which continue to grow. Using “dual-ray” analysis, the paper provides explicit expressions for these quantities and verifies the predictions empirically across both explicit $\\ell_p$ regression and diagonal linear networks (DLNs) trained by gradient descent. By calibrating the DLN initialization scale $\\alpha$ to an effective $p{\\text{eff}}(\\alpha)$, the authors demonstrate that DLNs inherit the same scaling laws, revealing a consistent connection between explicit and implicit bias. Overall, the work offers a clear, closed-form picture of how different norm measures behave with data growth and highlights that the predictive power of norm-based generalization proxies crucially depends on which ℓr norm is chosen.", "questions": "Can authors comment on the Gaussian design choice? Do they expect Gaussian universality, or can the results significantly change under other distributions, and how under other natural distributions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "See summary. The paper has a very clear and concrete contribution. The paper’s main strength lies in its clear and unified theoretical characterization of how all $\\ell_r$ norms scale under $\\ell_p$-biased interpolation. It successfully bridges explicit and implicit bias through an effective p_{\\text{eff}}(\\alpha) mapping in diagonal linear networks, and its empirical results convincingly validate the theoretical predictions across multiple regimes, offering practically valuable insights into the sensitivity of norm-based generalization measures.", "weaknesses": "I do not seen any major weaknesses of the study. The paper addresses a clear question. One minor weakness is that the connection to generalization is still less clear to me. It would be good to expand on this, and point out other work that also share the same weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fj4064vnjl", "reviewer": "Reviewer_MDQs", "summary": "This work studies the ell_r norm of the minimum ell_p norm interpolator for regression over gaussian data in the overparametrized setting where number of data n >> number of dimensions d. the setting assumes a sparse ground truth w^star.\n\nIn particular, they focus on how the ell_r norm grow when n grows when d/n has a limit kappa > 1.\n\n\nThe main finding (theorem 3.1) is that for r <= 2 ( p - 1 ), the ell_r norm of the min ell_p norm interpolator grows, while r > 2 (p -1), the same quantity plateaus.\n\nAnother finding is a \"phase transition\" from spiky-to-bulk dependency in terms of the number of samples (denoted n_star).", "questions": "While i understand the spiky-vs-bulk part of X^TY, it's not clear to me how why the RHS of eqn (4) is said to be \"spike-dominated\". By contrast, in eqn (5), kappa_bulk clearly shows up on the RHS, so that makes sense.\n\nLine 303 says middle panel exhibits a clear elbow near the predicted n_star. but it's not clear to me where is n_star on the middle panel plot.\n\nThe definition of t_star in eqn (1) is confusing. since it is a \"definition\", why is there a \"w.h.p\" at the end?\n\nMinor:\n\nLine 229 equation (8) links to the appendix when it could link to eqn (3) instead. Same issue in the caption of figure 1.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theory is well-supported as seen in Figure 1. The provable behavior from the linear model seems to also carry over empirically to diagonal linear networks.", "weaknesses": "Can the authors give a more explicit reference to previous work that uses ell_r for bounding generalization error where r is not in {1,2}? I'm not convinced whether ell_r for r not in {1,2}. A more specific reference to a paper + lemma number would be helpful.\n\nThe legend of figure 1 looks to suggest that there should be error bars/confidence bands, but it's not visible in the plot. Is the result replicated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2PrzVGkIl", "reviewer": "Reviewer_4hrx", "summary": "The paper proves a closed form for the $\\ell_r$ norm of the min $\\ell_p$-norm interpolator under certain standard assumptions. Through experiments, the paper demonstrates that the trends predicted by the theory for linear regression, carry over to a slightly broader class of predictors.", "questions": "- Is there an explanation for the non-monotonical trend of the $\\ell_{1.1}$ norm in figure 1b?\n- What is $d$ for figures 1-2? does varying $\\kappa=d/n$ change the trends in the figures? also would it be possible to indicate for each figure the predicted transition scale $\\n^\\star$?\n- In [1] the authors try to extend the observations from Donhauser et al, 2022 to inductive biases present in non-linear models. Would a similar extension be applicable to the results of this paper?\n\n[1] - https://arxiv.org/pdf/2301.07605", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Very well-motivated, and clearly written.\n- The mathematical derivations are sound and the application of the proof technique is, to my knowledge, novel in the context of analyzing min-norm interpolators.", "weaknesses": "- Theorem 3.1 is quite dense and could perhaps be split into the closed form result in eq 2, and the result characterizing the different sample size regimes\n- It is not particularly easy to grasp what the main terms that determine $||\\hat{w}_p||_r$ are. Would it be possible to give an interpretation of the quantities in equation 2?\n- To further demonstrate the significance of the result, it would be nice to have a concrete example (perhaps supported by experiments) where one can take advantage of the relationship between $r$, $p$, $n$ and $d$ proved in Theorem 3.1. \n\n**Minor remarks**\n- It would help carry the message across more easily if one could visualize the motivation in lines 55-66 with the help of a teaser plot (perhaps a condensed version of figures 1-2?).\n- Some of the markers are not visible in the figures e.g. dotted line in fig 1a.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUa4U9LTwN", "reviewer": "Reviewer_ksga", "summary": "This paper aims to characterize one interesting term in overparameterized linear (models) regression: the behavior of $p$-norm minimum interpolator $||\\hat{w}_{p}||_r$ against the number of data for different $r$ given $p\\in(1, 2]$. To achieve this, the authors propose a novel \"dual-ray\" analysis to derive the corresponding scaling laws. Then they identify a data-dependent transition point and a universal threshold. They conduct some numerical experiments to verify their theoretical claims.", "questions": "1. Why the authors do not include $p = 1$ (and other values of $p$) in the analysis?\n\n2. How can the authors make their actionable guidance more reliable?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "I think the most significant strength of this paper is the new \"dual-ray\" proof technique. I find it direct and intuitive, providing a better approach for the problem studied in this paper than approaches like GMT that might be too complex to be applied.\n\nFurthermore, the derived $r$-norm of $p$-norm solution $ || \\hat{w}_p ||_r $ as a function of several meaningful constants is also interesting. This form can somewhat support the authors' motivation \"using norm-based bounds or proxies should be cautious\".", "weaknesses": "However, I still have several concerns, and I will list them below.\n\n1. I find the primary objective of this paper a bit unconvincing: studying the $r$-norm of the $p$-norm interpolator solution. There lacks a compelling motivation for this focus. Because it seems that it is the $p$-norm of the $p$-norm interpolator solution that matters, beyond the cautioning that generalization bounds may depend on the choice of $r$. This makes the scope of this paper a bit narrow. \n\n2. The theoretical analysis heavily depends on the isotropic Gaussian features and $p \\in (1, 2]$. This does not explore the validity of the findings that violates this assumption, e.g., perhaps the most direct case $p = 1$.\n\n3. Theorem 3.1 is based on a general $w_{\\star}$ (neither single spike nor flat), but the corresponding scaling laws are only studied in two very extreme case (single spike and flat) to discuss the corresponding laws. I understand that this might be due to the clarity reason, but a thorough discussion for how the scaling laws behave for arbitrary $w_{\\star}$ should also be crucial. This also leads to the next weakness.\n\n4. The actionable guidance (line 463 - 467), together with the implications for practice (line 81-85), is a significant overclaim. The theoretical results are built upon linear regression, and there even lacks discussion (both theoretical and empirical) for general $w_{\\star}$ in this task, then I think claiming the corresponding actionable guidance for general settings is very misleading. \n\nMinor: I think this paper should be reviewed in a venue for statistical learning theory or statistics, rather than ICLR (the connection with diagonal linear networks seems unnecessary because the authors directly convert the diagonal linear network to a minimizer of certain norm, which almost has nothing to do with the model itself).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)