Generalization Below the Edge of Stability: The Role of Data Geometry
Authors: Tongtong Liang
Alex Cloninger
Rahul Parhi
Yu-Xiang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: neural networks
deep learning theory
gradient descent
representation learning
generalization
Primary area: learning theory

Abstract: Understanding generalization in overparameterized neural networks hinges on the interplay between the data geometry, neural architecture, and training dynamics. In this paper, we theoretically explore how data geometry controls this implicit bias. This paper presents theoretical results for overparametrized two-layer ReLU networks trained *below the edge of stability*. First, for data distributions supported on a mixture of low-dimensional balls, we derive generalization bounds that provably adapt to the intrinsic dimension. Second, for a family of isotropic distributions that vary in how strongly probability mass concentrates toward the unit sphere, we derive a spectrum of bounds showing that rates deteriorate as the mass concentrates toward the sphere. These results instantiate a unifying principle: When the data is harder to “shatter” with respect to the activation thresholds of the ReLU neurons, gradient descent tends to learn representations that capture shared patterns and thus finds solutions that generalize well. On the other hand, for data that is easily shattered (e.g., data supported on the sphere) gradient descent favors memorization. Our theoretical results consolidate disparate empirical findings that have appeared in the literature.

Published as a conference paper at ICLR 2026
GENERALIZATION BELOW THE EDGE OF STABILITY: THE ROLE OF DATA GEOMETRY
Tongtong Liang & Alexander Clonginer
Rahul Parhi & Yu-Xiang Wang
UC San Diego (ttliang, acloninger)@ucsde edu
UC San Diego {rahul, (ordangw)@uccs.co edu
ABSTRACT
Understanding generalization in overparameterizet neural networks hinges the interplay between the data geometry, neural architecture, and training dynamics. In this paper, we theoretically explore how data geometry controls this implicit bias. This paper presents theoretical results for overparametrized two-layer ReLU networks trained below the edge of stability. First, for data distributions sup- ported on a mixture of low-dimensional balls, we derive generalization bounds that provably adapt the intrinsic dimension. Second, for family of isotropic distributions that vary in how strongly probability mass concentrates toward the unit sphere, we derive spectrum of bounds showing that rates deteriorate the mass concentrates toward the sphere. These results instantiate unifying princi- ple: When the data harder to "shatter" with respect to the activation thresholds of the ReLU neurons, gradient descent tends to learn representations that capture shared patterns and thus finds solutions that generalize well. On the other hand, for data that is easily shattered (e.g., data supported on the sphere) gradient de- scent favors memorization. Our theoretical results consolidate disparate empirical findings that have appeared in the literature.
INTRODUCTION
How does gradient descent (GD) discover well- -ele-generaliiing representations in overparameterized neural networks, when these models possess more than enough capacity to simply memorize the training data? Conventional wisdom in statistical learning attributes this to explicit capacity control via regularization such as weight decay. However, this view has been profoundly challenged by empirical findings that neural networks generalize remarkably even without explicit regularizers, yet can also fit randomly labeled data with ease, even with strong regularization (Zhang et al., 2017).
This paradox forces critical re-evaluation of how we should characterize the effective capacity of neural networks, which appears to be implicitly constrained by the optimizer's preferences (Zhang et al., 2017; Arpit et al., 2017). A powerful lens for examining this implicit regularization is to inspect the properties of solutions to which GD can stably converge, since these stable points are the only solutions that the training dynamics can practically reach and maintain. This direction is strongly motivated by the empirical discovery of the "Edge of Stability" (EoS) regime, where GD with large learning rates operates
critical regime where the step size is balanced by the local loss curvature (Cohen et al., 2020). This observation is further supported by theoretical analyses of GD's dynamical stability (Wu et al., 2018; Nar and Sastry, 2018; Mulayoff al., 2021; Nacson et al., 2023; Damian et al., 2024), confirming that the curvature constraint imposed by stability provides a tractable proxy for this implicit regularization.
While the EoS regime offers a valuable proxy, fundamental question remains: how precisely does this stability-inuuced regularization lead to generalization? Recent breakthroughs have established that for two-layer ReLU networks, this implicit regularization acts like a data-dependent penalty on the network's complexity. Technically, this is captured by a weighted path norm, where the weight function is determined by the training dataset itself (Liang et al., 2025; Qiao et al., 2024; Nacson et al., 2023; Mulayoff et al., 2021). This resulting data dependent regularity provides an ideal theoretical microcosm probe how data geometry governs effective capacity (Arpit et al., 2017). For example, for the uniform distribution on a ball, implies generalization but also a curse of dimensionality (Liang et al., 2025). However, this prediction of curse is odds with the empirical

---

Published as a conference paper at ICLR 2026
success of deep learning. This contradiction forces the question: how can we predict which data geometries will generalize well under implicit regularization, and which will not?
Contributions. We argue that the effectiivneess of this data-dependent regularity is governed by a single geometric quantity, which we call data shatterability: qualitatively, how easily the data distribution be partitioned into many disjoint small regions by ReLU half-spaces. Informally, the less shatterable the data geometry, the stronger the implicit regularization at the EoS. We make this principle precise and obtain the following results.
A Spectrum of Generalization on Isotropic Data. For a one-parametee family of isotropic Beta(a))-aadee distributions, we derive generalization upper bounds and matching lower bounds that depend smoothly on the radial concentration parameter a (Theorems 3.4 and 3.5). As decreases, the probability mass moves towards the boundary and the gen- eralization guarantee degrades. In the limiting case where the support collapses to the unit sphere, we construct perfectly interpolating networks that still satisfy the BEoS stability condition (Theorem 3.6). In particular, the "neural shattering" phenomenon, identified by Liang et al. (2025) for the uniform ball distribution, represents one special point of the broader generalization spectrum we uncover.
Provable Adaptation to Low-dimensionality Under a mixture-of-ssuspaeee assumption where inputs are supported on a union of m-dimensional balls in IRd with m < d, we prove that depends on
that all -stalee solutions enjoy a generalization rate the intrinsic dimension m rather than the ambient dimension d (Theorem 3.10). The depen- dence on the number of mixture components is at most polynomial. Synthetic experiments confirm that gradient descent indeed behaves according to this intrinsie-dimension law.
Taken together, these results identify data shatterability as the key geometric quantity that controls the strength of implicit regularization for gradient descent below the edge of stability.
Technical novelty. Many classical uniform-convergee vonvergence-basee generalization bounds for overpa- rameterized neural networks are Ietriiuutionaagnosii one first controls the _ metric entropy of a function class, then controls the Rademacher complexity (or derive uniform concentration inequal- ity via a union bound). This approach is not available in our setting. The data-dependent regularity induced by the EoS condition defines a function class whose L metric entropy is infinite, and our spherical interpolation result shows that nontrivial distribution-agnosiic bounds cannot hold. The key observation is that stability-inducee regularity is highly inhomogeneous over the input domain: there are regions where the effective regularization is strong ("good" regions) and regions where it is extremely weak ("bad" regions) such that metric entropy explodes.
Our main technical innovation is to bypass global metric entropy control via a half-space-deree quantile partition of the input space. This technique allows to decouple the analysis based on the strength of regularization enforce strict complexity control, while in the "bad region", where complexity explodes, we abandon function-space covering arguments. Instead, we control the generalization error by bounding the probability mass of these regions, tying the error directly to the data geometry.
In the "good region", the implicit regularization is effective, allowing us to
Conceptually, this framework inverts the classical VC dimension perspective. While VC dimension characterizes a model's active capacity to shatter arbitrary data, our "data shatterability" principle characterizes the feasibility of specific dataset being shattered by the GD-trained network.
Related work. We build upon recent line of work (Wu and Su, 2023: Qiao et al., 2024; Liang et al., 2025) that theoretically study the generalization of neural networks in Edge-of-SStabllig regime (Co- hen et al., 2020) from a function space perspective (Mulayoff al., 2021; Nacson et al., 2023). We add to this literature by abstracting neural-shattering analysis on uniform-ball of Liang et al. (2025) satttrhhtter framework and extending to a broader class of distribu- tions that capture both radial concentration and low-dimensional structure, yielding distribution- dependent upper and lower bounds that make the role of data geometry explicit.
into a depth-based
More broadly. our work is inspired by the seminal work of Zhang al. (2017) on "rethinking generalization". Our results provide new theoretical justification that rigorously explains several curious phenomena (such as why real data are harder to overfit than random Gaussian data) reported therein. Compared to other existing work inspired by Zhang et al. (2017), e.g., those that study the implicit bias of gradient descent from various alternative angles (dynamics (Arora et al., 2019; Mei
2

---

Published as a conference paper at ICLR 2026
et al,, 2019; Jin and Montúfar, 2023), algorithmic stability (Hardt et al., 2016), large-margin (Soudry et al., 2018), benign overfitting (Joshi et al., 2024; Kornowski et al., 2024)), our work has more end- to-end generalization bounds and requires (morally, since the settings are not all compatible) weaker assumptions. On the practical front, we provide new theoretical insight into how "mix-up" data augmentation (Zhang al., 2018; 2021) and "activation-base" pruning" (Hu et al., 2016; Ganguli and Chong, 2024) work. A more detailed discussion of the related work and the implications of our results can be found in Appendix B.1.
Scope of analysis. Our theoretical framework is situated in the feature-lcarning (or "rich") regime of overparameterized networks. Rather than tracking the full gradient dynamics we focus on the regime in which gradient descent with a large step size operates for long stretches below around the edge-of-stabilics boundary. We therefore analyze the generalization behavior of all parameter vec- Soiw---de-of--taaaiiiee
(BEoS, see Definition 2.1), without assuming optimality or stationarity. Our bounds hold uniformly over this BEoS region and characterize a baseline form of implicit regularization that is enforced whenever training remains near the edge of stability.
tors satisfying
2 PRELIMINARIES AND NOTATIONS
Neural network, data, and loss. We consider two-layer ReLU networks
So( Σ ok bww bu) + B3, x(z) = max 03,
(1)
with parameters 0 = {(r, wk, b U {3) E R((+2)K+1 Let be the parameter set of such for arbitrary K N. We also assume wt # 0 for all & in this form, otherwise we may absorb it into
the output bias 3. Given data D = {(x;; == with æ; in a bounded domain Ω C Ied with d >> 1,
the training loss is C())
1 Σi= (f. - y:)² We assume ||> S R and y S D for all i. 2n
"Edge of Stability" regime. Empirical and theoretical research (Cohen et al., 2020; Damian et al., 2024) has established the critical role of the linear stability threshold in the dynamics of gradient de- scent. In GD's 's trajectory, there is an initial phase of "progressive sharpening" where Amax \ ((),( increases. This continues until the GD process approaches the "Edge of Stability", a state where Amax (V² ~ 2/11, where vy is the learning rate. In this paper, all the GD refers to vanilla GD) with learning rate v).
Definition 2.1 (Below Edge of Stability (Qiao al., 2024, Definition 2.3)). We define the trajectory of parameters (0t) generated by gradient descent with a learning rate as Below-Edge-od Stability (BBoS) if there exists a time 0 such that for all 1 2 1", Xmaxx (())( 3.- Any parameter state O, with t 2 t' thereby referred to as a BEoS solution.
This condition applies to any wvice-iifferentiabee solution found by GD, even when the optimization process does not converge to a local or global minimum. Moreover, BEoS is empirically verified to hold during both the "progressive sharpening" phase and the subsequent oscillatory phase.
Our work aims to analyze the generalization properties of any solutions that satisfy the BEoS con- dition (Definition 1). The set of solutions defined as:
+ BEOS (n, D) = AmaA V-L(()) <
(2)
Data-dependent weighted path norm. Given weight function g :dd-1 x K. - R, where Sd-1 (u Rdd || = =1). the g-weighted path norm of neural network fo( Ek= vkd(w]
by) + 3 is defined to be
by
Ilfe g
Iuall g [|
|| l12
The link between the EoS regime and weighted path norm constraint is presented in the following data-dependent weight function (Liang et al., 2025; Nacson et al., 2023; Mulayoff et al., 2021). Fix
Neverthelesss, there is informal and heuristic discussion from the perspective of gradient dynamics in Ap- pendix B.2.
3

---

Published as a conference paper at ICLR 2026
dataset D 9e) C R R. we consider a weight function gr - R - R defined by go(u,t)
u, t) == min u. t). go( (pp(_u, u. -)); where
ID ( u. () == P(X u 1) E x - x 4
V |
X X > f]]
(4)
Here, X is a random vector drawn uniformly at random from the training examples {a;}2_. Specif- ically, we may also consider its population level øp by viewing X as random variable.
Informally, @D (u, I) measures how hard for GD to place a ReLU ridge with normalized and threshold / under the BEoS constraint. Large frD corresponds to directions that sense that sense significant data activation, thereby yielding strong gradients and hence strong implicit regularization. A more detailed interpretation from the perspective of gradient dynamics is given in Appendix B.2.
Proposition 2.2. For any 0 E BESS (17, D), fl pach. -
(R + 2C((()
The proof of this proposition refers to (Liang et al., 2025, Corollary 3.3). The non-parametric version of the weighted path norm constrain can be found (Liang et al., 2025; Nacson et al., 2023).
Supervised statistical learning and generalization gap. We consider supervised learning prob- lem where i..dd samples D = {( y:) are drawn from an unknown distribution P. In this paper, we assume the feature space is a compact subset of Euclidean space, S2 C R the label space is R. and the data is supported on f2 x [-D, DJ. We use the squared loss, de- fined as {(f, x; y) = 2 (f(x)) -) y) The performance of a predictor f is measured by its pop- ulation risk Rp(s)
E(x,))~p YY), while we optimize the empirical risk RD(f) 2ff. The difference between these two quantities the generalization gap Capp(/:D) D) = [ (f), Our work focuses on the hypothesis classes the BEoS class θ BEOS (11, D) and the bounded weighted-path norm class θ (9; M, C),
(2; M,C) = (0 e θ II.a la /| S M, [I sc}.
(5)
where 9 can be the weight function IDD associated to the empirical distribution 'D or the weight function gp associated to the population distribution P, see Section E for more details.
DATA SHATTERABILITY PRINCIPLE AND GENERALIZATION BOUNDS
This section introduces notion of data shatterability and explains how it leads to unified proof strategy for both the generalization upper bounds (Theorem 3.10, Theorem 3,4) and the lower bounds (Theorem 3.5, Theorem 3.6).
We start with the classical notion of half-space depth (also known as Tukey depth).
Definition 3.1. Given a distribution Px on Rd. For any Rd its population half-space depth depth(æ, x Px) = infuega-: Px~Px (u (X-x) 2 0). On the other hand, given point cloud X {II,... In) C Bd that is also represented by the empirical distribution P(X) that assigns mass 1/n to each point, the empirical half-space depth depth(æ, XX) inf E 1 {uT(x. x) 20.
Half-space depth measures the centrality of a point by finding the minimum data mass (either population or empirical) on one side of any hyperplane passing through it (Tukey, 1975). For any T E [0, z we define 7-deep region lT set of the half-space depth function (we define l2(X') similarly).
{ar E R& depth(æ, Px) 2 T} as upper level
Now let X (æI,.... } be the point cloud of inputs in the dataset DD The key geometric observation that, for a ReLU neuron to create nonlincarity inside a T-deep region (x))), its activation boundary must intersect that region. Formally, let fe be network of the form (1) and
define Nr := {* [w]e by = 0} n lT( # 0 The neurons with k & Nr are either always active or always inactive on 2r hence contribute only an affine function on that region. c_ x: + b (which absorbs all neurons with k Nr) such that fo(x) = Σ "ko(w x by) + cTa x + b. V x E (x)).
Therefore there exists affine function
By the definition of half-space depth, every hyperplane passing through point in (x leaves at least a T-fraction of the data on each side. In particular, for any neuron whose boundary intersects
4

---

Published as a conference paper at ICLR 2026
(X),, its activation event has probability least T (population empirical, depending the context). This gives positive lower bound on the data-dependent weight function g. We denote 9D (T) Of @D(u. t) > O, 9p (T)
inf
inf O) 9P (u, t).
Thus, the neurons that generate nonlinearity on (2((X) are effectively regularized by the BEoS condition, with strength controlled from below by y3th (T) (and ggm (7) at the population level).
On the Tedeep region, only neurons in Nr contribute nonlinearity, and for these we have (Dl 1x) 2 (T). Hence their unweighted path norm satisfies ΣKEN, |ux| || s (o (1]) pppthhgpt so the restriction of so to (r(X)) lies in a standard (unweighted) path- norm ball with radius proportional to (T))-1 Generalization bounds for such unweighted path-norm classes on bounded inputs are known (Parhi and Nowak, 2023; Neyshabur et al., 2015) and yield the T'deep term in (6). Outside the T'deepp region, BEoS provides only weak control, so we bound the contribution by the worst-case 7 amplitude times the probability mass of the shallow region. Finally, since g and (2') are random, we replace them by their popula- tion counterparts gpin(() and 1r((xx and control the discrepancy using empirical process tools (Appendix E.2). This yields the following generic decomposition, holding with high probability
Gap(fo. D) I O & (px())))) +
Ô ooping (T) (T) x743
d+3
sup 0 @ (1.
shallow region
T-deop rogion
where Ô hides logarithmic factors and universal constants. This incquality is the main techni- cal bridge between data geometry and the behavior of BEoS solutions. Now we instantiate it for isotropic and anisotropic distributions to establish the corresponding generalization bounds.
3.1 SPECTRUM OF GENERALIZATION ON ISOTROPIC DISTRIBUTIONS
We now specialize to isotropic distributions with compact support and assume that the maximal support radius is 1. In this case, the depth of a point depends only on its radius. More precisely, consider distributions of the form X h(R) U, U Uniform(Sd
where h is radial profile and R is a scalar random variable. After rescaling, we may assume the support of Px is contained in Bd. Writing r ||æ||| and setting & --r, r, decompose By into (1) e-annulus || 2 1 - -c. (2) e-strict interior :== : B By A.G.
A = x E By
For isotropic Px, the weight function dp (u, t) depends only f, so we write gr(t). In this setting, gr(1 1
<) is a lower bound on the regularization strength for ReLUs whose activation boundaries For any fixede (0, 1), plugging this into (6) and rewriting in radial form,
lie at offsets &
we obtain
sup Gap(So. D) s (Ax + 0 - = e (e
záta n 4d+4
d+33 w.h.p.
):
We instantiate this decomposition on a flexible family of radial profiles that interpolate between a "thick" conter-concnntaation ball and thin spherical shell.
Definition 3.2 (Isotropic Beta-radial distributions). Let X be a d-dimensional random vector in Rdd
For any (0, oo), the isotropic Beta(a) (etada)-radial distribution is defined by
X = h(R)U ~ Px(a)),
where Uniform(( [0, 1), U Uniform(S and l((r) 1- (1-r) is a radial profile. Assumption 3.3. Fix a E oo). Let P(a) be a joint distribution over Rd x R whose marginal over x is Px.a). The corresponding labels y are generated from a conditional distribution P(y (z) and are bounded: |y| < D for some constant D > O.
This framework enables a generalization upper bound that depends explicitly on the parameter Theorem 3.4 (Spectrum of generalization on isotropic Beta-radial distributions). Fix a dataset D {()), where each (æ,, ve) is drawn i.i.d. from P(a) defined in Assumption 3.3. Then, with probability least 6. for any E OBEOS D),
( 1 4M)
2 ++14+3
ad
M 3' 4ad43a
2 + 4oo++30)
old 93
3d
2d-3
Gapp(fo: Edd
( - + 4M)
74445
M
2d
24 4add
n 40
α <
2d-3
3d
5

---

Published as a conference paper at ICLR 2026
where M == max Il bd - , 1} and Sd hides constants (which may depend d) and loga- rithmic factors and (1/6).
Sketch proof of Theorem 3.4. Suppose Px is Beta(a))radial distribution Px(a) with radial profile lu(r) = 1 (1 r) r/ (Definition 3.2). Then Lemma G.2 gives (x x e and
Proposition G.6 gives gp(1 E) x gdt2n Substituting into (7), the right-hand side becomes Ö(E") Ö(E- dcd+2m) 40te
Morally', optimizing over yields the upper bounds in Theo-
rem 3.4, see Appendix G for details.
In the proof of the upper bound using (7), we sacrifice the shallow region Ag, but this not just proof artifact. Our lower bound constructions place localized ReLU atoms disjoint spherical caps in the shallow shell where gp is very small, so these neurons can develop large path norm at small cost. Combinations of these coundary-supporte ReLUs form family of hard-to-distingut networks (see Construction H.4), yielding Theorem 3.5.
Theorem 3.5 (Generalization gap lower bound). Let be any joint distribution of (ar y) where the marginal distribution of x is Px (a) and y is supported on [-1,1]. Let Dn, {(2), })) be dataset of n. i.i.d. samples from P. Let RD. (f) be any empirical risk estimator for the true risk Rp(f) == (ka.))>Pl(((( 3) Then,
sup ec (1111.)) Re([o) - No. [ool Zd,a ni
inf sup ED, f P
1720.
2n
The smaller ar is, the more mass concentrates Ag, the more disjoint spherical caps that each carry comparable probability mass (see Figure 1(b)&(c), the more memorization capacity the network sustain. the extreme case where a + and the law of X approaches Uniform(sd-1) each datapoint can be isolated by network and fitted perfectly within the BEoS regime, see Appendix I.
Theorem 3.6 (Flat interpolation with width < n). Assume that {(a,, vi) is dataset with I; E g and pairwise distinct inputs. Then there exists width K S n network of the form (1) that interpolates the dataset and whose Hessian operator norm satisfies
D2
Amnax (V)C) 1
(10)
n2
If we remove the output bias parameter 3 in (I), then Amax (V)C) 12+2
Remark 3.7. If we remove the hidden bias term b and assume a uniform spherical distribution, the model gets harder to shatter the data, and thus yields generalization bound of approximately (nn-1/4) 1/44,, which is compatible with the rates in (Wu and Su, 2023).
3.2 UNIFIED FRAMEWORK: DATA SHATTERABILITY PRINCIPLE
The trade-off in (6) and (7) shows that generalization is governed by the probability mass located in the T-deep region. Intuitively, the more mass lies in deeper regions, the stronger the implicit regularization. To quantify this geometric distribution, we introduce the following scalar summary.
Definition 3.8. Given distribution Px Rd its half-space-depth quantile function is Px (T) = Px (depth(X, x,,,, Px) 2 T), for T E [0, 1/2]. We define the half-space-depth (.0
concentration index SpQ(Px)
1/2 Pp (T) aT) tte reciprocal of the under P
We propose using this concentration index as proxy for data shatterability, the feasibility of the data being partitioned into disjoint regions. In the isotropic case, the spherical symmetry ensures that one can always arrange large number of disjoint activation regions (spherical caps), quantity governed by the packing number the high-dimensional sphere (see Lemma H.3). Consequently, the feasibility of shattering is strictly determined by the radial concentration: does each such poten- tial cap contain sufficient probability mass? As illustrated in Figure 1(b)&(c), we require every cap contain fixed amount of data, the total number of disjoint caps we can pack becomes
"Since the empirical process only guarantee sup (gr(u, t) (p t) < Ō(n ((12 with n = [D], we cannot let ge(( - =") less than -1/3), ) there is two-case discussion in Theorem 3.4.

---

Published as a conference paper at ICLR 2026
0=0.1 fores 0.024
0-003 larea 0.064
Protebility Ness V7y07
Tukey depihT
(a) Px for (x(a) in 5
(b) More caps (a 0.5)
(c) Less caps (a = 2)
Figure I: Visualization isotropic data shatterability, (a) For isotropic Bela(a)) distribu- tions, larger implies mass is more concentrated centrally (larger area, smaller index). (b & e) Red dots depict data points and dashed lines represent neuron boundaries. We visualize the partitioning feasibility by fixing the probability mass within each cap and comparing the number of disjoint caps that can be packed. A distribution with mass near the boundary (small a) populates significantly more such disjoint regions than one concentrated the center (a 2).
measure shatterability. large index (small a)) implies mass accumulates at the boundary, pop- ulating these pre-existing packing slots and facilitating the construction of shattering functions for our lower bounds, see Construction H.4. Conversely, small index (large a) starves these regions of data, limiting the noise-fitting feasibility.
further consistency check comes from the spherical limit: for distributions on a sphere, the depth is zero everywhere. The quantile function collapses and the concentration index diverges, correctly matching the flat interpolation behavior in Theorem 3.6.
Remark 3,9 (Rectangle vs. Area). This picture also clarifies why our upper bounds are not tight. On the T Tedeep region, we bound the entire contribution of the network by the worst-case variation over (Px), and on the shallow region we use a crude worst-case 1 bound times Txx X & (r )) ignoring any additional regularization. Our analysis only exploits the area of a single inscribed rectangle under T + Px (T), and choosing the optimal T* in (6) is equivalent to picking the largest rectangle under the curve, which discards the rest of the underlying area. This gap is exactly where potential improvements to the bounds would have to come from
3.3 PROVABLE ADAPTATION TO INTRINSIC LOW-DIMENSIONALL.
1=10 (area 0.0933)
d=3 (area 0.026)
d= 0.0902)
0.039)
Probabiiitt Moss Mo,sss,,, (7)
1:000 (area 0.0890)
Probabiltty Mass 17,17)
d=3 (area 0.065)
d-500 (area 0.0884)
d=2 (area 0.124)
(area 0.250)
Tukey depth
Tukey depth T (c) Uniform (B4)
(a) ReLUs shatter lines by knots
(b) A union of 20 lines in RRd
Figure 2: Visualization of anisotropic data shatterability. (a) When data lies mixture of lines (black lines), the complex activation boundaries of ReLUs (translucent planes) reduce to finite set of knots (colored dots). This rigidity makes the data harder shatter than the ambient depth suggests. (b) For union of 20 embedded lines, the curves remain nearly identical across ambient dimensions, matching the prediction by Theorem 10. (c) As comparion, the curves for Uni degenerates and reveal the curse of dimensionality predicted in (Liang et al., 2025).
7

---

Published as a conference paper at ICLR 2026
We now return to anisotropic data. Here, the connection between the concentration index and gener- alization becomes heuristic because the geometric structure restricts the available partitioning direc- tions. Unlike the isotropic case, where we can leverage the packing number of the ambient sphere to form disjoint regions, low-dimensional structures constrain the effective directions. For instance, on ID line, the high-dimensional spherical caps effectively degenerate into simple intervals or knots (Figure 2a). This collapse drastically reduces the number of possible disjoint partitions, even if the data has low ambient depth (high concentration index). Consequently, for structured data, the index serves as a conservative estimate of difficulty. Nevertheless, the principle of depth adaptation still holds: the implicit regularization adapts the intrinsic structure rather than the ambient dimension.
Formalizing this, suppose the feature marginal is a finite mixture Px = Σj= Ty Px.j. Once the depth-quantile curve pxx, (T) of each component is understood, the law of total probability yields
Px (T) Px (PPxxdepphhha (x, 2) my Px (depth(() Px) 2 TIZ=
(11)
For any fixed j. the mixture cannot reduce all halfspace probabilities contributed by component j by more than a factor иy (i.e., depth(æx, Px) 2 Ty depth(2,, Px.)). Hence, at the level of depth- quantile curves, this shows that py is controlled by the collection { p 1/ and the area under Px behaves like a mixture-weighted average of the areas under the Pxx3"
In this heuristic picture, if each component is essentially low-dimensional, then the concentration index of px is governed by the embedded low-dimensional components and hence by the intrinsic dimension rather than the ambient one. Formally, we have the following theorem.
Theorem 3.10 (Generalization bound for mixture models). Given data distribution P on Rd x R whose feature marginal satisfies Pxx = E' RyPx. where each Px.5 is the uniform distribution on the unit ball in an m-dimensional affine subspace V3 C. Rd let = {(x y:) be a dataset of i.i.d. samples drawn from P. Then, with probability at least 1 8,
sup e B (n)
Gapp(So:D) Ea _
1 + 4M
M J J 2m+4 + M² J
(12)
2
2n.
where M == max DD ll bay |1,=,1} and Sa hides constants (which may depend on d) and loga-
rithmic factors in J/8 and n.
Sketch proof of Theorem 3.10. The detailed proof is in Appendix F. The argument instantiates the above shatterability picture the level of the BEoS-induced regularization. For each component j, we define a local weight function 9j using only samples with X E Vj. and Lemma F.4 formalizes this uniform domination 9 2 9j. Consequently, the restriction of fo to V, has controlled gy- weighted variation with the same order of bound. Crucially, if we restrict the network to a single m-dimeensionll subspace Vj. neuron's activation is governed not by its full weight vector wa. but solely by projv, Wy. since the component of wy that is orthogonal to Vy is "invisible" to the data on Vj. This projection mechanism mathematically formalizes how linearly low-dimensional structures limit the model's shatterability (as visualized by the knots in Figure 2), and thus refine the characterization of network's 's effective capacity, see Theorem F.3. Moreover, some insight of this low-dimensional adaptation from the perspective of gradient dynamics is stated in Appendix B.2.
4 EXPERIMENTS
In this section, we present empirical verification of both our theoretical claims and proof strategies.
4.1 EMPIRICAL VERIFICATION OF THE GENERALIZATION UPPER BOUNDS
We test two predictions of our theory using synthetic data and two-layer ReLU networks of width 1000 trained with MSE loss and vanilla GD with learning rate 0.4 for 20000 epochs. The synthetic training data is produced by fixing ground-truth function f (ReLU networks or quadratic functions) to noisy labels y/5 fxx) + &,, where & is an i.i.d Gaussian noise. Generalization gap is measured by the true MSE Ep (f(() - f on the training set. In other words, this measures the resistance memorize noise. Theory predicts Errr X -e with geometry depeedett exponent
8

---

Published as a conference paper at ICLR 2026
-02
Aphaa-8.1 (vkgpem6.006
10 Iolope-0 0 813)
Apha=0.33 (5kpp- 0.055)
30 (slopen-O.000 994
Aphaa11.5 (stope-- 6 102)
100 (Slopo 0.924)
Apham5,0 (viopem 0.174)
(slope: 0
-0.4
foat MDD
0.
-03
Regression statiailss Mean stope -0 0.451
Regression stativics:: Mean stope. 00 0.8575 Mean Rt: 0,5802
Mean RP: 0,7803
Sippe range: [-6.1715, 640499
Skope range: (-0 66.9200 0126
38
logiSample Steel
logiSample Steo]
(a) Radial concentration parameter
(b) Adaptation to intrinsic dimension
Figure 3: How data geometry controls generalization. (a) Fixed ambient dimension d = 5 with isotropic Beta-radial distributions (Definition 3.2) for
{0.1,, 0.3, 5.0) Larger yields steeper slopes in the log-log error curve, consistent with improved rates as probability mass con- centrates away from the boundary. (b) Union of J 20 lines (m 1) embedded in Rd with d E {10, 50, 100, 500}. The regression slopes remain nearly constant across different d, showing that generalization adapts to intrinsic rather than ambient dimension.
c, so we plot log(clean MSE) against log n and estimate the slope by OLS. For each sample size n, train i... examples and report their true MSE. For each setup, we ran experiments with 6 random seeds and averaged the results. The results are summarized in Figure 3.
4.2 HOw DATA GEOMETRY AFFECTS REPRESENTATION LEARNING
We study how data geometry shapes the representation selected by GD at the BEoS regime through data activation rate of neurons. Given neuron v bk) in the neural network, its data activation rate is defined as Σ i=1 1 ( æ, by J. which is exactly the probability term in the definition of the weight function g in (28). Low data activation rate means the neuron fires on a small portion of the data.
Uniform Sphere
Scher ARTIN STUTIOY N6.SM
Mixed Law Dimensional
Weight Coefficiient Magnitude
100
105
800
1g
400
200
Final Activation Rata Range
Data Activation Rate
(a) Activationnnnae rate histogram
(b) Weight magnitude vs. activation rate
Figure& 4: Neuron activation statisties under different geometries. (a) On the uniform sphere, most neurons fire on less than 10% of the data, indicating highly specialized ReLUs as we predict in Theorem 3.5 and Theorem 3.6. On the low-dimensional mixture, many neurons fire on 10-40% of the data, reflecting broader feature reuse. (b) Scatter of weight coefficient magnitude versus activation rate. On the sphere, GD produces many low-activation neurons with large coefficients. On the low-dimensional mixture, neurons spread to medium activation rates with moderate coefficients.
To verify it empirically, we compare two input distribution in R50, (i) uniform distribution on sphere and (ii) a union of 20 lines by training ReLU networks with the same recipe and initialization As result, the ReLU network trained on the sphere interpolates the noisy label quickly with final true MSE 1.0249 ze noise level (a² = 1), while the the ReLU network trained on a union of lines resist overfitting with final true MSE 0.07 as 0 (more details appear in Appendix C). Notably, the trained representations are presented in Figure 4. In particular, GD empirically finds our lower bound construction below the edge of stability.
9

---

Published as a conference paper at ICLR 2026
Training Curves Comparison
MNIST: Depth vs Prediction Error
HAST
23
Heacia via
Notee Lnei 2.0
Prodiction Eror (Tru MSE
0.25
0.00
10000
20000
30000
60000
20002
Spach
1850
Tukey Depth
Figure 5: Data geometry and memorization MNIST. Left panel: Comparison of training curves under the same ground-truth predictor with Gaussian inputs versus MNIST inputs (n 30000). GD the Gaussian data set quickly interpolates, while MNIST resists overfitting for tens of thousands of steps. Right panel: Prediction error against half-space depth for MNIST samples. Shallow points (low depth) exhibit larger errors. This region refers to "highly shatterable region".
4.3 EMPIRICAL EVIDENCE FOR THE DATA SHATTERABILITH PRINCIPLE
Our theory assumes data supported exactly mixture of dow-dimensional subspaces. In practice, real datasets are only approximately low-dimensional, as highlighted in the literature on subspace clustering (Vidal et al., 2016; Elhamifar and Vidal, 2013). For instance, MNIST images do not perfectly lie on a union of lines or planes, but still exhibit strong correlations that concentrate them near such structures. Our experiments (Figure 5, more details in Appendix C.2) show that even this approximate structure has a pronounced effect: compared to Gaussian data of the same size, GD on MNIST requires orders of magnitude more iterations before mildly overfitting solutions emerge. This demonstrates that our theoretical prediction is not fragile: generalization benefits from low- dimensional structure across a spectrum.
5 DISCUSSION AND FURTHER QUESTIONS
In this work, we present a mechanism explaining how data geometry governs the implicit bias of neural networks trained below the Edge of Stability. We introduce the principle of "data shatterabil- ity," demonstrating that geometries resistant to shattering guide gradient descent towards discovering shared, generalizable representations.
Limitations. While our theoretical framework instantiates the principle of data shatterability, there are limitations to our current analysis. First, the proposed half-space-dephe concentration index (Spq) serves as a proxy primarily for isotropic distributions and extending this scalar metric to quantify the shatterability of arbitrary, anisotropic, structured data remains a non-trivial challenge. Second, our results are derived for two-layer ReI.U networks to maintain tractability in the function- space analysis. Generalizing these bounds to deep and complicated networks presents substantial theoretical hurdles, particularly in characterizing how the EoS-induced regularity constraint propa- gates through hierarchical layers without becoming ensuring the bounds remain non-vacuous.
Further Questions. Our framework opens several promising avenues for future research. A central question is the connection between shatterability and optimization. The observation that a flip side high-
of being prone to overfitting is often faster optimization leads to natural hypothesis: shatterability distributions easier to optimize? This, in turn, raises further questions about the role of normalization techniques. For instance, do normalization techniques like Batch Norm accelerate training precisely by enforcing more isotropic, and thus more shatterable, representations at each layer? This line of inquiry extends naturally to deep networks, where hidden layers not only sense the initial data geometry but actively create new "representation geometry"". Can our principles be translated to the understanding of representation geometry? Finally, this framework may offer new lens to understand architectural inductive biases. For example, do CNNs generalize well precisely because their local receptive fields impose an architectural constraint that inherently reduces the model's 's ability to shatter the data, forcing it to learn local, reusable features? Answering such questions, alongside developing a quantifiable metric for shatterability, remains a key direction.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
The research was partially supported by NSF Award # 2134214. Tongtong Liang thanks Dan Qiao and Zihan Shao for providing helpful suggestions.
REFERENCES
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradi- ent descent. In International conference on machine learning, pages 247-257, PMLR, 2022.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference Machine Learning, pages 322-332. PMLR, 2019.
Sanjeev Arora, Zhiyuan Li. and Abhishek Panigrahi. Understanding gradient descent the edge of stability in deep learning. In International Conference on Machine Learning, pages 948-1024. PMLR, 2022.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization deep networks. In Proceedings of the 34th International Conference Machine Learning (ICML), pages 233-242. PMLR, 2017.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Ma- chine Learning Research, 18(1):629 681, 2017.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(49):30033-3007 2020.
Alexander Cloninger and Timo Klock. A deep network construction that adapts intrinsic dimen- sionality beyond the domain. Neural Networks, 141:404-419, 2021.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. International Conference on Learning Representations, 2020.
Jeremy M. Cohen, Alex Damian, Ameet Talwalkar, J. Zico Kolter, and Jason D. Lee. Understand- ing optimization deep learning with central flows. In The Thirteenth International Confer- Learning Representations, 2025. URL https: /wwp///eeeeeeeee///////eee net www.t/t///o/rr// forum?id= 5 SIE2rI3ZPS
Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: descent at the edge of stability. In International Conference on Learning Representations, 2024.
The implicit bias of gradient
Ronald DeVore, Robert D. Nowak, Rahul Parhi, and Jonathan W. Siegel. Weighted variation spaces and approximation by shallow ReLU networks. Applied and Computational Harmonic Analysis, 74:101713, 2025.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions on Puttern Analysis and Machine Intelligence, 35(11):2765-23) 2013.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis.
Journal of the American Mathematical Society, 29(4):983-104) 2016.
Tushar Ganguli and Edwin P. Chong. Activation-based pruning neural networks. Algorithms, 17(1):48, 2024.
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, 2019a. URL https: / /api. semanticschon .org/ CorpusTD:59291990
11

---

Published as a conference paper at ICLR 2026
Boris Hanin and David Rolnick. Deep ReLU networks have surprisingly few activation patterns. In Advances in Neural Information Processing Systems, volume 32, 2019b.
Boris Hanin, Ryan Jeong, and David Rolnick. Deep relu networks preserve expected
length. ArXiv, abs/2102.10492, 2021. URL https://apl. wwwpppsssaanniccccl....... CorpusID: 2TIID::33983000
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225-1234. PMLR, 2016.
David Haussler. -eeosicn- generalizations of the pac model for neural net and other learning applications. Information and Computation, 100(1):78-150, 1992.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint, 2016. URL
https: /ttp///rvvvv//////////e/////aev. www.eiiwwg....eee...eee.eeoooeee wtt.//ss//////oo/////////...//// 607 03250.
Hui Jin and Guido Montúfar. Implicit bias gradient descent for mean squared error regression with two-layer wide neural networks. Journal of Machine Learning Research, 24(137):1-97, 2023.
Nirmit Joshi, Gal Vardi, and Nathan Srebro. Noisy interpolation learning with shallow univariate ReLU networks. In International Conference Learning Representations, 2024.
Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. Annals of Statistics, 49(4):2231 2021.
Michael Kohler, Adam Krzyžak, and Sophie Langer. Estimation of a function of low local dimen- sionality by deep neural networks. IEEE transactions on information theory, 6836::032244422, 2022.
Lingkai Kong and Molei Tao. Stochasticity of deterministic gradient descent: Large learning rate for multiscale objective function. Advances in neural information processing systems, 33:2625-2638, 2020.
Guy Korowski, Gilad Yehudai, and Ohad Shamir. From tempered to benign overfitting in relu neural networks. Advances in Neurul Information Processing Systems, 36, 2024.
Tengyuan Liang Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, ge- ometry, and complexity of neural networks. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 888-896, Naha, Okinawa, Japan, 2019. PMLR. URL https: / liangl 9a html.
ml mlr.press/v89)
Tongtong Liang, Dan Qiao, Yu-Xiang Wang, and Rahul Parhi. Stable minima of relu neural networks suffer from the curse of dimensionality: The neural shattering phenomenon, 2025. URL https:
//arxiv. www.ii.org/aaaa.ooooooooooe abs 2....2022222222222222222222222.. 20779.
Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems, 34:1 16805-168117, 2021.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-fre free bounds and kernel limit. In Conference on Learning Theory, pages 2388-2464 PMLR, 2019.
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Neural Information Processing Systems, 2014. URL https: org/ Corpus 5941770.
/ //api. www.nticschoieeooooe......
Rotem Mulayoff, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability: A view from function space. Advances in Neural Information Processing Systems, 34:17749-17766
2021.
12

---

Published as a conference paper at ICLR 2026
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability in multivariate shallow ReLU networks. In International Con- ference Learning Representations, 2023.
Kamil Nar and Shankar Sastry. Step size matters in deep learning, Advances in Neural Information Processing Systems, 31, 2018.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 Proceedings of Machine Learning Research, pages 1376-1401, Paris, France, 03-06 Jul 2015. PMLR. URL https: httpss//ppcceeeeieeeeee mmr. press/v4D Neyshaburls 15. html.
Rahul Parhi and Robert Nowak. Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research, 22(43):1-40, 2021.
Rahul Parhi and Robert D. Nowak. What kinds of functions do deep neural networks learn? Insights from variational spline theory. SIAM Journal on Muthematics of Data Science, 4(2):46-4489 2022. doi: 10.1 0..1137/2MM111666 18642.
Rahul Parhi and Robert D. Nowak. Near-minimas optimal estimation with shallow ReLU neural networks. JEEE Transactions on Information Theory, 69(2): 122-1139, 2023.
Rahul Parhi and Robert D. Nowak. Compositional function spaces for deep learning. SIAM Review, 68(1):127-149, 2026. doi: 10. 10.1137725M1802248
Rahul Parhi and Michael Unser. Distributional extension and invertibility of the k-plane transform and its dual. SIAM Journal Mathematical Analysis, 56(4):4662-4686, 2024. doi: 10.1137/ 23M1556721.
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Rel- ative flatness and generalization. In Advances in Neural Information Processing Systems 34, volume 34. Curran Associates, Inc., 2021. URL https: / http:///ooecciggieeeee ngs .wurips..cooo ca/ paper/2021/har 2w/22021hasshh 995 58 (0b029865f402ee60
2d Abst html.
Tomaso Poggio and Qianli Liao. Theory ii: Landscape of the empirical risk in deep learning. arXiv preprint ar /X03:0/703.093333 2017.
Dan Qiao, Kaiqi Zhang, Esha Singh, Daniel Soudry, and Yu-Xiang Wang. Stable minima cannot overfit in univariate ReLU networks: Generalization by large step sizes. In Advances in Neural Information Processing Systems, volume 37, pages 94163-94208, 2024.
Johannes Schmidt-Hieber, Nonparametric regression using deep neural networks with ReLU acti- vation function. Annals of Statisticc,, -844):1875-1897, 2020.
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting lin- ear regions of deep neural networks. ArXiv, abs/1711.02114, 2017. URL semanticscher
org ort./oooosos:.131111555.
Jonathan W. Siegel and Jinchao Xu. Characterization of the variation spaces corresponding to shal- low neural networks. Constructive Approximation, pages 1-24, 2023.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70): 1-57, 2018
Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint arXiv: 2018.
Saket Tiwari and George Konidaris. Effects of data geometry in early deep learning. In Advances in Neural Information Processing Systems, volume 35, pages 25010-25023 2022.
John W. Tukey. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathematicians, volume 2, pages 523-531, Vancouver, 1975, Canadian Mathemati- cal Congress.
13

---

Published as a conference paper at ICLR 2026
Vladimir N. Vapnik. Statistical Learning Theory. Wiley, 1998.
René Vidal, Yi Ma, and Shankar Sastry. Generalized Principal Component Analysis. Springer, 2016.
Lei Wu and Weijie J Su. The implicit regularization of dynamical stability in stochastic gradient descent. International Conference on Machine Learning, pages 37656-37684. PMLR, 2023.
Lei Wu, Chao Ma, and Weinan How SGD selects the global minima in over-parameterivzed learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31, 2018.
Yunfei Yang and Ding-Xuan Zhou. Optimal rates of approximation by shallow ReLU* neural net- works and applications nonparametric regression. Constructive Approximation, pages 1-32, 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Rep- resentations (ICLR), 2017. URL https: oownnevveeeeeeeeeeeeeeeeeeeeeeee ew net/ forum ? i Sy
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empir- ical risk minimization. In International Conference on Learning Representations (ICLR), 2018. URL httpttt:////eeeeeeeeee
Kaiqi Zhang and Yu-Xiang Wang. Deep learning meets nonparametrie regression: Are weight- decayed DNNs locally adaptive? 2023.
In International Conference on Learning Representations,
Linjun Zhang, Kenji Kawaguchi, Amirata Ghorbani, Zhun Deng, and James Zou, How does mixup help with robustness and generalization? In International Conference Learning Representa- tions (ICLR), 2021.
Xiao Zhang and Dongrui Wu. Empirical studies on the properties of linear regions in deep neural networks. ArXiv, habs/200.001072. 2020. URL https: / /appipp.... semanticscholar org/ CorpusID:2098622988
14

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIALS
A The Use of Large Language Models (LLMs)
15
More Related Works and Discussions
16
B.1 More Related Works
16
B.2 A Gradient-dynamies Perspective on Stability and Data Geometry
17
C Details of Experiments
18
C.I Experimental Details for Section 4.2
18
C.2 Experimental Details of Section 4.3
20
D Functional Analysis of Shallow ReLU Networks
22
D.1 Path-norm and Variation Semi-norm of ReLU Networks
22
D.2 Total Variation Semi-norm on Radon Domain
23
D.3 The Metric Entropy of Variation Spaces
23
D.4
Generalization Gap of Unweighted Variation Function Class
24
Data-Dependent Regularity from Edge-of-Stability
25
E.1
Function Space Viewpoint of Neural Networks Below the Edge of Stability
25
E.2
Empirical Process for the Weight Function g
26
F
Generalization Upper Bound: Mixture of Low-Dimensional Balls
28
F1 Case: uniform distribution on unit disc of a linear subspace
28
F.2 Proof of Theorem 3.10
29
G Generalization Upper Bounds: Isotropic Beta Family
33
G.1 Characterization of the Weight Function for Custom Radial Distribution
34
G.2 Proof of Theorem 3,4
36
Generalization Gap Lower Bound via Poissonization
39
H.l Construction of "hard-to-learn" networks
40
H.2 Proof of Theorem
I Flat Interpolating Two-Layer ReL.U Networks the Unit Sphere
46
J Technical Lemmas
48
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In accordance with the ICLR 2026 policy on the responsible use of LLMs, we disclose the follow- ing. We employed commercial LLM services during manuscript preparation. Specifically, we used Gemini 2.5 Pro, ChatGPT 5, and DeepSeek to assist with language polishing, literature search, and consistency checks of theoretical derivations. We further used Claude 4 and Cursor to help generate experimental code templates. Importantly, all research ideas, theoretical results, and proof strategies
15

---

Published as a conference paper at ICLR 2026
originated entirely from the authors. The LLMs were used solely productivity aids and did not contribute novel scientific content.
B
MORE RELATED WORKS AND DISCUSSIONS
B. MORE RELATED WORKS
How "rethink" generalization, Our shatterability principle provides a theoretical account of the discrepancy noted by Zhang et al. (2017): networks fit Gaussian noise much faster than real images with random labels. Gaussian inputs concentrate on thin spherical shell and are highly shatterable, while CIFAR-10 10 exhibits unknown fow-dimensions structure that resists shattering. Strong general- ization arises in practice because gradient descent implicitly exploits this non-shateerable geometry of the real world data. We conduct a similar experiment from the perspective of generalization in Section 4.3 (see Figure 5).
Revisit data augmentation. Mixup forms convex combinations of inputs and labels and encourages approximately linear predictions along these segments (Zhang et al., 2018). The added in-between samples penalize solutions that memorize isolated points with sharply varying piecewise-lineal be- havior. For example, on spherical-like data that ReLU units can easily shatter, such memorization incurs high loss the mixed samples, which suppresses thattering-type separators. Prior work mostly views Mixup as a data-dependent regularizer that improves generalization and robustness (Zhang al., 2021). Our analysis complements this view by tracing the effect to the implicit bias of gradient descent near the edge of stability and by linking the gains to a reduction in data shatterability induced by interpolation in low-density regions.
Activation- network pruning, Empirical works have shown that pruning strategies based on neuron activation frequency, such as removing neurons with low activation counts, can even improve the test performance after retraining (Hu et al., 2016; Ganguli and Chong, 2024). This coincide with our theory: such rare-firing fir--fiing neurons may be harmful to generalization and pruning these neurons help models to learn more generalizable features.
Subspace and manifold hypothesis. A common modeling assumption in high-dimensions learn- ing is that data lies on or near one or several low-dimensional subspaces embedded in the ambient space, especially image datasets where pixel values are constrained by geometric structure and are well-approximmted by local subspaces or unions of subspaces (Vidal et al., 201 2). In particu- lar, results in sparse representation and subspace clustering demonstrate that such structures enable efficient recovery and segmentation of high-dimensional data into their intrinsic subspaces (Elham- ifar and Vidal, 2013). This also extends to a more general framework of the manifold hypothesis (Fefferman et al., 2016).
Capacity of neural networks. The subspace and manifold hypotheses have important implications for the capacity and generalization of neural networks. When data lies near low-dimensional sub- spaces and manifolds, networks can achieve expressive power with significantly fewer parameters, the complexity of the function be learned is effectively constrained by the subspace dimen- sion rather than the ambient dimension (Poggio and Liao, 2017; Cloninger and Klock, 2021; Kohler al., 2022). However, these results focus only on expressivity and the existence of neural networks to learn efficiently on this data.
Interpolation, Benign overfitting and data geometry. Benign-overfitting (Bartlett et al., 2020) studies the curious phenomenon that one can interpolate noisy labels (i.e., 0 training loss) while consistently learn (excess risk - 0 as n gets larger). Joshi al. (2024) establishes that overfitting in ReLU Networks is not benign in general, but it could become more benign as the input dimension grows (Kornowski et al., 2024) the isotropic Gaussian data case. Our results suggest that such conclusion may be fragile under low-dimensional or structured input distributions. On positive note, our results suggest that in these cases, generalization may follow from edge-of-stabiliyy which applies without requiring interpolation.
Implicit bias of gradient descent. A rich line of work analyzes the implicit bias of (stochastic) gradient descent (GD), typically through optimization dynamics or limiting kernels (Arora et al., 2019; Mei et al., 2019; Jin and Montúfar, 2023). In contrast, we do not analyze the time evolution per se; characterize the function spaces that GD tends realize at solutions. Our results highlight
16

---

Published as a conference paper at ICLR 2026
strong dependence the input distribution: even for the same architecture and loss, the induced hypothesis class (and thus generalization) changes as the data geometry changes, complementing prior cynamics-centrie views.
Edge of Stability (EoS) and minima stability. The EoS literature primarily seeks to explain when and why training operates near instability and how optimization proceeds there (Cohen al., 2020; Kong and Tao, 2020; Arora et al., 2022; Ahn et al., 2022; Damian et al., 2024). Central flows offer an alternative viewpoint on optimization trajectories that also emphasizes near-instability behavior (Cohen et al., 2025). Closest to our work is the line on minima stability (Ma and Ying, 2021; Mulayoff et al., 2021; Nacson et al., 2023), which links Hessian spectra and training noise to the geometry of solutions but largely leaves generalization out of scope. We leverage the EoS/minima- stability phenomena to define and analyze a cata-distiibutionaaaaa notion of stability, showing adaptivity to low-dimensional structure and making explicit how distributional geometry shapes which stable minima GD selects.
Nonparametric function estimation with neural networks. The notion of generalization gap is closely related to the estimation error. It is well known that neural networks are minimax optimal estimators for a wide variety of functions (Suzuki, 2018;
2020; Kohler and Langer, 2021; Parhi and Nowak, 2023; Zhang and Wang. 2023; Wu and Su, 2023; Yang and Zhou, 2024; Qiao et al., 2024). Outside of the univariate work of Qiao et al. (2024), all prior works construct their estimators via empirical risk minimization problems. Thus, they do not incorporate the training dynamics that arise when training neural networks in practice. In contrast, Qiao et al. (2024); Liang al. (2025) derive nonparametric guarantees for solutions selected directly by gradient descent dynamics, without assuming even stationary condition. Our work further develops in this direction by modeling how data geometry affect generalization for the gradient trajectories below the edga- of-stability.
Flatness vs. generalization. Whether (and which notion of) flatness predicts generalization re- mains debated. Several works argue sharp minima can still generalize (Dinh et al., 2017), propose information geemmerri or MatneerRRo-baarr notions (Liang et al., 2019), or develop relative/scale- invariant flatness measures (Petzka et al., 2021). We focus on the largest curvature direction (i.e., Amax) motivated by EoS/mmnnima-stailiit Our results rigorously prove that flatness in this notion does imply generalization (note that there is no contradiction with Dinh et al. (2017)), but it depends on data distribution.
Linear regions of neural networks. Our research connects to significant body of work that investigates the shattering capability of neural networks by quantifying their linear activation regions (Hanin and Rolnick, 2019a;b; Hanin et al., 2021; Montúfar et al., 2014; Serra et al., 2017). Other empirical work has meticulously characterized the geometric properties of linear regions shaped by different optimizers (Zhang and Wu, 2020). Particularly, (Tiwari and Konidaris, 2022) consider the how these linear regions intersect with data manifolds. These analyses primarily leverage the number of regions to characterize the expressive power of deep networks, while work shifts the focus on the generalization performance of shallow networks at the EoS regime.
B.2 A GRADIENT-DYNAMIC PERSPECTIVE ON STABILITY AND DATA GEOMETRY
We now discuss our results from the viewpoint of gradient dynamics.
Why Shattering Neural Networks May Be Dynamically Stable. Here we paraphrase and adapt the discussion from (Liang et al., 2025, Appendix A.4). The BEoS condition defines a set of dy- namically stable parameter states, and the data-dependent weight function gr g t) provides a static summary of how expensive it is to place a ReI.U ridge at orientation u and threshold (. Small values of gD (u. g t) indicate weak stability constraints in that region of parameter space, so a neuron aligned with (u, t) can carry a large coefficient while still satisfying the BEoS curvature bound. In highly shatterable geometries, the shallow shell contains many such directions with tiny gD, creating am- ple room inside the stable set for high-magnitude, sparsely activating neurons supported on disjoint caps.
This static picture is closely tied to the actual gradient dynamics. the dataset is highly shatterable in the sense of the salf-space-depthh concentration index, a neuron's 's activation boundary can easily drift toward regions where fires on only few data points. Once those few points already well-fitted,
17

---

Published as a conference paper at ICLR 2026
the gradient contributions in (13) become small and localized, so the neuron experiences almost no force pulling it back toward more central regions of the data cloud. Its parameters become effectively "stuck" near the boundary, and the corresponding directions in the loss landscape remain flat enough to satisfy the BEoS condition. Our lower bound constructions exploit exactly this mechanism: by neurons disjoint caps, obtain shattering
arranging many such trapped, houndary-supports networks that interpolate, remain dynamically stable, and yet are statistically hard to learn. Although our analysis is carried out for ReLU, where hard sparsity makes this effect particularly transparent, the underlying "weak-ge sarakggdieeett trapping" mechanism suggests that similar phenomena may persist for other activations with rapidly decaying gradients away from their transition region.
How GD Adatts to Low-dimensionality Dynamically. Recall our notations: for a two-layer ReLU network
fo( = Σ Uk (w bb) + k=1
trained on data {(ar; yr) )==1 with empirical loss L(0) 2n Ei= ((So(1))) yi), the gradient with respect to a hidden weight we has the form
Vw, (() -())=
1 n Σ e (fo(x.). un d' d'(w bb) x. ak x
(13)
Thus, at every step of gradient descent, the update of wt is a linear combination of input vectors. In informal terms, the shape of the gradient field is inherited from the shape of the data cloud: the dynamics cannot move in arbitrary directions in parameter space, but only along directions induced by the training inputs and the neuron-specific activation pattern.
In the idealized case where all x; lie in linear subspace of setrinice-dimension adaptation: the trajectory of each hidden weight lives in an affine translate of V. so any meaningful notion of effective complexity should depend on dim V rather than on the ambient dimension. However, as soon as we move to more realistic geometries affine subspaces + V or mixtures of m--dimensional components the global span of the data can easily be full- dimensional. In those settings, the observation that gradients lie in span(æ;) J is nearly vacuous: it no longer encodes the structured way which the data constrain the dynamics, and by itself it does not yield Matrisic-dimensinn generalization bounds.
R2 (13) already suggests form
Our contribution is to make this stability-based picture interact explicitly with the geometry of the data. For structured distributions such as mixtures of m-dimensional balls, the data-dependent weight function gD u. t) inherits a corresponding structure. A hyperplane that cuts through the thick interior of an m-dimensional component activates on many of its points. Neurons aligned with such hyperplanes receive large gradients and are strongly constrained by the stability condition, which is reflected in large values of go yu, l). In contrast, hyperplanes that only touch shallow caps or skim the boundary fire on very few points; neurons aligned with these directions see much weaker "gradient pressure" and correspondingly small values of gD (u, t). allowing them to carry high norm and implement localized features.
In this way, the gradient-dynamic perspective can be summarized as
data geometry
gradient geometry
stability-induees regularization,
with fD acting as the bridge between dynamics and capacity control. Stability does not merely say that gradients live in the span of the data; through fp, tells which parts of the data geometry are expensive to fit and which parts can host high-norm, sparsely supported neurons. This refinement allows our analysis to turn the qualitative statement that "gradient trajectories are shaped by the data geometry" into quantitative, distribution-depeedder generalization bounds that scale with the intrinsic dimension m rather than the ambient dimension d.
C DETAILS OF EXPERIMENTS
C.I EXPERIMENTAL DETAILS FOR SECTION 4.2
Here provide the full experimental details of the discussion in Section 4.2.
18

---

Published as a conference paper at ICLR 2026
We worked in ambient dimension d 50 with 2000 training examples. For the Sphere condition, samples were drawn uniformly from the unit sphere. For the Low-dimensional mixture, generated data from mixture of 20 randomly oriented 1-dimensional subspaces uniformly. Labels were produced by a fixed quadratic teacher function with added Gaussian noise of variance 1.
We trained a two-layer ReLU network with hidden width 1024. All models were trained with GD for 10000 epochs using learning rate 0.4 and gradient clipping at 50. The loss function the squared error against noisy labels, while generalization performance was evaluated by the frue MSE against the noiseless teacher. For comparability, both datasets shared the same initialization of parameters.
We monitored (i) training loss and true MSE, (ii) Hessian spectral norm estimated by power itera- tion on random minibatches, and (iii) neuron-level statisties such activation rate and coefficient magnitude. The training curves are shown in Figure 6 and Amax -curres are shown in Figure 7.
Training vs True MSE: Sphere l Distribution
Sphere: Training Loss
Sphere: True MSE
10
Low-Dim: Train Training Loss
Low-Dim True MSE
Noise Baseline
10a
Loss MSE
10
10-2
10
10
2000
4000
6000
8000
10000
Epoch
Figure 6: Training curves on different geometries. Training loss and clean MSE on Sphere vs. Low-dimensional mixture. We see GD sphere interpolate very quickly (before the 2000-th epoch) while the mixed fow-dimensional data resist to overfitting.
19

---

Published as a conference paper at ICLR 2026
Hessian Norm Evolution: Sphere vs Low-Dimensional Distribution
5.2
o 109 Mo Wo wy
Hessian Norm
4.8
4.6
4.4-
Sphere Distribution
4.0
Low- www-Dimensional Distribution
2000
4000
6000
8000
10000
Epoch
Figure 7: Amax ( -curveees Both of the curves oscillates around 2/n 5, signaling the edge of stability regime.
C.2 EXPERIMENTAL DETAILS OF SECTION 4.3
We complement the main experiments with a controlled comparison between real data (MNIST) and synthetic Gaussian noise under the ground-truth function. The goal is to illustrate how the geometry of real-world data affects the speed and nature of memorization by GD.
We fix ground-truth predictor two-layer ReLU network) and generate noisy labels
= f(x::) + Ei,
& ~ N00 1).
We then compare two input distributions of size 30000:
Gaussian inputs x; _N(0,, La) with 784, and
iii MNIST images x, E [0, 7)84 after normalization by 1/255.
Both datasets are trained with identical architecture (two-layer ReLU neuron network of 512 neu- rons), initialization, learning rate 7) 0.2, gradient clip threshold 50.
We track both the empirical training loss and the true MSE 1 Σn (f(a.) (xx:)) which mea- sures generalization. The horizontal dotted line at y corresponds to the noise variance and represents the interpolation limit.
n
Figure 8 shows training curves over the first 5000 epochs. On Gaussian inputs, GD rapidly inter- polates: the training loss vanishes and the clean MSE rises to the noise limit within few hundred steps. On MNIST inputs, GD initially decreases both training loss and clean MSE, entering pro- longed BEoS regime where interpolation is resisted. Only after thousands of epochs does the clean MSE start increase, suggesting that memorization occurs at much slower rate.
20

---

Published as a conference paper at ICLR 2026
Gencralization Comparison
Traiiiing Curves Comparison
[Clean MSE o" Train MSE)
1.4
HUST Trein locs
2.00
Naro 1itt 1.2
1.75
1.30
Generalbat Cap
1.25
0.8
1.00
0.6
0.25
ww M Mw
0.50
PRN1ST Damn Goe
Epoch
Epoch
Figure 8: Training curves for Gaussian noise MNIST the first 5000 epochs. Gaussian quickly interpolates, while MNIST remains in a BEoS regime where clean MSE stays well below the noise level.
Mean and Median Error by Depth Quintile
MNIST: Depth vs Prediction Error
Enror Median
Correlation: 0.069
0.14
Prediction Eror (True MSE)
2.0
0.12
0114
0.10
1.5
True MSE
0.08
1.0
0.06
0.04
0.5
0.02
0.0
0.00
01
Q5
0.000
0.005
0.010
0.015
0.020
Depth Quintile
Tukey Depth
(a) Meun/median
depth quintile (lower betler)
(b) Sample-wise bulf space depth epoch 5,000
Figure 9: MNIST at 5,000 epochs: deeper points have smaller error. The shallow region produces a long upper tail of errors, consistent with the inttrir- decomposition used in our upper bounds.
21

---

Published as a conference paper at ICLR 2026
D FUNCTIONAL ANALYSIS OF SHALLOW RELU NETWORKS
D.1 PATH-NORM AND VARIATION SEMI-NORM OF RELU NETWORKS
In this section, we summarize some result in (Parhi and Nowak, 2023) and (Siegel and Xu, 2023).
Definition D.I. Let fa( Σ Uk & ( x bk) B be two-layer neural network. The (unweighted) path-norm of fe is defined to be
K
flfe path
Uk 222122
(14)
Dictionary representation of ReLU networks. By the positive I-homogeneity of ReLU, each neuron be rescaled without changing the realized function:
Up ó(w] x by) as ou tw),
Us
|well2
Wk
E gd-1 to
=
1wxl12"
by
:= 2
Hence fo admits the normalized finite-sum form
f(x) = Σ au pb tk) + c.x c
(15)
Let the (ReLU) ridge dictionary be 2s == {Ø(ut T e E R}. We study the over- parameterized, width-agnostic class given by the union over all finite widths
Fin
U K21
Σ
au ou -t)) c c ()
(16)
and measure complexity by the minimal path-norm needed realize f:
Ilathhanm inf flllelpp fe f of the form (15))
From finite sums to a width-agnostic integral representation. To analyze Fün without com- mitting fixed width K, we pass a convex. measure-based description that represents the closure/convex hull of (16). Specifically, let be finite signed Radon measure on gd x [-R, R] and consider
f(x) f o( x -) du(u, t) + c. co.
(17)
Jgd-1x[-R,R
Any finite network (15) corresponds to the sparse measure v = Σ ag oies an): and conversely sparse measures yield finite networks. Thus, (17) is width-agnostic relaxation of (14), not an assumption of an width limit.
Definition D.2. The (unweighted) variation (semi)norm
Iflv == inf {[pll> admits (15) for some (v, (v.c,cu)}.
(18)
where || is the total variation of v.
For the compact region f2 = =B we define the bounded variation function class as
Vc(l))== f: f: [
ouu x t) dv((u,)) cT b, 1flv so
(19)
R.]
Specifically, by identifying (15) with the atomic measure = Ex ux (nd(u we have
fflv S Σ Jal = ll.e ll
hence
1flv S Il.f
Conversely, the smallest variation needed to represent equals the smallest path-norm across all finite decompositions,
|| I| = Iflv-
(20)
Thus, the variation seminorm (18) is the nonparametric counterpart the path-norm, which cap- tures the notion of complexity but without fixing the width K.
22

---

Published as a conference paper at ICLR 2026
Remark D.3 ("Arbitrary width" * "infinite width"). Our analysis concerns Fün in (16), i.e., the union over all finite widths. The integral model (17) is a comexifiicttioiillsure of this union that facilitatess analysis and regularization; it does not assume infinite-widd limit. In variational training with a total-variation penalty on v, first-order optimality ensures sparse solutions (finite support of v), which correspond to unite-width networks. Thus, all results in this paper apply to arbitrary (but finite) width, and the continuum measure is only device to characterize and control Ilf min.
D.2 TOTAL VARIATION SEMI-NORM ON RADON DOMAIN
We now connect the (unweighted) variation semi-norm of shallow ReLU networks to an analytic description on the Radon domain. Our presentation follows (Parhi and Nowak, 2021: 2022; 2023; 2026; Parhi and Unser, 2024).
Definition D.4. For a function f Rd R and (u, E gd-1 x ga-1 R, the Radon transform and its dual are defined by
/ [1: f(x) ds(x))
ff(utt)
/ (2) (a) Iga-1 liu, u x do(u).
Proposition D.5 (Filtered backprojection (Radon inversion)). There exists > 0 such that f ~0 (A--1111
where Ad-1 acts in the I-variable with Fourier symbol Ad-1 w) x |us|a- ((),w). Definition D.6 (Second-order Radon total variation (ReLU case)). The (second-order) Radon total- variation seminorm is
- =
where the fractional power is understood via the Fourier transform. Proposition D.7 (Equivalence of seminorms on bounded domains (Parhi and Nowak, 2023)). Let
B By For any B R with finite variation seminorm, its canonical extension fex to TR.
satisfies
Iflv @TV2 and, in particular, for any finite two-layer ReLU network in reduced form fo(x)
Ek 2k Φww.x bx) + cTx t co.
TTV = Σ loul |1
k=
which equals the minimal (unweighted) path-norm needed realize fo B Remark D.8. For ReLU networks on bounded domains, the three viewpoints:
path-norm || path < unweighted variation [flv second-order Radon-TV (TV
equivalent up to affine functions.
D.3 THE METRIC ENTROPY OF VARIATION SPACES
Metric entropy quantifies the compactness the definition of covering numbers and metric entropy.
A in a metric space (x). Below we introduce
Definition D.9 (Covering Number and Entropy). Let A be compact subset of metric space (x) For t > 0, the covering number N(A, px) radius t needed cover A:
) is the minimum number of closed balls of
N
A(1,A,px) = min NEN: AN E X S.I. A
U
(21)
i=1
where B(x,,t) = (y E : St.. The metric entropy of A at scale t is defined as: (A))) log N(1, A,px).
(22)
23

---

Published as a conference paper at ICLR 2026
The metric entropy of the bounded variation function class has been studied in previous works. More specifically, we will directly use the one below in future analysis. Proposition D.10 (Parhi and Nowak 2023, Appendix D). The metric entropy of (B(B)) (see Defi- nition D.2) with respect to the L (B), dissttnct || ll satisfies
2d
(
log (t, Vo(BB) 1lb0) Sd
(23)
where Ed hides constants (which could depend on d) and logarithmic factors.
D.4 GENERALIZATION GAP OF UNWEIGHTED VARIATION FUNCTION CLASS
As a middle step towards bounding the generalization gap of the weighted variation function class, first bound the generalization gap of the unweighted variation function class according metric entropy analysis.
< S M) with M 2 Then let D pen be sampled data set of size n, with probability at least 8,
Lemma D.11. Lei JM.C = (J (B%)
sup [R( Ro(S) Sa Cz M 20.3 n
4+3
+ M²
(log(4/0) 12
(24)
feFM,c
Proof. According to Proposition D.10, one just needs (() balls cover F in || Iloo with radius t > 0 such that where
2d
log (t)) Ed (
Then for any f.g E FM.C and any (x, v).
=) [(((x)) y)² - (g(x)) - [f(x) g(x) [(x) g(())=)))))))))))))) - 2yl S 4M || glloo-
Hence replacing f by centre fi within changes both the empirical and true risks by most 4Mt.
For any fixed centre f in the covering, Hoeffding's inequality implies that with probability at least have
1og(2/5)
[R(f) RD(f))= S 4M²
(25)
n
because each squared error lies in 10, JM Then we take all the centers with union bound to deduce that with probability at least - 5/2, for any center f in the set covering index, we have
log( log(4N(1)/8)
RR() R 4M²
n
(og(4/8) -
C
$ 2
+ M
(26)
t
4F3
Sa M
where Sa hides the logarithmic factors about 1/6 and constants.
According to the definition of covering sets, for any f E FM. we have that ||f flloo S t for some center f. Then we have
[B() - RD(1))
La [R(F) - RD) + O(MI)
(27)
Its n - O
Sa M
After tuning to be the optimal choice, deduce that (24).
24

---

Published as a conference paper at ICLR 2026
E DATA-DEPENDENT REGULARITY FROM EDGE-OF-STABILIN
This section summarizes the data-dependent regularity induced by minima stability for two-layer
ReLU networks.
E.1 FUNCTION SPACE VIEWPOINT NEURAL. NETWORKS BELOW THE EDGE STABILITY
Recall the notations: given a dataset D {(,,, v)) R2 x E, we define the data-dependent weight function ID :S R R
gr(u, 1) = min min (u, (),00(),, -1)),
where
gD(u, --
V ||
X 011
(28)
Here, X denotes random draw uniformly sampled from {ar; 1i=1 that Pp, ED refer to proba- bility and expectation under the empirical distribution > 8n, When the dataset D is fixed and clear from context, we will simply write in place of 9D-
Then the curvature constrain on the loss landscape of C is converted into weighted path norm constrain in the following sense.
Proposition E.1 (Finite-sum version of Theorem 3.2 in (Liang al., 2025)). Suppose that fo(æ)
Ek= Uk @w - by) + 3 is two-layer neural network such that the loss is twice differentiable
at O. Then
by
Amax (V)((0))
uull ||
+RR+ (C(())
(29)
[
2
k=1
we write fo into reduced form (15), then we have
Σ ak I uur tx) Amnac (2((())
+ (R (R+1))/2(())
(30)
2
2
Therefore, we bring up the definition the g-weighted path norm and variation norm are introduced prior work introduced (Liang al., 2025; Nacson et al., 2023).
Definition E.2. Let fo(x)) Ek= i uk ow - bk) + 3 be a two-layer neural network. The (g- weightedd path-norm of fe is defined to be
K
[ | path.g = Σ
o (1110111 lol 1lxxll
wk by
(31)
|[welll
Similarly, for functions of form
fu.e,eg /
- 6lu - t) du(u, t) + cTa + co. x DER"
(32)
where R> 0, CE Rd,, and co R. we define the g geweihtted variation (semi)norm as
Iflv, = VEM(S" x(-R.R)) (((R.R) BERd .oER
inf
11g . vlIu s.t. f = =f.
(33)
where, if there does not exist representation f in the form of (32), then the seminorm is un- derstood to take the value +00, Here, M(SS-1 x R, RJ) denotes the Banach space of (Radon) measures and, for p M(3d-1 x [-R, R)), || Jsa-1x R. dlp (u, f) is the measure-
theoretic total-van variation norm.
With this seminorm, define the Banach space of functions V (g((B4) the ball B = {a Rd ||æ||| < R} as the set of all functions such that Ifl,, is finite. When g 1, lv, and V, (TH)) coincide with the variation (semi) norm and variation space of Bach (2017).
25

---

Published as a conference paper at ICLR 2026
For convenience, we introduce the notation of bounded weighted variation class
( ; M, C) [ Q |Illvv, S S, 11 S <M.. In particular, for any 0 o (S2; M, have E Ff((2; Remark E.3. Vg is a weighted variation space in the sense of Dorere et al. (2025).
(34)
Within this framework together with the connection between llv and (-) as summarized in Section D.2, we show the functional characterization of stable minima.
Theorem E.4. For any fe E ④ BEoS(1)) D), Voolv, = llg ((-)) + Sellm S 1 + (R+
()())
The detailed explanation and proof be found in (Liang al., 2025, Theorem 3.2, Corollary 3.3, Theorem 3.4, Appendix C, D).
E.2 EMPIRICAL PROCESS FOR THE WEIGITT FUNCTION
The implicit regularization of Edge-of-Stabilie induces data-dependent regularity weight on the cylinder gd-1 x R := s [1,, Denote this empirical weight by gD for dataset D { Directly analyzing generalization through the random, data-dependent class weighted by ID is conceptually delicate, since the hypothesis class itself depends on the sample. To separate statistical from algorithmic randomness, we adopt the following paradigm.
()) Fix an underlying distribution for with only the support assumption supp(P) C B fo Rd ||ж||| s R}. Define population reference weight gp on Sd x R (see below). This anchors distribution-leve notion regularity independent of the par- ticular sample.
(ii) For a realized dataset D ~ pan, form the empirical plug-ins that define JD on the same index set gd-1 IR.
(iii Use empirical-process theory control the uniform deviation llgo gpll < with high prob- ability over the draw of 'D. After this step, we can condition on the high-probability event and regard D as fixed in any subsequent analysis.
Let X P with supp C By- For tu, gd-1 x R define
Pp p f) = P u > 1); sp(u, Exup [X u -]+]-
On the unit ball we have (XT l)+ < and |B X All S 1, which yields the pointwise equivalence
gr(u,t) 1) pp 1) (p (with absolute constants).
(35)
Given a dataset D (x; 1 let IPD, Ep denote probability and expectation under the empirical
distribution n Σズ 8n,, Define
(p(utt) t) = PD (
Σ 1 t). sp(u,t) = Ex ( u-+++++
1 n i=1
Σ
x u t)+
and the empirical weight
gD(u.t) x (w, sp(u.t).
(36)
Lemma E.5 (Uniform deviation for halfspaces). There exists a universal constant C > such that, for every 10,
CV c log(1/5) /1)
d
IP
sup
(D pp 1)
< 8.
wE3d-1 161-1,1]
Proof. The class ((x + 11 t}) sd-1, R} has VC-dimension d 1. Apply the
VC-uniform convergence inequality for {0, 1) 13-valued classes (e.g., Vapnik (1998)) to the index set gd-1 x [-1, to obtain the stated bound.
n
26

---

Published as a conference paper at ICLR 2026
Lemma E.6 (Uniform deviation for ReLU). There exists universal constant C 0 such that, for every 8 E (0,
sD(utt - sp(u, t) > Co d log(1/6)
sup wES4-1 6E|-1,11
S 8.
Proof. Let F == (fu.(() = (u"x - t)+ a E gú-1, t E [-1, 1]}. Since ||=|| t E [-,,1, every f E F takes values in [0, 2]. Consider the subgraph class
and
subG(F) = { (æ. v) E Rd x R: S (u x 1)+}
For any (x, 3) with S O, membership in subG(FF) holds for all parameters, hence such points do contribute to shattering. For points with y to u x - t 2 0, i.e. an affine halfspace in Rd+1 with variables (x, y). Therefore the family subG(F) (up to the immaterial fixed set {y S 0}) parametrized by affine halfspaces in Rd+1 whose VC-dimension is at most 2- By the standard equivalence Pdim(F) = VCdin(subG(F)), we obtain
the condition y S (u x t) is equivalent
Pdim(F) S d + 2.
Then by (Haussler, 1992, Theorem 3, Theorem 6, Theorem 7), we
sup |sp(u, f) sp(u, z ov log(1/6) (u.t)
n
with probability at least 5 for some universal constant C, which is the claimed bound.
Theorem E.7 (Distribution-free uniform deviation for 9.). There exists a universal constant C such that, for every ó 10,
d log(1/o)
sup ME31-1 (.11- 1,1]
1 () - gp( CV "
S 28.
Proof. By (35) and (36), suffices (up to a universal factor) to control 0 SD, sp and 0 S PD.PP 1,
|DAD - ppopl. Using
PDSD ppspl lpD
ppl sp + IsD sp pp
lPD
PPl IsD spl
Taking the supremum over (u, () E gd-1 1-1, and applying Lemmas E.5 and E.6 with union bound yields
d dd+ log(1/0)
P
sup u.!
PDOD ppspl Z
S 25.
Recall that for Q E (D,P).
go(u, u, t) = min t), 100000(-00 -t)}.
For any real numbers a, b, c,
min(a,b) b) minfc, d3/ I max{la - c), 1b d|}.
Hence
sup uesd nEgd-1,1E(-10
( - (v((1)))
sup uegd-1 1agd-1,EEf -
G () GD (u, 0)l-
Since (u, 1) + (-u, u. -t) preserves Sd-1 x [-1, bound for ÿD - fr applies to both terms inside the minimum, and therefore transfers gr gp with the same rate and failure probability.
1]. the previously established uniform deviation
27

---

Published as a conference paper at ICLR 2026
F GENERALIZATION UPPER BOUND: MIXTURE OF LOW-DIMENSIONAL BALLS
In this section, present the proof of Theorem 3.10. We formalize the mixture model by the following setting.
Assumption F.1 (Mixture of low-dimensional balls). Ler (Vz) be a finite collection of J distinct
m-dimensional (affine) linear subspaces within IR Let P be a joint distribution over I& x R. The marginal distribution of the features x under P, denoted Px, is mixture distribution given by
(x = Ep, Px)(x) = Px x Vi).
(37)
where Pj 0 are the mixture probabilities P(x V2) satisfying E/= Pj 1. Each component
distribution Pj is the uniform distribution the unit ball B {x E V, ||ا|| I 1}. The corresponding labels y are generated from a conditional distribution P(y x)) and are assumed to be
bounded, ie.. yl < D for some constant D O. Similarly, we define Py( (z, P(x, æ E V2).
First, we prove the simple case of singe-subspace assumption (J = 1) via Theorem F.3.
F.1 CASE: UNIFORM DISTRIBUTION ON UNIT DISC OF A LINEAR SUBSPACE
Fix an m-dimensional subspace V C Rd and write BY fa V ||>|| S 1}, the canon- ical linear projection projy Rd + V. Recall the notations in (1): the parameters 0 (vy, Wk. bk)bk)) 8}
with ur F O, define two-layer neural network
fo(
Σ uk o(w bu) B,
wh
W
by
bk
ww
l
1 l12
Then we define neuronwise projection operator from neural networks to neural networks
projv: fo(Φ) + Σ uk @(proovv wx) Jx bx) + B.
(38)
k=1
Lemma F22 (Projection reduction). Fix F hyothesis class of two-layer neural networks. Let P be joint distribution on (z, y) supported on Rd x [-D, D] such that the marginal distribution Px of a supports on V. For any dataset D = (x,; yi )i)) drawn ii.dd from P,
sup Gapp sup Gapp(projý J; fEF
(39)
fe
Proof. Because V almost surely and in the sample, have f(x) (f projy
(x) for every
I and every x E BY. Using the identity wi (projy x) (projy wxx ) xx we obtain o projv projv f pointwise on BY Hence for any f E F. (f: (f: D) = Gapo(projv f:D).
Theorem F.3. Let P denote the joint distribution of (x, y). Assume that P is supported on B x
[- D., D) for some D > and that the marginal distribution of x: is Uniform(Bi (B Fix dataset
D { y= where each (x Vs) is drawn i.i.d. from P. Then, with probability 2 1188 8,
(log( |log(4/0) (4/6)
Gapp(Ja; So Om2+tm+3 M n + M²
sup feE (3 M.C)
12
where M == max D, (B)) and Sa hides constants (which could depend on d).
Proof. By Lemma F22, it remains to consider projý fo. Let D C and let X be supported on V.
For any d with projy (w) + 0, set
w
projy
u =
w
uy =
projy (w)l
28

---

Published as a conference paper at ICLR 2026
Since for all a E V.
w x = projy(w)) x.
have
w
x > b
+ projr(w) x b.
After normalization,
b
b
u
x
u >
||
(rojv(w)]
Hence
w
projy
b
g
||w|||
projy (w)ll projy (w)ll
Therefore, when the data are supported on V, the weight function depends only on directions in V. It suffices to consider the class
JD BB V M.C) = ( (rrojt fo fo E 9 90. M, (B, C).
Therefore, just need consider the case where the whole algorithm with any dataset sample from V operates in V and we get the result from (Liang et al., 2025, Theorem F.8) by replacing TRd with V = m
F.2 PROOF OF THEOREM 3.10
In this section, we extend the generalization analysis from a single low-dimensional subspace to a more complex and practical scenario where the data supported on finite union of such sub- spaces. This setting is crucial for modeling multi-modal data, where distinct clusters can each be approximated by a oow-dimeniional linear structure. Our main result demonstrates that the sample complexity of stable minima adapts to the low intrinsic dimension of the individual subspaces, rather than the high ambient dimension of the data space.
F.2.1 ANALYSIS OF THE GLOBAL WEIGHT FUNCTION
A critical step in our proof is understand the relationship between the global weight function g(u,t), t), which is induced by the mixture distribution P, and the local weight functions gy gy(u, t), each induced by single component distribution P, defined on Vj. which should be understood as the distribution conditioned to E Vj- Fix a dataset D, the function class θ (E):(()) D) is defined by the properties of the global function g. To analyze the performance on a specific subspace Vj. must ensure that the global regularity constraint is sufficiently strong when viewed locally. The following lemma provides this crucial guarantee.
Lemma F44 (Global-to-Local Weight Domination). For any mixed distribution Px E P,P with supp(Px.,) Vz. Let g be the global weight induced by the mixture Px, and 93 the weight induced by Px.x For every j E [I....,J}.
p
g(u, t) 2
V2 22 (), for all (u, E 5d-1 x R.
(40)
Consequently, for any M, M,C> 0.
F ( c (B M, V2C/203)
(41)
Proof. Fix j and the activation event A:= = {x u x > t}. By definition of g (global) and gj (local) we can write
g(u,t) (
E. [ ||A]
1+
E 2~Px
X Alll
BNPx
/ +1+1
(u Px(A 2
E
|
A Vil+
A, x E Vyll X
B>Px
x~Px
-A)](A)2.
E xxPx,
A]
1 +
E
X
ALL
29

---

Published as a conference paper at ICLR 2026
Using the law of total probability and total expectation for the mixture distribution Px Ei P.Px,
x i and the non-negativity of (XTu 1) we get
(x(A) 2 Py (A),
E
[X u
t)1Al 2 Pj
E
[(x u t)1A].
P
Hence, by combining the first two terms of g(u, Px (x RaNPx [X t)la), we have:
g(u,t) (p,Px.s(A)) (p)
(X"u 1 p] PA)(A)
1x'uu
T-Pxx
amPxs
For the local weight function Jj. the same algebra gives
gy(u,t) = (A)
E [X u t)LAl
X
A]]
P Px.
Since the support of Px.j is BY we have || 1 almost surely under Px.. This implies
RewPPx [X] Allla I 1, and therefore / + EamPxx [
AA]l S v2.
Combining these results, we establish the lower bound:
Pj 2
PJ gu L),
g(u,() () 2
(A)
E [x u (()'=l)aa)
R xxPx
X
A]l
v2
xxPxx,
which proves (40). The class embedding (41) follows directly from the definition of the weighted variation seminorm.
Proposition F.5. Let P be a distribution defined in Assumption F.I and recall that Pa is P condi- tional to x Vj. Fix j E [I, and data DD pon Let Dj = D V3 and ny 1Dyl. Then with probability - 6,
4M 2
2m14 M² 1og(4/6)
sup fo BE
Gapp, (so: Do) Sa
M n
Pj
(42)
where M = max D, fo (z) 1) and Sa hides constants (which could depend on d).
Proof.
Note that the notation Gapp, (fo: Dj) can be expanded into
Gapp, (fo:
Rp, (fo) Ro, (fo)
([f((z)-))] -)] - -RDD (fo)
E (a.))>>;
(fo(z) y)² [ LeEV;] vs] RD, (fo)
(w.g)~p
Let C = 4M. According to (Liang al., 2025, Corollary 3.3), we have that
fe O (B M,C),
v0 E O BEoS((M) (7; D).
Then by Lemma F.4, we conclude that
M,)
where the weight functions g and gj can be either empirical or population.
Therefore,
sup @EO
Gapp, 5
sup see,, ( (2Ce/33
Gapp, (J:
30

---

Published as a conference paper at ICLR 2026
Then by Theorem F.3, may conclude that
1 1 + +M
mn 1.
sup fo 1B 2200/p3)
Gapp, (fo:D;) Sa
2
M² m 2m+4
pj
Theorem F.6 (Generalization Bound for Mixture Models). Let the data distribution P be as defined in Assumption I. Let 'D ((a),,;))) y:)) be dataset of ii.dd samples drawn from P. Then, with probability at least 26,
GSpp(So:D) Sa () ( +M)
1
20 m +im+3 M Jm
log (og(4.1/)
sup 0 (1.2)
2m44 + M2 J
2n
(43)
where M = max fl lay ||ا>0 1} and Sa hides constants (which could depend on d).
The proof proceeds in several steps. First, we establish a high-probabilityy event where the number of samples drawn from each subspace is close to its expected value. Second, we decompose the total generalization gap into several terms. Finally, we bound each of these terms, showing that the dominant term is determined by the generalization performance the individual subspaces, which scales with the intrinsic dimension m.
Proof. Let nj Σn (x,EV,) be the number of samples from the dataset 'D that fall into the subspace Vj. Each nj is a random variable following Binomial distribution, n, ~ Bin(n, We need to ensure that for subspaces simultaneously, the empirical proportion nj/n close to true probability P...
We use Hoeffding's inequality for each j JI. For any
0, P( n
2e 2nc². To ensure this holds for all subspaces at once, we apply a union bound. Let 8, be the failure probability allocated to the j-th subspace. The total failure probability is at most ΣØ=1 63
[og((2//)) 2m
5, so we set 6, = J/J and yields
Let be the event that n2 Pyl S c holds for 1....,J. We have shown that P(E) 2 6. The remainder of our proof is conditioned on this event E. A direct consequence of this event is lower bound on each n;
2J 8
n 2 np, npp
log
(44)
Now we decompose the generalization gap using the law of total expectation for the true risk and by partitioning the empirical sum for the empirical risk.
31

---

Published as a conference paper at ICLR 2026
Let Py denote the distribution P conditioned Vj, and let D, = D'v)).
Gapp(fo:D)
R(fo) - Roo(fo)
Σ Pj E [(f(x)) - -)) x I EEV
Bj Σ (f(x:) y6)
a~p
j=l
j [(fo(x)= - -))
Σ
Σ (fe(x;) - -vi)
Pj
E
Pj
==1
)=1
J
Σ (fo(z;) - -y:)² 2--
n2 Σ (fo(z;) - y6)
Pj
m2
n n2 (m..9))D))
(IL:
j=1
J
j
Pj
Rp, (fe) - RRo((fo)
+
Pj
n2
RD, (fe)
5=1
J
Pj Gapp, (f):
+
Pj
n2
RD, (fo)
1=1
==1
where Ro,(5) = N, n;
EOE,,EE (f(z;) - -,))
Bounding the Weighted Sum of Conditional Gaps (Term A): According to Proposition E.5, with probability at least 1 8, for each j,
+ 4M
2m44 + M² (og((4J/0)
Gapp, Dj) Ss
4.4m
M nj
P3
Conditioned on E, we use the lower bound on ny from (44), ny S np;(1 - e/pj).
Term A
pj Gapp, (fo:D,)
1=1
J
+ 4M
M (1-pp((1)) e/pp))
2m+4
Rdd
Pj
p3
-
/=1
log (2.(8)/))
2mm
+
4M
m 4m
M
2m
Pj (p) 2) m 3+im4
Pj
17
2
2n
j=1
Ed
1
1
+ 4M
m +3
m
2
2m47
Pj
1-
m 44m++ 2m44
2m
17
2
j=1
The exponent of pj simplifies to
2m
1
2m3 + 7m² 2 + 10m +
(45)
(m + 1) + 3)
2m + 4) 2(m 1)(m 2) + +3)
For positive integers m. (45) strictly increasing and bounded above by 1. In particular, when m 1, (45) 12. Therefore, brute-force upper bound is
7
2m 34m +-10m Hon T1X 12 137
Σ
Pj
<J
32

---

Published as a conference paper at ICLR 2026
and thus
1
2m m1 11-m4 13
Term A ad
+ 4M)
M n 2m44
Pj
17 2
f=1
2d
+ 4M
1m
M n2
2m+44
17
Note that the dependence of Term A on J is very mild. Indeed, if we denote
2m
a(m) = mn² + 4m +
2m + 4
then
a(m) Pj
S J (ma(m))
mm==mm+3
2me S J
a(m) since D, For large m, the exponent am)) is close 1, hence E, Pj remains essentially of order one. Consequently, the bound Term A grows most linearly with J. and in practice the J-dependence is negligible in high m. Here we use the power 4/m upper for clean format.
Bounding the Sampling Deviation Error (Term B): Conditioned on the event &, we have Pj ny/n| for all j. The empirical risk term is bounded because max {[f(z)),, [y] s y:)² I 4M2. Thus, Term B is bounded by:
M. which implies ب my (E,,)ED (f)())))
log (4J/6)
Term S Σ AMM 4M2 4JM²
(46)
2n
The total generalization gap is bounded by the sum of the bounds for Term A and Term B.
Gapp D) Sa _ 1 + 4M
(oJ(4)/))
M J th n 2m + M J
2
2n
This completes the proof.
G GENERALIZATION UPPER BOUNDS: ISOTROPIC BETA FAMILY
In this section, the data generalization process is considered to be family of isotropic Beta-radial distributions.
Definition G.1 (Isotropic Beta-radial distributions). Let X be a d-dimensional random vector in Rd For any E oo), the isotropic (etaaaa)-radilll distribution is defined by the generation process
X = h(RJU Px(a),
(47)
where Uniform(0, is random variable drawn from a continuous uniform distribution on the interval [0, U ~ Uniform(@d-1) is random vector drawn uniformly from the unit sphere gd-1 in Rd and h(r) = 1 (1 rr//// is radial profile.
Lemma G.2. Let Px (x) be the isotropic Beta(a)-radial distribution in Definition 3.2. For X t) = In particular, ||||| la a Beta(1, distribution.
(a(a) any E [0, 1]. P X|I
Proof. The proof follows from a direct calculation based on the properties of the data-generating process.
First, the norm simplifies to: probability that the scalar random variable h (R) greater than 1 - t:
X
| = (RR). Next, it equivalent to calculating the
P X
>
t) P (h 1)-
33

---

Published as a conference paper at ICLR 2026
To proceed, need to apply the inverse of the function h to both sides of the inequality. The func-
tion h((r is monotonically increasing for E [o, 11. so applying its inverse preserves the direction of the inequality. Note that the inverse function is h -(y)
y)"
Applying the inverse function h-1 1 to the inequality h(R) > t, we get
R> h 1- -
Substituting the expression for h h-1:
R> 1 - == ( 1
Finally, we compute the probability of this event for the random variable R. By our initial assump- tion, R is uniformly distributed the interval [0, 1], i.e., R Uniform(0, 1). The cumulative distribution function (CDF) of R is (x( = x for r E [0, 11. The tail probability is therefore.
F(R > x) = FR(x) = -1-6 xt
Applying this to our inequality
> 19:
P P > 1 (a) = =1 (1-
Combining all steps, we have rigorously shown that
TP ( >1
To show that this implies | is Beta(1, a) distribution, we can examine its cumulative distribu-
tion function (CDF). Let Y= || |. The CDF is (3) = P(Y s y). Substituting = 1 t, we have y. Then the tail probability becomes:
X[| v) = (1 (1-y)².
From this, the CDF can be derived as
(v) X| < 4) =1 P(| v) = 1 (1-
This is the characteristic CDF of Beta(1, a.) distribution, thus completing the proof.
Assumption G.3. Fix a (0, 20). Let P(a) be a joint distribution over Rdd x R such that the marginal distribution of the features is Px (a). The corresponding labels y are generated from a conditional distribution P(y(z) x)) and are assumed to be bounded, i.e., lyl S D for some constant D O. Similarly, we define Pz(æ, v) P(x, y Vj).
G.1 CHARACTERIZATTIO OF THE WEIGHT FUNCTION FOR A CUSTOM RADIAL DISTRIBUTION
In this section, we analyze the properties of the weight function (u, 9Px, (u, t) with respect to the population distribution Px,a defined in Definition 3.2 and Assumption G.3. Recall that 9. (u, t) = min (ō., (u. t), ( (-u,- -t))), where
aa (u. t) = Prx... (XT u > 1)2 Exx. [X u - x u > t) V Epx X
XTu u
(48)
Due rotational symmetry, analyze the projection X₂ XTe, without loss of generality. Our primary goal to establish rigorous bounds on the tail probability Q(1) == P(Xx > 1) and the conditional expectation for in a specific range close 1.
Proposition G.4 (Tail Probability). Let X be random vector from the distribution defined above. Let Xa be its projection onto a fixed coordinate, and let its tail probability be Q(t) = TPXX, > t) for E (-1,1). Then there exists a fixed E [0, 1) such that for all [to, 1):
c2(0, d)(1 + < Q(1) < ca(a, d)(1 10++ + +
where ca(a, d) and ca(a, d) are positive constants depending and d.
34

---

Published as a conference paper at ICLR 2026
Proof. The tail probability is Q(t) distribution R Uniform(0.. (0, [0, 1]:
P(h(R)Ua t). We compute this by integrating the
Q(1) / (1)
((Ua>1/h(r))
where the lower limit h- (t) 1 (1 t)a ensures h(r) t. The term P(U > x) is the normalized surface area a spherical cap on gd-1 For a [0, 1), this area can be bounded.
Let do ar)cos(x). The area is proportional to 00 (sin d) dø. For ø [0, 1/2). have 20/n S sin o S Φ. This provides lower and upper bounds on the cap area
- (a.((1=))(d-1 x) I P(Ua > x) S Ca,u(1 -
20) x)(d-1)/2
where Ca. and Cayy are constants depending on d. Let's apply this to our integral, substituting 1/h(r):
Q(1) 2 / h-1(1))
CA.L (1- (1-n(r)
(d-1)/2
dr.
h((r)))
We analyze this for t + 1 Let t = 1 - e. The lower limit is 1 --e For r E [1 - E², 1,, h(r) is close to 1. Let's choose to such that fort E [ta, 1), h(r) 2 h(tg) 1/2. Then h(r) is bounded away from O. The term - t/h(r) = (h(r) - t)/h(r). Let's bound the denominator: h(to) - h(r) s 1.
Q(1) 2 Ca. f 1-
(M(r) (1 - -() dr.
The integrand is h(r) - c) (1-r)2/a The integral becomes:
1-ca (e (--11 --(1-))/a) (d-1)/2
dr.
Let = (1 - r) 80 r 8 and dr = -aya-1 dy. Limits for y are [e, 0].
[ (e y) (d-1)// (-ayaa dy) 1/" (e w) dy.
Let y ez, dy dz. Limits for z are [0, 1].
a J. (c (e-ez)(d-1))) ((===))d-1/((z>))>-1 edz ac a+ 4
d d+1
B
2
Combining all constants, we establish the lower bound Q(t) 2 c2(a, d) +) The upper bound follows an identical procedure, absorbing the 1/h(r) term into the constant ca(a, d).
Proposition G.5 (Conditional Expectation). For E [oo 1), the conditional expectation E[Xa Xa t] is bounded by
(a, d) < E[Xa X. > 4 5 cala, 1),
where cafa, d) and ca d) positive constants.
q From Proposition G.4, we know Sx, (s) x - -s)
Proof. We analyze E[1 X. X 1
= S:((
s))x((s) where fx,(s) (s))s) d-3 The numerator is:
N(t) S. (11- s)xxe ds.
Bounding the constant of proportionality for fx,(() by Cr and cI
C15 S (1 s) E ds S N(I) s CTU f (1 - +f ds.
35

---

Published as a conference paper at ICLR 2026
The integral evaluates to (1-1)0+ 44 at das
So, N(t) x (1 - t) $
Dividing ((t) by Q(t) &
(1 t)at = we get:
(1 t)a+ 1=1
E - -x Xa t x
1 t.
(1 t)a+
By carefully tracking the constants C2, C3 from Proposition G.4 and the constants from the integra- tion of (s), we can construct explicit (though complex) expressions for c4 and that provide rigorous two-sided bounds for the specified range (o. 1).
Proposition G.6 (Asymptotic Behavior of go (t)). Let the function 9 (t) be defined in (48). Then fort E [to 1), we have:
(a, - 1)2atd S ot (1) s c (a, 1)2a+d
where cg (a,d) and d (a,d) are positive constants.
Proof. Let Q(1) P(Xa 1) and E(I) E|Xa Xa (]. The function is st Q(1)² (E(t) t) v1+ E(t)? We establish bounds for [to. 1) for sufficiently large to-
1. Bounds for Q(t)² From Proposition G.4, we have:
(c2(a, d))² - 2) < Q(t)² s (cs(a,d))²( - t) 20 Let A d) (c2(a, d))² and (v d) (ca(a,
2. Bounds for Et) t: This is E[Xa -t X t). From Proposition G.5, have
(1 (1-th t) cs 1 - t) & E(t) -t - (1 -) - c1(1 -t).
t). This gives:
By d)
-) S Ett) - t (a,,d))(-t))
where By, d) 1 co d) and (a, d) = 1 c( d). We can choose to close enough to to ensure these constants are positive.
3. Bounds for /1+ E(t)²; For t [to, 1). we have to St < E(t) 1. By choosing, for instance, to = 3/4, we have 3/4 I E(t) I 1. Thus,
V1+ (3/4)² - VI E(t)² & / 1
This gives constant bounds C₁, = 5/4 and Cv = V2.
Combining these three bounds, for E [lo, 1):
A₁ (1-CC(111atttt (1 t) S Ia (t) S Au -1 -t). t). This simplifies to the final result:
22tdd-111
(a d) - t) 2add < gt S d (a, (l.d)(1 - t)2a+d
(a,
where the bounding constants are given by (a, d) A,B and d (a, Au Cu.
G.2 PROOF OF THEOREM 3.4
Theorem G.7 (Restate Theorem 3.4). Fix a dataset D = {1 be) where each (æ,, Wi) is drawn i.i.d. from P(a)) defined in Assumption 3.3. Then, with probability at least for any fe OBBoS(n, D),
( [1- + 4M
2 Tod+
old+3)
Gapp(fo: D) Sd
"7
M
2
n
ald +od433))
a
2
2d-3i
(49)
(1 + 4M
2444d=3
od
2 Tod+6n
24
2d-31
and for where M = max D, llfel (.x 1} and Sa hides constants (which could depend on d) and logarithmic factors in and (1/8).
36

---

Published as a conference paper at ICLR 2026
Proof. For convenience, we let A = 1 - + 4M and have that 4) fe E F MM, C), 40 E BEOS ():
For any fixed 1, we may decompose BB into s-annulus A = for Bf the closure of its complement is called E-strict interior denoted by Ilg = BB
||æl|2 2 }) and
B - A U..
According to the law of total expectation, the population risk is decomposed into
(1(())
Ip E AZ) E& (f(())-)) - -)) + P(x E 12) Er [I(()) w)² (50) u)² -)²
E (m
where Eg means that {x, y is new sample from the data distribution conditioned E Ag and Eg means that (a, y) is a new sample from the data distribution conditioned on æ E Itd.
Similarly, also have this decomposition for empirical risk
(f(a)) =
1
Σ
(f(m)) v1)² + [(((m)) - vi)²
1=1
SE
JEA
(51)
n1
E(((o)) - wi))
nA
(f(x))) - y:)
n nJ
n
na
REI
jEA
where 7 is the set of data points with æ; Tg. and A is the set of data points with w; Ag Then the generalization gap can be decomposed into
R(f) - RD(f) < Pax Ag).
Eg (fo(z)- y)² -))
nA 1
E(f(x,) y1)²
(52)
n T2 jEN
n2 1
+ P( E (&)
Σ (s(z)) - y²
(53)
n
SEI
+ P(x 12)
Ej [(((x)) y)²]
(((x)) -i)
(54)
TE!
Using the property that the marginal distribution of x is Px Px and its concentration property, with probability at least 1 6,
(52) Sa O(M2g)),
(55)
where Ea hides the constants that could depend on and logarithmic factors of 1/8.
For the term (53), with probability 1 8
IP E T ny 1 (f (2x)))0) y1)
S
log(1/8)
n
(56)
4
so we may also conclude that
lloo(118/
53
S M²
(57)
For the part of the interior (54), the scalar IP(x E Ie) is less than I with high-probability. Therefore, we just need to deal with the term
E (f(z)) -y>²
1
(f(x)) - 2.
(58)
ni
37

---

Published as a conference paper at ICLR 2026
Since both the distribution and sample points only support in Ii may consider f by its restrictions in I2 which are denoted by f. Furthermore, according to the definition, we have
f(x) = [ S of u x t) dr(u, t) + +c
| -x--1x--1te--d
p(u' ()d (
6(u' 10-1
1) (u)
Annulus ReLU
+ x
(59)
where the Annulus ReL.U term is totally linear in the strictly interior i.e. there exists c', V such that ouu x 1) de(ee
dT b'
- J 1te)((e-e))
Vo E 1g.
(60)
Therefore, may write
f(x) (x) -
slu - t) +c + ny. (61)
According to the definition, we have that
1 & f 1xl-1x(-1te,122
|dv.
(62)
From empirical process we discussed in Section E.2, especically Theorem E.7, we know that with probability least 1 5,
d log(2/8) En.
sup (g((u,))) Ica (u, Sa V
(63)
n
This implies a lower bound on the empirical minimum weight in the core with probability at least 1 8/3,
gD inf go (u, () 2 inf (a, 1) 6n Jo.min Ep. |r|Si-e"
(64)
I4ST-e
Here, Io,min 20 is the minimum of the population weight function in the core.
For the bound lf S A/gD,min I A//Agammin - En to meaningful with high probability, we must operate in regime where mai Z En. We enforce stricter validity condition for our proof
+ 16g(6/8)
mi. 2 2Ep
d+22
(65)
Under this condition, we have JD, mim 2 9a,min Cn 2 9a.min/2 2 x gd220 Thus, for any f e + BEaS (1), D), its restriction fe has controlled unweighted variation norm with high probability:
A
A cd+2a
| Iv(E,_
Cc-
JD.min
maimin/
According to the assumption, we have that (nF) I A, and thus we have
f Jga-1x[-111] gol dul < A.
J=--xx(144.1- J gol dv <
(66)
Suppose the validity condition (65) holds (we will verify it later), we have g(u,t) Za ed+2a when t < 1 e with probability 1- 8/3, we may use (66) to deduce that
{ ggd 1x[-148,1-5] Mpl del S A.
2+
dul <
(67)
Jgd 1x(-1+
Combining (62) and (67), we deduce that
1f (flvvL Ea Ed+2a =: C.
38

---

Published as a conference paper at ICLR 2026
Therefore, may leverage Lemma D.11 to fo Vo((B_)). may conclude that with probability at least 1 6,
S(4+2) 4=3 (54) E Cais M
2013 n WाC
(68)
where Ed hides the constants that could depend d and logarithmic factors of 1/6.
Now we combine the upper bounds (55), (57) and (68) to deduce an upper bound of the generaliza- tion gap. We have for any fixed > 0, with probability 1 1 8,
RR() RD( La M
A Ed+20
3(4+2) M 2043
6+33 1646
(69)
Then we may choose the optimal 8* such that
A M = (E+)
3(4+2)
d+3
M 2213
4210
and by direct computation, we get
(A #41od430
443
M
2 #4iod430 n ((0-1nd+3))
To satisfy the validity condition (65), we require
(=")a+2a= = =0
4+3 2(22 2(d2+4dd+3a)
d+2a
2 Ô(n Ōtn
(70)
By adjusting some universal constants, it suffices to show whether
(d + 3) 2a)
(71)
2(d + 4ad + 3a)
After direct computation, (71) is equivalent to E (2d-3- co). With this assumption, we may evaluate the optimal E* in the inequality (69) to deduce the optimal results that
|R() ( {
1 + +MM
4043
M d'4ad-Jn
222 17ad
20d 4nd Se))
(72)
In the where <
3d 7=20
set
6" = कककनक
and adjust some universal constant to satisfy the validaty condition. Then (69) has the form
[R() - (o))) S (n
2a zatto
+
2446
Then assumption a
3d d4-22cr
implies that n 2d > n zdio and thus
R(f) RD(f)s δ
Note that the other constants in the front of 1/n does not change, so we finish the proof.
H GENERALIZATION GAP LOWER BOUND VIA POISSONIZATION
This section provides self-contained proof for a lower bound on the generalization gap in a noise- less setting. We employ the indistinguishability method, where the core technical challenge is to construct two functions that are identical on given training sample yet significantly different in population. The Poissonization technique is the key tool that simplifies the probabilistic analysis required to guarantee the existence such a pair. The paradigm is almost the same as the one in (Liang et al., 2025, Appendix H & I), but the assumption distributions are different.
39

---

Published as a conference paper at ICLR 2026
H.1 CONSTRUCTION OF
NETWORKS
Our strategy relies on functions localized on small, disjoint regions near the boundary of the unit ball. We first establish key geometric properties of these regions, called spherical caps. Let u E g be a unit vector. Let E E R4 be constant with - 1/2. Consider the ReLU atom:
Pu, o(u (1-1 (1--=)).
(73)
Lemma H.1. The (P (nrm of Pu,ea, where the measure Px (a) is defined in Definition 3.2, is given by
cr(d,a)e
443+20
S &lluee (PP.ee)) S (c, a.)e
(74)
where co (d, a) and cu cu(d, a)) are constants that depend on the dimension d and the parameter a.
Before the formal proof, we offer a geometric justification for the result. The squared norm an integral of ()², and we can estimate its value as the product of the integrand's average magnitude and the measure of the small domain where it is non-zero. We estimate the measure of =², using polar coordinate perspective.
this "active" domain, where ru U
Integrand's 's Magnitude: Within the active domain, the term ru (1- the positive "height" above the activation threshold. This height varies from O to maxi- mum on the order of (s²²). reasonable estimate for the squared term's 's average value is thus (((=2)²) (saa).
=²) represents
Measure of the Domain: We decompose the domain's volume into radial and angular parts.
The condition requires the radius r to be near 1. For the Px(a) distribution, this confines region of length Ar (
Radial Measure:
The vector U confined to a small spherical cap around u. A cap defined by "height" of h (sa2) has a surface area Sd-1 of order ((h(d-1)) 1)/2). This gives an angular measure of An (((=))dd-1)/²)
Angular Measure:
(ssd-1).
Combining these estimates, the squared norm scales as the product the integrand's 's magnitude and the two components of the domain's 's measure:
Rd (ce2a)
O -
(gd44342a)
Integrand
Radial
Angular
Taking the square root provides the claimed scaling for the noomm. The formal proof makes this geometric heuristie rigorous.
Proof. The squared L norm of Pu,c² over the distribution Px (a) is defined by the expectation
I ll [((()(())) Ex~Px(a) [|vu, s² [
Substituting the definition of 2 (x:) and using the property of the ReLU function, we get
I ER, (o(h(R)uTU - (1-=))))
(75)
= ER,U [LCCE))OSI-+ (h) - [1)=
where Uniform(0. 1] and Uniform(Sd-1)
Due to the rotational symmetry of the distribution of U, we can perform rotation of the coordinate system such that u aligns with the d-th standard basis vector ed (0, (0....,Q, 1) without changing the value of the integral. In these new coordinates, u TU Ua. The expectation becomes an iterated integral:
I = {' Eu (1a0)sss---)))(())) - dr
40

---

Published as a conference paper at ICLR 2026
Let Us. The probability density function of is p(2) Ca(1 2 22)(d-3)/2 for [-1, 1),
where Ca / The integral is non-zero only if h(r) > E², which implies >
r(d/2)
1 20
1 1 Sr
S 1-22 Tipoy
(h(r)=-(11-)1 - (1- (1- - =²) - dz dr
(76)
We perform change of variable = - 1, d= dtt and the integration limits change from
to [1 h(r] 0J.
Inner integration of (76) Ca / 1 -
(-d()
(77)
= Ca /
po(rr) (h(r))a(r)) - - - 12) dt
where (o(r) = 1 1-c² h(r)-1+e* Since r E c 1]
and for small 6, h(r) is close to 1, we know to is small. For sufficiently small E, we can ensure < to (r) < 1/4. Thus, we can bound
h[r]
h(r)
the term t as 7/4 S 2 5 2. This gives bounds on (2t (2) (d) ((2 (d-3/22,
(1)
S (2t- 2) 52
The integral I is therefore bounded by:
f 1-22 J(r) S STS I dr
Ca
(78)
11220
where Ca. Oa are new constants and J(r) Jo |( (h(F)to()) - h(r)))2+ x dt.
Consider the integral J(r) and change variable by setting = tu(r)s then dt = (r( ds.
J(r) [. (M()))()) (())6()))))((((())))) 1-2 (l)
(M))6)()) o to(r) / (1 5)3 1ي3 ds (1-))/0
(79)
= (((r)(()(r))) 4t (1))) a-3 ds
constant
To analyze to(r) 243 we let r 8, so dr dô and the integration limits for ó are 1s20 0j.
1-8-/a h(r) 1 (1 (1 - 0))2/a - -///a As + 0, h(r) + 1. to(r) --8-/-)1--- 8////
1-5+/a
For small 6, 1 - 81/a is close to I, providing upper and lower bounds. Thus / is bounded by integrals of the form
C e2a
( (8-88%)
(8
433 (-d6)= C S. (6 (8-
4+3
d5
for some mild constant C.
Now we perform a new change-of-varian by setting 81/a = 2v.. This gives 8 = (62r)a = g20
and dð = 2 aa dv. The limits for v become
10 (5² (52- (22-81/a)
( (62 =2)) 2/3 (ae aa du)
do =
4 2a (']
(e2)
(1 1- v) - am de
(80)
j0
2-3 a (1 - v) 4 -a- du
constont
41

---

Published as a conference paper at ICLR 2026
The squared norm I is bounded by constants times gd+3+2n The -2-orrm is the square root of I:
cr(d, a)) d+ < Il (| = vI s (d, 2+3-20
(81)
where cg(d, oi) and (d, or) constants that absorb all factors depending on and a from the bounds established in the derivation. This completes the proof.
Lemma H.2 (Cap mass at angular scale E). For (0, {) and gd-1, define the thin cap
C(u, E) = {a E B$ uJx > 1 - =²}.
There exist constants depending only on (d, a), such that Px (C(u,e)) x-d-1220
Sketch proof. The result and the proof almost the same as the ones about Lemma H.l. We omit the calculation details.
Lemma H.3 (Disjoint Cap Packing), For any E 1/21/2]. there exists set of N unit vectors (ui.... uN) C gd-1 with x e-(d-1), such that the caps (C(us, e) are pairwise disjoint.
Sketch proef. The angular radius of the cap C(u,E) is 2 arccos(1 - E²) x For two caps to be disjoint, the angular separation between their centers must be at least 28. The maximum number of such points is the packing number (S 20). A standard volumetric argument provides the upper bound M(dd1), 20) O(E-(d-1) (d-1).. The lower bound is established by relating the packing number the covering number (Sd-11, a), which is known to scale as N (Sd-11 a) x a (d-1) thus yielding the asserted scaling for N.
We now formally establish the family of functions used to construct the adversarial pair. This family resides within a function class F, 1, and is built upon normalized ReLU atoms localized on the disjoint spherical caps.
Construction H.4 (Adversarial Function Family). Recall that Pu (ax) slu (1 e²)).
We define its normalized version T u.,= 2u, By construction, T Pu (1 Il S
and 1l x = = 2(d-1+2u) We assume that these normalized atoms, for sufficiently small e, belong to our function class Fg (B 1)
Let (ui...., 3ND be the set of vectors from Lemma H.3 that define a disjoint cap packing. We define a family of candidate functions indexed by sign vectors & E [+1] For each &, the function fg F is given by:
fe(x)) = Σ (.)( where &; == Pu..e2.
As the atoms de have disjoint supports, the squared (2(Px(a)) distance between any two distinct functions fe and fe. can be computed as:
N Σ - - Σ | (2a()(()))
IIfe fe fell
1=1
Referring to the cap mass properties in Lemma H. (which implies || |0,,1((((xx()) ed-1+2a), this simplifies to the final distance scaling
[se Je (22 Saa e d- d-14+2 dnr(Ege),
where dir E') is the Hamming distance.
42

---

Published as a conference paper at ICLR 2026
H.2 PROOF OF THEOREM 3.5
A key step in our proof is to find a large number of caps that contain no data points from the dataset D. In the standard fxed-sample-size setting, the number of points in each disjoint cap, say Z == # # C(u,,E) }) follows a multinomial distribution. The counts (Zj...., Zv) are negatively correlated because their sum fixed n. This dependence complicates the analysis of finding many empty caps simultancously.
To circumvent this difficulty, we employ Poissonization. We replace the fixed sample size with random sample size Npoi drawn from a Poisson distribution with mean n. This means the occupancy counts Zy become independent Poisson random variables. This independence allows for the direct use of standard concentration inequalities like the Chernoff bound.
Proposition H.5 (Abundance of Empty Caps under Poissonization). Let {C(u;, } be the set of disjoint caps from Lemma H.3. Let the sample size be Npoi Poi(n). Let Ze be the number of samples falling into cap C(u,,E. Define the expected number of points per cap as X == n Px P ()) If we choose & such that 1, then there exists a constant e. > such that with probubility at least 1 exp(-cN):
#(i E (1....,,NN Z, =0 2 A
2
Proof. Under Poissonization, the random variables 2, = # # E E(u,,E)) are independent Pois- son variables with mean A; = P (C(u,,))) By Lemma H.2 and our choice scale, X x for all i.
Let Y, 1 1[2; 0} be the indicator that the i-th cap is empty. The variables Yı.... Yv are i..d Bernoulli random variables. The probability of success (a cap being empty) is:
e xo =
p = P P
0!
Since À x 1, p a positive constant. The expected number of empty caps is E| Np Ne A By a standard Chernoff bound on the sum of i..dd Bernoulli variables, we have that for any E (0, 1):
82 82NN
P Σ Y, < (1-)))p
S exp
2
Choosing 5 = 1/2, we find that the number of empty caps is at least !N = - with proba- bility at least exp(-cV) for some constant e 0.
The condition À is central. It balances the sample size with the geometric scale Using Lemma H.2, this balance is achieved when:
n d-142a
1
n 1d=1))))
(82)
With this choice, Proposition H.5 guarantees that a constant fraction of the x (dd-1) caps are empty with overwhelmingly high probability.. Informlly speaking, this hints appearance of the neural network with dedicated neurons, each of which has at most one activation point. This paradigm aligns with our construction stable/fiat interpolation neural network discussed in Appendix I.
Armed with the guarantee many empty caps, we can now construct our adversarial pair func- tions. and f'. These functions will be designed to agree on all non-empty caps but disagree large number empty caps. Since by definition no data lies in the empty caps, the functions will be identical on the training data. However, their disagreement on substantial portion of the space will create a large gap in their population risks.
Proposition H.6 (Indistinguishable yet Separated Pair). Work under the scale choice e x -1/(d-1420) and on the high-proban probabiiyyy event from Proposition H.5 where at least 3ec > caps are empty. There exist two functions f. f E F F Construction H.4 such that
1. Indistinguishabiiity on Data: f(x;) = (xx) for all points 2j in the Roisson-drawe sample.
43

---

Published as a conference paper at ICLR 2026
2. Separation in Population: ||
f'll f n 1925.
Proof. Let J C. [I,...,N} be the set of indices corresponding to empty caps, with [J] 2
26 N N. Construct sign vectors E, {+1] as follows:
For i E set ç, = and 5 =-1.
For i & J, set Ç = c
Let = fg and fe'
1. Indistinguishabiilty: The function difference is f -iEz 24, The support of this difference is Use.J C(u,, E). Since all caps indexed by J are empty, no data point æj falls into this support. Thus, (f - (z) = 0 for all j. which implies f(z,) = fxz)).
| x N. Using the result from 2. Separation: The Hamming distance is dH(E,E') Construction H.4:
|/ 1-11 x 1+20 ddr(s,5)) x x d-1+20 1+20 N d-11+20 . e (d-1) 20
Substituting our choice of scale e = - yields the desired separation:
|f (1-
2a
1726
The final step is to transfer the result from the Poissonized model back to the original fixed-sample- size model. This is justified by the strong concentration of the Poisson distribution around its mean.
Lemma H.7 (De-Poissonization), Let Npoi ~ Poi(n). For any (0, 1), P(Npoi & [(1 7)n, (1 + n)ml)= < 2 eexp(- ci) for some constant Cy > The conclusions of Proposition H.6 hold for
fixed sample size
Proof. The existence of a large fraction of empty caps is an event that monotone with respect to the sample size (fewer samples lead more empty caps). The gigh-probbbility conclusion from Proposition H.5 holds for any sample size & within the concentration interval [(1 17)n, (1 + n)n]. changing n to k only alters the key parameter À by a constant factor, which does not affect the asymptotic analysis. Since Npoi falls in this interval with probability 1 - o(1), the event of finding an indistinguishable pair also occurs with probability probability statement can be transferred back to the fixed-n setting, yielding the same rate for the lower bound.
o(1) for a Poisson sample. This high-
The existence of an indistinguishable pair allows to establish lower bound on the minimax risk for estimation in the noiseless setting. This intermediate result is the foundation for the final generalization gap bound.
Let Fpack be the adversarial class defined in Construction H.4 with defined Proposition H.6. Corollary H.8 (Minimax Lower Bound), the noiseless setting where y: f(x.), the minimax risk for any estimator f over the adversarial class Fpack is bounded below
inf sup E -I sulli((x((O)) Z 22 f f
Proof. Let be the event that an indistinguishable pair (f. Fpuck exists for a fixed sample size n. From Proposition H.6 and Lemma H.7, we know that PPF)) = 1 - o(1). On this event F, let the true function fo be chosen uniformly at random from {'}.
Any estimator f receives the dataset D, size n. Since f(x,) = (xx for all x; E D,,. the generated data is identical whether f or So f" The estimator thus has no information
44

---

Published as a conference paper at ICLR 2026
distinguish between f and f'. The expected risk of any estimator, conditioned the event E, can be lower-bounded
[l - foll [|] = - - f11 + - f11 f
where the inequality a standard result for a choice between two points. The worst-case risk for an estimator over fo { } } is thus at least 1 - f'1|2.
Taking the expectation over the sampling of Dn:
inf sup E BlIf foll
2 inf sup
E [lf - foll
E P(E)
f f
f foEF
1
E
||I 11 1 P(E).
Since on the event E, the separation |lf - (1 x a-142o and P(E) + n oc, the result follows.
Finally, we connect the minimax risk lower bound to the generalization gap. The argument reduces the problem of bounding the generalization gap to the minimax estimation problem just solved.
Theorem H.9 (Generalization Gap Lower Bound). Let P denote any joint distribution of (æ, w) where the marginal distribution of x is Px((x) and y is supported on [-1,1]. Let D, Ue 9;))) Mj)};= be a dataset of n i..dd samples from P. Let RD. (f) be any empirical risk estimator for the true risk Rp(J) = ((aa)) y) Then,
Rx( (Ro()--oo.. Ro. cnll Zda n fiz.
inf sup ED.
sup (EF,1
Proof. We lower-bound the supremum over all distributions P by restricting it to a worst-case family of deterministic distributions Poo. where labels given by fo(æ)) for some fo from adversarial packing set, Fpack The proof proceeds via a chain of inequalities.
Rp(f) RDD CO1|
inf sup E R
sup SEF
(83)
P
Ro,, (1) No. (n)
inf sup ñ JoEF
E
sup fEr
(84)
sup
inf E Rp, (FERM) Hp, (fo)
(85)
JoET ma
2
> inf sup
1
E Rp, (f) -Rp,, (fo)
(86)
2
fo
1
E BlIf fol(t((x((0)))
inf sup
(87)
2 f Fpac
Corollary H.8 = 2
x- 1क20
(88)
The steps justified as follows
Inequality (85): This step uses standard result relating the generalization gap to the excess risk of Empirical Risk Minimizer (ERM), fERM arg minser RD, (f). By definition, R((ERm) < R(foo. This leads to the decomposition
R(fERM) - R(fo) = (R(fERM) R( (E(F)))) + (f((FER)) R((o)) + (A(f)) - R(fo))
< sup [R(f) R((f)). fEF
45

---

Published as a conference paper at ICLR 2026
Inequation (86) The infimum over all risk estimators R (which induces corresponding ERM) is lower-bounded by the infimum over all possible estimators f of the function fo- This transitions the problem to the standard minimax framework.
Equation (87): In this noiseless setting with a deterministic labeling function fo- the pop- ulation risk of fo is Rp,, (fo) O. The excess risk Rp (f) is precisely the squared L2 distance f foll (x))))(Px(a))* The expression becomes the definition of the minimax risk over the class Fpack
This completes the proof.
I FLAT INTERPOLATING TWO-LAYER RELU NETWORKS ON THE UNIT SPHERE
Let {(x;, ui) be dataset with æ; E gd-1, d 1, and pairwise distinct inputs. Assume labels are uniformly bounded, i.e., M/a < D for all i. Consider width-K two-layer ReLU models
K
fo(x))
Σ
ue d(w b&) B.
(89)
Theorem I.1 (Flat interpolation with width n). Under the set-up above, there exists a width K < network of the form (89) that interpolates the dataset and whose Hessian operator norm satisfies
DD +
Amnax (V2C) S 1+
(90)
n
Construction 1.2 (Flat interpolation ReLU network). Let 140 == Pi # 0} and set the width K = |L00| S n. For each ls E 140 define
1 1 Pc
Pk
max ki
n. a0k
1,
be E (pk, 1) (e.g. by
Wk 2
(91)
Then for any sample index i.
1 by > 0,
i = k,
w vi
ba
(92)
Pk
bk
O, + k,
so the k-thh unit activates on æy and is inactive on all a; with + k. Set the output weight
Uk
Uk
(93)
bE
By (92) and (93), the model interpolates on nonzero labels because f(xx) = un(1 - bx) ys for I I4n, and it also interpolates zero labels since all constructed units inactive æ; when i & 140, hence f f(x;)= = Yi-
For each constructed unit, define
ūg = sign(vx) E {+1},
Wx : |vx| wk:
b = |ool bx-
(94)
Then for any imput x.
Ty o ((
(x) = sign(ve) K|0 (w..a - bx)) = Ug <lw) x by),
(95)
so interpolation is preserved. Moreover, the activation pattern on the dataset is unchanged because (92) has strict inequalities and |ul 0. At æ; we have the (post-rescaling) pre-activation
ãs = : Fs by lual (1 - bx) = 1y 0,
| =
(96)
In what follows we work with the reparameterized network and drop tildes readability, implicitly assuming [ax] for all & T and z w. mg by lynl
46

---

Published as a conference paper at ICLR 2026
Proposition L.3. Let θ be the model in Construction 1.2. Then
D +2
Amax (V2) S1+
n
Proof. By direct computation, the Hessian V C given by
Vac
Tof(x))Vof(an + -W
(97)
71
Since the model interpolates f(x;) = y: for all i, we have
n
V2C Vol(a:)Vol(a.).
(98)
Denote the tangent features matrix by
= [Vof(z1), Vef(2z), ... Vef(a,))
(99)
Then V2.C (98) can expressed by V2C <p T /n, and the operator norm is computed by
Amax (22)=
max 7ESÍ++2) n
1 >0>>===
max pu
(100)
WE3"
72
From direct computation we obtain
(w(f)
vo(f)
Val(#) =
(101)
Vu(S)
For the parameters (wg, bu, v&] associated the neuron of index j.
1(w > by) (wix bk). 0f(x)
0f(x;)
bx] 2k 22
00k
Ow
0f(x:) 0bg
0f(x;) 1.
1(wFx x > bp}
uk-
03
By the one-to-one activation property (92), each sample x; activates exactly one unit (the unit with the same index k when Tyo), and activates when & Tyn- Hence the sample-wise gradient (ol(#x) has support only on the parameter triplet (wx, by, Uk :3) for E Tyo, and is zero for other parameters. Writing the nonzero gradient block explicitly (recall |ux| 1).
V ((wz.be.et))
Uu 36
(102)
(k E Tx0),
V (wr. (o
0,
(k & 440),
47

---

Published as a conference paper at ICLR 2026
After permutation and subsistion by (102), (100) is of the form
V (10 b1 (1
(22,b2 b2))
So (x22
0
(103)
0
0
(eon bastn) ) Jo (3:n)
1
1
21
F1
0
0
y/
U2 002
1/
3/2
(104)
0
:
0
Un
0
0
Un
yn
1
1
Let (u,, Un) E -5-1 and plug (104) in (100) to have
Amax V2C)=
max
(105)
u1 (w, bo
U2 w2 b2 So (o (2x2)
1
max
71 4E3
Un V (wa, A.. So
D Ui
2
1
72
max
u
2 |
(w b. fe(;))
14
(106)
71 uE3n-1
Σ
2 u
[10: 112 + 1+ 11
+
(107)
uC8n-1
i=1
i=1
1
||x;; +1++++77 vi²
+
max
Ug
(108)
n
/E[n]
w
n
1
=1 D2 +
(D² + + n)
7
If we remove the output bias term @ from the parameters, then the bottom row of 104 will be remove and thus term Σ, us in (106) will be removed.
TECHNICAL LEMMAS
Lemma J.1 (Concentration of Poisson Random Variable). Let Np Poi(n) be Poisson random variable with mean n. Then for any 1) E (0, 1),
n'n
IP ( N poi
nl 2 1p)) S 2exp
3
48

---

Published as a conference paper at ICLR 2026
Proof. The proof employs the Chernoff bounding method. The Moment Generating Function (MGF) of Npoi ~u Poi(n) is given by:
[e = 11)
We will bound the upper and lower tails separately.
We want to bound P(Npoi 2 (1 + 7))) For any & > O, Markov's inequality implies:
P(NNoi 2 (1 + (-)))))=>-)) = PP cM 2 cl((+a)n)
E p
(1+n)n
e
ed(1n)) exp (n(e' -) tn(1 + 17)
To obtain the tightest bound, we minimize the exponent with respect The optimal is found by setting the derivative to zero, which yields e¹ = 1+ ort = In(1 + 11). Substituting this value back into the bound gives:
P(N (1+ s exp (n((1 + 1)) 1) n(1 17) 1n(1 n)) exp (n(n (1 + n) In(1 + n)))
We now use the standard inequality: In(1 - x) ス 2² 2 for 2 0. A more specific inequality for n this context is 1 (1 + 77) In(1 + 7) S 211+n33) For E (0, 1]. this further simplifies. A widely used bound derived from this expression is:
exp(n/1 (1 + In(1 + 7))) exp (- 12n
3
Next, we bound P(NNot (1 n))r). For any t 0, we have:
PPN pai $ (1 n7) P pm
c ((1-n)n
e 1ppp
((1-))n
c (11-nn = exp (n(e- - 1) + tn(1 - -)))
The optimal is found by setting -t 1/,
In(11 11). Substituting this value gives:
P(NNpoi (1-n)m)= s exp (1o(11 11) - 1) n(1 77) In(1 1)) exp (n|- (1 7) In(1 7)))
Using the inequality - (1 17) In(1 1) - _タं for 17 (0, we get simple bound:
exp (nl || - (1 17) In(1 7))) s exp (_n²n 122n
2
Since for E (0, we have exp(-n"2/2) < exp(-ri'n the lower tail is also bounded by exp(-172n/3). Using the union bound, we combine the probabilities for the two tails:
IP ((.Npoi nl 2 mp) = PPNp 2 (1 + 7)m) + P(Npoi S (1 - 7)))
n
1
< exp
+ exp
3
2
22
< exp
3
This completes the proof.
49


Reviews:
[{"review_id": "mRUf0Xaqk5", "reviewer": "Reviewer_h9Wg", "summary": "This paper investigates how the geometry of the training data fundamentally controls the implicit bias of gradient descent (GD) and the resulting generalization performance of overparameterized two-layer ReLU networks trained in the \"Below Edge of Stability\" (BEoS) regime.", "questions": "Same to the weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The author claims that  \"The less shatterable the data geometry, the stronger the implicit regularization of EoS becomes.\" and illustrates this observation via two specific example.\n\nI thought this is a very interesting result and made a serious try to understand the performance neural network comparing with other \"not even wrong\" work.", "weaknesses": "I have not checked the whole proof. The results sound reasonable to me. However, to broad its impacts, it would be more beneficial if the author could make more implications of their theoretical results. e.g., its connection with some exiting theories?  Moreover, the rates stated in theorems are more less to technical,  could the authors  make it more comparable with some existing results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7s2ppc1or8", "reviewer": "Reviewer_M2En", "summary": "This work studies the interaction between data geometry and the implicit bias of edge of stability.\nIt shows that EoS bias can drive two-layer ReLU networks to adapt low-dimensionality for mixture of low-dim balls data, and that for isotropic data, its shatterablity determines generalization.\nExperiments verify the theoretical predictions.\nThis work then proposes the principle of shatterablity, where the shatterable data points attract specialized neurons, which are less regularized by the implicit weighted path norm in below the edge of stability.", "questions": "- Some works have emphasized the surprising importance of (benign) memorization for generalization, especially under long-tailed data distribution. Then is there any connection from lon-tailedness and memorization to shatterablity and neuron specialization? If so, what benign memorization looks like in the framework of shatterablity? At what threshold does memorization becomes harmful?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Based on the algorithmic bias of EoS, this work provides the data dependence aspect of neural network generalization, a valuable problem in modern data- and algorithm-dependent theory for generalization. \n- The work picks two representative examples of interest, where the first one is related to how neural network overcomes curse of dimensionality and the second one novelly reveals the role of shatterablity. The results are verified by empirical results. \n- This work also reveals that under EoS regularization, the network may still overfits, and it is data geometry with low shatterablity that helps resisting overfitting. \n- This work provides principled lens for studying feature learning and data geometry reflected in it, eg, neuron activation rate that impacts regularization strength of EoS bias and affects generalization.", "weaknesses": "- The paper supports the shatterablity principle using two proved cases, followed by intuitive interpolation/extrapolation. However, a formal results is missing, leaving shatterablity relying on intuitive definition and restricting its application to more complicated data. Is it possible to derive formal definition of shatterablity and provide more abstract generalization bounds with shatterablity and BEoS as parameters?      \n- In experiments, the training data is constructed by perturbing the label instead of IID sampling. How does this setting fits into the assumption of theories? Under standard setting, what will be low-dimension adaptation like?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FO6yIMdTr", "reviewer": "Reviewer_PDwN", "summary": "This paper investigates the role of data geometry in shaping the generalization behavior of overparameterized two-layer ReLU networks trained below the edge of stability (BEoS). Building on recent studies on the implicit bias of gradient descent, the authors propose a unifying principle termed data shatterability, which measures how easily data geometry allows ReLU thresholds to separate samples. The main theoretical contributions include:\n(1) A generalization bound for data supported on a mixture of low-dimensional subspaces, showing adaptation to the intrinsic dimension (Theorem 3.2);\n(2) A family of generalization bounds for isotropic distributions parameterized by a concentration parameter α (Theorem 3.5), together with lower bounds (Theorem 3.6) and a constructive example demonstrating perfect interpolation on the sphere (Theorem 3.7).\nEmpirical experiments on synthetic data and MNIST illustrate how data geometry affects generalization and representation structure.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is clear and well-grounded in contemporary discussions around implicit regularization and the edge-of-stability regime.\n2. The notion of data shatterability offers an elegant conceptual synthesis that connects data geometry, implicit bias, and generalization.", "weaknesses": "1. Definition and formalization of “data shatterability.”\nWhile the paper emphasizes shatterability as the central concept, it is not clearly or formally defined in a mathematical sense. The text gives intuitive descriptions (“harder to shatter data generalizes better”), but the precise operational definition is vague. \n\n  (1) Can the authors introduce a rigorous definition or metric of shatterability, perhaps analogous to VC dimension or some geometric measure of separability?\n\n  (2) Why do the authors choose the beta-radial distributions with a parameter $\\alpha$ to characterize this data property?\n\n  (3) There is a gap between the rates in Theorem 3.5 and 3.6. How are they related to the general claim?\n\n  (4) Would a toy mode concretely showing how different data geometries affect generalization make the concept more intuitive?\n\n2. Clarity and logical structure of results.\nThe theoretical results (Theorems 3.2–3.7) are presented in isolation, and their interconnections are not fully clear. The reader may struggle to see how they jointly establish a unified principle.\n\n(1) How do the results for subspace mixtures and isotropic distributions fit into a single theoretical framework?\n\n(2) Is there an overarching theorem or lemma that ties them together through the concept of shatterability?\n\n(3) A high-level diagram or summary of theoretical dependencies would help to improve readability and logical coherence.\n\n3. Lack of dynamics analysis.\nThe paper claims to study generalization “below the edge of stability,” yet the analysis focuses entirely on static properties of stable minima rather than on the gradient descent dynamics that give rise to them.\n\n(1) Without examining the time evolution of GD (e.g., curvature oscillations, stability trajectories), the results seem closer to a stability condition rather than a genuine characterization of the EoS regime.\n\n(2) The current framework could be better described as a stability-based generalization bound rather than an analysis of edge-of-stability generalization.\n\n4. Relation to prior work.\nTheorems 3.2 and 3.5 resemble results in [Wu & Su, 2023] and related stability-based analyses, with the main difference being the explicit dependence on data geometry. However, the paper does not clearly articulate the essential technical innovation over these works.\n\n(1) What are the key mathematical difficulties introduced by considering non-isotropic or low-dimensional data distributions, and how are they overcome here?\n\n(2) A more explicit comparison or ablation (possibly in the appendix) would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dDY6qpP9TU", "reviewer": "Reviewer_M57z", "summary": "This paper studies how data geometry (especially intrinsic dimension and shatterablity) determines generalization under the edge of stability implicit bias.\nThe paper shows that (1) for features from mixtures of low-dimensional balls, such bias provably drives two-layer ReLU networks's generalization to be controlled by the intrinsic instead of the ambient dimension; (2) for isotropic data, its concentration toward boundary / shatterablity controls the generalization.\nSuch theoretical predictions are verified by experiments. \nIn a finer-grained level, in both proofs and empirical results, the implicit path norm regularization induced by EoS mainly regularizes the harder-to-shatter samples, improving generalization on them, and ignores the easier-to-shatter samples. As a result, data geometry of less shatterablity improves generalization.", "questions": "- In Figure 1a, the slope of theoretical prediction is not marked and compared to the actual slope. They seem to be actual≈-0.9 vs theoretical=-1/6, where a gap is still observed. What is the source of this gap? Does it come from looseness of bounds or that the experimented problem is not the worst case to reach the upperbound (eg, the directions of the $J$ lines are not worst-case)? Or is it beyond the (B)EoS bias, similar to Sec 3.3, and is governed by some bias else? Can this question be answered by some experiments, eg, searching the worst BEoS models and comparing their slopes with the actual and the theoretical? Maybe this question demands too much efforts. But I would greatly appreciate it because it may offer a clear view on the limit as well as relation of EoS biases with other biases.\n- The discussion in Sec 4.3 seems quite generic. Is it possible to develop more general generalization bounds for BEoS weights assuming shatterablity instead of specific assumptions like low-dimensionality that leads shatterablity.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proves the adaptation of low-dimensionality and the spectrum of generalization wrt to data concentration under EoS implicit bias. The latter is also equipped with a lower-bound to show tightness.\n- The results are verified by empirical results.\n- The sketch of proof is clearly discussed. Combined with the empirical results, it also clarifies on finer-grained roles of EoS implicit regularization and clarifies why and how data geometry affects controls generalization under EoS. Such results emphasize the data-dependent nature of EoS regularization and provides explanations on the highly data-dependent behaviour of overparameterized neural networks.\n- The two main theoretical results and empirical results point to a promising shatterablity principle.", "weaknesses": "- In Sec 4.1, the experiments use label noise instead of I.I.D. sampling to construct training set and measures MSE losses instead of difference between the empirical and population risks. Such choice makes it more difficult to compare with theoretical results. What is the motivation for such choices?\n- Minor:\n  - Line 289: \"g is the population version of the *weighted*.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99RofwdCId", "reviewer": "Reviewer_9RWh", "summary": "This paper proposes a novel prospective on how data geometry influences the implicit bias of overparameterized two-layer ReLU networks, with a number of theoretical results demonstrating that solutions below the edge of stability have various generalizability properties that can be desirable or undesirable depending on the setting. They provably show that below the edge of stability solutions can adapt to intrinsic lower dimension subspaces within an ambient space and that a spectrum of generalizability occurs determined by the implicit regularization induced by the \"shatterability\" of the data. Further empirical results are given to validate their theoretical claims and proof techniques.", "questions": "- Is it possible to run experiments on other architectures ideally to hypothesise how one could extend these results beyond the two-layer ReLU networks?\n- Could further experiments be conducted to help elucidate the potential benefit of batch normalization through data shatterability?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper provides a novel perspective on how the final solutions of the edge of the stability regime will generalize, while avoiding the dynamics of the rich regime. This formulation allows for intriguing new insights into how the structure of the data affects the implicit bias within the training of these models.\n- This work has interesting implications for a number of different areas as outlined in appendix B and helps foster further work into the emerging direction of data shatterability.\n- The empirical results are presented nicely with subsection 4.2 adding substantive support to practically verifying the effect of theorems on the form of the representations learnt in two-layer ReLU networks.\n- The discussion and further work section is well written and suggests good follow-up directions on the topic.", "weaknesses": "- Data shatterability is not concretely defined or explained as a concept in the paper. While it can be roughly gleaned from previous work, a clear and concrete definition or explanation of the concept within the paper would substantially improve it.\n- Definition 2.1 claims to be for \"Isotropic Beta-radial distributions\" but then proceeds to define \"Isotropic alpha-powered distributions\". It is unclear what isotropic beta-radial distributions are in this work as I do not believe they are defined.\n- The theoretical claims are stated, but not much intuition or interpretation is given other than the overall message stated in the abstract and the introduction. A more fleshed out narrative and explanation between the theorems would have helped a deeper understanding of the work presented.\n- The empirical results in subsection 4.1 could be improved by giving a more complete explanation of how to interpret them.\n- For the right panel in figure 3 it is unclear why the correlation coefficient is given when the magnitude of the coefficient is so small. Additionally, the current figure provides no sense of how many of the points are in the bottom left corner.\n- There are some linguistic issues such as: the first sentence in the \"Disclaimers and Limitations\" does not make sense, \"deffered\" on page 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)