Training-Free Determination of Network Width via Neural Tangent Kernel
Authors: Tatsumi Sunada
Toshihiko Yamasaki
Atsuto Maki
Venue: ICLR 2026 Poster (2026)
Keywords: neural tangent kernel
kernel regression
smallest eigenvalue
generalization error
Primary area: learning theory

Abstract: Determining an appropriate size for an artificial neural network under computational constraints is a fundamental challenge. This paper introduces a practical metric, derived from Neural Tangent Kernel (NTK), for estimating the minimum necessary network width with respect to test loss *prior to training*. We provide both theoretical and empirical evidence that the smallest eigenvalue of the NTK strongly influences test loss in wide but finite-width neural networks. Based on this observation, we define an NTK-based metric computed at initialization to identify what we call *cardinal width*, i.e., the width of a network at which generalization performance saturates. Our experiments across multiple datasets and architectures demonstrate the effectiveness of this metric in estimating the *cardinal width*.

Published as a conference paper at ICLR 2026
TRAINING-FREE DETERMINATION OF NETWORK WIDTH VIA NEURAL TANGENT KERNEL
Tatsumi Sumada
Toshihiko Yamasaki*
Atsuto Maki
'he University of Tokyo (sunada, yamasaki) (ova. u-tooko..acc ac. jp, stsuto@kth.se
2TTH Royal Institute of Technology
ABSTRACT
Determining appropriate size for artificial neural network under computa- tional constraints fundamental challenge. This paper introduces a practical metric, derived from Neural Tangent Kernl (NTK), for estimating the minimum necessary network width with respect to test loss prior to training. We provide both theoretical and empirical evidence that the smallest eigenvalue of the NTK strongly influences test loss wide but finite-width neural networks. Based on this observation, we define NTK-based metric computed at initialization identify what we call cardinal width, i.e., the width of network at which gen- eralization performance saturates. Our experiments across multiple datasets and architectures demonstrate the effectiveness of this metric in estimating the cardi- nal width.
INTRODUCTION
Selecting an appropriate width for modern deep networks is crucial challenge, especially under computational constraints. In overparameterized regimes, models tend to reduce generalization loss as the width increases (Belkin al. 2019; Nakkiran et al., (202]. In practice, however, the marginal improvement saturates once the width becomes sufficiently large. This motivates the need for principled criterion to determine the point where additional width no longer useful.
Beyond simple cross-vale validatiio for width selection, prior work has explored three main directions: adopting architectures tasks and computational resources while training the models (Liu al. 2017 Gordon et al., 2018 He et al.) 2018 Yang et al. 2018 Yu & Huang 2019 Yu et al. 2019 Cai et al.[0 neural architecture search (NAS) without training (Abdelfattah et al.) 2002 Mellor al. 2021), and scaling laws for the size of models (Kaplan al. [2200] Hoffmann et al.) 2022). Although these methods reduce search effort, they largely lack thcorr-ggrunnde criterion for width determination. Width selection remains ad hoc in the absence of such a theory. These methods can rank or constrain architectures, but they still lead to repeated trial-and-error without a clear rule indicating the point which increasing width stops improving performance.
To address this issue, we introduce a training-free method for width determination based on the smallest eigenvalue of the Neural Tangent Kernel (NTK), derived from our theoretical analysis. Specifically, we show that in the infinite-width regime, where gradient-based training with squared loss is equivalent kernel ridgeless regression via NTK linearization (Jacot et al., 2018 Lee al., 2019), an upper bound on the test error is governed by the smallest eigenvalue of the NTK. In addition, for sufficiently large but finite width, we show that an analogous bound holds with the empirical NTK. We then demonstrate the validity of the theory with an experiment on two-layer networks. These theoretical and empirical results suggest that one can find the minimum sufficient width by monitoring the smallest NTK eigenvalue across widths and when its growth saturates. Based on these findings, we propose a method for using the smallest eigenvalue of the NTK to identify the width at which generalization performance saturates, which we call cardinal width. Our experimental results show that the based method of determining appropriate width is effective across multiple architectures and datasets, Our contributions can be summarized as follows:

---

Published as a conference paper at ICLR 2026
1. Infinite-width Theory:
We show that the test enror of infinite- width networks (which is an- alyzed in the framework of kernel ridgeless regression) admits an upper bound proportional to Pmin' where Mmin denotes the smallest eigenvalue of the NTK.
2. Finite-width Theory: We show that for sufficiently wide but finite-width networks, the upper bound the test error is controlled by /umin
3. Training-free width selection: We propose a method to determine the cardinal width based on Mmin that provides principled pre-training stopping point for increasing width.
4. Experiments: Across multiple architectures and datasets, the predicted saturation width aligns with the cardinal width, indicating the effectiveness of the proposed method.
RELATION TO PRIOR WORKS
Width selection. Studies width determination can be divided into three categories. Many stud- ies address width selection through supernet training (Liu et al., 2017: Gordon et al., 2018 He
et al. 2018 Yang al., 2018, Yu & Huang, 2019 Cai et al. 2020). These methods change the model size during training or train multiple models of different sizes in parallel. Another line of re-
search employs training- frree metric to rank architectures design network before training (Chen et al. 2021, Mellor et al. 2021 Xu et al. 2021). Although Chen et al. (2021) utilize a theoretical tool (NTK) for the metric, they do not provide any explicit theory that connects NTK the model's performance. Recently, scaling laws for models have been proposed (Tan & Le 2019 Kaplan et al.)
2020 Hoffmann et al. 2022), which indicate how to scale the model size along with changing data size. These methods substantially reduce search costs but are generally based on empirical evi- dence and lack a theory-backed stopping rule that certifies when further widening ceases to improve generalization. Our work provides a clear rule for width selection via an NTK-based theoretical analysis.
Generalization of kernel ridge (less) regression (KRR). KRR theory characterizes excess risk through Mercer-eigenvalue decay and effective dimension (Caponnetto & De Vito [007] Steinwart al. 2009). More recent analyses provide precise risk characterizations terms of the full kernel eigenspectrum, including closed-fore form spectral decompositions and filters (Canatar et al. 2021 Si- mon et al. [2023] Cheng et al., 2024). These works leverage the entire spectrum of kernel matrix to obtain tight bounds. In contrast, we show that upper bound the test error can be controlled by the single smallest eigenvalue so that the width selection computationally efficient compared to using full eigenvalues. Since infinite-width neural networks coincide with kernel regression with
NTK (Jacot et al. 2018 Lee et al. 2019). our results of KRR directly extend to the setting in infinite-widhe neural networks.
Neural Tangent Kernel (NTK). The NTK connects gradient descent in very wide networks to kernel regression (Jacot al. 2018 Lee al. 2019). Classical results rely the infinite-width or lazy-training idealization, in which features remain close to initialization (Chizat et al.) 2019). This yields an elegant theory but overlooks effects that arise at realistic widths. To address this gap, recent work has extended NTK analysis to finite-width regimes, including finite- cor- rections (Hanin & Nica 2020 Huang Yau 2020), concentration results (Lee et al.. (019), and fluctuation analyses (Bordelon & Pehlevan 2023). Within this literature, several studies have high- lighted the importance of the smallest eigenvalue the NTK, since governs kernel conditioning and thus affects both optimization and generalization (Nguyen et al. 2021 Montanari & Zhong, 2022 Karhadkar et al. 2024). Although the smallest eigenvalue is widely known as an important factor for generalization thanks to these works, the prior work stops short of directly connecting the smallest eigenvalue to generalization error. To the best of our knowledge, our paper is the first to es- tablish such connection, showing that the generalization behavior of both infinite and wnith networks can be characterized via the smallest eigenvalue.
Summary of differences from related works
Unlike many of prior approaches to width selection that are empirical or limited in theory, we present a training-fre criterion that is grounded in NTK theory. It selects the cardinal width by comparing changes in the smallest eigenvalue of the NTK across widths.
2

---

Published as a conference paper at ICLR 2026
We analyze kernel ridgeless regression through the single smallest eigenvalue of the NTK, whereas prior bounds typically rely on the full spectrum. The smallest eigenvalue alone controls upper bound test eπrоr, enabling practical estimation via minimal eigenvalue computation instead of full eigendecompositions
To our knowledge, this is the first work that links the smallest eigenvalue of the empirical NTK at finite width directly to an upper bound of the generalization error. Prior works on the smallest eigenvalue mainly established the positive definiteness of NTK bounded the eigenvalue itself, while our analysis connects it to test performance.
3 THEORETICAL ANALYSIS
3.1 PROBLEM SETUP
In this study, we begin our theoretical analysis by exploring the relationship between the smallest eigenvalue of the NTK and test error in infinite-width networks, analyzed within the framework of kernel ridgeless regression (Jacot et al. 2018 Lee et al., 2019). Then, we extend this analysis to fintte-width networks trained by gradient flow. In this section, we formally define the setting of kernel ridge (less) regression, NTK, and finite-width neural networks by gradient flow.
Kernel ridge (less) regression. Let 2> be unknown distribution over X' Y S R R and let {(x, y1) ) ~ be a dataset of size n. Assume the outputs follow a target function f : + R with noise E; ~ NN o²), i.c. V/- (xx) + EG. Given ridge parameter 2 0, Kernel Ridge Regression (KRR) solves
f -rrgmin 1 (f(x.) - vi)² + allf
(1)
EEN T1
where H. is the Reproducing Kernel Hilbert Space (RKHS) induced by a symmetric positive (semi) definite kernel K x + TR. Since the infinite-width network in the NTK-regime analyzed in the framework of kernel ridgeless regression, we specifically focus the case a 0.
Let X = {m:) and y = (Vi, VT) Given this setup, KRR predicts function fas
f(x) k( X) (K na In) y.
where K denotes the kernel matrix with entries Kw x;)), k(æ, X) is the row vector with entries ((rr, 2.), and I, denotes the identity matrix.
Neural Tangent Kernel.
An infinite-widhe neural network is regarded the function Equa- tion 2 where K is replaced by NTK. Given the dataset and functional output fe(x) of neural network with parameters 0 for input æ. the NTK matrix Kx Raxa is defined as
(Kx)), ij Es, Vo (a;) T VoJo,(,;))
where the expectation is taken over the random initialization of 0..
Finite-width neural network. Here, we describe the training process of an m-width neural network fm on regression problem. The network is trained by gradient flow while minimizing the scaled mean squared eтror (MSE) loss updated by gradient flow de = -7 ae where 7j is the learning rate and 2 is the training time. Let (-, denote the network at time t, and consider the terminal time T. producing fmn T)-
(z;; 2n En_ (a, 0) 31)² The network parameters are
ac
We also define k(ft) as the empirical NTK matrix at time t on the training inputs, computed as in Equation 3 but without the initialization expectation, i.e.
(K(),,, = (ofo(e)(Z;) T (2))()(z)).
(4)
3.2
MAIN RESULTS
We now present the key theorems of this paper that establish the relationship between the smallest eigenvalue the NTK and the generalization error in both infinite- and finite-width networks, We begin with the infimitewidthh case via the framework of kernel ridgeless regression.
3

---

Published as a conference paper at ICLR 2026
Given dataset {(2,, 91) Pi- ~ D, let let. be the training inputs, K X''X R be a positive definite kernel and G := (X, E R= its Gram matrix with entries G. K (z; xy). Let the eigenvalues of the matrix be ordered as f4 2 (42 2 ... 2 an 2 O, and denote its smallest eigenvalue an by Mmin- Let f be the predictor obtained by kernel ridgeless least squares with kernel K on X. We measure generalization eтror as
Infinite-width case. We consider kernel ridgeless regression introduced in Section 3.1
Eg :
f (x:))
(5)
In the infinite-widhh NTK regime, gradient flow neural networks with squared loss corresponds to kernel ridgeless regression under NTK (Jacot et al. 2018 [Le et al. 2019). Thus, the following
results apply to infinite-width networks by taking K to be the NTK.
Assumption 3.1 (Exclusion of the interpolation peak). We exclude problem instances that arbi- trarily close to the double-descent peak. In the closed-form risk expression by Canatar et al. (2021) that we use later, scalar factor y E [0, 1) quantifies the closeness of a model to the interpolation peak. Here, we assume 1 - 2 co for some universal constant > 0. Intuitively, the safnite-width model has far more effective parameters than the size of data. The peak occurs at the point where these numbers are equal, supporting the validity of this assumption.
Theorem 3.2 (Infinite-width error bound via the smallest cigenvalue). Under Assumption 3.1 there exist absolute constants C1, C2 > 0 independent of Umin and the width, such that the test error Eg satisfies
Eg S C1 Mmmu - C202n Mmin'
This bound shows that the single spectral quantity jmin controls an upper bound on the test error. For a fixed dataset, both n and a can be regarded as constants, so the bound simplifies to Eg S C' H'mi for an absolute constant C. Practically, estimating only M'in is far cheaper than computing the full eigenspectrum. Thus, this simple upper bound with jmin provides a natural starting point for efficiently determining an appropriate width. We emphasize that the bound characterizes worst-case behavior. In practice, the empirical relationship between pmin and test error can be substantially milder, we demonstrate in Section 3.4
Finite-width case. Theorem 3.2 holds on an infinite-width network, which does not apply in practice. We now pass from the infinite-width NTK predictor to finite-width neural networks trained by gradient flow, following the setup in Sec. 3.1 For a network of width m evaluated at terminal time T, we measure generalization by
E(m) =
-m (m, T
We rewrite E((00) for the test error of the wiitt (NTK) regressor characterized in Theo- rem 3.2 Let k-(w) be the empirical NTK at time E [0, for width m, and let Koo be the indthitewwdtt NTK. For any kernel matrix K, let Mmin (K) denote its smallest eigenvalue. In the following, we relate ((m) to ((x) via the minimal eigenvalue of the empirical NTK.
Assumption 3.3 (Positive definiteness of NTK). There exists Gpos > O such that Mmin K!l 2 Cpos for all f E [o, 7], and there exists Coo > 0 such that Mmin (Koin(Ko)) 2 Coc. The constants Gposy Coo are independent of m, n in the regime of interest.
Assumption 3.3 is supported by the positive definiteness of the NTK in the wide networks (Nguyn eet al. 2021
Assumption 3.4 (Upper bound on diagonal kernel under input normalization). Assume inputs sat- isfy ||0|||2 < 1 for all æ considered. Then there exist finite constants Kmax Kimax > 0 such that
(2, x) I Kmax
(x, S rE[O,T]
sup
Kmmax
Assumption i.4 a benign condition, typically satisfied in practice with normalized inputs. Assumption 3.5 (Amin ratio). There exist universal constants C_,, C_, Coo > such that
C- E-mmmin ((00) S Mmmin (K(() < C+ /mmii (K(0)),
Mmin (Koo) Coo Mmnil K<
4

---

Published as a conference paper at ICLR 2026
Assumption 3.5 implies that the smallest eigenvalue remains up to constant factor with respect to time and width. This is empirically verified in Appendix B
Assumption 3.6 (Kernel drift). There exist positive constants Carin and Ciiit, and a non- non-increasing function (mm) & such that,
sup M [0.T]
(((")) |
< Carint d(m),
|
Kool
I Cinit ((m).
Under the standard NTK conditions Lee et al. (2019), one may take ((m) m m-1/2 for any depth of deep neural networks, but the rate is not required here. Note that we discuss the operator norm in this assumption for proof' later, instead of the Frobenius norm, which used [ee et al.) (2019). However, the operator norm is smaller than the Frobenius norm, directly replacing our results the operator norm to the Frobnius norm is valid. Assumption 3.6 the kernel stability in lazy training and is empirically validated in Appendix B
Then, we derive the key theorem in this paper: the generalization error of a finite-width network Egm) by evaluating [Egm] F. via the smallest eigenvalue of the empirical NTK.
Theorem 3.7 (Finite-width error via the empirical NTK). Let Assumptions
3.3 3.5
hold. Then,
Sup, k
(00) + |K k-00
|
Kooll)
m) < E(xO) C3
(8)
Pmin 1
where the constant C3 is independent of m or
This theorem evaluates the difference between the generalization error of finite-width and infinite- width networks via the smallest eigenvalue and the norm of the empirical NTK. Note that it does not assume lazy training like Assumption 3.6 Intuitively, the functional difference |E( E. ((²)
small in lazy training since the NTK is stable during the training, whereas it becomes larger in feature learning since the kernel change || ((( Km becomes large in the regime.
Theorem 3.8 (Fitite-width test error in lazy training). Under Assumptions 3.7
3.3
3.6
the test
error En with constants C₁ and Css independent of m satisfies
C4 Mmmin (0))
Φ(m)
E(m)
Css Munin K 1
This is special case of Theorem 3.7 which applies Assumption 3.6 representigg the kernel change during training with the width m in sufficiently wide networks. In Theorem.3.7 an upper bound on (mm) is governed by the smallest eigenvalue of the empirical NTK at initialization, with width- dependent correction dd()) that vanishes as m -> co. This theorem establishes that single quantity, Amin of the empirical NTK at initialization, controls not only the infinite-width but also the finite- width generalization. These results also indicate that the generalization error decreases as the suf- ficiently large width increases. It is observed in experiment that Jmii increases as the width increases, and the increase saturates sufficiently large width. This shows that Fn(") decreases in wider networks, but the decrease saturates at sufficiently large networks. Practically, this means we can decide when widening a network no longer yields effective improvement by monitoring the saturation of Mmmin ((0)) across widths.
Note on theoretical novelty. Although the smallest eigenvalue the NTK is already recognized as important for generalization, to the best of our knowledge, this is the first work to explicitly charac- terize the direct relationship between generalization error and the smallest eigenvalue the finite- width NTK. For example, combining optimization analysis under PL condition (Liu et al.) 2022) and PL-based generalization bound (Lei & Ying 2021) in the NTK regime implies the relationship between generalization eтrоr and the smallest eigenvalue of the NTK. However, the combination quires resolving nontrivial mismatches between their assumptions, highlighting that our contribution provide explicit and self-contained description of this relationship.
5

---

Published as a conference paper at ICLR 2026
3.3 SKETCH OF PROOF
In this section, we provide a sketch of proof for Theorems 3.2 found in Appendix A
3.7 and 3.8 A detailed proof can be We start the argument by introducing the necessary tools.
Tool I: Mercer expansions.
On the empirical measure Pa. 1/ Σÿ= Says let G K(X, ( X) E R=x= with eigenpairs Gu, Pud and uk Un SEN, and let define on Φx(zz) vn ug [j]-
Then, on supp(pn).
k(x,x')
Ms (x(2))x(x').
(10)
Equation i0 the finite-sample Mercer expansion obtained by replacing the population measure px in the classical Mercer theorem Minh et al. (2006) with pa and working with the Gram spectrum of G. In the population statement, the kernel expands in eigenfunctions {0o1p21 of the integral operator (Tx (x)
J px (z') x') f()((x')))))) dæ' with eigenvalues {Ap}p21. The discrete ver- sion above uses the L (pm. basis induced by {uR} and empirical eigenvalues {uR J We adopt Equation to rewrite the following test error predicted Canatar et al. (2021) into a finite-dataset form.
Tool 2: Closed-form test error for KRR. Specializing Canatar et al. (021) to a finite dataset and the empirical Mercer basis in the Tool the test error Eg of KRR is expressed as:
Eg
Pk
// + pm).
(11)
n (K + p2)
with fxee-pooint scalars
1 n
M
7
(12)
+ pk
p2)
All symbols follow prior definitions. Here μı are the Gram eigenvalues from Equation 10 n is the sample size, is the ridge, and σ² is the noise variance. The new quantities K, the effec- tive ridge determined by the fixed point in Equation 12 E [o, 1), a shrinkage factor that increases toward the interpolation peak, and te, the target weights in the empirical basis associated with Equa- tion TO Equation I1 regarded as a mode-wise biri-variance decomposition in the Gram spectrum. /kk/ (K + p acts spectral filter, u contributes bias, and o² #4 contributes variance. The prefactor (1 - 7)- amplifies risk as + 1, quantifying proximity to interpolation.
Theorem 3.2 (infinite width).
Write the generalization error Eg of kernel regression model by
Canatar et al. 2021):
Eg =
B + V)
T #k
x² u
P
B
V=
n
(K + ph )2
n(K + mx)
A=1
As for the first term, since the function x + ( kttz
)2 monotonically decreases on [0, oo),
R²
n
x²
B S (( + pmin)²
Σ mmwi
Il. 12
(n + Mmin )2
In the ridgeless case, K < 1 Gr) < Kx sup,, K (xx x) and, away from the interpolation peak (Assumption 3.I. 1 - = (1). Hence, the incquality below holds using constant C₁:
n
B
C1 3
Regarding the second term, from Equation
12
Σ
2 S (I + pminn² }
Σ 22 S
nk2 (k + pmmi)
= n
(x + )2

---

Published as a conference paper at ICLR 2026
Therefore, using constant C2
V
o²n n Mmnn
C2
1-7
Combining B and V gives Theorem
3.2
Theorem 3.7 (finite width). Let fm( be the width-m network, K be its empirical NTK
X, and Im be the ridgeless regressor for Kit With Soo the regressor for Koo
mm (7) fool s
m (T) ((2) 1
((0)
Jon
(13)
(GI)
Using Duhamel's principle for (G1) and the closed solution of KRR for (G3), under Assump- tions 3.5 we obtain
(T)
1fm - -FT)) S C
(G1)
Mmin (K!1))
(7 - -
I
Civ
|| [K(T) k(0)
(G2)
Mmnin (Km)) [100g
)2
Cinit
P Sool
| Kxll
(G3)
Pmin (Km) )2
These inequalities are described in formal proofs. Combining (GI)-(G3) gives
C.
Im (T) - fool I
sup (|(( K + ( K || + ||
K (14)
Mmin ( (K(4)
Define E [(xx))=D(((a)) fm (m, T] and E analogously. By Bauchy-Schward in- equalities,
9 - - s C3s fm (T) - fall
(0)
(15)
Assumption 3.5
lets us express the order / (K! ) yielding Theorem 3.7
Theorem 3.8
(Finite-width test error in lazy training). Under lazy-training stability of As- each kernel difference is @(m), so the second term of Equation [4 is smaller than
sumption 3.6
C (mmm// Pmin ) The first term comes from Theorem 3.2
3.4 EXPERIMENTAL VERIFICATION
We now present series of experimental results to verify our key theorem, Theorem 3.8 in
a two-layer network. Figure 0 we train two-layer fully connected networks with a ReLU
activation function the following synthetic data generated via mankereegressoor from scikit-leazn library while changing the width of the network in the range: (16,32, 1024).
The dataset comprises 2,000 samples, each with 20 input features, and includes Gaussian noise with a standard deviation 10.0. All the weights are initialized as what Lee et al. call the NTK parame- terization. Then, the models are trained with optimizer SGD, learning rate 0.001, and batch size 32 for 1000 epochs.
The top plots in Figure establish that both test loss and the NTK's smallest eigenvalue stabilize at sufficiently large width. We therefore treat the largest width in our sweep as the asymptotic anchor. In the bottom plot, from that point (star), slope- line corresponds the theoretical bound Eg = O( Mmin ). All empirical points lie below this black dashed line across widths, consistent with Theorem 3.8 the same time, the trend of the scatter is noticeably shallower and is well captured by red dashed line. In short, the data respect the worst-case bound while exhibiting milder, approximately L/V/min dependence in practice.
'The document is at: h 1ear-eeann.oom waw.-llean.org/taate... e/modul es e//modules/eenerrteed es/ggeeettee//////////eeee// sklearn. www.stta.t...... wew.eetresseeeererrsssssss html

---

Published as a conference paper at ICLR 2026
1.25
102
1.00
10
Test Loss
10
0.75
10
0.50
10-1
0.25
10
200
1000
600
Width
Width
10"
11000
800
Test Loss
200
10
k<))
-2
Pmin
Figure 1: Mmin and test loss two-layer networks. From the top plots, that the decrease in the test loss and the increase in Jamin saturate in wide networks. The bottom plot illustrates that the test loss upper bounded by ((min) (consistent with Theorem 3.8) but is closer to ( ((1/V/min)) The scope of this plot is the point with sufficiently large width (closer to yellow).
-2
4 METHOD FOR THE CARDINAL WIDTH
In this section, we propose method to determine the cardinal width. The theoretical and empirical results in Section show that test error is governed by the smallest NTK eigenvalue. Leveraging this, we read off the width at which the minimum eigenvalue saturates and take it as training-free proxy for the width where the test error saturates, i.c., the cardinal width.
4.1 ALGORITHM
We now present our algorithm for determining the cardinal width. See Algorithm [ pseudocode. First, given a fixed architecture form with flexible width, dataset X = ( and initialization scheme I, we sean candidate widths m M. At each width, we compute the empirical NTK at initialization to estimate its smallest eigenvalue. For this step, we use Locally Optimal Block Pre- conditioned Conjugate Gradient (LOBPCG) (Knyazev 2001), which finds a few extreme eigenpairs by minimizing the Rayleigh quotient over small subspaces. The resulting scatter {(m, Punia (m)) exhibits a saturating trend as m grows (see Figure 3 saturating curve:
To make this trend operational, we fit simple
g xx)
b+ +c
(16)
by least squares. We emphasize, however, that this selection does not exclude the possibility of alternative functional forms which could also provide suitable fit. We then declare the cardinal width as the smallest m at which the fitted slope becomes negligible.
4.2 EXPERIMENTS
We evaluate the proposed method by applying it across several architecture dataset pairs. In con- crete, we train deep neural networks (DNNs) on the UCI Diabetes and California Housing datasets, and CNN and ResNet (He et al. 2016) MNIST (Lecun et al. 1998) and CIFAR-10 (Krizhevsky & Hinton 2009). The term width refers to the number of channels in convolutional layers (Arora et al. 2019) mn CNNs and the number of neurons each layer in ResNet (Huang al. 2020). For each pair, we follow Algorithm to compute the cardinal width. We then train the same models across range of widths under fixed training protocol and draw test loss curve with respect to width.
8

---

Published as a conference paper at ICLR 2026
Algorithm 1 Training-freee cardinal width via the smallest eigenvalue of the NTK
Inputs: Architecture (fixed form), dataset X { (labels not used), initialization I, width grid M C N, threshold ò > O, fitting family g(m; 0) +
Output: Cardinal width Meand
1: for m E M do
2: 8g ((m)
3: Compute empirical NTK at initialization: (Km) j3j Voloo ( VoJo, (zz)
4: Estimate the smallest eigenvalue of Km using LOBPCG and record Pmiin (m)
5: end for
6: Fit g to ((m,,miin (m))]}: & s- arg ming ΣmEM (Amin (m) - (mm; 0))
7: Define g(m) == g(m; Q) and compute its derivative dğ(m)/dm
d
8: Meard < min{ m E M: dm (mm) < <8}
9: return Mcard
In this paper, we focus regression tasks, so these models predict the exact value of the index. Evaluation proceeds by comparing the predicted cardinal width with the empirical saturation point where additional widening yields negligible improvement. For all the experiments, both Mmnn and the test loss are computed over five independent runs, and the reported results are then averaged. We provide experimental details such as architectures and hyperparameters for training in Appendix C
Figure illustrates that the behavior of pmin as a function of network width m closely aligns with the trend observed in the test loss across all architectures. The green vertical lines indicate the widths at which the derivative of the fitted function drops below the predefined threshold 8. These points correspond to the saturation of the smallest eigenvalue, and notably, they also coincide with the point where the test loss ceases to improve significantly. This alignment supports the validity of pmin as a proxy for identifying the cardinal width prior training. Below are some notes about the results.
Note 1: Lazy training/ feature learning. The experiments are conducted without any constraints on lazy training (e.g.. step sizes or initialization methods). Theorem 3.8 formally applies to the lazy training regime and therefore the theoretical validity of Pmin as a proxy in determining width does not hold in feature learning. Interestingly, however, the saturation of Jamin coincides with the saturation of test loss in the model operating in the feature learning regime. This result indicates that the smallest eigenvalue of the NTK can act as a robust indicator of the cardinal width even when features evolve during training.
We assume lazy training to derive Theorem 3.8 for theoretical rigor, but potential explanation that the theorem holds outside lazy training can be provided. [Bordelon & Pehlevan (2023) show that in the Dynamical Mean Field Theory (DMFT), the fluctuations over random initialization of NTK are of order 0m -1/2), and the variance is ((1/m) for a finite width m even in the Peature-learning regime. The authors show that richer feature learning leads to closer agreement with infinite-width mean field behavior. Thus, the kernel change || k=!' would converge
constant (not necessarily zero as in lazy training). We assume lazy training to ensure (sup,, || | x<(0)
Kooll) Il) + 0 in sufficiently wide networks and derive Theorem 3.8 However, even in the feature-learming regime, this part becomes a small constant in sufficiently wide networks if we apply the result of Bordelon & Pehlevan (2023). Therefore, the main message of our theory that "the smallest eigenvalue of the NTK controls the upper bound of generalization error" remains valid in feature learning. Thus, Mmin would work as an indicator for cardinal width in both lazy-training and feature-learning regimes.
! [0)) || + |
Note 2: Fluctuation among training recipes. Since the NTK depends on the architecture, dataset, and initialization, our width-selection problem is posed after these choices are fixed. In other words, the recommended width is determined per-architecture, par-dataset, and per-initialization. While the appropriate width would differ among training recipes (optimizer, learning rate, etc.), empirically, the cardinal width varies little across such training recipes (See Appendix [.II] D.I.) Thus, our predicted saturation width remains broadly stable over practical parameters.

---

Published as a conference paper at ICLR 2026
DNN/ Diabetes
CNN MNIST
ResNet/ MNIST
2.5
Test honeaaa Mad
-2.5
-5.0
Fitted g0s3
Fitted gle:
Fiited ple)
coniteol withh 533.29
candica wish 291.0
cordiai 5
Test loss
2.5
5.0
2.5
0.0
0
100
2001
300
400
SAN
0
25
100
125
25
50
100
DNN California
CNN CIFARIO
ResNet / CIFAR10
0.0
-2.5
-5.0
glr)
Filled
-4
candiaal willies 121.9
nambioal waskk
combinll
Text lads
0.35
Test loes
Teet losg
7.5
0.30
5.0
0.25
100
200
300
400
500
25
125
25
50
75
Width
Width
Width
Figure 2: The cardinal width identified using fitted jminn Green lines indicate where the growth of Pmni slows down, suggesting saturation point. At these widths, the test loss also plateaus, validating the proposed criterion for determining the cardinal width.
Note 3: Computational cost. While the NTK computations be expensive, our procedure only requires the smallest eigenvalue of the initialization NTK per width, which we obtain with the iterative eigensolver LOBPCG. For example, samples), one estimate of Mmin per width finishes in on the order of minute, well within realistic cost. In addition, computing umin on subset further reduces cost; we observe that the relative saturation across widths is stable under moderate subsampling. See Appendix D.2
DNN with California Housing (with about 20K
CONCLUSION AND LIMITATIONS
We propose training-free free method for estimating the cardinal width of neural network by tracking the smallest eigenvalue pmin of the NTK at initialization. We theoretically show that in the infinite- width regime, the test error admits upper bound proportional to "min: and for sufficiently wide finite models, we obtain an analogous proportionality via the empirical NTK, elevating the spectral edge practical predictor of generalization. By leveraging this relationship. we design a practical metric that identifies the width where further increases no longer yield significant generalization benefits. We validate our approach on multiple datasets and architectures, illustrating that the sat- uration point of the smallest eigenvalue of the NTK corresponds well with the point of test loss saturation. Our method enables model designers to determine the cardinal width with theoretical foundations and without any training, thus saving intensive trial-and-error.
Our main limitations can however be seen the theoretical constraints, i.e., the theoretical results are derived under the lazy training regime with squared loss and gradient flow, and thus may not directly extend to settings involving strong feature learning, alternative losses, or practical optimiz- ers. In addition, our results are largely restricted to regression tasks, leaving open questions about the applicability of the approach to classification tasks. While the NTK can also be computed for classification problems and extending method to this setting is technically feasible, it lacks theoretical guarantee at present. A natural next step is to extend the analysis to theory on feature learning and classification problems.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
The experimental settings are fully described in Section 3 Section 4 and Appendix C available at https://gitthii com Suna D/cardinaraaaaaaa width
Our code is
ACKNOWLEDGEMENTS
The authors thank Matteo Gamba for helpful discussions during the early development of this work. We also thank the anonymous reviewers for their constructive feedback.
REFERENCES
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
Andrew V. Knyazev. Toward the optimal preconditioned eigensolver: Locally optimal block pre- conditioned conjugate gradient method. SIAM Journal on Scientific Computing, 2372:517--41, 2001.
Ha Quang Minh, Partha Niyogi, and Yuan Yao. Mercer's theorem, feature maps, and smoothing. In Proceedings of the Annual Conference on Learning Theory, pp. 154-168, 2006.
Andrea Caponnetto and Ereesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3)3311-66,, 2007.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Ingo Steinwart, Don Hush, and Clint Scovel. Optimal rates for regularized least squares regression. In Proceedings of the Annual Conference on Learning Theory, pp. 79-93, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026-1034 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Mor- phnet: Fast simple resource-constrained structure learning of deep networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1586-1595, 2018.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li. and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision, pp. 784-800 800, 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in Neural Information Processing Systems, pp. 8580- 8589, 2018
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-awasd neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision, pp. 285-300, 2018.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances Neural Information Processing Systems, pp. 8141-8150, 2019.
11

---

Published as a conference paper at ICLR 2026
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical varianee trade-off. Proceedings of the National Academy 2019.
of Sciences, 116632):1544--1585
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, pp. 2937-2947, 2019.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, pp. 8572 8583, 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114 14, 2019.
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: generalization and op- timization of neural nets v.s. their induced kernel. In Advances in Neural Information Processing Systems, pp. 9712-9724, 2019.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In International Conference on Learning Representations, 2020.
Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hier- archy. In International Conference on Machine Learning, pp. 4542-4551, 2020.
Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize neural tangent kernel perspective. In Advances in
better than deep feedforward networks? Neural Information Processing Systems, pp. 2698-2709 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. International Conference on Learning Representations, 2021.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Commu- nications, 12(1):2914, 2021.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in
four gpu hours: theoretically inspired perspective. In International Conference on Learning Representations, 2021.
Yunwen Lei and Yiming Ying. Sharper generalization bounds for learning with gradient-dominated objective functions. International Conference on Learning Representations, 2021.
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley, Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598, 2021.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12): 2021.
12

---

Published as a conference paper at ICLR 2026
Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning. pp. 8119-8129, 2021.
Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: green neural architecture search. In International Conference on Machine Learning, pp. 11613-11625, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Advances Neural Information Processing Systems, pp. 30016-30030, 2022.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116, 2022.
Andrea Montanari and Yiqiao Zhong. The interpolation phase transition neural networks: Memo- rization and generalization under lazy training. The Annals of Statistics, 50(5):2816-2848 2022.
Blake Bordelon and Cengiz Pehlevan. Dynamics of finite width kernel and prediction fluctuations in mean field neural networks. In Advances in Neural Information Processing Systems, pp. 9707- 9750, 2023.
James Simon, Madeline Dickens, Dhruva Karkada, and Michael R DeWeese. The eigenlearn- ing framework: A conservation law perspective on kernel regression and wide neural networks. Transactions on Machine Learning Research, 50, 2023.
Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, and David Belius. Characterizing overfitting in kernel ridgeless regression through the eigenspectrum. In International Conference on Machine Learning, pp. 8141-8162, 2024.
Kedar Karhadkar, Michael Murray, and Guido F Montufar. Bounds for the smallest eigenvalue of the ntk for arbitrary spherical data of arbitrary dimension. In Advances in Neural Information Processing Systems, pp. 138197-138249, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
3.7 and 3.8 in Appendix A along with the experimental verification of the assumptions in the proot. Details the experiment for estimating the cardinal width are provided in Appendix [C] Then, supplementary notations are given in Ap- pendix D Further, will discuss the extension of the proposed method classification tasks in Appendix E
In this appendix, we provide the proof of Theorem 3.2
Finally, we provide our LLM usage in Appendix
A FORMAL PROOFS
this section, we provide formal proofs the theorems in the main paper. We develop the proofs in single flow that mirrors the setup in Sec. 3.1] Beginning from kernel ridge (less) regression, we first make Mercer expansions operational on a finite dataset. We then recall the closed-form generalization error of Canatar al. (202I), specialize it to the finite-data Mereer basis, and use it to prove Theorem 3.2 Finally, we pass from the infinite-width predictor to finite-width networks trained by gradient flow, proving Theorems 3.7 and 3.8. Throughout, all symbols in this section follow the definitions in the main paper.
A.1 DETAILS OF MAIN TOOLS
Tool Mercer's theorem. On the population distribution px. define the integral operator
(Tkf)(x)= = px (xx) (xx, æ²) f(x') dx'.
Assume TK has (pxx)) orthonnrmall eigenfunctions (4ppp21 with eigenvalues {Ap}p21. Mercer's theorem gives
K(x, = Σ oo (p) in (2xxx) 021
(17)
We work with finite dataset X = {0, } and replace px by the empirical measure Pa Σド= Sæ,- Let G XX Rax= and write its eigenpairs Guę uau with Ug ux* = dRs. Induce an (.²(p)) Forthoonalt basis on supp(px) by
bx(xx) = v un bil,
Σ (o = 6Rs..
Define 0 as Au = uk/n. The kernel then admits the expansion to the finite number data:
K(x, x') = Σ A (z on supp(p.).
(18)
which is the empirical counterpart of Equation 17 under Pn.
Define the feature functions
(x)(x) = V Ax (x) so that vx(x;)) Vux uu [T].
We expand the target and the learned predictor as
fst x
W
(x),
f(x) Σ WA t/x (x)
Tool 2: The generalization error of Canatar et al. (2021) Specializing Canatar et al. (2021) to our notation, generalization error of KRR with ridge a 2 0 can be written the Mercer basis as
A
E Σ
k 22 + +2nn))
(19)
k == a + Σ
kd:
y= Σ
nd? (n + n
K + +nn
14

---

Published as a conference paper at ICLR 2026
Intuitively, Ax/(N + n2 2 act spectral filters. The directions with nλx > R. shrunk, while directions with small A are relatively amplified. Using the finite-data expansion Equation 18 and A LK,, Equation 19 can be expressed using empirical eigenvalues in the main paper: n
Eg
(FF + +02 us))
Ti (K + p4)
k=1
(20)
P.k
R 0
+
7
R
n2
(K + pkk
A.2 PROOF OF THEOREM
3.2 (INFINITE WIDTH)
Under Assumption 3.1 the KRR model is away from the interpolation peak, so there exists Oo E (0, 1) with 1 - 7 co. Decompose Eg of Equation 20 into bias B and variance terms,
1
2
R2W
o² p
1-7 (B+V), Eg
B
n (k p2
n(n + kb)
Bias. The function + ( )² 2 monotonically decreasing for 2 0, hence
2
x2
B S n(K +
(n + Mmin ) 11 1122(p))
pmin
In the ridgeless
0, the fixed point gives IS < } tr((G) Koc. With 1- co, obtain
B
1 S C1 -2
(21)
Co (k + Mmin]2
Variance.
From Equation 20
tr(
22) S (K + Mmin )2
2
n(k + Mmin )2
By the Cauchy-Schwarz Schwarz bound K(x,, )2 < (ax,
S K2., we have tr(Ga)
Ens K ((x;,, x2,)2 n² K2 and thus tr(G22 (/)))//nn///)/ S nK² Therefore,
V
tr(G9)
22n
S C2
(22)
co (k + Mmin)
P'min
Combining Equation 21 and Equation
22 yields Equation and proves Theorem
3.2
A.3 PROOF OF THEOREM 3.7 (FROM INFINITE TO FINITE WIDTH)
We now relate the finite-width network trained by gradient flow to infinite-width network (kernel ridgeless regressor). Let (m(:;)) () be the width-m network at time / E [o, 7). and k<(t) its empirical NTK Gram on X. Write Al for the ridgeless KRR predictor built from K(!)) and let foo be the NTK regressor associated with K Our goal here is to evaluate [fm (T) fool.
A three-term decomposition. Using the triangle inequality, we decompose [fm(T) - focl into three terms and derive an upper bound for each term:
(T) fool
Im
20))
foo
(23)
(GI) linearization gap
(G2) fimeviaration gup
(G3) finite vs. infinite width
We now bound (G1)-(G3) under Assumptions
3.5
15

---

Published as a conference paper at ICLR 2026
(G1) Linearization gap. Let (t) R" be the vector of training predictions at time t, and define the difference r(t) = F(t) y. The gradient flow dynamics on the training points satisfy
d - r(t),
di r()
F(0) y. r(0)
(24)
Fix T' > 0 and introduce the residual (t)() of the frozen kernel Kmn) as the unique solution of
d Pr (t) K (t((),
Pr(() r(),,
(25)
dt
so that the solution of Equation 25 is:
(t) K (2 (T()).
(26)
Define the deviation (tt) == rt)) (t). Subtracting Equation 24 and 25 above yields
h(t) = K ( htt
(K K
(t)), (()) = O.
(27)
dt
m
By applying Duhamel's principle to Equation 27 we obtain
h(t)
f
k (t (t-s) (K K((')) r(s) ds.
(28)
Since K (2) is positive semidefinite, ||e - (t-e)) S for 1 Thus,
[n(1I) S ( [| - [|((} ds
(29)
|K - - / [r(s) uE[0,T]
$ sup
For test input æ, let (x)) IR" be the kernel vector with entries K!l (ax, a;) and fr(x,) :) be ridgeless KRR model with the frozen (x)) The function space dynamics obey
d
di fm (x, t) = (x T rtt),
dt [r (2x, 1) (x) T (t))),
(30)
0x) = fm,
Let ((x,t) == (z t) - fx t). Then,
d
di g(a 1) (Aa A (T)) r(t) (x)
(31)
g)x,0)= 0.
Integrating Equation cquality,
31 from O to T and applying the triangle inequality and Schwarz in-
1g(0, 7)1 < (x) - (x)))))))) + /' (T)
/
T
(32)
By Assumption 3.4 (1 (x) / K Kmax for all I, æ. Since (xx) consists entries of k(t m follows from the definition of the operator norm that
kk(0 (x) (TT)
(x)] |K( K(T')
(33)
Combining Equation
32 Equation
29 and Equation 33 yields
1g(x, T) < (1
|K[()) K(T' L." [|r(t)]dtt.
K max T
sup
(34)
we [
16

---

Published as a conference paper at ICLR 2026
To control the residual integral, note that
d ||(()]]==
2r)()) d r(t)
dt
dt
-2((t) r(t)
(35)
< - umin km) [|]())
Under Assumption 3.3 and 3.5 there exists constant C, > 0 that satisfies Mmin
k=(t)
Or Mminl (m))) for all E [0, Thus, applying Grönwall's inequality to Equation 35
gives
Cr Mmiin x (())))))))) Therefore, the integral fo ||](t]]]tt can be upper bounded as ||]()]]| S e-
follows:
L [r(())lat S Or Mmmnl Km)))
(36)
Thus, Equation 34 is upper bounded by:
(9( 0) 1 (2. [r(
$ + Kmaxx T).
sup |K - 1( ue
(37)
C Mmiul (0)
Furthermore, the gradient flow with frozen kernel converges exponentially: rI (t)
-K ( (t-T) (T))) implies
F( fr(,T)) S K m
max
(r)
K max
m
(T)
Mmin K (T))
(38)
Kmax
<
|r (T)Il-
Pmin K
where we use Assumption 3.5 to compare Mmin (Kmh) and Mmiin (0()) Combining Equation 37
and Equation 38 therefore yields
Clin (0)
If
PT
sup wE(0,Ty
|K K(T) 11.
(39)
Mmin ((
with Ciin independent of Mmin and
(G2) Time-variation of the empirical KRR predictor. Using the ridgeless representation
f(x) = k K-1 and the resolvent identity A-1 B-1 A A A)B-1, together with ||K||| ||| 1/ (K) and ||k S Kmax we obtain
Cev
(AT)
FO0 <
(K
K [|.
(40)
2
(K!0) (0)
(G3) Initialization vs. infinite width. Applying the same resolvent argument K! and K and using Assumption 3.5 to compare their smallest eigenvalues via Manin ( ), one gets
Cinit
( - fool S 122 (K(O))
|| m
Kooll.
(41)
Synthesis of (G1)-G33.. A simple row-sum bound using Assumption 3.4 implies Mmax (A) <
Cspec for any relevant PSD matrix A, 1 H/ (A) Csper/ pmin (A)² This enables us to factor
out 1/ 1/min (K110 (0))2 Combining Equation 39 Equation 41 we obtain
C.
(T) - facl S
sup ue(0,T]
| (T) m
+ ( K²O0) + |||[ - Ksoll). (42)
/ (K(0)
m
17

---

Published as a conference paper at ICLR 2026
From functional difference to test error.
Define
g m Lxx fm (x, 2))
and E (00) analogously. A Cauchy- Schwarrz inequality yields
E ) < v2 (mm fool (E) )
Considering that the generalization errors are uniformly bounded by positive constants, substituting
Equation 42 and using Theorem 3.2 to control F)(e) produces Equation 8 and completes the proof
Theorem 3.7
A.4
PROOF OF THEOREM 3.8 (LAZY TRAINING)
Under Assumption 3.6
K
K (((()))
K m Kx = (dmm)))
sup Me (1.7)
Hence
| 0 < sup
|
k || {(((m)),
and
sup |K(
K
I sup | K
||
K100 ((p(m)).
Therefore,
sup |KT
+
| K
/ Kool ((o(m))
Inserting this bound into Equation yields
((m)
(T) - fool S
Amin
K
[0(0)
Combining with the infinite-wa width control from Theorem 3.2 gives
C4
((m)
E(m)
Cg Mmnnl (!!)) )
Mmin )0)
which is Equation and completes the proof.
The three proofs above controlled by single quantity, the smallest eigenvalue. For the infinite- width predictor, (mmin (G) controls both bias and variance through Equation [9] For finite width,
Mmnin k(00)) governs the stability of kernel linearization, the time variation of the empirical KRR predictor, and the proximity to the infinite-width kernel. In the lazy regime, these effects incur only an @(m) correction, smoothly connecting practice to the NTK theory.
B VERIFICATION OF THE ASSUMPTIONS
In this section, we validate Assumption synthetic datasets exactly in Section 3 For each width m. form the empirical NTK at initial- ization (! and after T=1000 epochs, (T) We then evaluate the following quantities used the assumptions:
3.5 and 3.6 on a two-layer ReLU regressor trained on
Punnn K-!
suPwEJo.T K K!
Ko0
Mminn K(00
||-100
KK
Figure shows that the smallest eigenvalues at initialization and after training both increase with width and then saturate, approaching the infinite-width limit. Moreover, their ratio remains bounded away from and oc across the sweep, indicating relative spectral stability. Figure 4 reports the relative kernel change: both the pathwise supremum sup=e]o,r] || k || and the terminal difference | K decrease as m grows, supporting the assumption on the kernel change. ||<( k=(0| follows O(m 1/2), which reproduces the result of Lee al. (2019), while the right figure indicates the decrease of || K along width is far sharper. In either case, these kernel changes would be upper bounded by ((m). These plots support the assumptions used in our fiith bounds.
The left plot shows sup
O(m-1/2),
18

---

Published as a conference paper at ICLR 2026
109
smallest sigenvaluus
10a1
10-1
10
10
10-3
10-7
10-4
10-10
10-5
(n))(K
10a
10a
10a
10a
Width
Width
Figure 3: Smallest NTK eigenvalue width (two-layer ReLU). Left: comparison Pmiin (m))((11)))
Mmin (m (TT) and the infinite-width limit Amin a) Right: spectral ratio of these eigenvalues. Both eigenvalue curves increase with width and then saturate toward (mni((K))) while Pm remains bounded away from and oc, supporting relative spectral stability of Assumption 3.5
109
1/
1/m
10
1624
10
1/ym
10-2
10
1/
10a
10a
10a Width
10
Width
Figure 4: Relative kernel change (two-layer ReLU), Pathwise drift sup (, || initialization drift | - K both decrease as width grows, consistent with width-dependent decay d(m) &. and supporting Assumption 3.6
- and
C EXPERIMENTAL SETTINGS
C.I TRAINING
The case of DNN:
We construct 10-layer fully connected ReLU network, where the width is
consistent among layers. The width m is swept over m E (4.8....,511 512). The network trained on the Diabetes Dataset and the California Dataset. Each dataset is divided into 80% for training and
20% for test samples, Models are initialized with He initialization (He et al. 2015) and optimized
with Adam (learning rate 10-3), mini-batch size 32, for 500 epochs. Unless otherwise stated, report test performance from the final checkpoint.
The convolutional baseline is small two-stage network with 3x3 convolutions, batch normalization, and ReLU, followed by global average pooling and linear output head. Spatial resolution preserved inside the convolutional stack and reduced only by the global pooling. We treat the channel count as the width parameter and sweep E {4, 8, ..... 128] while keeping depth and other components fixed. Training uses the same protocol as the DNN: Adam with learning rate 10-3, batch size 32, for 500 epochs, with He initialization. During training, batch normalization uses batch statistics; during evaluation, it uses fixed stored statistics. We report test performance at the final epoch. The CNNs are trained CIFAR-10 10 and MNIST. Each dataset is divided into 80% for training and 20% for test samples. The models are trained with the same hyperparameters as DNNs.
The case of CNN:
19

---

Published as a conference paper at ICLR 2026
The case of ResNet:
The residual baseline uses 3x3 stem followed by fixed number of residual blocks. Each block has two 3x3 convolutions with batch normalization and ReLU, and an identity skip connection. A global average pooling layer feeds a linear head with single output unit. We control width through the base channel count c and sweep e E (4,8,... 128), holding the number of blocks constant. Optimization and initialization follow the same settings above (Adam 10-3, batch size 32, 500 epochs, He initialization). Batch normalization uses batch statistics for training and stored statisties for evaluation. Results are taken from the final checkpoint. CIFAR-IO and MNIST are also employed for training these ResNets. Additionally, we train the models with the same hyperparameters as DNN and CNN.
Each network is trained on an NVIDIA 40GB A100 GPU.
C.2 A
We follow Algorithm to compute the empirical NTK at initialization and to estimate its spectral edge, Architectures, datasets, width grids, and initialization match the training setup in Sec. Sec.B.I for fully connected networks, sweep E {4, ..... 512]; for convolutional and residual networks, we sweep channel counts (4.8..... 128} while keeping depth fixed. Labels are not used. Batch normalization is evaluated in inference mode when forming the kernel to avoid dependence on batch statistics.
For each width m, we construct the empirical kernel K x0) at initialization and estimate its smallest eigenvalue using the LOBPCG solver with a tolerance 10- and at most 500 iterations. Denote the resulting estimate by Jmminn(1 (m). We then fit smooth parametric curve g(m; 1) {(n, Pmin (m)) by least squares and determine the cardinal width with ē = 0.01.
D NOTES FOR THE CARDINAL WIDTH
D.1 INSENSITIVITY OF THE CARDINAL WIDTH TO TRAINING RECIPES
We assess the robustness of the cardinal width on a DNN regressor (UCI Diabetes) under the same setup as Section E varying the optimizer, learning rate, and initialization methods: Adam with learning rates 10 3 and 5 10 for 500 epochs, and SGD with step decay schedules 0.1/ V and 0.2/v f/ffoo for 1000 epochs, where denotes the number of steps. The networks are initialized with He initialization and NTK initialization. The batch size is 32 throughout.
Figure 3 shows that the test loss rapidly drops for small widths followed by a flat region beginning widths on the order of 102 for all the settings. The location of the cardinal width varies little across recipes. For Adam (both learning rates, both He and NTK initializations), the cardinal width concentrates in a narrow band around 100. For SGD, slightly to the right but remains within the same order, and the plateau level shifts modestly without materially moving the cardinal width. Naturally, under extreme hyperparameters, including very large learning rates, the cardinal width shift. However, within the practically reasonable settings used in Figure it remains within narrow range. In short, the predicted saturation width is broadly stable over practical optimizer and learning-ra rate choices, consistent with Note in the main paper.
D.2 SUBSAMPLING NTK COMPUTATION
We repeat the experiments of Section 4 computing Mmiul K!1)) on the full training set and on ran- dom subsets while holding all other settings fixed. For each subsample rate, we sweep the width m, fit g(m; v),, and estimate the cardinal width via Algorithm| | Figure 6 summarizes the outcomes. On Diabetes dataset, the full-data estimate is 94.40, and rates 0.10 and 0.50 give 90.58 (-4.0%) and 67.66 28..%.). On California Housing dataset, the full-data estimate is Meard Rs 114.77, and subsample rates 0.10 and 0.50 yield 89.30 (-22.15 22.1%) 1%) and 98.22 (-14.4%) respectively. Note that the estimated width in full sampling differs from the result of Figure 2 due to the randomness the fitting curve. In both datasets, the cardinal width remains reasonably stable under 10-50% subsam- pling, indicating that training-fre width selection can be performed on a reduced fraction of inputs with only modest loss in accuracy and with computation roughly proportional to the subsample rate.
20

---

Published as a conference paper at ICLR 2026
Adam
1.25
Test Loss
opt=Adam, 1r=0,0005, init=He, epochs=500, bs=32
1.00
opt-Adam, Ir-0,0005, Init-NTK, epochs-500, 500, hs-32
opt Ir-0.001, Init-He, epochs-500, bs-32
0.75
optmAdam, 1r-0.001, Init=NTK, epochs=500 bsm32
0.50
SGD
Test Loss
1.5
opt=SGD, 1r=0.1 init=He, epochs=1000, bsm32
opt-SGD, Ir-0. 1/V, init-NTK, epochs-1000 1000, bs-32
1.0-
opt=SGD, 1r=0.2 init=He, epochs=1000 bs=32
ogt=SGD, 1r-0, init=NTK, epochs- 1000, bs-32
100
200
300
400
500
Width
Figure 5: Cardinal width is stable across training recipes (DNN Diabetes). Test loss versus width for Adam (top) and SGD (bottom) with two learning rates/schedules and two initializations (He, NTK). All curves exhibit sharp drop followed by plateau beginning at widths ~ 100, with only minor shifts the cardinal width across recipes.
DNN Diabetes
10a
Computed (nte(
100
LOBPCG, cardinal width 94.440
10-3
Subsample rate: 0.10, cardinal width 90.58
Subsample rate: 0.50, cardinal width 67.
10-6
10
100
200
300
400
500
DNN
California
102
Computed (ama(
100
LOBPCG, cardinal width 114.77
10
Subsample rate: 0.10, cardinal width 89.30
Subsample rate: 0.50, cardinal width = 98.22
10
10-6
100
200
300
400
500
Width
Figure 6: Subsampling preserves the cardinal width. Computed Mminn [ DNNs. The curves fits of g(m; i)) and vertical dashed lines mark the estimated cardinal widths. The predicted cardinal widths are broadly stable under 10-50% subsampling, supporting lower- cost, training-free free width selection.
versus width for
E EXTENSION TO CLASSIFICATION
While our theory focuses on regression tasks following previous NTK-based works, we now discuss the potential extension of method to classification tasks. We apply our Algorithm i to determine
21

---

Published as a conference paper at ICLR 2026
DNN XOR
DNN MNIST
3
log 1o (K
log to Mmis (1)0))
2
Fitted g(m)
Fitted g(m)
cardinal width m su 12.86
cardinal width 131.19
Test loss
Test loss
0.3
0.5
0.2
0.0
0.1
10
20 Width
30
40
0
50
100
150 Width
200
250
Figure 7: The predicted cardinal width in classification tasks. At the cardinal width identified using algorithm, the test loss correspondingly saturates, indicating the potential extension of our method to classification tasks.
the cardinal width for 10-layer DNNs trained solve the classification task the stylized XOR data introduced in (Wei et al.) 2019) and MNIST. We prepare XOR data with 2000 samples for training and 100 for testing. The input dimension is 20.
In our experiment, the network is optimized via Adam with learning rate 10-3 and trained for 20 epochs on the XOR data and 50 epochs on MNIST. The batch size is 32 for both datasets. Figure shows that the estimated cardinal width is aligned with the width at which the test loss no longer improves for both the XOR data and MNIST. These results imply that our algorithm can potentially be applied to classification problems, although it is not theoretically guaranteed.
F LLM USAGE
We used large language model tool solely for editing during manuscript preparation, such gram- mar corrections, wording clarification, and minor condensation of prose. All the text suggested by the tool was reviewed and revised by the authors. All ideas, methods, experiments, and conclusions are developed by the authors.
22


Reviews:
[{"review_id": "wB4lE721bH", "reviewer": "Reviewer_thWr", "summary": "This paper introduces a principled, training-free method to determine an appropriate network width, termed \"cardinal width,\" where generalization performance is expected to saturate. The core contribution is a novel metric based on the smallest eigenvalue ($\\mu_{min}$) of the Neural Tangent Kernel (NTK) computed at initialization. The authors provide a theoretical justification, linking an upper bound on the test error to $\\mu_{min}^{-2}$ in both infinite-width and, under certain assumptions, finite-width regimes. The proposed algorithm identifies the cardinal width by finding the point at which the growth of $\\mu_{min}$ saturates as width increases, offering a practical tool to avoid unnecessary computational costs from over-provisioning network width without requiring any training.", "questions": "- Could you please elaborate on the observed alignment between the saturation of $\\mu_{min}$ and test loss in the feature learning regime, despite Theorem 3.8 relying on the lazy training assumption? Is there an intuition or a potential theoretical argument that could bridge this gap?\n\n- Regarding scalability: Could you provide a more quantitative analysis of the trade-off between the data subsampling rate, the computational cost, and the stability/accuracy of the predicted cardinal width? How sensitive is the method to the random choice of the subset?\n\n- How do you envision this method being extended to classification tasks? What are the primary theoretical hurdles (e.g., defining the NTK for cross-entropy, the validity of the error bounds) and have you conducted any preliminary experiments in this direction?\n\n- While the method provides an absolute value (\"cardinal width\"), how does it perform as a relative ranking metric on a standard NAS benchmark? For instance, does ranking architectures by their proximity to the predicted cardinal width yield a strong correlation with final accuracy compared to established proxies like Synflow or GraSP?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **Theoretical Rigor:** The paper's primary strength lies in its strong theoretical foundation in NTK theory. Unlike many zero-cost (ZC) proxies that are based on heuristics derived from pruning or empirical observations, this method provides a clear, analytical connection between a spectral property of the network at initialization ($\\mu_{min}$) and its generalization capability. This adds a significant degree of interpretability and trustworthiness.\n\n- **Actionable and Concrete Goal:** Most ZC proxies provide a relative ranking of architectures, which is useful for search but does not answer the fundamental question: \"How wide is wide enough?\". This paper's proposal of a \"cardinal width\" offers a specific, actionable target for practitioners, which is a distinct and highly valuable contribution.\n\n- **Novel Theoretical Connection:** To our knowledge, this is the first work to formally connect the smallest eigenvalue of the empirical NTK in finite-width networks directly to an upper bound on generalization error. This provides a new and potentially powerful analytical tool for the community.", "weaknesses": "- **Fragile Theoretical Assumption for Practical Regimes:** The key theoretical result for finite-width networks (Theorem 3.8) explicitly depends on the \"lazy training\" assumption (Assumption 3.6), a regime where the network behaves like a linear model and does not perform significant feature learning. The authors commendably acknowledge their experiments operate outside this regime. However, this creates a critical disconnect between the theory that provides the method's justification and its demonstrated practical application. The method's success in the feature learning regime is thus an empirical observation, not a theoretically guaranteed outcome.\n\n- **Significant Scalability Concerns:** The methodology requires constructing an $N \\times N$ NTK matrix for a dataset of size $N$, a process with at least $O(N^2)$ complexity. This makes the approach computationally infeasible for large-scale datasets (e.g., ImageNet), where many modern ZC proxies that operate on a single minibatch (e.g., Synflow, GraSP) are orders of magnitude faster and more scalable. While subsampling is suggested as a remedy, its impact on the accuracy of the estimated cardinal width is not fully explored.\n\n- **Narrow Scope of Application:** The paper focuses exclusively on selecting a single hyperparameter (width) for regression tasks. The broader and more common application of ZC proxies is in Neural Architecture Search (NAS) across complex search spaces, often for classification tasks. The paper does not demonstrate the method's utility as a general-purpose NAS proxy, and its theoretical underpinnings for classification losses are not provided.\n\n- **Static Nature:** The proposed method is static, determining the width before training commences. This overlooks recent dynamic approaches that learn or adapt network width during training (e.g., Adaptive Width Networks) or allow for efficient width reduction after training (e.g., Triangular Dropout), which may find more data-efficient architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gCITg9cyZF", "reviewer": "Reviewer_M1eP", "summary": "This paper proposes a training-free criterion for selecting an appropriate network width. The key observation is that, when the width is sufficiently large, test error is governed by the smallest eigenvalue of the NTK and authors justify this both theoretically and empirically. Based on this, authors introduce a method that estimates the cardinal width, the point at which generalization performance saturates, by tracking the smallest NTK eigenvalue at initialization through Locally Optimal Block Preconditioned Conjugate Gradient . Experiments across multiple architectures and datasets validate effectiveness of this method.", "questions": "Considering the eigenvalues of the NTK is a nice approach, but eigenvalues are sensitive quantities in numerical analysis. Did the authors report any uncertainty in calculating this eigenvalue? If not, do you have insights on how to compute it robustly in this setup?\n\nIn the EXPERIMENTAL VERIFICATION subsection, the weights seem to be initialized as ( \\mathcal{N}(0,1) ), which appears different from NTK initialization. Could the authors explain this point?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper first makes an non trivial claim, test error is controlled by the smallest NTK eigenvalue, and supports it both theoretically and empirically. Leveraging this insight and pointing out the highlighting of using the smallest NTK eigenvalue rather than the full spectrum, the authors devise a practical procedure to estimate the cardinal width and experimentally validate the approach on two-layer networks. This line of research is compelling and provides well-balanced results.", "weaknesses": "Assumption 3.6, invariance of NTK, is central to the NTK framework, yet the authors assume it without proof. Although the empirical evidence is supportive, a full theoretical justification would substantially strengthen the work.\n\nAlthough the effectiveness of the cardinal width is demonstrated experimentally, I’m concerned about how closely the experimental setup reflects real-world training conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eiopFAQe9J", "reviewer": "Reviewer_fdRP", "summary": "The paper studies generalization loss of neural nets in the kernel regime and derives a generalization bound that depends on the minimum eigenvalue of the kernel at initialization ($\\mu_{min}$). In particular their results indicate an upperbound proportional to $1/\\mu_{min}^2$ for the test loss. Moreover, the result holds for both infinite and finite width neural networks. This leads to an algorithm for a training-free selection for the network width where the point where increasing the width does not significantly help with increasing $\\mu_{min}$ is selected as the best choice for width.", "questions": "Please see above section for questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Rigorous theory and several experiments to verify the effectiveness of the proposed algorithm and theory. Bounding the test loss based on a single computable parameter can be insightful even in the restricted setup of NTK regime and gradient flow considered in this paper. The authors also present a simple algorithm for width selection which is computationally efficient and does not require training.", "weaknesses": "-The paper mentions the importance of kernel condition number for both optimization and generalization as specified in literature. Can the authors please explain more the relation to prior works in line 90 of the paper? Moreover, the authors claim that they are the first to show the relation between $\\mu_{min}$ and test error in the kernel regime. In my opinion, it needs more clarification by authors as there is already a link between these two in the literature. In particular, it is known that the neural network objectives of sufficiently large width with square-loss satisfy a PL condition in the kernel regime (e.g, Thm 4. in [1] where the authors show the relation between PL parameter and the minimum eigenvalue of kernel) and therefore one can obtain test loss bounds based on $\\mu_{min}$ (e.g., see [2]). The minimum eigenvalue also seems to be proportional to class margin for classification tasks as discussed by [3] (section 5). \n\n-How does $\\mu_{min}$ scale with number of samples ($n$)? Also, It's better to clarify whether $C_1,C_2$ in theorem 3.2. scale with $n$. can the authors please clarify how the bound scales with $n$, or specify relevant work in literature?\n\n-What is the benefit of bounding test loss only based on $\\mu_{min}$ beyond computational efficiency reasons? can including more information beyond the minimum eigenvalue (while not using the whole spectrum) help with generalization bounds? I seems the bounds based only on $\\mu_{min}$ can be loose (or at least the authors do not characterize tightness) and moreover the authors observe a tighter $1/\\sqrt{\\mu_{min}}$ relation in their experiments. \n\n-Do the authors suspect that the theory can be extended to the feature-learning regime using DMFT framework of [Bordelon and Pehlevan, 2023] and the resulting equations to specify whether there is relation between generalization gap and $\\mu_{min}$ of dynamical (time-varying) kernel? Has such a relation been observed in experiments? \n\n-Can the authors verify their theory on stylized data (e.g. boolean XOR data or multi-index models (see references[3-4])) to see how their theory predicts compared to current theoretical lower bounds for the network width in the kernel regime? \n\n-Does Thm 3.8 hold for any $m$? or Should the network be sufficiently wide for this results to hold? It seems to me that this detail is not emphasized in the statement of the theorem. \n\n1- Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, Liu et al, 2021\n\n2- Sharper Generalization Bounds for Learning with Gradient-dominated Objective Functions, Lei and Ying, ICLR 2021. \n\n3- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. Ji and Telgarsky, ICLR 2020.\n\n4- Feature selection and low test error in shallow low-rotation relu networks, Telgarsky, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sArXkyZIy8", "reviewer": "Reviewer_khFg", "summary": "The paper shows that the test error of infinite and finite-width networks is upper bound by a value related to the smallest eigenvalue of the (empirical) NTK. Based on the theory, the authors provide a training-free method to determine the cardinal width, which is the critical width where the test loss saturates, through examining the saturation of the empirical NTK smallest eigenvalue at initialization. Experiments show that the proposed method can predict cardinal width pretty well.", "questions": "In scaling law studies (Kaplan et al. 2020), the authors show that width and depth have minimal effects within a wide range compared to overall parameter count. Can the authors provide more insights on the superiority of the proposed methods in practical uses in more details, for example, how can the proposed method be used in real world experiment settings, like scaling law experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. There are many empirical works trying to utilize the empirical NTK spectrum to study the neural network properties at initialization. The theory derived in the paper offers a principled way of choosing the proper width without model training, which could be useful for practitioners in the field.\n2. The paper is well-organized. The theory is accompanied with proper experiments.", "weaknesses": "1. The main concern of the paper is that whether the cardinal width is useful in practice. In several plots of figure 2, in CNN/MNIST from width 50 to 125 I do not observe obvious slope change. Similar observations can be seen in CNN/CIFAR10 width 50-100 and  ResNet/MNIST width 25-75. Hence, it makes the “plateau point” ambiguous. In such cases, a coarse manual random guess could possibly achieve similar conclusions. In conclusion, I am unsure about the superiority of the proposed method compared to heuristic rule.\n2. Based on the figure 1, the observed plot is closer to $1/\\sqrt{\\mu_{min}}$, raising the concern of whether the bound is loose in real settings. Could the authors provide a detailed comparison/discussion with existing theoretical results regarding the NTK-based test error bound?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)