On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory
Authors: Guhan Chen
Yicheng Li
Qian Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: neural tangent kernel
random initialization
non-parametric regression
reproducing kernel Hilbert space
Primary area: learning_theory
TL;DR: This paper explores the impact of random initialization in Neural Tangent Kernel (NTK) theories and argues that NTK theories fail to explain the superior performance of neural networks.

Abstract: This paper aims to discuss the impact of random initialization of neural networks in the neural tangent kernel (NTK) theory, which is ignored by most recent works in the NTK theory. It is well known that as the network's width tends to infinity, the neural network with random initialization converges to a Gaussian process \(f^{\mathrm{GP}}\), which takes values in \(L^{2}(\mathcal{X})\), where \(\mathcal{X}\) is the domain of the data. In contrast, to adopt the traditional theory of kernel regression, most recent works introduced a special mirrored architecture and a mirrored (random) initialization to ensure the network's output is identically zero at initialization. Therefore, it remains a question whether the conventional setting and mirrored initialization would make wide neural networks exhibit different generalization capabilities. In this paper, we first show that the training dynamics of the gradient flow of neural networks with random initialization converge uniformly to that of the corresponding NTK regression with random initialization \(f^{\mathrm{GP}}\). We then show that \(\mathbf{P}(f^{\mathrm{GP}} \in [\mathcal{H}^{\mathrm{NT}}]^{s}) = 1\) for any \(s < \frac{3}{d+1}\) and \(\mathbf{P}(f^{\mathrm{GP}} \in [\mathcal{H}^{\mathrm{NT}}]^{s}) = 0\) for any \(s \geq \frac{3}{d+1}\), where \([\mathcal{H}^{\mathrm{NT}}]^{s}\) is the real interpolation space of the RKHS \(\mathcal{H}^{\mathrm{NT}}\) associated with the NTK. Consequently, the generalization error of the wide neural network trained by gradient descent is \(\Omega(n^{-\frac{3}{d+3}})\), and it still suffers from the curse of dimensionality. Thus, the NTK theory may not explain the superior performance of neural networks.

On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory
Guhan Chen
Yicheng Li Department of Statistics and Data Science
Department of Statistics and Data Science
Tsinghua University
Tsinghua University Beijing, China
Beijing, China
chen- bten-gh23@maile tsinghua.ede edu. en
liyc22@mails s0sc22mmails.tssinnnuaaaaa tsinghua. edu. cn
Qian Lin"
Department of Statistics and Data Science Tsinghua University Beijing, China
gianlin@tsinghua.cm edu. cn
Abstract
This paper aims to discuss the impact of random initialization of neural networks in the neural tangent kernel (NTK) theory, which ignored by most recent works the NTK theory. It is well known that the network's width tends to infinity, the neural network with random initialization converges to Gaussian process fGP, which takes values in L (X), where X' is the domain of the data. In contrast, adopt the traditional theory of kernel regression, most recent works introduced a special mirrored architecture and mirrored (random) initialization ensure the network's 's output is identically zero at initialization. Therefore, it remains a question whether the conventional setting and mirrored initialization would make wide neural networks exhibit different generalization capabilities. In this paper, we first show that the training dynamics of the gradient flow of neural networks with random initialization converge uniformly to that of the corresponding NTK regression with for any s J+1 and P(GGP E [HNT]') = 0 for any 2 d+I where [HNT]= is the real interpolation space of the RKHS H associated with the NTK. Consequently, the generalization error of the wide neural network trained by gradient descent is 2(n कंं) and it still suffers from the curse of dimensionality. On one hand, the result highlights the benefits mirror initialization. On the other hand, it implies that NTK theory may not fully explain the superior performance of neural networks.
random initialization fGP We then show that P(fGP [HNT]s)
3
Introduction
In recent years, the advancement of neural networks has revolutionized various domains, including computer vision, generative modeling, and others. Notably, large language models like the renowned GPT series [8, 51] have shown exceptional proficiency language-related tasks. Similarly, neural networks have achieved significant successes in image classification, as evidenced by works such as [27, 34, 37]. This proliferation of neural networks spans wide range of fields. Despite these
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

impressive achievements, a comprehensive theoretical understanding of why neural networks perform so well remains elusive in the academic community.
Several studies have delved into the theoretical properties of neural networks. Initially, researchers were keen on exploring the expressive capacity of networks, demonstrated in seminal works like [17,28 28]. These studies established the Universal Approximation Theorem, asserting that sufficiently wide networks can approximate any continuous function. More recent research, such as [15, 26, 43] extended this exploration to the effects of deeper and wider network architectures. However, a significant challenge remains in these studies: they often do not fully explain the generalization power of neural networks, which is crucial for evaluating the performance of a statistical model.
Recently, some researchers have examined the generalization properties of networks. Bauer and Kohler [5], Schmidt- [46] showed the minimax optimality of networks with various activation functions for specific subclasses of Hölder functions, within the nonparametric regression framework. In contrasts to the static ERM approach, some studies made more attention to the dynamics of neural networks, particularly those trained using gradient descent (GD) and stochastic gradient descent (2 13, 20j.
With similar insights. Jacot et al. [31] explicitly introduced the Neural Tangent Kernel (NTK) concept, demonstrating that there exists a time-aarying neural network kemel (NNK) which converges to a fixed deterministic kernel and remains almost invariant during training network width approaches infinity. And thus NTK theory proposes that network training can be approximated by a kernel regression problem [4, 29, 39, 50]. As general case, fully-connected networks directly trained by GD, Lai et al. [35], Li et al. [41] showed the generalization ability of two-layer and multi-layer networks, respectively.
This paper mainly follows [4, 35, 41J, and explores the impact of initialization in the NTK theory. Prior research [35, 41] which verified the minimax optimality of network utilized the so-called mirrored initialization setting. It refers to a combination of mirrored structure and mirrored initial value of parameters, which results in zero initial output function. However, the assumption divates from the commonly used initialization strategy in real-world applications, whose initial output is actually non-zero. To bridging the gap, in this study we explore the generalization ability of standard non-zero initialized network, within the NTK theory framework. Our findings reveal that the vanilla non-zero initialization will theoretically results in poor generalization ability of network, especially when the data has relatively large dimension. If that is true, it suggests a divergence between theoretical models and real-world applications, highlighting a potential limitation in the current understanding of the NTK theory. Therefore, we arrive at a critical problem central to this study:
Does initialization significantly impact the generalization ability of networks within the kernel regime?
1.1 Our contribution
Network converges to a NTK predictor uniformly. We show that under standard initialization, the network function converges to the NTK predictor uniformly over the entire training process and over all possible input in the domain. The convergence is essential in the study of the generalization ability of network in NTK theory. However, in previous work, the initial values of network has long been overlooked. Under mirrored initialization which leads to zero initial output function, Arora et al. [4], Lai al. [35], Li et al. [41] demonstrated the point-wise convergence and the uniform convergence of network, respectively. More recently, Xu and Zhu [54] studied the uniform convergence of NTK under standard initialization, but did not study the convergence of the network function. Why the initial output of the network is ignored is not that it is insignificant, but rather because it a stochastic function, making it challenging to analyze in convergence. Our findings make it valid to approximate the network's 's generalization ability based on the corresponding NTK predictor's 's performance.
The generalization ability of standardly non-zero initialized fully-connected network. Our research explores the impact of standard non-zero initialization in NTK theory. At this issue, Zhang al. [56] proposes the existence of implicit bias induced by non-zero initialization, when the neural network is completely overfitted. We delve deeper into this argument, studies the exact formula of the bias at any stage of training, within the framework of NTK theory. Additionally, we established
that the (optimally tuned) learning rate of network is n 4+3, even when the regression function is

---

sufficiently smooth. This insightful discovery implies notable limitation in the generalization ability of networks with non-zero initialization, if NTK theory can precisely upproximate the performance of real network. Consequently, we need to reconsider the weakness of NTK in the study of network theory. Also, the results show that mirrored initialization is superior to standard initialization in practical applications.
1.2 Related works
Our research is conducted within the framework of NTK theory. This type of research, in general. can be categorized into two main steps: the approximation of the network trained by GD through a kernel regression problem, and the evaluation on the corresponding kernel regression predictor. Several studies [2, 4, 19, 31] which focused on the former step, illustrated the point-wise convergence of NTK for multi-layer ReLU networks. Additionally, [39] demonstrated the point-wise convergence of the kernel regression predictor to the network. Furthermore, Lai et al. [35], Li et al. [41], Xu and Zhu [54] demonstrated the uniform convergence result with respect to all input and all time on two-layer and multi-layer networks. As to the latter step, few researchers have analyzed the spectral properties of the NTK [6. 7] as well as kernel regression [40, 55J. Building upon these findings, Lai et al. [35] and Li et al. [41] demonstrated that early-stopping GD induces minimax optimality of the network. is worth noting that the setting in these works assumes mirrored initialization, which may not be well-aligned with real-world scenarios. When it comes to initialization, Zhang et al. [56] provided insights into the impact of initialization under kernel interpolation, which a special case of our results at t
2 Preliminaries
2.1 Model and notations
Suppose that ((*;, are i.i.d. drawn from unknown distribution p which is given by
fx) +
where (x) is the regression function and e a centered random noise. Suppose that the marginal distribution (xx;) of the radom variable is supported in non-empty bounded subset X' of IRd with C smooth boundary. The generalization error of an estimator of f is given by excess risk
1*) 1100=(li--- [2)
We introduce the following standard assumption the noise(e.g.. [21, 42]). It is clear that sub- Gaussian noise satisfying this assumption.
Assumption 1 (Noise). The noise term satisfies the following condition for some positive constant o, L, and m 2 2:
1 ml m-2 E.
[z)
Given a set of samples pairs {(14, W:) we denote X and Y to be vector (z), )" and (y1..... "","""""""""" respectively. In a similar manner, (f(z1), ... (((x), f(x)))" (f(n)),... ff.))) are represented as f(X) and f(Y), where f(-) : Rd + R is an arbitrary given function. Regarding a kernel function k(.,.) : Rd x Rd + R. we use k(z, X) to denote the vector (k(I, x1), kxx, 22), k(2, 2n)) and ((X,)) ) to denote the matrix [k(2,, x,)) nxn. For real number sequences such as }a,} and {bn J. we write On = (b,,) (or a, = o(b,)) )). there exists absolute positive constant C' such that Ja, < C/bb, holds for any sufficiently large (or la..1/16, approaches zero). We also denote a₂ x bn if there exists absolute positive constant c abd C' such that (b,l < |a,,| Clbb,,l holds for any sufficiently large
Notations
and
2.2 Reproducing kernel Hilbert space
Suppose that is kernel function defined on the domain X' satisfying that || S x². Let He be the reproducing Hilbert space associated with & which is the closure of linear span of (k(z -), E X3
3

---

under the inner product induced by (k(x, ky, -)) = k(z, y). Given a distribution paz) on X, we can introduce an integral operator Ta L P)) + (2' p):
Txf(2) k(æ, fv) du(y).
(4)
The celebrated Mercer's 's decomposition [12] asserts that
Tel = Σ ()) k(x = [xer(t)e:(v))
(5)
KEN
IEN
where fei and {Aī JiN are the orthonormal basis of (2 pi) and Hu respectively, It well known that Ha can be canonically embedded into (2(2, p))
If the eigenvalues X; of l are polynomially decaying at rate 3 ( i.e X; i we can further introduce concept of the relative smoothness of function f E 1. (2, More precisely, let us recall the concept of real interpolation space [48] ( Please see more detailed information in the Appendix).
Real interpolation space
The real interpolation space [H&]] S is given by
[He]* Easfecaa Σ ag < )
(6)
VENN
EEN
with the inner product (EeN a. (1)), EVEN (=))(())))
EvEn a,b; for 2 0.
It is clear that [He]* is separable Hilbert space and is isometric to the l2 space. With the definition above, we can see that [H& (,,, μu) and [Hx]] H4. Also, for any 82 81 2 0, we know
[Hu]"* C [Ha] with compact embedding. Let
inf s [74]] C (X))
which is often referred the embedding index of RKHS Hd [21]. is well known that og 2 B and the equality holds for large class of usual RKHSs if the eigenvalue decay rate is B. We further define the relative smoothness of a given function f:
Definition 2.1 (Relative smoothness). Given kernel k on X' with respect measure μ4, the smoothness of function f is defined as
a(, sup a 0
Σ 17 )
iEN
where c, (2(x
2.3 Kernel gradient flow
For positive definite reproducing kernel k. the dynamic of kernel gradient flow (KGF) [22] is
d GF (x:) - k(, X) (J (x) (X)
(8)
dt
where fCP is the KGF predictor. In kernel gradient flow, the performance of kernel predictor depends on the relative smoothness of regression function. People often consider the case that a(f, k) 2 1 [10, 11]. When the smoothness satisfies atf,, k)) 1, the regression function is said to be poorly smooth and belongs to the so-called misspecified spectral algorithm problem. We collect the related result in Zhang et al. [55] and apply it to case, to derive the following proposition:
Proposition 2.2. Suppose the eigenvalue decay rate of 3 and the embedding index is 1/4 with respect to μ.. Suppose the noise term e satisfies Assumption 1. Let the dynamic (8) starts from fCP 0. Also, suppose the regression function satisfies f E [7*x] and Ilf [[pa]] R, for some
8 >> 0. Let 7 S a and 0 < S 1. By choosing x n for any fixed 8 E (0, 1), when n is sufficient large, with probability at least 1 - 6, we have
||
pm [He]] s
<1n In [n
6) 2 9) R²Cn
f0-228
where C' is a positive constant.

---

Network and Neural Tangent Kernel
3.1 Network settings
consider the fully-connected network with hidden layers. As is commonly-used in deep learning.
we consider the ReLU activation [44] defined by a(x) = max( 0). Denote f(;;)) R + R as the network output function, where representing the column vector that all parameters flattened into. We write the recursive structure of network following:
(x)
2
[1(0) x xb00))
m
2
(9)
a (x)
W (x))(((0-1)
1= 2,3,
my
f(;; 0) a(
The parameter matrix for the 1-th layer denoted as W Their dimensions are of m4+1 x mi, where my is the number of units in layer / and M1+1 is that of layer (+ 1. Also, the bias term of the first layer is denoted as (0)) R' The setting of bias term is to make sure the positive definiteness of NTK [41]. We further assume that the number of units in each layer is at the same order while the ,.,-1) width comes to infinity, as cm S min(m1, , S max(m,, ; S Cm where c, C are some absolute positive constants.
Standard initialization variables:
At initialization, the parameters are randomly set as i.i.d. standard normal
Lid. N(O, 1), 1 1= 0,0,,.... w Dg
m1
(10)
Remark 3.1 (Mirrored initializaation). As to the mirrored initialization considered in [4, 35, 41], part of the network f (1:66)) undergoes standard initialization, while the other complicated corresponding part ((2) 0(3) holds the same structure as f()) (:; (())), with parameters initialized
to the same values as ((2) (()) Lastly, the neural network output function is defined as /(=;0))
v2
(1: (1:66))))) (:: (:)) This setup ensures that f(: 0o) is constantly zero.
The network is trained under the mean square loss function. If we suppose ((xs, W:) be the training data, then the loss function specified as
((0) = 1 Σ (f(x;;0) - v1)
(11)
2n
(x;x;;)) 0t). The training process for the network is For notational simplicity, we denote by (NN(x) (x) = fax performed by gradient flow, where the parameters updated through the differential equation:
_ 0: = -
- NN (X)) ( (x) -X)-
- 10.
(12)
where 0o Oo (X) is matrix with dimensions n x M, with M being the length of the parameter vector O. This matrix represents the gradient of the network output (NN(X) with respect to the parameters 0 at time f. Incorporating the chain rule, we can formulate the gradient flow equation for the network function follows:
d NN
(x) = - AN {z) "F -Y..
(13)
di
3.2 Network at initialization
In order to state the properties of wide network with standard initialization, we need introduce the concept of Gaussian process.
5

---

Gaussian process Gaussian process is stochastic process for which every finite collection of random variables follows multivariate Gaussian distribution. Let X be Gaussian process with index & E T. If the mean and covariance are given by the mean function m and the positive definite which holds for any t, 1 E T,
kernel such that ( = m(t) and (t) = k(t,
then we say X ~ GP(m,)) k).
In standard initialization (10), the parameters of the neural network are i.i.d samples from a standard normal distribution. If the network contains only one hidden-layer (that is, if L 2), it is direct to prove that fo (a)) converges to a centered Gaussian distribution by CLT, for any fixed point X. As to the multi-layer network, prior research [25] also proved that such initialized network converges Gaussian process, as following:
Lemma 3.2 (Limit distribution of initialization). As the network width m tend to infinity, the sequence of nelwork stochastic process {Jo Jm== converges weakly in C(X, RR) to a centered Gaussian process fGP, The covariance function the so-called random feature kernel (RFK), which is denoted by KREK (z, x²) as defined in (42) in Appendix C.I.
3.3 The kernel regime
As the gradient descent of neural network involves high non-linearity and non-convexity, it is difficult to study the training process. However, Jacot et al. [31] introduced the Neural Tangent Kernel (NTK) theory which provides connection between network training and a class of kernel regression problems, when the network width comes to infinity. To demonstrate this, we first define a Neural Network Kernel (NNK):
KT" (2, x²) [
(14)
Using this notation, we reformulate (13) in a kernel regression format:
d NN (x) = n2
K," (f, IN (X) - Y).
(15)
dt
NTK theory shows, if the network width m tends to infinity, then the random kernel KT' (---) will converge to a time-invariant kernel KNTK (...). : Rd x Rd > IR which is referred to as the NTK of network. The phenomenon is the so-called NTK regime [2. 39J. The fixed kernel KNTK only depends on the structure of the neural network and the way of initialization. To get more knowledge of NTK, we present the explicit expression of NTK in Appendix C. In NTK theory, the dynamic of network (15) can be approximated by kernel gradient flow equation:
d NTK (x) di f
--KK KNK (x,, XCU - Y), 72
(16)
which starts from Gaussian process NT fGP In this way, if we aims derive the generalization property of sufficiently wide network, we can achieve by considering the corresponding kernel gradient flow predictor. Such approximation is strictly ensured by uniform convergence of SNN and ,NTK over all x E X and all t 2 0 as m + 00, since we use L excess risk to evaluate the generalization ability. Actually, we have the following theorem, whose proof is given in Appendix B.
Proposition 3.3 (Uniform convergence). Given training sample pairs { v For any d (0, 1) and E > 0, when network width m is large enough, we have
sup suplft 2,''Ex 120
NN (a:) ft JNTK (x)) Se
holds with probability at least 1 8 8.
In this theorem, we show the uniform convergence of network under standard initialization. Previous related studies [4, 35, 41, 54] always utilized delicately designed mirrored initialization (as shown in Remark 3.1) to avoid the analysis on the initial output function of network, since it will lead to the challenging problem that fNN and f ANTK are both random, unlike that #NTK is a fixed function in the case of mirrored initialization. However, as shown in Section 3.2, the initial output function is near a Gaussian process that can not be overlooked. To under the performance of neural networks commonly used in real world, it is necessary to analyzing the network initialization. To the best of our knowledge, we are the first to consider the initial output function of network in uniform convergence. This comprehensive result allows to study the generalization error of network more preciscly.

---

4 Impact of Initialization
4.1 Impact of standard initialization on the generalization error
The standard kernel gradient flow is always considered to start from zero, as in Proposition 2.2. Therefore, we need do transformation since the initial value of predictor fNTK is actually fGr instead of zero. Firstly, we can yield a solution of (8) in matrix form:
(x) fo (z) + k(z, X) e ((X.X)) XX] (FOP (X) (x) -xx), (17)
where ex is employed represent the n x 1 column noise term vector Y Y fx). We denote by JCP the kernel gradient flow predictor under initial function fo and denote by ft the KGF predictor under initialization f = O. If we plug them into (17) and excess risk (2), respectively, we directly have the following theorem:
Proposition 4.1 (Impact of initialization in kernel gradient flow), Denote f f fo as the biased regression function. For the KGF predictor J,F and J[GF defined above, have
EUL ; (
(18)
The theorem establishes the equivalence of the generalization properties between the KGF predictor with initial value fo. regression function f and the KGF predictor with initial value zero, regression function f - foo Back to the network case, combining uniform convergence result in Proposition 3.3, it suggests that, compared to mirrored initialization, the impact of standard initialization which has non-zero initial output function is equivalent to introducing a nume-valued implicit bias to the regression function. This is generalization of the main result in Zhang et al. [56], which only focused on case / oo. To summarize, Proposition 4.1 provides convenient approach to quantify the impact of standard initialization in early-stopping neural networks.
4.2 Smoothness of Gaussian process
Building upon the analysis above, our focus now turns illustrating the smoothness of the Gaussian process fCP, as it is the limit distribution of foN. AAtually, we can derive the following theorem:
Theorem 4.2 (Smoothness of Gaussian Process). Suppose that foP is a Gaussian process with mean function 0 and covariance function K The following statements hold:
P ( E EIHNTT00 = 1,
<<
d d+1
(19)
P (j E [ANTT])= = O. d d+1
We furnish a comprehensive proof for Theorem 4.2 in the Appendix C.
Let us now turn our attention to the implications established by this theorem. Recall that Proposition 4.1 has shown that, in KGF, the existence of initialization function fGP is equivalent to adding same-valued bias term to the regression function ʃ". Consequently, the poor smoothness of initialization function causes the high smoothness assumption on the regression function meaningless. Regardless of how smooth we assume the regression function to be (e.g., af K > 2). the value of (relavtive) smoothness aff* - fGr K KNTK) ) will always be at most प Namely, the biased regression function / JGP is always poorly smooth. In this specific case, we could hardly expect the KGF predictor to have fine performance.
4.3 Upper bound
Now we are ready to provide the upper bound of generalization error of network. With the help of Proposition 2.2. Proposition 3.3, Proposition 4.1 and Theorem 4.2, we derive the following theorem:
Theorem 4.3 (Generalization error upper bound). Assume that the regression function f NTs for some s > O, and IIll paNTt- < R where R is positive constant. Assume the marginal probability measure μ with density p(x) satisfies p(z) C for some positive constant and C.
7

---

For the case of 2 3
d+1 a4i. for any å (0, 1) and E (0, 243). ) by choosing certain t = t(n) + co (as shown in Appendix), when n is sufficiently large and m is sufficiently large, with probability 1 we have
| f = (m
(R (R+ CR+ atste
(20)
where Ce is a positive constant related to
2il
For the case of < for any ö (0, by choosing t z nad+i)+d when is sufficiently large and is sufficiently large, with probability 1 we have
|| M 1 S (m ") (R C.+ a(a41)ad
(21)
where Cs is a positive constant related to 8.
The proof is provided in Appendix D. This result shows the generalization error upper bound for network with standard initialization and demonstrate its negative effect. Even if the goal function f" is quite smooth, the generalization error upper bound n अोअ remains to be quite low rate, particularly considering that the dimension d of data is usually large in real world. It suggests that the network no longer generalizes well, even if we adopt the once useful early stopping strategy in Li al. [41].
4.4 Lower bound
From the analysis above, we can see the poor generalization ability of network under standard initialization. Furthermore, in this section, take spherical data example and provide the lower bound of generalization error. Namely, we presume the input vectors I are distributed on the sphere St with probability measure μ4, which is common assumption in NTK theory [6, 31, 36, 53]. We also slightly change the network structure. Compared to the network (9), we eliminate the bias term of the initial layer, shown in (40) in Appendix. In this case, the NTK of new network is denoted by K and the RKHS H. ( is abbreviated as H6 whose detailed properties is also given in Appendix C.I. Additionally, we make more assumption on the noise of data. We assume the noise term e in (1) to have a constant second moment, as E [|<|]]r]] = a² for E 3d a.e.. Under these conditions, with the help of method in Li et al. [40], we derive the theorem:
Theorem 4.4 (Generalization error lower bound). We assume that the regression function f e [H for some s 3. and denote by || (11gTTT S R where R is a positive constant. Assume that p is the uniform measure. For any 8 (0, 1), when n is large enough and m is large enough, for any choice oft = t(n) + co, with probability at least 5 we have
E f115.T I [x] = S (n 213
(22)
The proof is given in Appendix E. Through Theorem 4.4, we derive I as the generalization lower bound of standardly random-initializee network in NTK theory, even if the regression function is quite smooth. The rate 2ेΣ means model suffers notably from data that has large dimension: If is relatively large, then this rate of convergence can be extremely slow. This a manifestation of the curse of dimensionality. fact, it contrasts with the fact that neural networks excel at high- dimensional problems. This contradiction underscores the limitation of NTK theory for interpreting network performance.
5 Experiments
Our numerical experiments are conducted two aspects to fully understand the impact of standard initialization. First, show the performance of standard initialized network is indeed worse than the mirrored initialized case, on the aspect of learning rate. The phenomenon is in line with our theoretical analysis. Second, the smoothness of regression function of real data is significantly larger than 3 + which suggest the bad effect of non-zero intial output function of standard initialization will indeed destroy the performance of network if NTK theory holds. It demonstrates the drawback of NTK theory through contradiction.
8

---

5.1 Artificial data
In the first experiment, we employ artificial data to show the negative effect of standard initialization on the generalization error of network. The detailed settings shown in Appendix F.
Learning rate of network under different initialization The experiments are conducted for both d = 5 and d = 10, contrasting network performance subject to mirrored and standard initialization strategies. We choose a relatively smooth goal function to emphasize the impact of initialization. Specifically, we use m = 20n, epoch = 10n. and the gradient learning rate (r = 0.6. The networks are made sufficiently wide to ensure the overparametrization assumption is met. Additionally, we implement the early-stopping strategy mentioned in Theorem 4.3, that is, selecting the minimum loss across all epochs as the generalization error. Finally, we test the network's generalization error on different levels of sample size n, and plot the log value of the generalization error corresponding to log(r) (n) as shown in Figure 1. As we expected, the points in Figure fits linear trend. Moreover, the figure highlights the difference in learning rate under different initialization methods. This aligns with our theoretical results.
d
-1.5
Mimored intiiicasion, alppe-1 1.20
Standard ippial uston, alopee-2.43
-22
Mirooee infolization slsper11 09 Standard intakastoon, -0p
-1.8
loguferort
ingunerro))
-2.0
-26
-22
-28
-24
-28
-32
20
29
3.0
3.1
3.2
2.80
2.85
2.90
2.05
3.00
3.05
3.10
3.15
100zeln)
l0gi0(n)
Figure 1: Generalization error decay curve of network. The scatter points show the averaged log error over 20 trials. The dashed lines are computed through least-squares. The scale of is not broad because a larger n requires larger m, which would induce higher computational costs.
5.2 Real data
In this subsection, we focus on datasets from the real world and estimate the smoothness of function. Although we could not know the goal function that the real data is generated from, there exists a way to estimate its smoothness [16]. We show the technical details in Appendix G.
Table 1: Smoothness of goal function
Dataset
Name
Dimension
Smoothness
MNIST
28 x 28 X
0.40
CIFAR- 10
32 32 x
0.09
Mashion-MNIST
28 28 X 1
0.22
Smoothness of goal function in real datasets We employed the MNIST, CIFAR-10 and Fashion- MNIST datasets[33, 38, 52]. In the experiments, we evaluate the smoothness of goal function of the datasets, with respect to the one-hidden layer NTK. The results are presented in Table 1. With the input dimension d = 784, 3072, 784, we can compute that the smoothness of initialization function is equal to 3 1+1 = 0. However, the smoothness of goal function is far better than 3 4+1 which implies that standard initialization will indeed destroy the generalization performance, under NTK theory. The contradiction between NTK theory and the real situation shows its limitation and once again confirms our conclusion.

---

Discussion
To summarize, this research focuses on the impact of standard random initialization on generalization property of fully-connected network in the NTK theory, which makes up the gap in this field. Many previous work [35, 41] verified the statistical optimality of neural network under delicately designed mirrored initialization, whose initial output function of network is zero. However, through our study, we pinpoint that we consider the commonly-used standard initialization, the learning rate of network is notably slow when the dimension of data is slightly large, which fails to explain network's s favorable performance overcoming the curse of dimensionality. is the superiority of mirror initialization over standard initialization, which suggests direction for future improvements. On a deeper level, although NTK theory can describe many properties of network, at least for the fully connected networks with Gaussian initialization discussed in this paper, we can explore better theoretical frameworks to characterize their generalization ability the future.
direct implication of our work
10

---

Acknowledgments
Lin's research was supported in part by the National Natural Science Foundation of China (Grant 92370122, Grant 11971257).
References
[1] Robert A Adams and John JF Fournier. Sobolev spaces. Elsevier, 2003.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via paeeeearaaettiiiiinnn In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
[3] N. Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Sociery, (8(3):337-404 1950. URL http: //xp::///xx// doi. org/ 00t.22307/99940
[4] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems, 32, 2019.
[5] Benedikt Bauer and Michael Kohler. On deep learning as remedy for the curse of dimension- ality in nonparametric regression. 2019.
[6] Alberto Bietti and Francis Bach. Deep equals shallow for relu networks in kernel regimes. arXiv preprint arXiv:2009. 14397, 2020.
[7] Alberto Bietti and Julien Mairal, On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 33:187771101, 2020.
[9] William Elwood Byerly. An elemenatary treatise on Fourier's 's series, and spherical, cylindrical, and ellipsoidal harmonics, with applications to problems mathematical physies. Dover Publicatioons, 1893.
[10] Andrea Caponnetto. Optimal rates for regularization operators in learning theory. 2006.
[11] Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized ceast-squares algorithm. Foundations of Computational Mathematics, 7:331-368, 2007.
[12] Claudio Carmeli, Ernesto De Vito, and Alessandro Toigo. Reproducing kernel hilbert spaces and mercer theorem. arXiv preprint math/0504071, 2005.
[13] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. Advances in neural information processing systems, 32, 2019.
[14] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
[15] Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on learning theory, pages 698-728. PMLR, 2016.
[16] Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34: 10131-10143, 2021.
[17] George Cybenko. Approximation by superpositions sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[18] Eleonora Di Nezza, Giampiero Palatucci, and Enrico Valdinoci. Hitchhiker's guide to the fractional sobolev spaces. Bulletin des sciences mathématiques, 136(5):521-573, 2012.. 2012.
[19] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[20] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterizerr neural networks. In International Conference on Learning Representations, 2018.
11

---

[21] Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. The Journal of Machine Learning Research, 21(1):8464-855 8501, 2020.
[22] L Loo Gerfo, Lorenzo Rosasco, Francesca Odone, E De Vito, and Alessandro Verri. Spectral algorithms for supervised learning. Neural Computation, 20(7):1833-1877, 2008.
[23] Nadine Große and Cornelia Schneider. Sobolev spaces on riemannian manifolds with bounded geometry: general coordinates and traces. Mathematische Nachrichten, 286(16):1586-166) 2013.
[24] Moritz Haas, David Holzmüller, Ulrike von Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. arXiv preprint arXiv:2305.14077 2023.
[25] Boris Hanin. Random neural networks in the infinite width limit as gaussian processes. arXiv preprint arXiv:2107.00 0156,, 2021.
[26] Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint
2017.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
[29] Tianyang Hu, Wenjia Wang. Cong Lin, and Guang Cheng. Regularization matters: A non- parametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics, pages 829-837. PMLR, 2021.
[30] Simon Hubbert, Emilio Poreu, Chris Oates, Mark Girolami, et al. Sobolev spaces, kernels and discrepancies over hyperspheres. arXiv preprint arXiv:2211.09115 2022.
[31] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[32] Vladimir Koltchinskii and Evarist Giné. Random matrix approximation of spectra of integral operators. Bernoulli, pages 113-167, 2000.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
[35] Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on arXiv preprint 232.v:230..05933, 2023.
[36] Jianfa Lai, Zixiong Yu, Songtao Tian, and Qian Lin. Generalization ability of wide residual networks. arXiv preprint 1.Xii:2305.18506 2023.
[37] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(1 86(11):227)) 1):227)-22333 1998.
[38] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yam..ecum.comeeee com/exdb/nnnistt 2, 2010.
[39] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32:8572-8583, 2019.
[40] Yicheng Li, Weiye Gan, Zuoqiang Shi, and Qian Lin. Generalization error curves for analytic spectral algorithms under power-law decay. arXiv preprint arXiv:2401.01599 2024.
[41] Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. On the eigenvalue decay rates of class of neural-network related kernel functions defined on general domains. Journal of Machine Learning Research, 25(82):1-47) 2024.
12

---

[42] Junhong Lin and Volkan Cevher. Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. Journal of Machine Learning Research, 21(147): 1-63, 2020.
[43] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017.
[44] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-1O), pages 807-814, 2010.
[45] Yoshihiro Sawano et al. Theory of Besov spaces, volume 56. Springer, 2018.
[46] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. 2020.
[47] Alex Smola, Zoltán Ovári, and Robert C Williamson. Regularization with dot-product kernels. Advances in neural information processing systems, 13, 2000.
[48] Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and rkhss. Constructive Approximation, 35:363-417, 2012.
[49] Robert S Strichartz. guide to distribution theory and Fourier transforms. World Scientific Publishing Company, 2003.
[50] Namjoon Suh, Hyunouk Ko, and Xiaoming Huo. A non non-parametric regression viewpoint: Generalization of overparametrized deep relu network under noisy observations. In International Conference on Learning Representations, 2021.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1ee.17708777447 2017
[53] Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, and Jeffrey Pennington. Precise learning curves and higher-order scalings for dot-product kernel regression. Advances in Neural Information Processing Systems, 0114558-45700 2022.
[54] Jiaming Xu and Hanjing Zhu. Overparametrized multi-layer neural networks: Uniform con- centration of neural tangent kernel and convergence of stochastic gradient descent. Journal of Machine Learning Research, 25(94):1 2024.
[55] Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms. arXiv preprint arXiv:2303. /4942, 2023.
[56] Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo, and Zheng Ma. A type of generalization error induced by initialization in deep neural networks. In Mathematical and Scientific Machine Learning, pages 144-164, PMLR, 2020.
13

---

A Further notations
In appendix, we will provide many technical proofs. Before that, let us provide more notations. For two sets A and B with mapping function ø A + B, the notation ((A) is used to denote the image set of A under Φ. For two random variable sequences {un] and { J. we denote by Un = op (v,) (or (2,,) if the ratio un, Cn approaches zero (or 2 cUn for some positive constant c) in probability as n + oo with respect to probability measure P. For two real number sequence {an} and ), we denote by an ()(b,) if there exists positive constant and no such that [a,] 2 clbnl holds for any n 2 nig. For two sequences of real numbers {an) and {bx) such that dn = 2(bb,) (or an = (b,)), we also denote by on 2 bn (or on S bn ) If On 2 bn and an & bR then we denote by an x bn-
B Proof of uniform convergence
In this section, we demonstrate the uniform convergence from J.N to PNTK.
B.1 Initialization
The following a direct proposition based on Lemma H.2 and Lemma 3.2,
Proposition B.1. For the random network function sequence { with probability measures on (C(X, C) there exists {Xm} and XGP defined on new probability space (S', F),,, on which we have
P lim X - X = 0) 11
where X, and XGP has the same distribution as Jö" and JGP, respectively.
Remark B.2. The separability of (C(X,R), C) can be derived by the density of polynomials. Therefore, it satisfies the requirement of Lemma H.2. In the context of our study, our reliance is only on the distribution of {fōm} for each given value of m. Consequently, it reasonable to reconstruct in the new probability space. For convenience, we directly denote X²" as fo (or f ) and denote and GG as fGr, respectively. In other words, we are considering the network function new probability space, even though this approach may result in a moderate abuse of notation.
B.2 Uniform convergence of network
Our aim is give the uniform convergence between NTK regressor fNTK and network function JNN. Note that the NTK regressor is trained by NTK, and the network function is trained by NNK, which is denoted by Kmo Here first show the uniform convergence between NNK and NTK m comes to infinity.
Lemma B.3. For any 8 E (0, 1), suppose m is large enough, then with probability at least 1 - 8, we have
sup sup [KTh [KT"(,, (x,2²) - K-TK (I, xx, s O(m _ /log m). 120 x.
Proof. The proof is similar to that in Li et al. [41], while the difference is the way of initialization. So only provide the sketch of proof. In Li al. [41], the uniform convergence of NTK is proved through a standard c - net argument, which divided into point-wise convergence and continuity of both NTK and NNK. Namely, the following decomposition:
[K (2,
KNTK (z, x')l S|KT (2,x²) KW" (2, 2')
+ |K? (2, NTK (2, (2,2))++
(K NTK (2,2') - KNNK (x,
(23)
where 2, 2' are the points in the c - net which divides X.
Back to our case, in non-zero initialization, the structrue of NTK and NNK remain the same, as well as the continuity property. Consequently, the effect of initialization reflects on the wise convergence from (2, z') to KNTK =²), or more precisely, the NTK regime [2]. NTK regime requires that the residual decays to near zero and thereby the parameters will not deviate too far from their initial values in the training process, which holds under mirrored initialization. Standard
14

---

initialization lets the residual at time 0 be | (X) 12 instead of || Therefore, there is slight risk that the residual too large decay to near zero during training. However, since
(x)(()) - O(n m²)
(24)
holds with high probability when m is large through Proposition B.1 and direct analysis on fGP we can verify that the residual | (X) - Y|2 1|2 is still not large enough to break the stable lazy regime. Namely, the control on parameter matrix that
w wo'|| l| Om
sup t20
(25)
still holds. In this way, we can finish the proof.
Then, we can derive the uniform convergence of network function.
Proof of Proposition 3.3. The proof is also similar to that of uniform convergence under mirrored initialization. Therefore, we only exhibit the sketch of different part. Define event A as
A = {15o NN - - S om (1)) n ([][CP(()))) s C&}
(26)
where Cs is some constant related to 6, such that event A holds with probability at least 1 1- when m is large enough. Such a constant Cs is ascertainable, as fi converges to f for by Proposition B.1 and f fGr Gaussian process with finite second moment. Define event B
B sup sup |K?" (x,,") KNTK (sr, (x,, om (n) 120 a.
(27)
We have event B holds with probability least 1 when m is large enough. Conditioned event A and B, we do kernel gradient flow by K7" and K KTK on ANN and jGP respectively. Let event C be
C C= sup |S 120
J.NTKIle S om (mm(1))
(28)
Conditioned on event A and B, we can prove that event C' holds by Gronwall's inequality, as the same method Lai et al. [35]. In this way, we can finish the proof.
After we get the uniform convergence of network function, we can obtain the proposition on the convergence of excess risk:
Proposition B.4. Suppose f L2 p)). For any å 10 and 0, when m large enough,
with probability at least - 8, we have
sup 1>0
NN ||
11 12
-11 - 11 Se
(29)
Proof. Recall the dynamic equation of fNTK, we have
|SNT (x:) < || K (z, X) (X, NT (X) Y11l.
(30)
Since the kernel function KNTK(, (..)) is bounded, there exists some positive constant C, such that
K NNT (x. x)T S CVn.
(31)
The initial function of kernel gradient flow yT GPP follows Gaussian process with mean and covariance kernel function KRFK, By the boundness of KRFK we can also bound fNNK, That is, for any 8 (0, 1), there exists positive constant M, such that with probability least 1 8/2,
|| NTK (X)1, S VnMs.
(32)
Denote Ag == Amin (X, We have À >>0 0 since KNTK is strictly positive definite [41]. Thus we have
< CoV/AAi'(VrNN + (1211)).
(33)
15

---

The excess risk
|EU 1*) E(JETK ; 1/x JNN S.TK p du + / ( - S(TK) dj
(34)
Since f E (2( where is probability measure, also have f E (2), Denote
Mp. = 11f and A = SuPpEx.120 If, (x)- - J.TT]. We have
; f E(fT 1 S
CCCAA'//VVAi + (12112) + My.))
(35)
By Proposition 3.3, when m is large enough, with probability at least 1 5 we have A s <5((1+
(VTM ( + My.). Thus the proposition is proved.
C Proof of the Theorem 4.2
Before the proof, first we introduce some basic properties of NTK and RFK, as well as some technical properties of Sobolev space. We say that two Hilbert space Hi, H2 are equivalent they are equal as sets and share equivalent norm. If Hi and H2 are equivalent, we denote by Hi = H2.
C.1 Basic properties of NTK and RFK
Dot-product kernel A reproducing kernel function k is dot-product if its value only depends on the dot-product of inputs. That is, there exists function such that
k(a, = =()x)=²)).
(36)
A dot-product kernel on sphere can be decomposed with spherical harmonic polynomials as the cigenfunction:
k(x, y)
pm
2a (z)),
(37)
==
1=1
where spherical harmonic polynomials (Yn,i, 1.... an are also the orthonormal basis of L L3ddd o), with denoting the uniform measure on Sd [47]. This is also its Mercer decomposition.
Now come back to our network case.
We first define two dot- not-product kernels on Sd
L-1 No (t) (u)),
KNN v) = Σ r
(w) II
K KRFK (2, = x] (3) (u).
(38)
0=
where u = (x. v) = x² and
1
1 / 2² + (I arccos u). π
ko(u)= ( arccos u)
ki(u)
(39)
π
The definition of (8) L1 is given by the composition Ki o Ki Ki total of compositions). The explicit expression indicates that KNTK and KRFK are dot-product kemels on gd.
KNTK and KRPK is the homogeneous NTK and RFK of a homogeneous fully-connected network S defined on 8d [6, 14], whose structural difference from (9) is the removal of the bias term in the first layer. Specifically, the network is structured as follows:
Homogeneous fully-connected network on sphere recursive formula:
The network is constructed using the following
a (x)
w()
mi (0)))) w (x)
(40)
1= 2,3, .
my
(x;;0) = (L)) ((())()))(()))
16

---

where the function o is entrywise ReLU activation. The parameter matrix for the 1-th layer is denoted as W whose dimensions are of m4+1 x my where my is the number of units in layer and m4+1 that of layer / + 1 for / {0, 1............... 1-13. 1]. We also set mo be equal d + 1 and mL+1 equal to 1. The network is also random initialized as (10).
We easily build a connection between our NTK and RFK for network (9) and the homogeneous kernels defined in (38). For æ E X, let H = (z, 1) which means add 1 as the new last component (x.1)
of Define Φ(z) == I| being isomorphism from open set X' to subdomain of positive hemisphere shell S = (x) C $4. Then we have
f(z) (((x),
(41)
where f is network (9) and f is network (40). Actually, we can thus verify that
KNT xx, v) | K KI (6(x), (()),
K (x, v) | K (δ(x), ((v)).
(42)
We denote by HDT and HBP the RKHS on Sd with respect to KNIK and KBPK Their eigenvalue decay rates are well known:
Lemma C.1 (Bietti and Bach [6], Haas et al. (24)), For k and K on s with uniform measure o, the decay rate of spherical harmonics coefficients satisfy
Pn (K nn" (d+1) and (K(KFF x= (d43).
(43)
while the eigenvalues satisfy
X Xe( NTK S 3) xi
d+1 and A.(KEEK g8t o) x xi.
d+3
(44)
Additionally, we list some further result on the eigenvalue decay rate of NTK and RFK provided by Li et al. [41], which will be used later:
Lemma C.2. Denote Q as a non-empty subdomain of For Ko KNTK and K we have eigenvalue decay rate:
X, -2,0) = =1,0)= and A 02 x1=) 44
where a is the uniform measure on Lemma C.3. For K NTK and KRFK
we have eigenvalue decay rate:
NTK ,,, x and (KKREK .X,H) = 1- d43 a
where measure on bounded domain X C Rd has density c S p(x) S C with respect to the Lebesgue measure.
C.2 Basic concepts of Sobolev space
Soboley Space of integer power Let X' be a open subset of Rd. Let m E N. N.152 S < +00. Sobolev space WmmP(( (X') is defined of function such that
| SllE < <xx,
(45)
where is vector with length and f is the weak a-th partial derivative of f. In other words, the definition of W/mP is:
Wm (X) (f E IP(X) E LP(X), Vaa S m),
(46)
where 1 S S 0o. Conventionally, when the index p is equal to 2, we denote Wm by H", since is a Hilbert space. Further, if the index m g. the Sobolev space Hmh qualifies as RKHS and thus embraces the properties of RKHS. In our work, we mainly utilize its property of interpolation as
defined in (6). Consequently, we first introduce generalized concept of real interpolation [1], as an expansion to the definition in (6).
17

---

Real interpolation For two Bananch spaces H1 and H2, we use interpolation space to represent space that lies in between them in some specific way. We introduce the commonly-used K-method to define real interpolation. Suppose 0 < < 1, q 2 1. The space generated by their real interpolation H (H1, H2)s,g is defined by following:
((;==)=
inf I= tag
|| + 111x221992
(47)
and
| ( ((()))()))))))))) :
(48)
Based on the definition of real interpolation, we introduce some basic concepts about fractional power Sobolev space.
Sobolev space of fractional power Suppose X' E Rd is bounded domain with smooth boundary and denote Lebesgue measure by μ. We can define fractional power Sobolev space through real interpolation (we refer [45] Chapter 4.2.2 for more details):
(x) (L2 p1), 11
HT (X) 2
(49)
The fractional power Sobolev space H'(() with r 2 d is also RKHS [1]. Specifically, Steinwart and Scovel [48] reveals that for < 8 <
[7| = (X, 2))
(50)
for RKHS H and the interpolation defined in (6). Therefore, the results above directly implies that [H] (x))* (x)
(51)
holds for any r 2 g and >
Up now, we have introduced the basic properties of Sobolev spaces on X, an open subset of IR For Sobolev spaces defined on more intricate manifolds, such as hyperspheres, owing to the intricate property of Sobolev spaces, numerous equivalent definitions emerges [1, 18].
We now delineate a kind of definition that will facilitate our subsequent proofs, since we will consider RKHSS Sd, like HBT and HRF, which are the RKHSs associated with KNTK and KRFK, respectively. Such definition can form a linkage with the Sobolev spaces defined on g and on domain X' C Rd, which is also utilized in Haas et al. [24]. Our exposition begins with the characterization of a manifold.
Trivilization Define a trivialization of a Riemannian manifold (M, g) with bounded geometry of dimension d, which consists three part. The first part is some locally finite open covering (U, Joel-
The second part is the charts (Ka) Jaes which consists of smooth diffeomorphism Ka V. Rd - Ua. The third part is a partion of unity ha such that supp(ha) C Ua. EaEl ha = 1 and 0 S ha S 1.
In our case, we write a trivialization of Sd, which, is a manifold of dimension d. We write U1 { < e|x E 8d and U2 {I > f/ Sd for a small fixed > 0. Let Φ1 U1 + Rd and Φ U2 Rd be stereographic projections with respect to æ (0,0, 0, 1) and 32 (0, 0, ... -1) respectively. Namely, they are
Φ : (01, F2, ... Id+11) +
(21, .. (I)
(52)
1 + xd+1
and
1
2 (Z1, .2. 1a411
(21, x2, x.))
(53)
2d+1
Finally, we can find 8 smooth functions h1 and h2 such that halge 1. For the simple trivialization above, we can directly verify that it meets the admissible trivialization condition (details see Große and Schneider [23]). Thus we can apply Theorem 14 of [23] to define the norm of Sobolev space on gd:
|| (894)) (1((21))
"I + (122f) o5 '12 (
(54)
for distribution f D'() [49]. It gives kind of equivalent definition of Sobolev space on gd
18

---

C.3 Relationship between dot-product kernel and Sobolev space
Previous work observed that, for dot- not-product kernels defined on sphere with polynomial eigenvalue decay rate, their RKHSs are equivalent to Sobolev spaces:
Lemma C.4 (Hubbert et al. [30] Section 3). For a dot-product kernel k defined on S and its RKHS
Hg. if the coefficients of spherical harmonic polynomials satisfies pn X n' for some t z d, then there exists an equivalence between RKHS and Sobolev space:
H₂ _ H
Recall that KTTK and KRFK are both dot-product kernels with polynomial eigenvalue decay rate by Lemma C.I. Therefore, Lemma C.4 provides the equivalence between HbT HDP and the corresponding Sobolev spaces on Sd We have the following proposition: Proposition C.5. We have the following equivalence:
H
11 + (Sd)
and
HD
RF
= H 7 (Sa)
(55)
C.4 Interpolation of H and HDp
In this subsection, we aims to provide the interpolation relationship between RKHSs associated with KN and KRFK, on subdomain of g We remind that if we consider the case on S i.c. H.T and HRF the conclusion is direct since they are both dot-product kernels and share the same orthogonal basis {Y,,,} as introduced in (37).
Suppose 2 d and Ω be a subdomain of gd with C smooth boundary. With little abuse of 2 notation, we define (S2) as the RKHS H**) restricted to Q in the way of Lemma H.3. For an injection
Ω + Rdd we define (*(()) o == {f o op E H²((()))) with norm IIʃ Pll = |I =) Recall that Φi is the stereographic projection defined in (52), now let us show the equivalence of (S ) and ((((((4)) Φ1.
Lemma C.6 (Equivalence as sets). Suppose 2 $. At the aspect of sets, we have H* H (e1(()) 01.
Proof. For a H²()), have extension (1 co such Mls J. Thus
[[h)f") or '1 11 $ ||f o Φi (- <
(56)
which implies (hi of E H((()) Then we have [(hif') of (T'11o1184) E ($))). Since f f= f= and h lsg 1, we have f fo Φ E (01
In the converse direction, we assume f E (((())))) Then know there exists E H (RR)) such that I' (e (84) = f. Now we want to show fo E (SL).. Define a dp E C (RR) such that (i (84 = 1 and (U1)))))) O. According (54), we have
Ilf dill (1)) < |1( f') 01 @1ll = =|( o ) . f (IIR <0
(57)
Thus we finish the proof.
Lemma C.7 (Equivalence as space). Suppose 2 2. At the aspect spaces, we have H*($4) (Ø (8)) o Φ1.
Proof. By Lemma C.6, we know ($4) & ()1 Φ1 sets. Since H*($4) and ( ($4) Φi are both RKHSs, we can finish the proof by closed graph theorem.
( ( 10 Define the canonical map 1 Hi + H2 as I h +> h. Let (hn, )eN be a sequence such that there exists Hi and E H2 where h, - h in 74 and h, Th, + g in H2. It implies that h 9. Therefore, closed graph theorem shows that the linear operator holds for some positive constant C and any h E Hi. We can also prove |T S C' C' for any h in the same way. Consequently, the lemma is proved.
For notational simplicity, denote by Hi = ( and H2
bounded, which means that ||Ail2,, < <CllAll H
L
19

---

Now we come back to our network case. Let = ((x) C 8 where X' is the set from which data sampled, and Φ used in (42). Since the boundary of X is C smooth, we know that S is C smooth. If we combine Lemma C.4, Lemma C.7 and Proposition H.4, then we can directly show the following lemma:
Lemma C.8. Define X1 (i( For KNTK and KRFK defined on S. we have the following equivalence:
H (S) = (X1) o Φ1, and
(58)
= H (xi) o $1
Now we can obtain the interpolation relationship between Hi (S) and Lemma C.9. Suppose 2 We have
[A) = (HD (5)]
Proof. Define Xi = (S)... Let a be the uniform measure on Sd, Recalling (51), we have the interpolation on 21 with lebesgue measure denoted by ju:
o(d+1) 4+3
r(d+1)
(X
(Xi) = H (x)))*
(59)
that is
(1,, M2), H 443 (2i)
(1i, H 242 (21) s,2
(60)
ا 22
Since f + fo Φi is an isometric isomorphism, we have
(1'( Φ1, H 1 (2i) 01) ((ي 2
(1) H + (Ri) ( (21) 00
(61)
Recall that X' is bounded and thus Xi = (d(())) is bounded. Therefore, the Jacobian JØ1 satisfies S [J < C for some constant c and C. It is easy to verify that (21,),))) o Φ1 = (3 μn o Φ1) = L (S, Finally, with Lemma C.8, Lemma H.5 and Lemma H.6, we have
[7R (S) oidi
[M
(62)
with respect to the uniform measure on S.
C.5
Smoothness of Gaussian process
Lemma C.9 provides the interpolation relationship between HD HNT((S) (S) and HBF(S). By the kernel transformation relationship of NTK and RFK from IR and to Sd as described in (42), we can also derive the interpolation relationship of NT and HRF It will help for to derive the smoothness of fGP.
Lemma C.10 (Interpolation of RKHSs). Suppose O. We have
[XTT())* = [R (x)) (d+1) 413
with respect to measure μ on X which has Lebesgue density S p(x) C.
Proof. Define a function p(x) ||ا|| on X. Define measure v on X' such that the Radon-Nikodym
derivative satisfies du = p². We consider measure o & on S well as measure p on X, and then du
NT (5)]* - NT ( define map T: [HN
I: l:f---(foo).
(63)
Now we prove an isometric isomorphism. We first show that for any eigen pair (f, A) of (K SS Φ), (Lf is also an eigen pair of (KNTK p). Actually, for eigen pair (f, we have
K KNTK (x,v))(v) d(v Φ) (v) = Af(z).
(64)
20

---

We perform a transformation of the integral domain,
J
Kg K
(0()))))))) Af(o(z))
(65)
Ko
Ki
Recalling the transformation between K and KNTK (42), have
(y)dp() = {o(z){(((x))
(66)
= =/e K(x (f
These transformations are both reversible. Therefore, through the structure of real interpolation space as described in (6), we can see I is an isometric isomorphism. In the same way, there exist isometric isomorphism /' (7O (S))
a(d++))
[HRF aid+1)
f.
(67)
Combined the result in Lemma C.9, the Lemma is proved.
Now we are ready to give the smoothness of Gaussian process fGP. We remind the reader that HNT and HRF are abbreviations used for denoting HNT and HRF (X), respectively.
Proof of Theorem 4.2. Let t
s(d+1) d+3
simplify the notation. By Lemma C.10, we have [H = [HRF)]
(68)
Recalling the structure of interpolation space, we suppose RF can be written as
RFF
Σ 2 <
(69)
iEN
iEN
Recall that f fGP represents a random function defined on (Ω, F, P), where each w E Ω corresponds to a path function fCP X R. We can express this in the orthonormal basis as f Eiena: az(w) es, where
(ı(w) = ( A pe)pp = =,, f (ePe())ddul))
Recall that as defined in Lemma 3.2, fCP has the distribution GP(O, KRFK). From this, we can acquire the joined distribution for a, Firstly, let us compute the covariance: Cov(a, aB) = E(a,, az]
x 1/2 x3
/ f (x)) ())))())))) (
=E
-,,1122
E [s (w) (x())) dn(z)dp(y)
(70)
x
X x/, 1/2 / KRFK (z. (2. (1) dn(x)dp())
A;- (1-02/ x, (1-1)/2 (1=z})
1(=0))
The exchange of integration is accomplished by Fubini's theorem since KRFK is a bounded kernel function, and both e; and e, are L2 integrable. Moreover, as fGP is a Gaussian process, we finally get a; (O, x²-1) for i E N, and az 0 are independent for any * j. Consequently, we can directly derive that
GP
Σ Ex-1Z2
(71)
iEN
where } indicates collection of independent and identically distributed standard Gaussian random variables. Finally, as Lemma C.3 establishes the eigenvalue decay rate as
A X
(72)
it is direct to prove the theorem.
21

---

Part 1.
When s <
3 d+1
we have d-3 (1 - t) and thus
E
GP
Jia (1-1) < t00.
(73)
[NNT]
iEN
Consequently we have P
GP
x0 1.
[MST]*
Part 2.
When 2 3 ascertain that d+3 (1 (1- -1-t) t) S and consequently
E
G
(1-( +00.
(74)
[NT]
EEN
Denote by X, = - ZZ We then obtain
EXa = EN..
VarX, = Σ 21
(75)
1=1
1=1
We can thus derive that
EXP,
EX,,
4VarX,
8
P(X, S
S P( EX, 2
I
(76)
2
EX,, 2
E A
1--7
Given that GP [MxT]] Xn for any n E N.,, we have
P([ 2P!2 [HNT]* -00 lim -
P(
[NT]]* 2 MM21-
lim n-000
P(Xn S 2
EX,
=. (77)
This completes the proof.
D Proof of Theorem 4.3
With the findings from Theorem 4.2, Proposition 4.1, and Proposition B.4, the influence of non-zero initialization could be interpreted in terms of a misspecified spectral algorithms problem. To apply Proposition 2.2, only remains determine the embedding index of HN, Now, let's proceed to do so.
D.1 Embedding index of NT
Recall that the Proposition 2.2 requires the embedding index of HT X' under the probability measure μ. Fortunately, the embedding index of the Sobolev space has been previously established by Zhang et al. [55], which is helpful to simplify our proof.
Lemma D.1 (Zhang et al. [55] Section 4.2, Embedding index of Sobolev space). Suppose r 2. For a bounded open set C TRd and Lebesgue measure p. the embedding index of H" HD (X) equals 2r-
Since we have established the relationship between HNT and the Sobolev space, we can easily get the embedding index through similar way used in the proof of Lemma C.10.
Lemma D.2 (Embedding index of NTK). Suppose that the density function p(x) of probability measure μ satisfies the condition embedding index of NT (X) with respect to is concluded to be d
p(z) S C, where and C. are positive constants. The
ă+I.
We omit this proof as it can be carried out in the same manner as Lemma C.10. Here, we provide only d the structure. First, the embedding index of H d=1 (S) is Z+T (Lemma D.I and Lemma C.7). Second, the embedding index of NT (S) is स41 (Lemma C.4). Third, the embedding index of (ENT(X)) is d since I : f + p (f o) is isometric isomorphism both from (TT(S) HN (X) and from d+I
L (S.v' () to _ (X, 3 p), where measure / is defined on de' du
p (an argument similar to that in
the proof of Lemma C.).).
22

---

D.2 Proof of Theorem 4.3
Proof of Theorem 4.3. Recall that Proposition 4.1 elucidates the impact of non-zero initialization. Namely, the generalization error of the kernel gradient flow with an initialization of fo and a regression function fr, consequently equivalent
that of kernel gradient flow with initialization at 0 and regression function of I* fo- On the other hand, Proposition B.4 demonstrated the uniform convergence from the network function to the kernel gradient flow predictor as the network width m tends to infinity. Lemma D.2 verify the embbeding index condition in Proposition 2.2. Thus, we only need verify the source condition that fGP order to derive the generalization error of the kernel gradient flow.
fulfills and to incorporate it with Proposition 2.2 in
Now we start the proof. Since the proofs for the cases same, will only provide the proof for the former case here. Through Theorem 4.2, we know for any 0 < r < 3
ब and < a+1 are exactly the
d+1- it follows that E [||| [[ANNT] = Ex < co. Let C2 = E|| [HNTT By
the Markov inequality, for any 8' E (0, 1), we have with probability exceeding 1 8', that
R C.
1 -f -f 1
(78)
8'
Recall that the eigenvalue decay rate for KNTK is d+1 have for any d E (0, 1) and any (0, d+3 ), there exists < 7 such that r8 3
d as mentioned in Lemma C.2. Therefore, we
3
a+3 (i.e.,
rp+1
r 3(4+))+3 (4+ (2+))333)) Denote by f f - PGP and JNTK be the kernel gradient flow predictor
12 6d 34
starts from initial value O. Through Proposition 2.2, We thus have
NNTK - jm 1 (3) 8'
In 6 2) (R C.+CC'Cn-- A
(79)
holds with probability at least 1 - 28' when x TI Through Proposition 4.1, also we have
g) 2
| 1 s 8
(R (R+ CR+CC)O''n atste
(80)
holds with probability at least - 28'. Through uniform convergence in Proposition B.4, we have
1 in )) (R C.+CC A
| NN - fzd 12 || |
sup E20
(81)
with probability least 1 - 8' when m is large enough. Therefore, with appropriate choice of 0' and C², we can finish the proof.
E Proof of Theorem 4.4
In this section, we establish the generalization error rate lower bound in our problem. We incorporate result delincated in [40], which systematically studies the learning rate of kernel regression. Prior to this, we take some preparatory work.
We assume l is dot-product kernel on Sd with eigenvalue decay rate 3, with respect to the uniform measure. We notate the corresponding RKHS as H4. Then, we can verify that Hg satisties the definition of regular RKHS, as detailed in [40]. Subsequently, the main theorem in [40] can be applied under our proposed settings, since K a dot-product kernel defined on S engenders the following lemma.
Lemma E.1 (Generalization error lower bound). Assume l is a dot-product kernel defined on S we have the interpolation space of its RKHS as [Hx]* {EiEN es EiEN of < oo) where vee the orthonormal busis of La (2d o) and denotes the uniform measure. Decompose the regression function f* over the series of basis:
fee
(82)
23

---

We assume that f E [H]] holds for any t < ffo given s > 0. Also, we we assume that
(83)
GAZA:
We also assume that the noise term satisties main bias term in generalization error by
B||c = o² holds for E Sd, a.e. Then, we define the
(t:
Σ iEN
2tA: X
(84)
and define the variance term by
N(1) Σ Pie 12.
(85)
VEN
Fix the given input vectors of samples X. Consider the kernel gradient flow process detailed in (8) and let it start from For any choice of = t(n) - DG, we have
E
|
1
Spp
R (1= N
(86)
n
With the lemma above, now we are ready to prove Theorem 4.4.
Proof of Theorem 4.4. By Proposition 4.1, we know the initial output function introduce an implicit bias term to the regression function. And thus the original problem is same as to consider a standard kernel gradient flow problem start from initial output zero with regression function f== =f for Recall that y is the uniform measure. On sphere, the RKHSs of dot-product kernels K and KB KRFF are equivalent to corresponding Sobolev spaces through Lemma C.4. More precisely, suppose that we have chosen orthonormal basis {er consisting of spherical harmonic polynomials. Then we have
[74
NT
Σ iEN
1/2 C4
00
SEN
(87)
(HBF)
x0000
SEN
SEN
for any t 2 0. Through Lemma C.2, we have the eigenvalue decay rate:
w, i
and
A A
(88)
We denote by 81 d+1 and 32 Kosambi-Karrunnn-LLorv expansion of f
d+3
Similar to the proof of Theorem 4.2, we write the
d
jr = Fre
Σ (b: - a;)a
(89)
iEN
iEN
where
04 i.d. N(O, 1), and f= -' b,2b/20 E [H
(90)
iEN
Here a; is sequence of independent standard Gaussian variables, and b, represents sequence derived from the decomposition of f*. With such decomposition, we can verify that (83) holds with probability 1 8' for any 8' (0, 1). Denote by g()) Eaxso, 1 Firstly, we have
E[g(A)] E
Σ
E
Σ
1
(91)
LAA
We also have the variance
Var [A() Var
Σ
1
Σ
x
Σ
(92)
24

---

Where the second term is controlled by the source condition assumption on f*. Therefore, we have
9(A) ) -
[Varlg(A]] O(
P (O(()) - Elg(AMI 2 E
(93)
2
(Eg((A)))
Define event A()) (1g(A) ([D( Elg())1/2). For any 8' E (0, 1), choose sequence Aj. such that A = C" 1)
Then we have
P (USENA (2,)) 21 2
(94)
JEN
We can choose appropriate - 0 such that U(E))(()) holds with probability at least 1 8', denote by event A. Conditioned on event A, for any ds+1 S S Az. we have
.(A) ((xg+1) 2 / (( o
X--1
264+13
and
(95)
5+1
Aj
which shows that
( 2 2 (A5+1))
1 a
dti.
(96)
[j+1
Therefore, we finish the proof of (83).
Then, we turns the calculation of generalization error lower bound. First, we plug the decompo- sition and calculate the bias term R(t; f)):
R1:
f Σ
2th: (a - -bb,a + b2).
(97)
EN
Recalling that the eigenvalue decay rate is denoted by 3, it follows that
E
R² ftt
21d, X
Σ
t -
(98)
BEN
VEN
Also, the variance of (1; )+) follows that
Var(R" (t: (t: f*)) S Σ
X2 + Σ
e -4Ai
(99)
iEN
iEN
Here we introduce the denotations:
V0 = Σ
and
V/2 Σ
-ttd 6? X2
(100)
iEN
iEN
We then have
Vi= = Σ
1th
Ati 91
20 x 11 2
(101)
iEN
IEN
As to V2, we first recall that the smoothness of f lead to the following inequality:
xo,
(102)
which implies that
Σ b < x0.
(103)
AEN
Now we turn to the evaluation of V2:
V2
4th
13
185
12.1-22
-11 12i-11;22044
VEN
iEN
NEN
(104)
<
8ti
j 42
bli-2 <
2 + +
VEN
1EN
25

---

It is worth noting that we use Cauchy's 's inequality to derive the upper bound above. With the control of V and V2, we have
(tr(R = Vo + V/2 S S 2
(105)
Consequently, by Chebyshev's inequality, we directly have
P f*) E R$ (t: ft);) [
12) =
(aa = 0 (
(106)
E
(2: ft)
Since t = t(n) - +00, we have
(²: f*)) = Slp (t $
(107)
In the same way, we also have the bound of variance term (t).
- N(t)
(108)
71
Finally, apply Lemma E.1 and Proposition 3.3. We derive that for any d >> O, long as n is large enough and m is large enough, for any choice of t t(n) + oc, with probability at least - å we have
E [IIA 11 a R² ===22
52
22 p1
1 +
1
1 10nnaaat) a43
(109)
n1
Thus the theorem is proved.
Remark E.2. In Proposition 3.3, we consider the situation that both input and output Y of samples are fixed, while in the proof above we require that only X is fixed. However, the conclusion of Proposition 3.3 still holds when of samples is random. This because the noise term has a finite second moment. Also, the change of domain from X' to Sd will not affect the uniform convergence result.
F
Details in artificial data experiments
Fixing the dimension of data as d 5, 10. We draw samples for variable from the standard Gaussian distribution (0, Ia), which are consequently standardized to lie on the surface of the unit hypersphere $d. The dependent variable y formulated as:
y = f(x) EE,,
(110)
where (xx -Eiz1
o() and 0.2. The function f exhibits notable smoothness,
since it can be linearly represented in terms of the first few spherical harmonic polynomials on gd S
[9] and the fact that KNTK is dot-product kernel. We consider fully-connected network with one singular hidden layer, choosing m 20 n to ensure large enough width. Consistent to previous sections, we choose ReLU function as the non-linear activation and train the network using Gradient Descent for a sufficiently long time. We record the generalization error at each moment and define the moment of minimum generalization error as the final generalization error. This is done to align with the early stopping strategy mentioned in the Theorem 4.3.
G Details in real data experiments
In this subsection, we will provide the theoretical basis of the method which approximates the smoothness the goal function of real dataset. Let X' C Rd be a bounded domain. Given reproduce kernel k(-,-) on X' and probability measure p. Denote the RKHS by H4
{ErEN aide es Ever of < }. We assume that there is function f X + R and proba- bility density a on X. Suppose that the samples satisfies fx)), then f has the decomposition:
Σ 0,e1-
(111)
26

---

The smoothness ay = =(f.k) depends on the coefficients if we have 0; xi and X x xidd, ds, then we derive the smoothness: af
2d.
We consider n samples ((ri, (:) The Gram matrix k(X, can be decomposed as
k(X, X) DEΦ and
666 = In.
(112)
In this regard, we can utilize the eigenvalue of the empirical kernel matrix estimate the eigenvalue of the kernel function, since previous work has shown the convergence of eigenvalue when n is large enough [32]. Through the decomposition Y de (i.e., c Y and the approximation ez 0,, we can roughly estimate the eigenvalue decay rate when is large enough with respect to i:
Σ axrrd
(113)
In our experiments, we let n 3000. Namely, an arbitrary selection of 3000 samples was made from the each dataset. We did not use all samples in the datasets, because n 3000 already sufficient to calculate the decay rate of eigenvalues. We consider the NTK of a one-hidden-layer fully connected network as the kernel The results shown Figure 2, Figure and Figure 4 for the three datasets, respectively. In each figure, the scatter plot shows the log value of the summed squares of each ck (for S k S n as per equation (113)) against log1n on the x-axis. Also, the dashed line represents the corresponding leaat-square regression fitting using index smaller than 2700. Theoretically, the slope of the dashed line will be -ayd3-
MN
Desay Fale
lggol
Figure 2: Decay curve of the logarithm of sum of squared coefficients for NMIST.
fashicn-MNiST
Daray rate +0.2
05-
-05
-10
0.0
0.5
10
15
2.0
2.5
3.0
3.5
Figure 3: Decay curve of the logarithm of sum of squared coefficients for Fashion-NMIST.
27

---

CIAR-10
Decay 0999
10-
0.-
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
lsgul0
Figure 4: Decay curve of the logarithm of sum of squared coefficients for CIFAR-10.
H Technical Lemmas
In this section, introduce series of technical lemmas. These will be helpful in our proof, and many of the lemmas have been established by prior researchers.
Lemma H.1 (Change of measure [41]). ). For a positive definite kernel defined on a compact set X, it has the same eigenvalue decay rate under two measure and
X,(K .V = >,(Ko ,.)
if the Radon derivative du exists and S C holds for some positive constant and C. do
Lemma H.2 (Skorohod's Representation Theorem). Suppose that a sequence of probability distribu- tion {F converges weakly to F and F has separable support. Then there exist random variables X, and X, defined on a new probability space (,',F, P), such that the distribution of Xn, is Fn, the distribution of X is P, and X,, - X holds almost surely.
Lemma H.3 (Restriction of RKHS [3]). Suppose H4 is RKHS defined on É with the norm Illl
then le a restricted to subset Ω C E is the reproducing kernel of space {f' = flal, E Ho? with norm defined by
|| = min
(114)
fla=f'
The following proposition is direct proposition of Lemma H.3:
Proposition H.4 (Equivalence of RKHS under resriction). Assume RKHSs Hi _ H2 defined E. Write Ω as a subset of E. Then we have Hi la H2ls2
The following two lemmas are common in real interpolation:
Lemma H.5 (Equivalence of interpolation spaces), Suppose 0 < < 1. Denote L = (2 for abbreviation. If we have RKHSs H. = H2, then (L², (2,,)) (L.2, H2)
Proof. To prove the lemma, we only need to prove that the embedding L2),2 (L2, H2) and (L2, H2)s.2 + (L?, H2),.2 are both bounded.
First we prove that [|,2,2 (1)s,,2 > (22,H)) < C1 where C1 is an absolute positive constant.
For any x E _ + Hi, define the K-functional
inf (1(1;))= xn =0+21=x:0E on EH
(1|=ollıa
inf K2 2a4z1 2 a EH2 (1|001112 + f|| ll
Since H41 H2, for any x E H1, we have || S Cl| Thus we have (llzollza + Ct ll,,) Ki(Ct;));
K2(t;o) S
inf
= EHE
28

---

Then we have
00
dt [t= "=([k=))]] -
J0
- t ['(((Ct dt
t
(115)
S C I [(Ct) d(Cl)
Ct
C -=ll
Let C1 C²2 have the canonical injection satisfies [|(L2, H1) + (L2, (22)22)s S C1. Also, since H1 = H2, for any x E H2, we have || S c|alll can prove ((I2,H2)ss > (I?, H1) H1)s.2llon S C2 in the same way. Then, we finish the proof.
Lemma H.6 (Equivalence of interpolation spaces), Suppose O and p1 V be measures on set X. If we have (2, = (2(X, then (I, p), H)s22 = (L²(X,)), v), H)
Denote H be a RKHS
Proof. The proof in accomplished in the same way as Lemma H.5.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and the introduction, we have summarized the background of the NTK theory and the influence of initialization neural networks under this background. We have also discussed the results, which reflects the contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Compared to the other sections, stronger assumptions were used in the lower bound section (data distributed on the sphere). Although this is common in kernel regression, we still reminded in the paper that this is a more strict assumption.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
30

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have provided assumptions in the main body of the paper, and have also given comprehensive theoretical background information and proofs in the appendix.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In our experiments, our model is simple and uses public datasets. The experimental results are easy reproduce.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path to reproducing verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, the datasets we utilized are publicly accessible for everyone. We have comprehensively outlined the experimental parameters and model settings within the experi- mental section our paper. Notwithstanding the low complexity of our model, we are more than willing to post our code on GitHub should there be demand for it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
http://gguiieeeeeeeeuabiiii sot.uubmiiiiooooicccooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: hwwbubliigguids....e /uodee///mmdSddmiiiiaaa ssisnnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper has indeed provided all necessary training and testing details for understanding the results. We leveraged the several datasets which are publically accessible to everyone. Furthermore, we have comprehensively outlined all the experimental parameters and model settings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the artificial data experiment, decay rate in the figure. In the real data experiment, we do a least square regression to calculate the smoothness of a function.
directly see the generalization error
Guidelines:
32

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the computational resources needed are minimal, which means the experiments can easily conducted on nearly any GPU without any additional requirements.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://nnnt http:://nuuippppe cccpppplicc/chhccccuueeellllelle
Answer: [Yes]
Justification: Yes, we conducted original theoretical research, which is in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
33

---

Justification:
Our work is mainly theoratical work focusing on learning theory, so this question is not applicable to our paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is mainly theoratical work focusing learning theory, so this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have explicitly referenced the datasets in our paper, MNIST, CIFAR-10 and -NIIS..
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
34

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not utilize new assets in our work.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments or research related to human subjects, therefore this question is applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any study participants, there is no need to discuss potential risks they might face. Therefore, there also no need for an Institutional Review Board (IRB) approval. Hence, this question is not applicable to our paper.
35

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "1MpiOeTcqG", "reviewer": "Reviewer_43yC", "summary": "This paper studies various kernel theories of neural networks, both random feature and neural tangent kernels. The main approach is to use the decay rates of the target function and kernel eigenvalues to get generalization error rates. The paper's novel theoretical contribution is to show uniform convergence of the network's input-output function $f$ to that of a kernel regressor with NTK kernel. The setting is that the weights are scaled to be in the NTK regime and when both are trained under gradient flow. Previously known results are used to characterize the RKHS associated with the NTK and \"interpolations\" of this space (which is useful if the target function isn't actually in the RKHS). It is argued that, since the network initialization with random weights is not smooth, that the network overall will have a slow error rate. These theoretical arguments are backed up with experiments on synthetic data comparing the standard initialization, that was studied theoretically, with a \"mirrored\" one. Also, the smoothness of the MNIST, CIFAR-10, and Fashion-MNIST classification problems is estimated and found to be quite different from the basic bound based on input dimensionality.", "questions": "* Please clarify the definition of the interpolation space (Eq 6 and around it). Your notation for the space of functions is unclear to me. Some explanation should be given. It is unusual to see a function written as a series $\\sum_i a_i \\lambda_i^{s/2} e_i(x)$\n* Fix the formatting issue in line 104\n* Prop 2.2 \"satisfis\" typo\n* Do you think it would be more consistent to use the NTK superscript everywhere you treat it rather than f^NTK and K^NT ?\n* Line 267 \"network on longer\" typo\n* Does the mirror initialization lead to a significantly different error bound than yours, Thm 4.3?\n* Can you include the method of estimating the smoothness of the various classification problems (Table 1) in the main text? Looking at the appendix, I see it depends on an empirical kernel/Gram matrix, but under what kernel is not clear. NTK?\n* You report various least-square fits on log-log plots. You should be clear about how these were performed. Linear fits in log-space are known to have problems versus least-square fitting with power-laws. See the work by Clauset et al.\n* In your notation section, you use standard $O, \\Omega, o, \\omega$ symbols but do not define the frequent $\\asymp$ symbol. This should be explained.", "limitations": "The work here is very focused on the NTK theory of a multi-layer perceptron/fully-connected network. Strong claims are made about the weakness of such theories. However, different architectures lead to different kernels with different spectral properties and generalization performance. For instance, convolutional architectures lead to convolutional kernels which are known to work better in image classification and other spatially structured settings. It is also known that initialization with weights that are not Gaussian white noise can lead to better performance (see the work on \"structured random features\" and \"rainbow networks\").\n\nI did not check the appendix proofs, but I read all of the main text results.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The work here brings together a bunch of connected kernel theory and neural networks results in a (relatively) readable treatement. The uniform convergence of the network's output to a Gaussian process is good to know. The overall goal of understanding the effect of initialization is certainly important, as is highlighting the weaknesses of kernel frameworks in understanding neural networks.", "weaknesses": "* In the introduction and experimental sections, the standard and mirror initializations are contrasted. From the introduction, I was expecting to see theoretical results that differentiated between the mirror and standard initializations. The authors could be clearer about what their contributions are and more clearly state what is known about the mirror initialization. \n* As someone who is fairly familiar with kernel theory of neural networks, I still found it hard to identify which theorems and other results presented by the authors here are actually novel and which are applications of known results to their setting. For instance, isn't the decay rate of $1/(d+3)$ classical and known from Sobolev theory? I would suggest that the authors clarify this. \n* The bulk of the paper is really just setting the stage for your results. I would have liked to see more explanation of how to interpret your results and discussion of how they fit into the greater understanding of these networks/kernels. I would also have liked more detail on the experiments within the main text.\n* The synthetic experiments (Fig 1) only cover less than an order-of-magnitude in $n$. They would be much more convincing if they covered more. However, I can imagine you'd run into issues with double-descent and other factors that your asymptotic theory could have trouble with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Brief Introduction of Mirror Initialization\n\n*\"In the introduction and experimental sections, the standard and mirror initializations are contrasted. ...... what is known about the mirror initialization.\"*\n\nThank you for your suggestion. Here we provide a brief introduction of the existing theoretical results on mirror initialization.\n\n- Under mirror initialization, [lai2023generalization], [li2023statistical], and [lai2023generalization2] studied the generalization ability of single-layer fully connected, multi-layer fully connected, and multi-layer residual network, respectively, and proved the minimax optimality of network. For example, [li2023statistical], which studied the generalization ability of multi-layer fully connected mirror-initialized networks, concluded that for $f^* \\in [\\mathcal{H}^{NT}]^s, (s \\geq 1)$, the $L^2$ generalization error is $n^{-\\frac{s(d+1)}{s(d+1)+d}}$. This differs significantly from the $n^{-\\frac{3}{d+3}}$ we obtained, reflecting the theoretical differences between different initialization.\n\n### More Explanation of Our Contribution\n\n\n*\"As someone who is fairly familiar with kernel theory of neural networks, I still found ...... I would suggest that the authors clarify this.\"*\n\n\nBefore the response, we apologize for a typo. As mentioned in Theorem 4.2, the smoothness of the Gaussian process is $\\frac{3}{d+1}$. Therefore, the smoothness in lines 250, 260, and 281 in the Theorem should also be $\\frac{3}{d+1}$ instead of $\\frac{1}{d+3}$. This typo does not affect our results.\n\nThanks for your reminder and we are willing to respond to your questions. Since NTK and Gaussian process are well-studied objects, we do need to clarify what the novel contribution we have done here.\n\nThe major contribution appears in Theorem 4.2 where we showed that $P(f^{GP} \\in [\\mathcal H^{NT}]^{t}) = 1_{t \\textless \\frac{3}{d+1}}$. Consequently, the generalization error of the gradient descent with standard random initialization is $n^{-\\frac{3}{d+3}}$.\n\nTo the best of our knowledge, this cannot be simply concluded from classical Sobolev theory.\n\n- **Challenge of Theorem 4.2:**\n  - If the input is uniformly distributed on the sphere $\\mathbb{S}^d$ instead of $\\mathcal{X}$, both NTK and RFK are inner-product kernels and their Mercer's decomposition shares the same eigen-function (i.e., spherical harmonic polynomials). In this situation, simple calculations on the eigenvalues would show that $f^{GP} \\in [\\mathcal{H}^{NT}]^{\\frac{3}{d+1}}$.\n  - However, when the data is a general distribution supported in a bounded open set $\\mathcal{X} \\subset \\mathbb{R}^d$, NTK and RFK will not possess the nice property anymore which makes the claim if $f^{GP} \\in [\\mathcal{H}^{NT}]^{\\frac{3}{d+1}}$ unclear.\n  - Before we clarify our solution and technical contributions, we first recollect some notations. For a sub-region $S \\subset \\mathbb S^d$, let $\\mathcal{H}_{0}^{NT}(\\mathbb{S}^d)$ be the RKHS associated with the homogeneous NTK defined on $\\mathbb S^d$ and $\\mathcal{H}_0^{NT}(S)=\\mathcal{H}_0^{NT}(\\mathbb{S}^d) |_S$ be its restriction on the sub-region $S \\subset \\mathbb{S}^d$. We define $\\mathcal{H}_0^{RF}(\\mathbb{S}^d)$ and $\\mathcal{H}_0^{RF}(S)$ similarly.\n  - We first show that, on a general $\\mathcal{X}$, $\\mathcal{H}^{RF} \\cong [\\mathcal{H}^{NT}]^\\frac{d+3}{d+1}$ (where the $\\cong$ is isomorphic as Hilbert space not RKHS).\n  - Let us fix a continuous and isomorphic mapping from $\\mathcal{X} \\subset \\mathbb{R}^d$ to a subregion $S \\subset \\mathbb{S}^d$. We then show the equivalence (as RKHS) $$\\mathcal{H}^{RF} \\cong \\mathcal{H}_{0}^{RF}(S), \\mathcal{H}^{NT} \\cong \\mathcal{H}_0^{NT}(S).$$   This equivalence is not as trivial as it appears.\n  - We then need to show that $[\\mathcal{H}_0^{RF}(S)] \\cong [\\mathcal{H}_0^{NT}(S)]^{\\frac{d+3}{d+1}}$.\n  - Notice that the Mercer's decomposition of RFK and NTK share the same eigenfunctions (i.e., the spherical harmonic polynomials), we know that $[\\mathcal{H}_0^{RF}(\\mathbb S^d)] \\cong [\\mathcal{H}_0^{NT}(\\mathbb{S}^d)]^{\\frac{d+3}{d+1}}$. If the restriction $\\mid_S$ and the interpolation $[]^{\\frac{d+3}{d+1}}$ are commutative, then we are done. Unfortunately, this commutativity is not easy to verify in general.\n  - Fortunately, by a recent result [haas2023mind], we have $\\mathcal{H}_0^{RF}(\\mathbb{S}^d) \\cong W^{\\frac{d+3}{2},2}(\\mathbb{S}^d)$ and $\\mathcal{H}_0^{NT}(\\mathbb{S}^d) \\cong W^{\\frac{d+1}{2},2}(\\mathbb{S}^d)$ where $W^{s,2}$ denotes the usual fractional Sobolev space. Since Sobolev space is a more tractable object, we then carefully verify that $\\mathcal{H}_0^{RF}(S) = [\\mathcal{H}_0^{NT}(S)]^{\\frac{d+3}{d+1}}$ and finish the proof. We would like to emphasize that there are several different definitions for the fractional Sobolev spaces and there are no similar results explicitly stated in the literature.\n  - By Lemma H.5, we have $[\\mathcal{H}^{RF}]^{\\frac{3}{d+3}} \\cong [[\\mathcal{H}^{NT}]^\\frac{d+3}{d+1}]^{\\frac{3}{d+3}} (\\cong [\\mathcal{H}^{NT}]^{\\frac{3}{d+1}})$.\n  - If we denote $\\mathcal{H}^{RF} = \\left\\lbrace \\sum_{i \\in \\mathbb{N}} a_i \\lambda_i^{\\frac{1}{2}} e_i | \\sum_{i \\in \\mathbb{N}} a_i^2 < \\infty \\right\\rbrace$, then the K-L expansion gives us that $f^{GP} = \\sum_{i \\in \\mathbb{N}} Z_i \\lambda_i^{\\frac{1}{2}} e_i$ where $\\{Z_i\\}$ are i.i.d. standard Gaussian variables. We can verify that $\\mathbf{P}(f^{GP} \\in [\\mathcal{H}^{RF}]^{t}) = 1$ for $t < \\frac{3}{d+3}$ and that $\\mathbf{P}(f^{GP} \\in [\\mathcal{H}^{RF}]^{t}) = 0$ for $t \\geq \\frac{3}{d+3}$. i.e., we have\n  $\\mathbf{P}(f^{GP} \\in [\\mathcal{H}^{NT}]^{t}) = 1$ for $t < \\frac{3}{d+1}$ and that $\\mathbf{P}(f^{GP} \\in [\\mathcal{H}^{NT}]^{t}) = 0$ for $t \\geq \\frac{3}{d+1}$.\n\n(We apologize for the inconvenience, due to the 6000-word limit, the second half is provided in the comment.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTTgPwprXT", "reviewer": "Reviewer_iQcQ", "summary": "This paper explores the standard random mode of initialization of neural networks through the lens of the neural tangent kernel (NTK). This connection is made by showing that a randomly initialized neural network does indeed converge to the NTK uniformly during training thus allowing to analyze the generalization ability of a neural network with more relaxed assumptions on initialization. Furthermore, this paper finds that neural network performance under this regime comes at odds with the NTK theory which indicates that networks should perform poorly for high dimension inputs.", "questions": "Line 186 - Can you specify what space $C(\\mathcal{X}, \\mathbb{R})$ refers to?\n\nLines 193-194 - (Less of a question and more of a comment) I am assuming NNK is just your notation for the standard empirical NTK...\n\nLines 274-275 - Why is it necessary to remove the bias term for the derivation of the lower bound?", "limitations": "Everything is adequately addressed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is sufficiently original in that it explores the more conventional initialization schemes used with neural nets and shows that, despite uniform convergence to the NTK, the kernel itself contradicts real world performance of the networks. This is a continuation of the theme in NTK theory where the infinite width regime fails to fully capture the finite's performance.\n\nThis is presented in a clear and thorough manner with rigorous proofs and straightforward, but convincing experiments. In regards to significance, again, this further builds on top of work that has \"poked holes\" in the practical usage of NTK theory and as such is a relatively significant work in that direction.", "weaknesses": "There are a few typos/grammer mishaps that I caught in the paper:\n\nProposition 2.2 - \"...noise term $\\epsilon$ **satisfis**...\"\n\nLines 126-127 - \"...*let us reall the* concept...\"\n\nLine 233 - \"This is **an** generalization...\"\n\nLines 266-267 - \"...the network **on longer generalizes** well...\" (does not make sense)\n\nLine 314 - \"...datasets *from real world* and estimate the *smoothness of function*.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Minors\n\nThank you for your thorough review and for pointing out the typos and grammatical errors in our paper. We appreciate your feedback, and we have made the necessary corrections as follows:\n\n1. **Proposition 2.2:** The typo \"satisfis\" has been corrected to \"satisfies\".\n    - Original: *\"...noise term $\\epsilon$ satisfis...\"*\n    - Corrected: *\"...noise term $\\epsilon$ satisfies...\"*\n\n2. **Lines 126-127:** The typo \"reall\" has been corrected to \"recall\".\n    - Original: *\"...let us reall the concept...\"*\n    - Corrected: *\"...let us recall the concept...\"*\n\n3. **Line 233:** The grammatical error \"an generalization\" has been corrected to \"a generalization\".\n    - Original: *\"This is an generalization...\"*\n    - Corrected: *\"This is a generalization...\"*\n\n4. **Lines 266-267:** The phrase \"on longer\" has been corrected to \"no longer\" to clarify the meaning.\n    - Original: *\"...the network on longer generalizes well...\"*\n    - Corrected: *\"...the network no longer generalizes well...\"*\n\n5. **Line 314:** The phrase has been corrected for clarity and grammatical correctness. The article \"the\" has been added before \"real world\" and \"function\" has been pluralized to \"functions\".\n    - Original: *\"...datasets from real world and estimate the smoothness of function.\"*\n    - Corrected: *\"...datasets from the real world and estimate the smoothness of functions.\"*\n\nWe have carefully reviewed the entire manuscript to ensure that no additional errors are present. Thank you for your attention to detail, which has helped us improve the quality of our paper.\n\n### Response to Questions\n\n1. **Meaning of** $C(\\mathcal{X},\\mathbb{R})$: \n$C(\\mathcal{X},\\mathbb{R})$ includes all continuous functions from $\\mathcal{X}$ to $\\mathbb{R}$ with norm defined as $\\lVert f \\rVert = \\sup_{x \\in \\mathcal{X}} |f(x)|$ for $f \\in C(\\mathcal{X},\\mathbb{R})$. In fact, Lemma 3.2 on Line 186 is the Theorem 1.2 in [hanin2021random]. The definition of $C(\\mathcal{X},\\mathbb{R})$ is consistent with that of $C^0(T,\\mathbb{R}^{n_L+1})$ in the latter. For more information on weak convergence of continuous random processes, we can refer to Section 7 of [billingsley2013convergence].\n\n2. **NNK:** The Neural Network Kernel (NNK) we define here is indeed the standard empirical NTK defined in some other literature. Thank you for your feedback and we apologize for the confusion caused by our notation. We will specifically mention this point in the main text to avoid potential confusion caused by the name.\n\n3. **Why Remove the Bias Term When Obtaining the Lower Bound:** This technical setting is for convenience in the derivation of our proof. After removing the bias term in network structure and further assuming the data is distributed on the sphere, the NTK is a dot-product kernel defined on sphere and the corresponding RKHS meets the so-called Regular RKHS condition (refer to Assumption 2 and Example 2.2 of [li2024generalization]). In general, this technical setting helps us obtain the lower bound of the generalization error.\n\n### References   \n[billingsley2013convergence] Billingsley, P. (2013). “Convergence of probability measures”.\n\n[hanin2021random] Hanin, B. (2021). “Random neural networks in the infinite width limit as Gaussian processes”.\n\n[li2024generalization] Li, Y., et al. (2024). “Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnfc6SjBeZ", "reviewer": "Reviewer_XKWd", "summary": "The paper aims to study the impact of standard random weight initialization - as opposed to mirror initialization - on NTK theory.  The key observation is that, for standard initialization, the operation of the network at initialization $f_0$ acts as a bias on the regression function $f^*$.  When analyzing the convergence of the excess risk in the kernel gradient flow model the convergence rate depends on the smoothness of the regression function, which is (effectively) $f_0+f^*$.  The paper demonstrates that, regardless of how smooth $f^*$ is, the smoothness of $f_0+f^*$ is bounded above by the smoothness of $f_0$ (theorem 4.2).  Non-zero initialization thus places upper and lower bounds on the generalization error predicted by KGF (theorem 4.3 and 4.4).  This is claimed to bring into focus the shortcomings of KGF in the NTK context for analyzing the convergence of neural networks even in the infinitely wide limit.", "questions": "My main question is whether or not it would be feasible to run mirror-initialization vs standard-initialization experiments on the real-world dataset, or even a high-dimensional artificial dataset.  My guess would be that the results would replicate the Figure 1, which could have interesting practical implications.", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "It is obviously important to consider the role of standard random initialization on the training and convergence of neural networks.  The argument presented in the paper is clear and gives an interesting insight into the predictions of NTK and KGF for understanding generalization and convergence.  Mathematically the results are strong (as best as I can tell, having skimmed the supplementary).", "weaknesses": "Perhaps I am misreading the results, but it would seem to me that, far from demonstrating the downsides of NTK theory, you have in fact shown that NTK theory correctly predicts that standard initialization can degrade performance (in terms of the generalization error during training) compared to mirror initialization for the artificial data.  With regard the real data, all I can see is that you have shown that standard initialization will indeed decrease (effective) smoothness; however your predictions are asymptotic, so without a side-by-side comparison of standard and mirror initialization training (as per the artificial data) I don't see how this demonstrates anything at all.  It would certainly be interesting to see if the results for real data are the same as for artificial data as this would (presumably) indicate that mirror initialization is to be preferred where possible.\n\n(fwiw this is the core reason for the discrepancy between my evaluation of soundness and my recommendation.  Mathematically the paper seems sound, but the interpretation seems incorrect to me: far from showing the downsides of NTK theory, the results would appear to show its power.  If I'm wrong or the paper is suitably modified then I am happy to upgrade my recommendation).\n\nMinor point: please run this paper through a spell checker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your valuable suggestion. It greatly helps us re-interpret our main results. More precisely, Theorem 4.3 and 4.4 actually tell us that the generalization ability depends on the smoothness of the target function $f^*$ and the initialization function. In particular, given the target function $f^* \\in [\\mathcal H]^{s}, (s \\geq \\frac{3}{d+1})$,\n\n- The mirror initialization $f_0 = 0 \\in [\\mathcal H]^{\\infty}$  and thus the generalization error is $n^{-\\frac{s(d+1)}{s(d+1)+d}}$ ([li2023statistical]);\n- The standard initialization $f_0 \\in [\\mathcal H]^{\\frac{3}{d+1}}$ and thus the generalization error is $n^{-\\frac{3}{d+3}}$ (our result).\n\nThus, the $L^2$ generalization error with mirror initialization is $n^{-\\frac{s(d+1)}{s(d+1)+d}}$ (which is minimax optimal) and even if the target function $f^*$ is sufficiently smooth, the $L^2$ generalization error with standard initialization is $n^{-\\frac{3}{d+3}}$ (which suffers the curse of dimensionality). i.e., the implicit bias introduced by initialization significantly impacts the generalization ability and we should prefer mirror initialization in practice.\n\nWe reviewed previous literature and found that [zhang2020type] compared mirror initialization with standard initialization. In Section 7.2.3 and 7.2.4 of [zhang2020type], Figures 3 and 4 show experiments on the Boston house price dataset and the MNIST dataset, demonstrating that mirror initialization (i.e., referred to as Anti-Symmetric Initialization in his notation) indeed outperforms standard initialization in terms of generalization ability. This confirms that NTK theory correctly predicts the impact of initialization in real applications, and we will emphasize this point in our paper.\n\nIn summary, we appreciate your highly valuable suggestion. We will revise our paper to emphasize the impact of different methods of initialization in the NTK theory. Thank you again for your insightful comments.\n\n\n\n### References\n\n  \n\n[li2023statistical] Li, Y., et al. (2024). “On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains”.\n\n  \n\n[zhang2020type] Zhang, Y., et al. (2020). “A type of generalization error induced by initialization in deep neural networks”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jtuXN9wIbT", "reviewer": "Reviewer_Yc4R", "summary": "In this paper, the authors discuss the impact of random initialization of deep neural networks in the neural tangent kernel (NTK) regime.\nPrecisely, the authors prove that in the case of standard random initialization, the network function in the finite wide limit, converges to the NTK predictor uniformly.\nThen, by studying the behavior of the underlying Gaussian process, the authors establish upper and lower bounds for the generalization errors of DNNs in the NTK regime, in terms of the real interpolation space of the RKHS.", "questions": "* Is Proposition 2.2 novel result? If yes, where is its proof?\n* it appears that the notations of expectation are not aligned in Assumption 1 and line 178.\n* The results of Theorem 3.3 and 4.1 are not very surprising and their proofs also look more or less standard\n* The result in Theorem 4.2 looks interesting, but essentially focuses on the Gaussian process with zero mean and random feature kernel of covariance function. Also, is this the FIRST time such results are established (e.g., in terms of real interpolation space of the RKHS)? How should we compare this result to other kernels/methods/models? It may be helpful to compare this behavior to the literature.\n\n---\nI thank the authors for their detailed reply and clarification. This helps me have a better grasp of this presented results, I will increase my score accordingly.\n\nI would like to ask the authors to carefully include the above discussions in a revised version of the paper, as also suggested by Reviewer 43yC.", "limitations": "I do not see any potential negative social impact of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper studies an important problem.\nThe paper is in general well written and not difficult to follow.\nThe numerical results, at least those with artificial data, look compelling.", "weaknesses": "It is a bit difficult for me to evaluate this paper. \nThe technical results of Theorem 3.3 and 4.1 are not very surprising and their proofs also look more or less standard.\nThe result in Theorem 4.2 appear novel and interesting, but it is a bit difficult to position this in the existing literature of, kernel or DNN. I believe that the authors should make more efforts on that to make the message of this paper more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and for raising these questions. We will summarize our main technical contribution and address the questions:\n### **Summary of our technical contribution**\n*\"The result in Theorem 4.2 appears novel and interesting, but it is a bit difficult to position this in the existing literature on kernel methods or deep neural networks. I believe that the authors should make more efforts to make the message of this paper clearer.\"*\n\nThank you for your review and thoughtful comments. Your intuition is correct. Our paper discusses the impact of initialization on the generalization ability of network under NTK theory, with the most critical technical contribution being Theorem 4.2. This theorem addresses the smoothness of the Gaussian process approximated by the initial output network function with respect to the NTK. Now we discuss the challenges and contributions of Theorem 4.2:\n- **Challenge of Theorem 4.2**:\n  - If the input is uniformly distributed on the sphere $\\mathbb{S}^d$ instead of $\\mathcal{X}$, both NTK and RFK are inner-product kernels and their Mercer's decomposition shares the same eigen-function (i.e., spherical harmonic polynomials). In this situation, simple calculations on the eigenvalues would show that $f^{\\mathrm{GP}}\\in [\\mathcal{H}^{\\mathrm{NT}}]^{\\frac{3}{d+1}}$.\n  - However, when the data is a general distribution supported in a bounded open set $\\mathcal{X} \\subset \\mathbb{R}^d$. NTK and RFK will not possess the nice property anymore which makes the claim if $f^{\\mathrm{GP}}\\in [\\mathcal{H}^{\\mathrm{NT}}]^{\\frac{3}{d+1}}$ unclear.\n- **Solution and Technical Contributions**:\n  - Before we clarify our solution and technical contributions, we first recollect some notations. For a sub-region $S\\subset \\mathbb{S}^d$, let $\\mathcal{H}_0^{\\mathrm{NT}}(\\mathbb{S}^d)$ be the RKHS associated with the homogeneous NTK defined on $\\mathbb{S}^d$ and $\\mathcal{H}_0^{\\mathrm{NT}}(S)=\\mathcal{H}_0^{\\mathrm{NT}}(\\mathbb{S}^d) |_S$ be its restriction on the sub-region $S\\subset \\mathbb{S}^d$. We define $\\mathcal{H}_0^{\\mathrm{RF}}(\\mathbb{S}^d)$ and $\\mathcal{H}_0^{\\mathrm{RF}}(S)$ similarly.\n  - We first show that, on a general $\\mathcal{X}$, $\\mathcal{H}^{\\mathrm{RF}} \\cong [\\mathcal{H}^{\\mathrm{NT}}]^\\frac{d+3}{d+1}$ (where the $\\cong$ is isomorphic as Hilbert space not RKHS).\n    - Let us fix a continuous and isomorphic mapping from $\\mathcal{X} \\subset \\mathbb{R}^d$ to a sub-region $S \\subset \\mathbb{S}^d$. We then show the equivalence (as RKHS):\n      $$\n      \\mathcal{H}^{\\mathrm{\\mathrm{RF}}}\\cong \\mathcal{H}_0^{\\mathrm{\\mathrm{RF}}}(S), \\quad \\mathcal{H}^{\\mathrm{NT}}\\cong \\mathcal{H}_0^{\\mathrm{NT}}(S).\n      $$\n      This equivalence is not as trivial as it appears.\n    - We then need to show that $[\\mathcal{H}_0^{\\mathrm{RF}}(S)]\\cong [\\mathcal{H}_0^{\\mathrm{NT}}(S)]^{\\frac{d+3}{d+1}}$.\n      Notice that the Mercer's decomposition of RFK and NTK share the same eigenfunctions (i.e., the spherical harmonic polynomials), we know that $[\\mathcal{H}_0^{\\mathrm{RF}}(\\mathbb{S}^d)]\\cong [\\mathcal{H}_0^{\\mathrm{NT}}(\\mathbb{S}^d)]^{\\frac{d+3}{d+1}}$. If the restriction $\\mid_S$ and the interpolation $[]^{\\frac{d+3}{d+1}}$ are commutative, then we are done. Unfortunately, this commutativity is not easy to verify in general.\n    - Fortunately, by a recent result [haas2023mind], we have $\\mathcal{H}_0^{\\mathrm{RF}}(\\mathbb{S}^d) \\cong W^{\\frac{d+3}{2},2}(\\mathbb{S}^d)$ and $\\mathcal{H}_0^{\\mathrm{NT}}(\\mathbb{S}^d) \\cong W^{\\frac{d+1}{2},2}(\\mathbb{S}^d)$ where $W^{s,2}$ denotes the usual fractional Sobolev space. Since Sobolev space is a more tractable object, we then carefully verify that $\\mathcal{H}_0^{\\mathrm{RF}}(S) = [\\mathcal{H}_0^{\\mathrm{NT}}(S)]^{\\frac{d+3}{d+1}}$ and finish the proof. We would like to emphasize that there are several different definitions for the fractional Sobolev spaces and there are no similar results explicitly stated in the literature.\n  - By Lemma H.5, we have $[\\mathcal{H}^{\\mathrm{RF}}]^{\\frac{3}{d+3}} \\cong [[\\mathcal{H}^{\\mathrm{NT}}]^\\frac{d+3}{d+1}]^{\\frac{3}{d+3}} (\\cong [\\mathcal{H}^{\\mathrm{NT}}]^{\\frac{3}{d+1}})$.\n  - If we denote $\\mathcal{H}^{\\mathrm{RF}} = \\left\\lbrace \\sum_{i \\in \\mathbb{N}} a_i \\lambda_i^{\\frac{1}{2}} e_i \\mid \\sum_{i \\in \\mathbb{N}} a_i^2 < \\infty \\right\\rbrace $, then the K-L expansion gives us that $f^{\\mathrm{GP}} = \\sum_{i \\in \\mathbb{N}} Z_i \\lambda_i^{\\frac{1}{2}} e_i$ where $\\lbrace Z_i\\rbrace_{i \\in \\mathbb{N}}$ are i.i.d. standard Gaussian variables. We can verify that $\\mathbf{P}( f^{\\mathrm{GP}}\\in [\\mathcal{H}^{\\mathrm{RF}}]^{t}) = 1$ for $t < \\frac{3}{d+3}$ and that $\\mathbf{P}( f^{\\mathrm{GP}}\\in [\\mathcal{H}^{\\mathrm{RF}}]^{t}) = 0$ for $t \\geq \\frac{3}{d+3}$. i.e., we have\n  $$\n  \\mathbf{P}( f^{\\mathrm{GP}}\\in [\\mathcal{H}^{\\mathrm{NT}}]^{t}) = 1 \\text{ for } t < \\frac{3}{d+1} \\text{ and } \\mathbf{P}( f^{\\mathrm{GP}}\\in [\\mathcal{H}^{\\mathrm{NT}}]^{t}) = 0 \\text{ for }t\\geq \\frac{3}{d+1}.\n  $$\n### **Response to the Questions**:\n  1. **Proposition 2.2**: Proposition 2.2 is a direct corollary of Theorem 1 in [zhang2023optimality]. Theorem 1 in [zhang2023optimality] provides an upper bound on the generalization error for general spectral algorithms. We directly applied this to the kernel gradient flow, a special type of spectral algorithm.\n  2. **Consistency in Notation**: Thank you for your correction. We will use the expectation symbol $\\mathbf{E}$ consistently.\n  3. **Theorem Designation**: Thank you for pointing out the issues with Theorem 3.3 and Theorem 4.1. These two theorems respectively provide conclusions on uniform convergence and the impact of the initial output function in kernel gradient flow. The proofs of them are standard and we will change their designation from Theorem to Proposition.\n  4. **Explanation of Theorem 4.2**: The explanation of the technical contribution of Theorem 4.2 is detailed in our response above. Thank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies various kernel theories of neural networks, both random feature and neural tangent kernels. The presentation is clear, and contributions are incremental. However, the authors are still asked to polish this paper carefully based on responses.
Decision: Accept (poster)