On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory

Authors: guhan  chen
yicheng  li
qian  lin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:59:29

Abstract: This paper aims to discuss the impact of random initialization of neural networks in the neural tangent kernel (NTK) theory, which is ignored by most recent works in the NTK theory. It is well known that as the network's width tends to infinity, the neural network with random initialization converges to a Gaussian process \(f^{\mathrm{GP}}\), which takes values in \(L^{2}(\mathcal{X})\), where \(\mathcal{X}\) is the domain of the data. In contrast, to adopt the traditional theory of kernel regression, most recent works introduced a special mirrored architecture and a mirrored (random) initialization to ensure the network's output is identically zero at initialization. Therefore, it remains a question whether the conventional setting and mirrored initialization would make wide neural networks exhibit different generalization capabilities. In this paper, we first show that the training dynamics of the gradient flow of neural networks with random initialization converge uniformly to that of the corresponding NTK regression with random initialization \(f^{\mathrm{GP}}\). We then show that \(\mathbf{P}(f^{\mathrm{GP}} \in [\mathcal{H}^{\mathrm{NT}}]^{s}) = 1\) for any \(s < \frac{3}{d+1}\) and \(\mathbf{P}(f^{\mathrm{GP}} \in [\mathcal{H}^{\mathrm{NT}}]^{s}) = 0\) for any \(s \geq \frac{3}{d+1}\), where \([\mathcal{H}^{\mathrm{NT}}]^{s}\) is the real interpolation space of the RKHS \(\mathcal{H}^{\mathrm{NT}}\) associated with the NTK. Consequently, the generalization error of the wide neural network trained by gradient descent is \(\Omega(n^{-\frac{3}{d+3}})\), and it still suffers from the curse of dimensionality. Thus, the NTK theory may not explain the superior performance of neural networks.

Main paper: 
 introduction: 
 In recent years, the advancement of neural networks has revolutionized various domains, including computer vision, generative modeling, and others. Notably, large language models like the renowned GPT series [8, 51] have shown exceptional proficiency in language-related tasks. Similarly, neural networks have achieved significant successes in image classification, as evidenced by works such as [27, 34, 37] . This proliferation of neural networks spans a wide range of fields. Despite these impressive achievements, a comprehensive theoretical understanding of why neural networks perform so well remains elusive in the academic community.Several studies have delved into the theoretical properties of neural networks. Initially, researchers were keen on exploring the expressive capacity of networks, as demonstrated in seminal works like [17, 28] . These studies established the Universal Approximation Theorem, asserting that sufficiently wide networks can approximate any continuous function. More recent research, such as [15, 26, 43] extended this exploration to the effects of deeper and wider network architectures. However, a significant challenge remains in these studies: they often do not fully explain the generalization power of neural networks, which is crucial for evaluating the performance of a statistical model.Recently, some researchers have examined the generalization properties of networks. Bauer and Kohler [5] , Schmidt-Hieber [46] showed the minimax optimality of networks with various activation functions for specific subclasses of Hölder functions, within the nonparametric regression framework. In contrasts to the static ERM approach, some studies made more attention to the dynamics of neural networks, particularly those trained using gradient descent (GD) and stochastic gradient descent (SGD) [2, 13, 20] .With similar insights, Jacot et al. [31] explicitly introduced the Neural Tangent Kernel (NTK) concept, demonstrating that there exists a time-varying neural network kernel (NNK) which converges to a fixed deterministic kernel and remains almost invariant during training as network width approaches infinity. And thus NTK theory proposes that network training can be approximated by a kernel regression problem [4, 29, 39, 50] . As a general case, fully-connected networks directly trained by GD, Lai et al. [35] , Li et al. [41] showed the generalization ability of two-layer and multi-layer networks, respectively. This paper mainly follows [4, 35, 41] , and explores the impact of initialization in the NTK theory. Prior research [35, 41] which verified the minimax optimality of network utilized the so-called mirrored initialization setting. It refers to a combination of mirrored structure and mirrored initial value of parameters, which results in a zero initial output function. However, the assumption divates from the commonly used initialization strategy in real-world applications, whose initial output is actually non-zero. To bridging the gap, in this study we explore the generalization ability of standard non-zero initialized network, within the NTK theory framework. Our findings reveal that the vanilla non-zero initialization will theoretically results in poor generalization ability of network, especially when the data has relatively large dimension. If that is true, it suggests a divergence between theoretical models and real-world applications, highlighting a potential limitation in the current understanding of the NTK theory. Therefore, we arrive at a critical problem central to this study: Does initialization significantly impact the generalization ability of networks within the kernel regime?1.1 Our contribution• Network converges to a NTK predictor uniformly. We show that under standard initialization, the network function converges to the NTK predictor uniformly over the entire training process and over all possible input in the domain. The convergence is essential in the study of the generalization ability of network in NTK theory. However, in previous work, the initial values of network has long been overlooked. Under mirrored initialization which leads to zero initial output function, Arora et al. [4] , Lai et al. [35] , Li et al. [41] demonstrated the point-wise convergence and the uniform convergence of network, respectively. More recently, Xu and Zhu [54] studied the uniform convergence of NTK under standard initialization, but did not study the convergence of the network function. Why the initial output of the network is ignored is not that it is insignificant, but rather because it is a stochastic function, making it challenging to analyze in convergence. Our findings make it valid to approximate the network's generalization ability based on the corresponding NTK predictor's performance.• The generalization ability of standardly non-zero initialized fully-connected network. Our research explores the impact of standard non-zero initialization in NTK theory. At this issue, Zhang et al. [56] proposes the existence of implicit bias induced by non-zero initialization, when the neural network is completely overfitted. We delve deeper into this argument, studies the exact formula of the bias at any stage of training, within the framework of NTK theory. Additionally, we established that the (optimally tuned) learning rate of network is n -3 d+3 , even when the regression function is sufficiently smooth. This insightful discovery implies a notable limitation in the generalization ability of networks with non-zero initialization, if NTK theory can precisely approximate the performance of real network. Consequently, we need to reconsider the weakness of NTK in the study of network theory. Also, the results show that mirrored initialization is superior to standard initialization in practical applications.Our research is conducted within the framework of NTK theory. This type of research, in general, can be categorized into two main steps: the approximation of the network trained by GD through a kernel regression problem, and the evaluation on the corresponding kernel regression predictor. Several studies [2, 4, 19, 31] which focused on the former step, illustrated the point-wise convergence of NTK for multi-layer ReLU networks. Additionally, [39] demonstrated the point-wise convergence of the kernel regression predictor to the network. Furthermore, Lai et al. [35] , Li et al. [41] , Xu and Zhu [54] demonstrated the uniform convergence result with respect to all input and all time on two-layer and multi-layer networks. As to the latter step, a few researchers have analyzed the spectral properties of the NTK [6, 7] as well as kernel regression [40, 55] . Building upon these findings, Lai et al. [35] and Li et al. [41] demonstrated that early-stopping GD induces minimax optimality of the network.It is worth noting that the setting in these works assumes mirrored initialization, which may not be well-aligned with real-world scenarios. When it comes to initialization, Zhang et al. [56] provided insights into the impact of initialization under kernel interpolation, which is a special case of our results at t = ∞. 
model and notations: 
 Suppose that {(x i , y i )} n i=1 are i.i.d. drawn from an unknown distribution ρ which is given byEQUATIONwhere f * (x) is the regression function and ϵ is a centered random noise. Suppose that the marginal distribution µ(x) of the radom variable x is supported in a non-empty bounded subset X of R d with C ∞ smooth boundary. The generalization error of an estimator f of f * is given by excess riskEQUATIONWe introduce the following standard assumption on the noise(e.g., [21, 42] ). It is clear that sub-Gaussian noise satisfying this assumption. Assumption 1 (Noise). The noise term ϵ satisfies the following condition for some positive constant σ, L, and m ≥ 2:EQUATIONNotations Given a set of samples pairs {(x i , y i )} n i=1 , we denote X and Y to be vector (x 1 , • • • , x n ) T and (y 1 , • • • , y n ) T , respectively. In a similar manner, (f (x 1 ), • • • , f (x n )) T and (f (y 1 ), • • • , f (y n )) T are represented as f (X) and f (Y ), where f (•) : R d → R is an arbitrary given function. Regarding a kernel function k(•, •) : R d × R d → R, we use k(x, X) to denote the vector (k(x, x 1 ), k(x, x 2 ), • • • , k(x, x n )) and k(X, X) to denote the matrix [k(x i , x j )] n×n .Suppose that k is kernel function defined on the domain X satisfying that ∥k∥ ∞ ≤ κ 2 . Let H k be the reproducing Hilbert space associated with k which is the closure of linear span of {k(x, •), x ∈ X } under the inner product induced by ⟨k(x, •), k(y, •)⟩ = k(x, y). Given a distribution µ(x) on X , we can introduce an integral operatorEQUATIONThe celebrated Mercer's decomposition [12] asserts thatEQUATIONwhere {e i } i∈N and {λ1 2i e i } i∈N are the orthonormal basis of L 2 (X , µ) and H k respectively. It is well known that H k can be canonically embedded into L 2 (X , µ).If the eigenvalues λ i of k are polynomially decaying at rate β ( i.e, λ i ≍ i -β ), we can further introduce a concept of the relative smoothness of a function f ∈ L 2 (X , µ). More precisely, let us recall the concept of real interpolation space [48] ( Please see more detailed information in the Appendix).The real interpolation space [H k ] s is given byEQUATIONwith the inner product ⟨ i∈N a i λs 2 i e i (x), i∈N b j λ s 2 j e j (x)⟩ [H k ] s = i∈N a i b i for s ≥ 0. It is clear that [H k ]s is a separable Hilbert space and is isometric to the l 2 space. With the definition above, we can see that[H k ] 0 = L 2 (X , µ) and [H k ] 1 = H k . Also, for any s 2 ≥ s 1 ≥ 0, we know [H k ] s1 ⊆ [H k ] s2 with compact embedding. Let α 0 = inf s s | [H k ] s ⊆ C 0 (X )which is often referred to the embedding index of an RKHS H k [21] . It is well known that α 0 ≥ 1 β and the equality holds for a large class of usual RKHSs if the eigenvalue decay rate is β. We further define the relative smoothness of a given function f : Definition 2.1 (Relative smoothness). Given a kernel k on X with respect to measure µ, the smoothness of a function f is defined asEQUATIONwhere c i = ⟨f, e i ⟩ L 2 (X ,µ) .For a positive definite reproducing kernel k, the dynamic of kernel gradient flow (KGF) [22] isEQUATIONwhere f GF t is the KGF predictor. In kernel gradient flow, the performance of kernel predictor depends on the relative smoothness of regression function. People often consider the case that α(f, k) ≥ 1 [10, 11] . When the smoothness satisfies α(f, k) < 1, the regression function is said to be poorly smooth and belongs to the so-called misspecified spectral algorithm problem. We collect the related result in Zhang et al. [55] and apply it to our case, to derive the following proposition: Proposition 2.2. Suppose the eigenvalue decay rate of k is β and the embedding index is 1 β with respect to µ. Suppose the noise term ϵ satisfies Assumption 1. Let the dynamic (8) starts from f GF 0 = 0. Also, suppose the regression function satisfiesf * ∈ [H k ] s and ∥f * ∥ [H k ] s ≤ R, for some s > 0. Let γ ≤ s and 0 ≤ γ ≤ 1. By choosing t ≍ n β sβ+1, for any fixed δ ∈ (0, 1), when n is sufficient large, with probability at least 1 -δ, we havef GF t -f * 2 [H k ] γ ≤ ln 6 δ 2 R 2 Cn -(s-γ)β sβ+1 ,where C is a positive constant.3 Network and Neural Tangent Kernel 
network settings: 
 We consider the fully-connected network with L hidden layers. As is commonly-used in deep learning, we consider the ReLU activation [44] defined by σ(x) := max(x, 0). Denote f (•; θ) : R d → R as the network output function, where θ representing the column vector that all parameters flattened into. We can write the recursive structure of network as following:EQUATIONThe parameter matrix for the l-th layer is denoted as W (l) . Their dimensions are of m l+1 × m l , where m l is the number of units in layer l and m l+1 is that of layer l + 1. Also, the bias term of the first layer is denoted as b (0) ∈ R m1×1 . The setting of bias term is to make sure the positive definiteness of NTK [41] . We further assume that the number of units in each layer is at the same order while the width comes to infinity, as cm ≤ min(m1 , • • • , m L+1 ) ≤ max(m 1 , • • • , m L+1 ) ≤ Cm where c, Care some absolute positive constants.Standard initialization At initialization, the parameters are randomly set as i.i.d. standard normal variables:EQUATIONEQUATIONRemark 3.1 (Mirrored initialization). As to the mirrored initialization considered in [4, 35, 41] , part of the network f (1) EQUATION0 ) undergoes standard initialization, while the other complicated corresponding part f (2) (•; θ (2)0 ) holds the same structure as f (1) (•; θ f (1) (•; θEQUATION0 ) . This setup ensures that f (•; θ 0 ) is constantly zero.The network is trained under the mean square loss function. If we suppose {(x i , y i )} n i=1 be the training data, then the loss function is specified asEQUATION)For notational simplicity, we denote by f NN t (x) = f (x; θ t ). The training process for the network is performed by gradient flow, where the parameters are updated through the differential equation:EQUATIONwhere ∂ θ f NN t (X) is a matrix with dimensions n × M , with M being the length of the parameter vector θ. This matrix represents the gradient of the network output f NN t (X) with respect to the parameters θ at time t. Incorporating the chain rule, we can formulate the gradient flow equation for the network function as follows:EQUATION)In order to state the properties of wide network with standard initialization, we need to introduce the concept of Gaussian process.Gaussian process Gaussian process is a stochastic process for which every finite collection of random variables follows a multivariate Gaussian distribution. Let X be a Gaussian process with index t ∈ T . If the mean and covariance are given by the mean function m and the positive definite kernel k such that E[X(t)] = m(t) and Cov[X(t)X(t ′ )] = k(t, t ′ ), which holds for any t, t ′ ∈ T , then we say X ∼ GP(m, k).In standard initialization (10) , the parameters of the neural network are i.i.d. samples from a standard normal distribution. If the network contains only one hidden-layer (that is, if L = 2), it is direct to prove that f NN 0 (x) converges to a centered Gaussian distribution by CLT, for any fixed point x ∈ X . As to the multi-layer network, prior research [25] also proved that such initialized network converges to a Gaussian process, as following: Lemma 3.2 (Limit distribution of initialization). As the network width m tend to infinity, the sequence of network stochastic process {f NN 0 } ∞ m=1 converges weakly in C(X , R) to a centered Gaussian process f GP . The covariance function is the so-called random feature kernel (RFK), which is denoted by K RFK (x, x ′ ) as defined in (42) in Appendix C.1.As the gradient descent of neural network involves high non-linearity and non-convexity, it is difficult to study the training process. However, Jacot et al. [31] introduced the Neural Tangent Kernel (NTK) theory which provides a connection between network training and a class of kernel regression problems, when the network width comes to infinity. To demonstrate this, we first define a Neural Network Kernel (NNK):EQUATIONUsing this notation, we reformulate (13) in a kernel regression format:EQUATIONNTK theory shows, if the network width m tends to infinity, then the random kernel K m t (•, •) will converge to a time-invariant kernel K NTK (•, •) : R d × R d → R, which is referred to as the NTK of network. The phenomenon is the so-called NTK regime [2, 31, 39] . The fixed kernel K NTK only depends on the structure of the neural network and the way of initialization. To get more knowledge of NTK, we present the explicit expression of NTK in Appendix C.1. In NTK theory, the dynamic of network (15) can be approximated by a kernel gradient flow equation:EQUATIONwhich starts from Gaussian process f NTK 0 = f GP . In this way, if we aims to derive the generalization property of sufficiently wide network, we can achieve by considering the corresponding kernel gradient flow predictor. Such approximation is strictly ensured by uniform convergence of f NN t and f NTK t over all x ∈ X and all t ≥ 0 as m → ∞, since we use L 2 excess risk to evaluate the generalization ability. Actually, we have the following theorem, whose proof is given in Appendix B. Proposition 3.3 (Uniform convergence). Given training sample pairs {(x i , y i )} n i=1 . For any δ ∈ (0, 1) and ε > 0, when network width m is large enough, we havesup x,x ′ ∈X sup t≥0 |f NN t (x) -f NTK t (x)| ≤ εholds with probability at least 1 -δ .In this theorem, we show the uniform convergence of network under standard initialization. Previous related studies [4, 35, 41, 54] always utilized delicately designed mirrored initialization (as shown in Remark 3.1) to avoid the analysis on the initial output function of network, since it will lead to the challenging problem that f NN t and f NTK t are both random, unlike that f NTK t is a fixed function in the case of mirrored initialization. However, as shown in Section 3.2, the initial output function is near a Gaussian process that can not be overlooked. To under the performance of neural networks commonly used in real world, it is necessary to analyzing the network initialization. To the best of our knowledge, we are the first to consider the initial output function of network in uniform convergence. This comprehensive result allows us to study the generalization error of network more precisely. 
impact of initialization 4.1 impact of standard initialization on the generalization error: 
 The standard kernel gradient flow is always considered to start from zero, as in Proposition 2.2. Therefore, we need to do a transformation since the initial value of predictor f NTK t is actually f GP instead of zero. Firstly, we can yield a solution of (8) in matrix form:EQUATIONwhere ϵ X is employed to represent the n × 1 column noise term vector Y -f * (X). We denote by f GF t the kernel gradient flow predictor under initial function f 0 and denote by f GF t the KGF predictor under initialization f GF 0 ≡ 0. If we plug them into (17) and excess risk (2), respectively, we directly have the following theorem: Proposition 4.1 (Impact of initialization in kernel gradient flow). Denote f * = f * -f 0 as the biased regression function. For the KGF predictor f GF t and f GF t defined above, we haveEQUATION)The theorem establishes the equivalence of the generalization properties between the KGF predictor with initial value f 0 , regression function f * and the KGF predictor with initial value zero, regression function f * -f 0 . Back to the network case, combining uniform convergence result in Proposition 3.3, it suggests that, compared to mirrored initialization, the impact of standard initialization which has non-zero initial output function is equivalent to introducing a same-valued implicit bias to the regression function. This is a generalization of the main result in Zhang et al. [56] , which only focused on case at t = ∞. To summarize, Proposition 4.1 provides a convenient approach to quantify the impact of standard initialization in early-stopping neural networks.Building upon the analysis above, our focus now turns to illustrating the smoothness of the Gaussian process f GP , as it is the limit distribution of f NN 0 . Actually, we can derive the following theorem: Theorem 4.2 (Smoothness of Gaussian Process). Suppose that f GP is a Gaussian process with mean function 0 and covariance function K RFK . The following statements hold:EQUATIONWe furnish a comprehensive proof for Theorem 4.2 in the Appendix C.Let us now turn our attention to the implications established by this theorem. Recall that Proposition 4.1 has shown that, in KGF, the existence of initialization function f GP is equivalent to adding a same-valued bias term to the regression function f * . Consequently, the poor smoothness of initialization function causes the high smoothness assumption on the regression function meaningless.Regardless of how smooth we assume the regression function to be (e.g., α(f * , K NTK ) ≥ 2), the value of (relavtive) smoothness α(f * -f GP , K NTK ) will always be at most 3 d+1 . Namely, the biased regression function f * -f GP is always poorly smooth. In this specific case, we could hardly expect the KGF predictor to have fine performance. • For the case of s ≥ 3 d+1 , for any δ ∈ (0, 1) and ε ∈ (0, 3 d+3 ), by choosing certain t = t(n) → ∞ (as shown in Appendix), when n is sufficiently large and m is sufficiently large, with probability 1 -δ we haveEQUATIONwhere C ε is a positive constant related to ε.• For the case of 0 < s < 3 d+1 , for any δ ∈ (0, 1), by choosing t ≍ n d+1 s(d+1)+d , when n is sufficiently large and m is sufficiently large, with probability 1 -δ we haveEQUATIONwhere C s is a positive constant related to s.The proof is provided in Appendix D. This result shows the generalization error upper bound for network with standard initialization and demonstrate its negative effect. Even if the goal function f * is quite smooth, the generalization error upper bound n -3 d+3 remains to be a quite low rate, particularly considering that the dimension d of data is usually large in real world. It suggests that the network no longer generalizes well, even if we adopt the once useful early stopping strategy in Li et al. [41] .From the analysis above, we can see the poor generalization ability of network under standard initialization. Furthermore, in this section, we take spherical data as example and provide the lower bound of generalization error. Namely, we presume the input vectors x are distributed on the sphere S d with probability measure µ, which is a common assumption in NTK theory [6, 31, 36, 53] . We also slightly change the network structure. Compared to the network (9), we eliminate the bias term of the initial layer, as shown in (40) in Appendix. In this case, the NTK of new network is denoted by K NTK 0 , and the RKHS H NT 0 (S d ) is abbreviated as H NT 0 , whose detailed properties is also given in Appendix C.1. Additionally, we make more assumption on the noise of data. We assume the noise term ϵ in (1) to have a constant second moment, as E |ϵ| 2 |x = σ 2 for x ∈ S d , a.e.. Under these conditions, with the help of method in Li et al. [40] , we derive the theorem: Theorem 4.4 (Generalization error lower bound). We assume that the regression function f * ∈ [H NT 0 ] s for some s > 3 d+1 , and denote by ∥f * ∥ [H NT 0 ] s ≤ R where R is a positive constant. Assume that µ is the uniform measure. For any δ ∈ (0, 1), when n is large enough and m is large enough, for any choice of t = t(n) → ∞, with probability at least 1 -δ we haveEQUATION)The proof is given in Appendix E. Through Theorem 4.4, we derive n -3 d+3 as the generalization lower bound of standardly random-initialized network in NTK theory, even if the regression function is quite smooth. The rate n -3 d+3 means model suffers notably from data that has large dimension: If d is relatively large, then this rate of convergence can be extremely slow. This is a manifestation of the curse of dimensionality. In fact, it contrasts with the fact that neural networks excel at highdimensional problems. This contradiction underscores the limitation of NTK theory for interpreting network performance. 
experiments: 
 Our numerical experiments are conducted in two aspects to fully understand the impact of standard initialization. First, we show the performance of standard initialized network is indeed worse than the mirrored initialized case, on the aspect of learning rate. The phenomenon is in line with our theoretical analysis. Second, the smoothness of regression function of real data is significantly larger than 3 d+1 , which suggest the bad effect of non-zero intial output function of standard initialization will indeed destroy the performance of network if NTK theory holds. It demonstrates the drawback of NTK theory through contradiction.In the first experiment, we employ artificial data to show the negative effect of standard initialization on the generalization error of network. The detailed settings are shown in Appendix F.The experiments are conducted for both d = 5 and d = 10, contrasting network performance subject to mirrored and standard initialization strategies. We choose a relatively smooth goal function to emphasize the impact of initialization. Specifically, we use m = 20n , epoch = 10n , and the gradient learning rate lr = 0.6. The networks are made sufficiently wide to ensure the overparametrization assumption is met. Additionally, we implement the early-stopping strategy as mentioned in Theorem 4.3, that is, selecting the minimum loss across all epochs as the generalization error. Finally, we test the network's generalization error on different levels of sample size n, and plot the log value of the generalization error corresponding to log(n) as shown in Figure 1 . As we expected, the points in Figure 1 fits a linear trend. Moreover, the figure highlights the difference in learning rate under different initialization methods. This aligns with our theoretical results.Figure 1 : Generalization error decay curve of network. The scatter points show the averaged log error over 20 trials. The dashed lines are computed through least-squares. The scale of n is not broad because a larger n requires a larger m , which would induce higher computational costs.In this subsection, we focus on datasets from the real world and estimate the smoothness of function. Although we could not know the goal function that the real data is generated from, there exists a way to estimate its smoothness [16] . We show the technical details in Appendix G. We employed the MNIST, CIFAR-10 and Fashion-MNIST datasets [33, 38, 52] . In the experiments, we evaluate the smoothness of goal function of the datasets, with respect to the one-hidden layer NTK. The results are presented in Table 1 . With the input dimension d = 784, 3072, 784, we can compute that the smoothness of initialization function is equal to 3 d+1 ≈ 0 . However, the smoothness of goal function is far better than 3 d+1 , which implies that standard initialization will indeed destroy the generalization performance, under NTK theory. The contradiction between NTK theory and the real situation shows its limitation and once again confirms our conclusion.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Yes, the datasets we utilized are publicly accessible for everyone. We have comprehensively outlined the experimental parameters and model settings within the experimental section of our paper. Notwithstanding the low complexity of our model, we are more than willing to post our code on GitHub should there be a demand for it. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our work is mainly a theoratical work focusing on learning theory, so this question is not applicable to our paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion: 
 To summarize, this research focuses on the impact of standard random initialization on generalization property of fully-connected network in the NTK theory, which makes up the gap in this field. Many previous work [35, 41] verified the statistical optimality of neural network under delicately designed mirrored initialization, whose initial output function of network is zero. However, through our study, we pinpoint that if we consider the commonly-used standard initialization, the learning rate of network is notably slow when the dimension of data is slightly large, which fails to explain network's favorable performance in overcoming the curse of dimensionality. A direct implication of our work is the superiority of mirror initialization over standard initialization, which suggests a direction for future improvements. On a deeper level, although NTK theory can describe many properties of network, at least for the fully connected networks with Gaussian initialization discussed in this paper, we can explore better theoretical frameworks to characterize their generalization ability in the future.In appendix, we will provide many technical proofs. Before that, let us provide more notations. For two sets A and B with a mapping function ϕ : A → B, the notation ϕ(A) is used to denote the image set of A under ϕ. For two random variable sequences {u n } and {v n }, we denote by u n = o P (v n ) (or u n = Ω P (v n )) if the ratio u n /v n approaches zero (or u n ≥ cv n for some positive constant c) in probability as n → ∞ with respect to probability measure P. For two real number sequence {a n } and {b n }, we denote by a n = Ω(b n ) if there exists positive constant c and n 0 such that |a n | ≥ c|b n | holds for any n ≥ n 0 . For two sequences of real numbers {a n } and {b n } such that a n = Ω(b n ) (ora n = O(b n )), we also denote by a n ≳ b n (or a n ≲ b n ) . If a n ≳ b n and a n ≲ b n , then we denote by a n ≍ b n .In this section, we demonstrate the uniform convergence from f NN t to f NTK t .The following is a direct proposition based on Lemma H.2 and Lemma 3.2, Proposition B.1. For the random network function sequence {f m 0 } with probability measures on (C(X , R), C) , there exists {X m } and X GP defined on a new probability space (Ω ′ , F, P), on which we haveP( lim m→∞ X m -X GP ∞ = 0) = 1.where X m and X GP has the same distribution as f m 0 and f GP , respectively. Remark B.2. The separability of (C(X , R), C) can be derived by the density of polynomials. Therefore, it satisfies the requirement of Lemma H.2. In the context of our study, our reliance is only on the distribution of {f m 0 } for each given value of m. Consequently, it is reasonable to reconstruct it in the new probability space. For convenience, we directly denote X m 0 as f m 0 (or f NN 0 ) and denote and X GP as f GP , respectively. In other words, we are considering the network function in a new probability space, even though this approach may result in a moderate abuse of notation.Our aim is to give the uniform convergence between NTK regressor f NTK t and network function f NN t . Note that the NTK regressor is trained by NTK, and the network function is trained by NNK, which is denoted by K m t . Here we first show the uniform convergence between NNK and NTK as m comes to infinity. Lemma B.3. For any δ ∈ (0, 1), suppose m is large enough, then with probability at least 1 -δ, we have supt≥0 sup x,x ′ ∈X |K m t (x, x ′ ) -K NTK (x, x ′ )| ≤ O(m -1 12 log m).Proof. The proof is similar to that in Li et al. [41] , while the difference is the way of initialization. So we only provide the sketch of proof. In Li et al. [41] , the uniform convergence of NTK is proved through a standard ϵ -net argument, which is divided into point-wise convergence and continuity of both NTK and NNK. Namely, as the following decomposition:EQUATION) where z, z ′ are the points in the ϵ -net which divides X . Back to our case, in non-zero initialization, the structrue of NTK and NNK remain the same, as well as the continuity property. Consequently, the effect of initialization reflects on the point-wise convergence from K m t (z, z ′ ) to K NTK (z, z ′ ), or more precisely, the NTK regime [2] . NTK regime requires that the residual decays to near zero and thereby the parameters will not deviate too far from their initial values in the training process, which holds under mirrored initialization. Standard initialization lets the residual at time 0 be f NN 0 (X) -Y 2 , instead of ∥Y ∥ 2 . Therefore, there is a slight risk that the residual is too large to decay to near zero during training. However, sinceEQUATIONholds with high probability when m is large through Proposition B.1 and direct analysis on f GP , we can verify that the residual f NN t (X) -Y 2 is still not large enough to break the stable lazy regime. Namely, the control on parameter matrix that 4 ).EQUATIONstill holds. In this way, we can finish the proof.Then, we can derive the uniform convergence of network function.Proof of Proposition 3.3. The proof is also similar to that of uniform convergence under mirrored initialization. Therefore, we only exhibit the sketch of different part. Define event A asEQUATIONwhere C δ is some constant related to δ, such that event A holds with probability at least 1 -δ 2 when m is large enough. Such a constant C δ is ascertainable, as f NN 0 converges to f GP by Proposition B.1 and f GP is a Gaussian process with finite second moment. Define event B asEQUATION)We have event B holds with probability at least 1 -δ 2 when m is large enough. Conditioned on event A and B, we do kernel gradient flow by K m t and K NTK on f NN 0 and f GP respectively. Let event C beEQUATIONConditioned on event A and B, we can prove that event C holds by Gronwall's inequality, as the same method in Lai et al. [35] . In this way, we can finish the proof.After we get the uniform convergence of network function, we can obtain the proposition on the convergence of excess risk: Proposition B.4. Suppose f * ∈ L 2 (X , µ). For any δ ∈ (0, 1) and ε > 0, when m is large enough , with probability at least 1 -δ, we haveEQUATIONProof. Recall the dynamic equation of f NTK t , we haveEQUATION)Since the kernel function K NTK (•, •) is bounded, there exists some positive constant C, such thatEQUATIONThe initial function of kernel gradient flow f NTK 0 = f GP follows a Gaussian process with mean 0 and covariance kernel function K RFK . By the boundness of K RFK , we can also bound f NTK 0 . That is, for any δ ∈ (0, 1), there exists a positive constant M δ such that with probability at least 1 -δ/2,EQUATIONDenote λ 0 := λ min K NTK (X, X) . We have λ > 0 since K NTK is strictly positive definite [41] .Thus we haveEQUATIONThe excess risk|E(f NN t ; f * ) -E(f NTK t ; f * )| = X |f NN t -f NTK t | 2 dµ + X (f NTK t -f * )(f NN t -f NTK t ) dµ (34) Since f * ∈ L 2 (X , µ)where µ is probability measure, we also haveEQUATIONBy Proposition 3.3, when m is large enough, with probability at least 1 -δ we have ∆ 2 ≤ ε/(1 + C √ nλ -1 0 ( √ nM δ + ∥Y ∥ 2 ) + M f * ).EQUATIONA dot-product kernel on sphere can be decomposed with spherical harmonic polynomials as the eigenfunction:EQUATIONwhere spherical harmonic polynomials {Y n,l , l = 1, • • • , a n } are also the orthonormal basis of L 2 (S d , σ), with σ denoting the uniform measure on S d [47] . This is also its Mercer decomposition. Now come back to our network case. We first define two dot-product kernels on S d ,EQUATIONwhere u = ⟨x, y⟩ = x T y andEQUATIONThe definition of κ(t)1 is given by the composition is the homogeneous NTK and RFK of a homogeneous fully-connected network f S defined on S d [6, 14] , whose structural difference from ( 9) is the removal of the bias term in the first layer. Specifically, the network is structured as follows:κ 1 • κ 1 • • • • κ 1 (Homogeneous fully-connected network on sphere The network is constructed using the following recursive formula:EQUATIONwhere the function σ is entrywise ReLU activation. The parameter matrix for the l-th layer is denoted as W (l) , whose dimensions are of m l+1 × m l , where m l is the number of units in layer l and m l+1 is that of layer l + 1 for l ∈ {0, 1, • • • , L -1}. We also set m 0 to be equal to d + 1 and m L+1 equal to 1. The network is also random initialized as (10) .We can easily build a connection between our NTK and RFK for network (9) and the homogeneous kernels defined in (38) . For x ∈ X , let x = (x, 1) which means add 1 as the new last component of x. Define ϕ(x) := (x,1) ∥(x,1)∥ being an isomorphism from open set X to a subdomain of positive hemisphere shell S = ϕ(X ) ⊂ S d + . Then we haveEQUATIONwhere f is network (9) and f S is network (40) . Actually, we can thus verify that . Their eigenvalue decay rates are well known:EQUATIONLemma C.1 (Bietti and Bach [6] , Haas et al. [24] ). For K NTK 0 and K RFK 0 on S d with uniform measure σ, the decay rate of spherical harmonics coefficients satisfyEQUATIONwhile the eigenvalues satisfyEQUATIONAdditionally, we list some further result on the eigenvalue decay rate of NTK and RFK provided by Li et al. [41] , which will be used later:Lemma C.2.Denote Ω as a non-empty subdomain of S d . For K NTK 0 and K RFK 0 , we have eigenvalue decay rate:λ i (K NTK 0 , Ω, σ) ≍ i -d+1 d and λ i (K RFK 0 , Ω, σ) ≍ i -d+3 dwhere σ is the uniform measure on S.Lemma C.3. For K NTK and K RFK , we have eigenvalue decay rate:λ i (K NTK , X , µ) ≍ i -d+1 d and λ i (K RFK , X , µ) ≍ i -d+3 dwhere measure µ on bounded domain X ⊂ R d has density c ≤ p(x) ≤ C with respect to the Lebesgue measure.Sobolev Space of integer power Let X be a open subset of R d . Let m ∈ N, 1 ≤ p ≤ +∞. Sobolev space W m,p (X ) is defined as a set of function such thatEQUATION)where α is a vector with length n and D α f is the weak α-th partial derivative of f . In other words, the definition of W m,p is:EQUATIONwhere 1 ≤ p ≤ ∞. Conventionally, when the index p is equal to 2, we denote W m,p by H m , since it is a Hilbert space. Further, if the index m > d 2 , the Sobolev space H m qualifies as a RKHS and thus embraces the properties of RKHS. In our work, we mainly utilize its property of interpolation as defined in (6) . Consequently, we first introduce a generalized concept of real interpolation [1] , as an expansion to the definition in (6) .Real interpolation For two Bananch spaces H 1 and H 2 , we use interpolation space to represent a space that lies in between them in some specific way. We introduce the commonly-used K-method to define real interpolation. Suppose 0 < s < 1, q ≥ 1. The space generated by their real interpolation H = (H 1 , H 2 ) s,q is defined by following:EQUATIONandEQUATION)Based on the definition of real interpolation, we introduce some basic concepts about fractional power Sobolev space.Sobolev space of fractional power Suppose X ∈ R d is a bounded domain with smooth boundary and denote Lebesgue measure by µ. We can define fractional power Sobolev space through real interpolation (we refer to [45] Chapter 4.2.2 for more details):EQUATIONThe fractional power Sobolev space H r (X ) with r ≥ d 2 is also a RKHS [1] . Specifically, Steinwart and Scovel [48] reveals that for 0 < s < 1,EQUATIONfor RKHS H and the interpolation defined in (6) . Therefore, the results above directly implies thatEQUATIONholds for any r ≥ d 2 and s > 0. Up to now, we have introduced the basic properties of Sobolev spaces on X , an open subset of R d . For Sobolev spaces defined on more intricate manifolds, such as hyperspheres, owing to the intricate property of Sobolev spaces, numerous equivalent definitions emerges [1, 18] .We now delineate a kind of definition that will facilitate our subsequent proofs, since we will consider RKHSs on S d , like H NT 0 and H RF 0 , which are the RKHSs associated with K NTK 0 and K RFK 0 , respectively. Such definition can form a linkage with the Sobolev spaces defined on S d and on domain X ⊂ R d , which is also utilized in Haas et al. [24] . Our exposition begins with the characterization of a manifold.Trivilization Define a trivialization of a Riemannian manifold (M, g) with bounded geometry of dimension d, which consists three part. The first part is some locally finite open covering {U α } α∈I . The second part is the charts {κ α } α∈I which consists of smooth diffeomorphism κ α : V α ⊂ R d → U α . The third part is a partion of unity h α such that supp(h α ) ⊂ U α , α∈I h α = 1 and 0 ≤ h α ≤ 1.In our case, we write a trivialization of S d , which, is a manifold of dimension d. We write U 1 = {x d+1 < ϵ|x ∈ S d } and U 2 = {x d+1 > ϵ 2 |x ∈ S d } for a small fixed ϵ > 0. Let ϕ 1 : U 1 → R d and ϕ 2 : U 2 → R d be stereographic projections with respect to x 1 = (0, 0, • • • , 1) and x 2 = (0, 0, • • • , -1) , respectively. Namely, they areEQUATIONandEQUATIONFinally, we can find C ∞ smooth functions h 1 and h 2 such that h 1 | S d + = 1. For the simple trivialization above, we can directly verify that it meets the admissible trivialization condition (details see Große and Schneider [23] ). Thus we can apply Theorem 14 of [23] to define the norm of Sobolev space onEQUATIONfor distribution f ∈ D ′ (S d ) [49] . It gives a kind of equivalent definition of Sobolev space on S d .We perform a transformation of the integral domain,EQUATIONRecalling the transformation between K NTK 0 and K NTK in (42), we haveEQUATIONThese transformations are both reversible. Therefore, through the structure of real interpolation space as described in ( 6), we can see I is an isometric isomorphism . In the same way, there exist isometric isomorphismI ′ : [H RF 0 (S)] s(d+1) d+3 → [H RF (X )] s(d+1) d+3 : I ′ : f → ρ • (f • ϕ).(67) Combined the result in Lemma C.9, the Lemma is proved. Now we are ready to give the smoothness of Gaussian process f GP . We remind the reader that H NT and H RF are abbreviations used for denoting H NT (X ) and H RF (X ), respectively.Proof of Theorem 4.2. Let t = s(d+1) d+3 to simplify the notation. By Lemma C.10, we have[H NT ] s ∼ = [H RF ] t .(68) Recalling the structure of interpolation space, we suppose [H RF ] t can be written asEQUATIONRecall that f GP represents a random function defined on (Ω, F, P), where each ω ∈ Ω corresponds to a path function f GP ω : X → R. We can express this in the orthonormal basis asf GP ω = i∈N a i (ω)λ t 2t e i , wherea i (ω) = ⟨f GP ω , λ t 2 i e i ⟩ [H RF ] t = λ -t 2 i f GP ω e i (x)dµ(x).Recall that as defined in Lemma 3.2, f GP has the distribution GP(0, K RFK ). From this, we can acquire the joined distribution for a i . Firstly, let us compute the covariance:EQUATIONThe exchange of integration is accomplished by Fubini's theorem since K RFK is a bounded kernel function, and both e i and e j are L 2 integrable. Moreover, as f GP is a Gaussian process, we finally get a i ∼ N (0, λ 1-t i ) for i ∈ N, and a i , a j are independent for any i ̸ = j. Consequently, we can directly derive thatEQUATIONwhere {Z i } indicates a collection of independent and identically distributed standard Gaussian random variables. Finally, as Lemma C.3 establishes the eigenvalue decay rate asEQUATION)Consequently we haveP f GP 2 [HNT] s < ∞ = 1.Part 2. When s ≥ 3 d+1 , we ascertain that d+3 d • (1 -t) ≤ 1 and consequentlyEQUATION)Denote byX n = n i=1 λ 1-t i Z 2 i .We then obtainEQUATION)We can thus derive thatEQUATION)Given that f GP 2 [HNT] s ≥ X n for any n ∈ N + , we haveEQUATION)This completes the proof.With the findings from Theorem 4.2, Proposition 4.1, and Proposition B.4, the influence of non-zero initialization could be interpreted in terms of a misspecified spectral algorithms problem. To apply Proposition 2.2, it only remains to determine the embedding index of H NT . Now, let's proceed to do so.Recall that the Proposition 2.2 requires the embedding index of H NT on X under the probability measure µ. Fortunately, the embedding index of the Sobolev space has been previously established by Zhang et al. [55] , which is helpful to simplify our proof. Lemma D.1 (Zhang et al. [55] Section 4.2, Embedding index of Sobolev space). Suppose r > d 2 . For a bounded open set X ⊂ R d and Lebesgue measure µ, the embedding index of H r (X ) equals d 2r .Since we have established the relationship between H NT and the Sobolev space, we can easily get the embedding index through a similar way used in the proof of Lemma C.10. Lemma D.2 (Embedding index of NTK). Suppose that the density function p(x) of probability measure µ satisfies the condition c ≤ p(x) ≤ C, where c and C are positive constants. The embedding index of H NT (X ) with respect to µ is concluded to be d d+1 .We omit this proof as it can be carried out in the same manner as Lemma C.10. Here, we provide only the structure. First, the embedding index of H I : f → ρ • (f • ϕ) is isometric isomorphism both from H NT 0 (S) to H NT (X ) and from L ∞ (S, ν ′ • ϕ) to L ∞ (X , µ), where measure ν ′ is defined on dν ′ dµ = ρ (an argument similar to that in the proof of Lemma C.10).Proof of Theorem 4.3. Recall that Proposition 4.1 elucidates the impact of non-zero initialization. Namely, the generalization error of the kernel gradient flow with an initialization of f 0 and a regression function f * , is consequently equivalent to that of kernel gradient flow with initialization at 0 and a regression function of f * -f 0 . On the other hand, Proposition B.4 demonstrated the uniform convergence from the network function to the kernel gradient flow predictor as the network width m tends to infinity. Lemma D.2 verify the embbeding index condition in Proposition 2.2. Thus, we only need to verify the source condition that f GP -f * fulfills and to incorporate it with Proposition 2.2 in order to derive the generalization error of the kernel gradient flow. Now we start the proof. Since the proofs for the cases s ≥ 3 d+1 and 0 < s < 3 d+1 are exactly the same, we will only provide the proof for the former case here. Through Theorem 4.2, we know for any 0< r < 3 d+1 , it follows that E f GP 2 [H NT ] r = λ 1-r i < ∞. Let C t = E f GP [H NT ] r. By the Markov inequality, for any δ ′ ∈ (0, 1), we have with probability exceeding 1 -δ ′ , thatEQUATION)Recall that the eigenvalue decay rate for K NTK is d+1 d as mentioned in Lemma C.2. Therefore, we have for any δ ∈ (0, 1) and any ε ∈ (0, 3 d+3 ), there exists r < 3 d+1 such thatrβ rβ+1 = 3 d+3 -ε (i.e., r = d 2 -6d-3d(d+3)ε 3(d+1)+ε(d+1)(d+3)). Denote by f * = f * -f GP and f NTK t be the kernel gradient flow predictor starts from initial value 0. Through Proposition 2.2, We thus haveEQUATIONholds with probability at least 1 -2δ ′ when t ≍ n β rβ+1 . Through Proposition 4.1, also we haveEQUATIONholds with probability at least 1 -2δ ′ . Through uniform convergence in Proposition B.4, we haveEQUATIONwith probability at least 1 -δ ′ when m is large enough. Therefore, with appropriate choice of δ ′ and C ′ , we can finish the proof.In this section, we establish the generalization error rate lower bound in our problem. We incorporate a result delineated in [40] , which systematically studies the learning rate of kernel regression. Prior to this, we take some preparatory work.We assume k is a dot-product kernel on S d with eigenvalue decay rate β, with respect to the uniform measure. We notate the corresponding RKHS as H k . Then, we can verify that H k satisties to the definition of regular RKHS, as detailed in [40] . Subsequently, the main theorem in [40] can be applied under our proposed settings, since K NTK 0 is a dot-product kernel defined on S d . It engenders the following lemma. Lemma E.1 (Generalization error lower bound). Assume k is a dot-product kernel defined on S d , we have the interpolation space of its RKHS as[H k ] s = i∈N a i λ s 2 i e i | i∈N a 2 i < ∞ where {e i } i∈N is the orthonormal basis of L 2 (S d , σ) and σ denotes the uniform measure. Decompose the regression function f * over the series of basis:EQUATIONWe assume that f * ∈ [H] t holds for any t < s for a given s > 0. Also, we we assume that i:λ>λiEQUATIONWe also assume that the noise term satisties E[|ϵ| 2 |x] = σ 2 holds for x ∈ S d ,a.e. Then, we define the main bias term in generalization error byEQUATIONand define the variance term byEQUATION)Fix the given input vectors of samples X. Consider the kernel gradient flow process detailed in (8) and let it start from 0. For any choice of t = t(n) → ∞, we haveEQUATIONWith the lemma above, now we are ready to prove Theorem 4.4.Proof of Theorem 4.4. By Proposition 4.1, we know the initial output function introduce an implicit bias term to the regression function. And thus the original problem is same as to consider a standard kernel gradient flow problem start from initial output zero with regression functionf * = f * -f GP .Recall that µ is the uniform measure. On sphere, the RKHSs of dot-product kernels K NTK 0 and K RFK 0 are equivalent to corresponding Sobolev spaces through Lemma C.4. More precisely, suppose that we have chosen an orthonormal basis {e i } i∈N consisting of spherical harmonic polynomials. Then we haveEQUATIONfor any t ≥ 0. Through Lemma C.2, we have the eigenvalue decay rate:EQUATIONWe denote by β 1 = d+1 d and β 2 = d+3 d . Similar to the proof of Theorem 4.2, we write the Kosambi-Karhunen-Loève expansion of f * :EQUATION∼ N (0, 1), andEQUATION)Here a i is a sequence of independent standard Gaussian variables, and b i represents a sequence derived from the decomposition of f * . With such decomposition, we can verify that (83) holds with probability 1 -δ ′ for any δ ′ ∈ (0, 1). Denote by g(λ)EQUATION)We also have the varianceEQUATION)Where the second term is controlled by the source condition assumption on f * . Therefore, we haveP |g(λ) -E[g(λ)]| ≥ E g(λ) 2 ≤ 4Var[g(λ)] (Eg[(λ)]) 2 = O(λ 3 d+1 ). (93) Define event A(λ) = {|g(λ) -E[g(λ])| ≤ E[g(λ)]/2}. For any δ ′ ∈ (0, 1), we choose a sequence λ j , such that λ j = C ′ j -2(d+1). Then we haveEQUATION)We can choose appropriate C ′ > 0 such that ∪ j∈N A( λ j ) holds with probability at least 1 -δ ′ , we denote by event A. Conditioned on event A, for any λ j+1 ≤ λ ≤ λ j , we haveEQUATIONwhich shows thatEQUATIONTherefore, we finish the proof of (83).Then, we turns to the calculation of generalization error lower bound. First, we plug in the decomposition and calculate the bias term R(t; f * ):EQUATIONRecalling that the eigenvalue decay rate is denoted by β, it follows thatEQUATIONAlso, the variance of R 2 (t; f * ) follows thatEQUATIONHere we introduce the denotations:EQUATIONWe then haveEQUATIONAs to V 2 , we first recall that the smoothness of f * lead to the following inequality:EQUATIONwhich implies thatEQUATIONNow we turn to the evaluation of V 2 :EQUATIONIt is worth noting that we use Cauchy's inequality to derive the upper bound above. With the control of V 0 and V 2 , we haveVar(R 2 (t; f * )) ≍ V 0 + V 2 ≲ t 3 2β 1 -2 β 2 β 1 (105)Consequently, by Chebyshev's inequality, we directly haveEQUATIONSince t = t(n) → +∞, we haveEQUATIONIn the same way, we also have the bound of variance term N (t).EQUATIONFinally, apply Lemma E.1 and Proposition 3.3. We derive that for any δ > 0, as long as n is large enough and m is large enough, for any choice of t = t(n) → ∞, with probability at least 1 -δ we haveEQUATION)Thus the theorem is proved.Remark E.2. In Proposition 3.3, we consider the situation that both input X and output Y of samples are fixed, while in the proof above we require that only X is fixed. However, the conclusion of Proposition 3.3 still holds when Y of samples is random. This is because the noise term has a finite second moment. Also, the change of domain from X to S d will not affect the uniform convergence result.Fixing the dimension of data as d = 5, 10. We draw samples for variable x from the standard Gaussian distribution N (0, I d ), which are consequently standardized to lie on the surface of the unit hypersphere S d . The dependent variable y is formulated as:EQUATIONwheref (x) = d j=1 x j 2, ε ∼ N (0, σ 2 ) and σ = 0.2. The function f exhibits notable smoothness, since it can be linearly represented in terms of the first few spherical harmonic polynomials on S d [9] and the fact that K NTK 0 is a dot-product kernel. We consider fully-connected network with one singular hidden layer, choosing m = 20 * n to ensure large enough width. Consistent to previous sections, we choose ReLU function as the non-linear activation and train the network using Gradient Descent for a sufficiently long time. We record the generalization error at each moment and define the moment of minimum generalization error as the final generalization error. This is done to align with the early stopping strategy mentioned in the Theorem 4.3.In this subsection, we will provide the theoretical basis of the method which approximates the smoothness of the goal function of real dataset. Let X ⊂ R d be a bounded domain. Given a reproduce kernel k(•, •) on X and a probability measure µ. Denote the RKHS byH k = i∈N a i λ 1 2i e i i∈N a 2 i < ∞ . We assume that there is a function f : X → R and a probability density µ on X . Suppose that the samples satisfies y = f (x), then f has the decomposition:EQUATIONThe smoothness α f = α(f, k) depends on the coefficients c i : if we have θ i ≍ i -dc and λ i ≍ i -d λ , then we derive the smoothness: α f = 2dc-1 d λ . We consider n samples {(x i , y i )} n i=1 . The Gram matrix k(X, X) can be decomposed as k(X, X) = ϕΣϕ T , andEQUATIONIn this regard, we can utilize the eigenvalue of the empirical kernel matrix to estimate the eigenvalue of the kernel function, since previous work has shown the convergence of eigenvalue when n is large enough [32] . Through the decomposition Y = ϕc (i.e., c = 1 n ϕ T Y ) and the approximation c i ≈ θ i , we can roughly estimate the eigenvalue decay rate when n is large enough with respect to i:EQUATION)In our experiments, we let n = 3000. Namely, an arbitrary selection of 3000 samples was made from the each dataset. We did not use all samples in the datasets, because n = 3000 is already sufficient to calculate the decay rate of eigenvalues. We consider the NTK of a one-hidden-layer fully connected network as the kernel k. The results is shown in Figure 2 , Figure 3 and Figure 4 for the three datasets, respectively. In each figure, the scatter plot shows the log value of the summed squares of each c k (for i ≤ k ≤ n as per equation ( 113)) against log 10 i on the x-axis. Also, the dashed line represents the corresponding least-square regression fitting using index i smaller than 2700. Theoretically, the slope of the dashed line will be -α f d λ . In this section, we introduce a series of technical lemmas. These will be helpful in our proof, and many of the lemmas have been established by prior researchers. Lemma H.1 (Change of measure [41] ). For a positive definite kernel k defined on a compact set X , it has the same eigenvalue decay rate under two measure ν and σ:λ i (K NTK 0 , X , ν) ≍ λ i (K NTK 0 , X , σ)if the Radon derivative p = dν dσ exists and c ≤ p ≤ C holds for some positive constant c and C. Lemma H.2 (Skorohod's Representation Theorem). Suppose that a sequence of probability distribution {F n } converges weakly to F and F has a separable support. Then there exist random variables X n and X, defined on a new probability space (Ω ′ , F, P), such that the distribution of X n is F n , the distribution of X is P , and X n → X holds almost surely. Lemma H.3 (Restriction of RKHS [3] ). Suppose H k is a RKHS defined on E with the norm ∥•∥ H k , then k| Ω restricted to a subset Ω ⊂ E is the reproducing kernel of spaceEQUATION)The following proposition is a direct proposition of Lemma H.3: Proposition H.4 (Equivalence of RKHS under resriction). Assume RKHSs H 1 ∼ = H 2 are defined on E. Write Ω as a subset of E. Then we haveH 1 | Ω ∼ = H 2 | Ω .The following two lemmas are common in real interpolation: Lemma H.5 (Equivalence of interpolation spaces). Suppose 0 < s < 1. Denote L 2 = L 2 (X , µ) for abbreviation. If we have RKHSsH 1 ∼ = H 2 , then (L 2 , H 1 ) s,2 ∼ = (L 2 , H 2 ) s,2 .Proof. To prove the lemma, we only need to prove that the embedding(H 1 , L 2 ) s,2 → (L 2 , H 2 ) s,2 and (L 2 , H 2 ) s,2 → (L 2 , H 2 ) s,2 are both bounded.First we prove that (L 2 , H 1 ) s,2 → (L 2 , H 2 ) s,2 ≤ C 1 where C 1 is an absolute positive constant.For any x ∈ L 2 + H 1 , define the K-functionalK 1 (t; x) = inf x0+x1=x;x0∈L 2 ,x1∈H1 (∥x 0 ∥ L 2 + t∥x 1 ∥ H1 ); K 2 (t; x) = inf x0+x1=x;x0∈L 2 ,x1∈H2 (∥x 0 ∥ L 2 + t∥x 1 ∥ H2 ).Since H 1 ∼ = H 2 , for any x ∈ H 1 , we have ∥x∥ H2 ≤ C∥x∥ H1 . Thus we haveK 2 (t; x) ≤ inf x0+x1=x;x0∈L 2 ,x1∈H1 (∥x 0 ∥ L 2 + Ct∥x 1 ∥ H1 ) = K 1 (Ct; x);Then we haveEQUATIONLet C 1 = C 2s , we have the canonical injection satisfies (L 2 , H 1 ) s,2 → (L 2 , H 2 ) s,2 op ≤ C 1 . Also, since H 1 ∼ = H 2 , for any x ∈ H 2 , we have ∥x∥ H1 ≤ c∥x∥ H2 . We can prove (L 2 , H 2 ) s,2 → (L 2 , H 1 ) s,2 op ≤ C 2 in the same way. Then, we finish the proof.Lemma H.6 (Equivalence of interpolation spaces). Suppose 0 < s < 1. Denote H be a RKHS and µ, ν be measures on setX . If we have L 2 (X , µ) ∼ = L 2 (X , ν), then (L 2 (X , µ), H) s,2 ∼ = (L 2 (X , ν), H) s,2 .Proof. The proof in accomplished in the same way as Lemma H.5. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work is mainly a theoratical work focusing on learning theory, so this paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have explicitly referenced the datasets in our paper, as MNIST, CIFAR-10 and Fashion-MNIST. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: Our work does not involve any study participants, so there is no need to discuss potential risks they might face. Therefore, there is also no need for an Institutional Review Board (IRB) approval. Hence, this question is not applicable to our paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper studies various kernel theories of neural networks, both random feature and neural tangent kernels. The presentation is clear, and contributions are incremental. However, the authors are still asked to polish this paper carefully based on responses.

Reviews: review: paper_summary: paper_summary: This paper studies various kernel theories of neural networks, both random feature and neural tangent kernels. The main approach is to use the decay rates of the target function and kernel eigenvalues to get generalization error rates. The paper's novel theoretical contribution is to show uniform convergence of the network's input-output function $f$ to that of a kernel regressor with NTK kernel. The setting is that the weights are scaled to be in the NTK regime and when both are trained under gradient flow. Previously known results are used to characterize the RKHS associated with the NTK and "interpolations" of this space (which is useful if the target function isn't actually in the RKHS). It is argued that, since the network initialization with random weights is not smooth, that the network overall will have a slow error rate. These theoretical arguments are backed up with experiments on synthetic data comparing the standard initialization, that was studied theoretically, with a "mirrored" one. Also, the smoothness of the MNIST, CIFAR-10, and Fashion-MNIST classification problems is estimated and found to be quite different from the basic bound based on input dimensionality.
strength_weakness: strength_weakness: * In the introduction and experimental sections, the standard and mirror initializations are contrasted. From the introduction, I was expecting to see theoretical results that differentiated between the mirror and standard initializations. The authors could be clearer about what their contributions are and more clearly state what is known about the mirror initialization. 
* As someone who is fairly familiar with kernel theory of neural networks, I still found it hard to identify which theorems and other results presented by the authors here are actually novel and which are applications of known results to their setting. For instance, isn't the decay rate of $1/(d+3)$ classical and known from Sobolev theory? I would suggest that the authors clarify this. 
* The bulk of the paper is really just setting the stage for your results. I would have liked to see more explanation of how to interpret your results and discussion of how they fit into the greater understanding of these networks/kernels. I would also have liked more detail on the experiments within the main text.
* The synthetic experiments (Fig 1) only cover less than an order-of-magnitude in $n$. They would be much more convincing if they covered more. However, I can imagine you'd run into issues with double-descent and other factors that your asymptotic theory could have trouble with.
questions: questions: * Please clarify the definition of the interpolation space (Eq 6 and around it). Your notation for the space of functions is unclear to me. Some explanation should be given. It is unusual to see a function written as a series $\sum_i a_i \lambda_i^{s/2} e_i(x)$
* Fix the formatting issue in line 104
* Prop 2.2 "satisfis" typo
* Do you think it would be more consistent to use the NTK superscript everywhere you treat it rather than f^NTK and K^NT ?
* Line 267 "network on longer" typo
* Does the mirror initialization lead to a significantly different error bound than yours, Thm 4.3?
* Can you include the method of estimating the smoothness of the various classification problems (Table 1) in the main text? Looking at the appendix, I see it depends on an empirical kernel/Gram matrix, but under what kernel is not clear. NTK?
* You report various least-square fits on log-log plots. You should be clear about how these were performed. Linear fits in log-space are known to have problems versus least-square fitting with power-laws. See the work by Clauset et al.
* In your notation section, you use standard $O, \Omega, o, \omega$ symbols but do not define the frequent $\asymp$ symbol. This should be explained.
limitations: limitations: The work here is very focused on the NTK theory of a multi-layer perceptron/fully-connected network. Strong claims are made about the weakness of such theories. However, different architectures lead to different kernels with different spectral properties and generalization performance. For instance, convolutional architectures lead to convolutional kernels which are known to work better in image classification and other spatially structured settings. It is also known that initialization with weights that are not Gaussian white noise can lead to better performance (see the work on "structured random features" and "rainbow networks").

I did not check the appendix proofs, but I read all of the main text results.
paper_summary: paper_summary: This paper explores the standard random mode of initialization of neural networks through the lens of the neural tangent kernel (NTK). This connection is made by showing that a randomly initialized neural network does indeed converge to the NTK uniformly during training thus allowing to analyze the generalization ability of a neural network with more relaxed assumptions on initialization. Furthermore, this paper finds that neural network performance under this regime comes at odds with the NTK theory which indicates that networks should perform poorly for high dimension inputs.
strength_weakness: strength_weakness: There are a few typos/grammer mishaps that I caught in the paper:

Proposition 2.2 - "...noise term $\epsilon$ **satisfis**..."

Lines 126-127 - "...*let us reall the* concept..."

Line 233 - "This is **an** generalization..."

Lines 266-267 - "...the network **on longer generalizes** well..." (does not make sense)

Line 314 - "...datasets *from real world* and estimate the *smoothness of function*."
questions: questions: Line 186 - Can you specify what space $C(\mathcal{X}, \mathbb{R})$ refers to?

Lines 193-194 - (Less of a question and more of a comment) I am assuming NNK is just your notation for the standard empirical NTK...

Lines 274-275 - Why is it necessary to remove the bias term for the derivation of the lower bound?
limitations: limitations: Everything is adequately addressed.
paper_summary: paper_summary: The paper aims to study the impact of standard random weight initialization - as opposed to mirror initialization - on NTK theory.  The key observation is that, for standard initialization, the operation of the network at initialization $f_0$ acts as a bias on the regression function $f^*$.  When analyzing the convergence of the excess risk in the kernel gradient flow model the convergence rate depends on the smoothness of the regression function, which is (effectively) $f_0+f^*$.  The paper demonstrates that, regardless of how smooth $f^*$ is, the smoothness of $f_0+f^*$ is bounded above by the smoothness of $f_0$ (theorem 4.2).  Non-zero initialization thus places upper and lower bounds on the generalization error predicted by KGF (theorem 4.3 and 4.4).  This is claimed to bring into focus the shortcomings of KGF in the NTK context for analyzing the convergence of neural networks even in the infinitely wide limit.
strength_weakness: strength_weakness: Perhaps I am misreading the results, but it would seem to me that, far from demonstrating the downsides of NTK theory, you have in fact shown that NTK theory correctly predicts that standard initialization can degrade performance (in terms of the generalization error during training) compared to mirror initialization for the artificial data.  With regard the real data, all I can see is that you have shown that standard initialization will indeed decrease (effective) smoothness; however your predictions are asymptotic, so without a side-by-side comparison of standard and mirror initialization training (as per the artificial data) I don't see how this demonstrates anything at all.  It would certainly be interesting to see if the results for real data are the same as for artificial data as this would (presumably) indicate that mirror initialization is to be preferred where possible.

(fwiw this is the core reason for the discrepancy between my evaluation of soundness and my recommendation.  Mathematically the paper seems sound, but the interpretation seems incorrect to me: far from showing the downsides of NTK theory, the results would appear to show its power.  If I'm wrong or the paper is suitably modified then I am happy to upgrade my recommendation).

Minor point: please run this paper through a spell checker.
questions: questions: My main question is whether or not it would be feasible to run mirror-initialization vs standard-initialization experiments on the real-world dataset, or even a high-dimensional artificial dataset.  My guess would be that the results would replicate the Figure 1, which could have interesting practical implications.
limitations: limitations: n/a
paper_summary: paper_summary: In this paper, the authors discuss the impact of random initialization of deep neural networks in the neural tangent kernel (NTK) regime.
Precisely, the authors prove that in the case of standard random initialization, the network function in the finite wide limit, converges to the NTK predictor uniformly.
Then, by studying the behavior of the underlying Gaussian process, the authors establish upper and lower bounds for the generalization errors of DNNs in the NTK regime, in terms of the real interpolation space of the RKHS.
strength_weakness: strength_weakness: It is a bit difficult for me to evaluate this paper. 
The technical results of Theorem 3.3 and 4.1 are not very surprising and their proofs also look more or less standard.
The result in Theorem 4.2 appear novel and interesting, but it is a bit difficult to position this in the existing literature of, kernel or DNN. I believe that the authors should make more efforts on that to make the message of this paper more clear.
questions: questions: * Is Proposition 2.2 novel result? If yes, where is its proof?
* it appears that the notations of expectation are not aligned in Assumption 1 and line 178.
* The results of Theorem 3.3 and 4.1 are not very surprising and their proofs also look more or less standard
* The result in Theorem 4.2 looks interesting, but essentially focuses on the Gaussian process with zero mean and random feature kernel of covariance function. Also, is this the FIRST time such results are established (e.g., in terms of real interpolation space of the RKHS)? How should we compare this result to other kernels/methods/models? It may be helpful to compare this behavior to the literature.

---
I thank the authors for their detailed reply and clarification. This helps me have a better grasp of this presented results, I will increase my score accordingly.

I would like to ask the authors to carefully include the above discussions in a revised version of the paper, as also suggested by Reviewer 43yC.
limitations: limitations: I do not see any potential negative social impact of this work.

Comments: title: Refernces of Rebuttal
Refernces of Rebuttal
The second part of Rebuttal
comment: Thank you for your valuable suggestions and for considering an increase in the score.  We appreciate your thoughtful review. (We noticed that the current score still appears as the previous one on our end, in case there might be an error.)

We agree with your suggestions. After presenting Theorem 4.2, we will provide a brief introduction about the proof process of Theorem 4.2 and the previous results it builds upon. This will help clarify the challenges addressed by Theorem 4.2 and highlight our technical contributions . Also, we will include the above discussions in the revised version, and we would like to re-iterate the changes we will incorporate:

1. The proofs of Theorems 3.3 and 4.1 are standard, and therefore, we propose renaming them as propositions rather than theorems.

2. We will enhance the discussion of related work on mirror initialization in the introduction, ensuring that our findings are better integrated with existing research. 

3. The revised version will emphasize the generalization advantages of mirror initialization over standard initialization, as indicated by our results. 

4. We will correct minor errors, including spelling and grammar mistakes, and clarify any ambiguous expressions in the text.

Thank you again for your thorough review and constructive feedback. If you have any further comments or questions, please feel free to ask. Thank you.
Thank you for your feedback and for taking the time to review our rebuttal. We are pleased to hear that our clarifications helped to address your concerns, and we greatly appreciate your willingness to increase the score. In response to your suggestions, we reconsider the interpretation of our results to ensure that the connection between NTK theory and the observed effects on generalization performance is well presented, and will highlight the potential of mirror initialization in the revised version. 

Thank you again for your thoughtful review and for helping us to improve the quality of our paper.
Thank you for this this clarification, I have raised my score accordingly.
Thank you for your thoughtful feedback. I appreciate your agreement with the other reviewers regarding the additional context provided in the rebuttal and its potential to enhance the quality of the paper. I will carefully address the points raised and incorporate the necessary revisions.

Additionally, thank you for pointing out the typo on line 115. I will correct the phrase to “absolute positive constant $c$ and $C$” in the revised version.

I look forward to making these improvements and resubmitting the revised paper. Thank you for your careful review again.
Hi, thank you for your message. I apologize for any confusion. I have now modified the comment to ensure that the second half of the rebuttal is visible to all reviewers. Please feel free to read through it, and I would greatly appreciate any further feedback you may have.
Hi, you mentioned that you included a second half to the rebuttal in another comment, but I don't think I can see it. It would be nice to read through it if possible/relevant.
Thank you for your comments and clarification. I concur with the other reviewers that the additional context you provided in your rebuttals would improve the quality of the paper. I will keep my score the same currently and await the revisions. 

I also found another typo on line 115 "... absolute positive constant $c$ **abd** $C$...".
Thank you very much for your feedback and valuable suggestions on my rebuttal. I will take this opportunity to clarify the relevant issues in the final paper. Specifically, I will focus on strengthening the discussion of related work on mirror initialization in the introduction and better integrating my findings with existing research. I truly appreciate your guidance and will make the necessary improvements in my revisions. Thank you again for your review and support.
Thank you very much for your feedback. We are grateful for the chance to improve the presentation of the current paper. For example, we will include a brief summary of the challenge and technical contribution after presenting Theorem 4.2. We will certainly carefully incorporate other relevant discussions into the revised version of the paper.

We are especially grateful for your decision to increase the score, which is very encouraging for us.
I thank the authors for their detailed reply and clarification.
This helps me have a better grasp of this presented results, I will increase my score accordingly.

I would like to ask the authors to carefully include the above discussions in a revised version of the paper, as also suggested by Reviewer 43yC.
Thanks for your responses. I think it would be good if you used this time to clarify things in the final paper. I suggest you strengthen this context in the intro when you discuss related work and how your results fit in. I hope you plan to do this. I'll keep my score the same.
(We apologize for the rebuttal reaching the 6000-word limit. We will provide the references below and appreciate your understanding for any inconvenience caused.)

### **References**


[haas2023mind] Haas, M., et al. (2023). “Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension”.

  

[zhang2023optimality] Zhang, H., Li, Y., and Lin, Q. (2023). “On the optimality of misspecified spectral algorithms”.
### References

[haas2023mind] Haas, M., et al. (2023). “Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension”.

[lai2023generalization] Lai, J., et al. (2023). “Generalization ability of wide neural networks on \(\mathbb{R}\)”.

[lai2023generalization2] Lai, J., et al. (2023). “Generalization ability of wide residual networks”.

[li2024saturation] Li, Y., Zhang, H., and Lin, Q. (2024). “On the saturation effect of kernel ridge regression”.

[li2023statistical] Li, Y., et al. (2024). “On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains”.
(We apologize for reaching the word limit earlier, and we provide the second half of the rebuttal here.)
### Summary of the Theoretical Significance of the Paper
*"The bulk of the paper is really just setting...... the experiments within the main text."*

Thank you for your suggestion. We will summarize the theoretical significance of the paper within the background of NTK theory.
- Background of our work: For a long time, NTK theory has played an important role in the study of network. Under NTK theory, a series of works tried to explain the remarkable generalization ability of network [lai2023generalization], [li2023statistical], [lai2023generalization2], which applied mirror initialization and derived the minimax optimality of network.
- Impact of standard initialization: However, we wonder how the actual results would be if the standard initialization are considered. Under standard initialization, our result shows that the generalization ability of network under NTK theory is actually quite poor. It is a surprising result and prompts us to think further. 
- Significance of our work: On one hand, this tells us that the generalization ability of network is not only influenced by the smoothness of the goal function but also by the way of initialization. Thus, we have to carefully take the way of initialization into consideration. On the other hand, we can consider whether the inconsistency between the theory and reality is due to the overly strong assumptions of NTK theory (e.g., the width of network is sufficient large). In the future, it may be worthwhile to explore how NNK changes during training under finite width to advance the existing NTK theory.
- Experiment: In the experiment of Section 5.1, we chose a single-layer network with width $m =20n$ to ensure the width is sufficiently large. At each $n$, we conducted 10 experiments with a learning rate of 0.6, and each training was run for 10n epochs to ensure sufficient time. We will add these details to the main text. Thank you for your suggestion.
### The Problem of Fig 1
*"The synthetic experiments (Fig 1) only cover ..... theory could have trouble with."*

Thanks for pointing this out and we appreciate your correction. Indeed, the range of our $n$ is not large enough. In kernel theory, it is common for the generalization error to decrease polynomially with respect to $n$ (e.g., Fig 1 in [li2024saturation]). In Fig 1, we aim to demonstrate a similar phenomenon. However, in fact, it indeed may encounter the problem of double-descent as you mentioned. To ensure the convergence of NTK, the width of the network $m$ needs to increase with $n$, but it will result in a high computational cost. The reasons above are why we didn't increase $n$. Thank you for correction again.
### Response to Questions
1. The notation $\sum a_i \lambda_i^{s/2}e_i(x)$ is mainly because $\lbrace\lambda_i^{s/2}e_i(x)\rbrace$ forms an ONB of the interpolation space. Thank you for pointing this out. We will explain this in more detail.
2. Under mirror initialization, for $f^*\in[\mathcal{H}^{NT}]^s$ ($s \geq 1$), the generalization error of network is $n^{-\frac{s(d+1)}{s(d+1)+d}}$, as shown in [li2023statistical]. We appreciate your reminder and we will add this point to the discussion of Theorem 4.3 for the convenience of comparison by readers.
3. Thank you for suggestion on the notations in the experimental part. We estimated the smoothness of real datasets with respect to the NTK of single-layer fully connected network. We will add this point in our paper. 
4. Regarding the estimation of decay rate in the log-log plot, there may be some miscommunication. As a function of $n$, e.g., in Fig 1, we denote the experimentally measured generalization error as $e_i(n)$, where $n$ is the sample size and $i$ represents the $i$-th experiment. Then $e_i(n)$ is approximately of the form $e_i(n) \propto n^{-\alpha}Z_{n,i}$, where $Z_{n,i}$ denotes a random variable with values around 1, and $\alpha$ is the decay rate to be estimated. In this case, a linear fit after a log transformation may not be a bad choice. We found that Clauset et al. studied estimating $a$ through i.i.d. samples $x_{i} \sim p(x) (\propto x^{-a})$, which is slightly different from our problem. We guess there may be some miscommunication here.
5. Thanks for your careful reading again. We apologize for our annoying typos. We have tried our best to read through the paper and will make several modifications in the new revision.
### Impact of Network Architecture on Our Results
Our results indeed focus on fully connected network. Thanks for pointing this out. For other architectures, like ResNet or CNN, if we know the properties of the corresponding NTK, RFK, and initial output function, it is also reasonable to apply our framework. This requires considerable preliminary work, but in the future, we are willing to explore whether our results hold under different network settings. Again, thank you for highlighting this point.