Title: On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory
Venue: NeurIPS 2024
Accepted: yes
Keywords: neural tangent kernel

random initialization

non-parametric regression

reproducing kernel Hilbert space
Abstract: This paper aims to discuss the impact of random initialization of neural networks in the neural tangent kernel (NTK) theory, which is ignored by most recent works in the NTK theory. It is well known that as the network's width tends to infinity, the neural network with random initialization converges to a Gaussian process \(f^{\mathrm{GP}}\), which takes values in \(L^{2}(\mathcal{X})\), where \(\mathcal{X}\) is the domain of the data. In contrast, to adopt the traditional theory of kernel regression, most recent works introduced a special mirrored architecture and a mirrored (random) initialization to ensure the network's output is identically zero at initialization. Therefore, it remains a question whether the conventional setting and mirrored initialization would make wide neural networks exhibit different generalization capabilities. In this paper, we first show that the training dynamics of the gradient flow of neural networks with random initialization converge uniformly to that of the corresponding NTK regression with random initialization \(f^{\mathrm{GP}}\). We then show that \(\mathbf{P}(f^{\mathrm{GP}} \in [\mathcal{H}^{\mathrm{NT}}]^{s}) = 1\) for any \(s < \frac{3}{d+1}\) and \(\mathbf{P}(f^{\mathrm{GP}} \in [\mathcal{H}^{\mathrm{NT}}]^{s}) = 0\) for any \(s \geq \frac{3}{d+1}\), where \([\mathcal{H}^{\mathrm{NT}}]^{s}\) is the real interpolation space of the RKHS \(\mathcal{H}^{\mathrm{NT}}\) associated with the NTK. Consequently, the generalization error of the wide neural network trained by gradient descent is \(\Omega(n^{-\frac{3}{d+3}})\), and it still suffers from the curse of dimensionality. Thus, the NTK theory may not explain the superior performance of neural networks.

On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory

Abstract

This paper aims to discuss the impact of random initialization of neural networks in the neural tangent kernel (NTK) theory, which is ignored by most recent works in the NTK theory. It is well known that as the network's width tends to infinity, the neural network with random initialization converges to a Gaussian process f GP , which takes values in L 2 ( X ) , where X is the domain of the data. In contrast, to adopt the traditional theory of kernel regression, most recent works introduced a special mirrored architecture and a mirrored (random) initialization to ensure the network's output is identically zero at initialization. Therefore, it remains a question whether the conventional setting and mirrored initialization would make wide neural networks exhibit different generalization capabilities. In this paper, we first show that the training dynamics of the gradient flow of neural networks with random initialization converge uniformly to that of the corresponding NTK regression with random initialization f GP . We then show that P ( f GP ∈ [ H NT ] s ) = 1 for any s < 3 d +1 and P ( f GP ∈ [ H NT ] s ) = 0 for any s ≥ 3 d +1 , where [ H NT ] s is the real interpolation space of the RKHS H NT associated with the NTK. Consequently, the generalization error of the wide neural network trained by gradient descent is Ω( n -3 d +3 ) , and it still suffers from the curse of dimensionality. On one hand, the result highlights the benefits of mirror initialization. On the other hand, it implies that NTK theory may not fully explain the superior performance of neural networks.

1 Introduction

In recent years, the advancement of neural networks has revolutionized various domains, including computer vision, generative modeling, and others. Notably, large language models like the renowned GPT series [8, 51] have shown exceptional proficiency in language-related tasks. Similarly, neural networks have achieved significant successes in image classification, as evidenced by works such as [27, 34, 37]. This proliferation of neural networks spans a wide range of fields. Despite these

impressive achievements, a comprehensive theoretical understanding of why neural networks perform so well remains elusive in the academic community.

Several studies have delved into the theoretical properties of neural networks. Initially, researchers were keen on exploring the expressive capacity of networks, as demonstrated in seminal works like [17, 28]. These studies established the Universal Approximation Theorem, asserting that sufficiently wide networks can approximate any continuous function. More recent research, such as [15, 26, 43] extended this exploration to the effects of deeper and wider network architectures. However, a significant challenge remains in these studies: they often do not fully explain the generalization power of neural networks, which is crucial for evaluating the performance of a statistical model.

Recently, some researchers have examined the generalization properties of networks. Bauer and Kohler [5], Schmidt-Hieber [46] showed the minimax optimality of networks with various activation functions for specific subclasses of Hölder functions, within the nonparametric regression framework. In contrasts to the static ERM approach, some studies made more attention to the dynamics of neural networks, particularly those trained using gradient descent (GD) and stochastic gradient descent (SGD)[2, 13, 20].

With similar insights, Jacot et al. [31] explicitly introduced the Neural Tangent Kernel (NTK) concept, demonstrating that there exists a time-varying neural network kernel (NNK) which converges to a fixed deterministic kernel and remains almost invariant during training as network width approaches infinity. And thus NTK theory proposes that network training can be approximated by a kernel regression problem [4, 29, 39, 50]. As a general case, fully-connected networks directly trained by GD, Lai et al. [35], Li et al. [41] showed the generalization ability of two-layer and multi-layer networks, respectively.

This paper mainly follows [4, 35, 41], and explores the impact of initialization in the NTK theory. Prior research [35, 41] which verified the minimax optimality of network utilized the so-called mirrored initialization setting. It refers to a combination of mirrored structure and mirrored initial value of parameters, which results in a zero initial output function. However, the assumption divates from the commonly used initialization strategy in real-world applications, whose initial output is actually non-zero. To bridging the gap, in this study we explore the generalization ability of standard non-zero initialized network, within the NTK theory framework. Our findings reveal that the vanilla non-zero initialization will theoretically results in poor generalization ability of network, especially when the data has relatively large dimension. If that is true, it suggests a divergence between theoretical models and real-world applications, highlighting a potential limitation in the current understanding of the NTK theory. Therefore, we arrive at a critical problem central to this study:

Does initialization significantly impact the generalization ability of networks within the kernel regime?

1.1 Our contribution

· Network converges to a NTK predictor uniformly . We show that under standard initialization, the network function converges to the NTK predictor uniformly over the entire training process and over all possible input in the domain. The convergence is essential in the study of the generalization ability of network in NTK theory. However, in previous work, the initial values of network has long been overlooked. Under mirrored initialization which leads to zero initial output function, Arora et al. [4], Lai et al. [35], Li et al. [41] demonstrated the point-wise convergence and the uniform convergence of network, respectively. More recently, Xu and Zhu [54] studied the uniform convergence of NTK under standard initialization, but did not study the convergence of the network function. Why the initial output of the network is ignored is not that it is insignificant, but rather because it is a stochastic function, making it challenging to analyze in convergence. Our findings make it valid to approximate the network's generalization ability based on the corresponding NTK predictor's performance.

· The generalization ability of standardly non-zero initialized fully-connected network . Our research explores the impact of standard non-zero initialization in NTK theory. At this issue, Zhang et al. [56] proposes the existence of implicit bias induced by non-zero initialization, when the neural network is completely overfitted. We delve deeper into this argument, studies the exact formula of the bias at any stage of training, within the framework of NTK theory. Additionally, we established that the (optimally tuned) learning rate of network is n -3 d +3 , even when the regression function is

sufficiently smooth. This insightful discovery implies a notable limitation in the generalization ability of networks with non-zero initialization, if NTK theory can precisely approximate the performance of real network. Consequently, we need to reconsider the weakness of NTK in the study of network theory. Also, the results show that mirrored initialization is superior to standard initialization in practical applications.

1.2 Related works

Our research is conducted within the framework of NTK theory. This type of research, in general, can be categorized into two main steps: the approximation of the network trained by GD through a kernel regression problem, and the evaluation on the corresponding kernel regression predictor. Several studies [2, 4, 19, 31] which focused on the former step, illustrated the point-wise convergence of NTK for multi-layer ReLU networks. Additionally, [39] demonstrated the point-wise convergence of the kernel regression predictor to the network. Furthermore, Lai et al. [35], Li et al. [41], Xu and Zhu [54] demonstrated the uniform convergence result with respect to all input and all time on two-layer and multi-layer networks. As to the latter step, a few researchers have analyzed the spectral properties of the NTK [6, 7] as well as kernel regression [40, 55]. Building upon these findings, Lai et al. [35] and Li et al. [41] demonstrated that early-stopping GD induces minimax optimality of the network. It is worth noting that the setting in these works assumes mirrored initialization, which may not be well-aligned with real-world scenarios. When it comes to initialization, Zhang et al. [56] provided insights into the impact of initialization under kernel interpolation, which is a special case of our results at t = ∞ .

2 Preliminaries

2.1 Model and notations

Suppose that { ( x i , y i ) } n i =1 are i.i.d. drawn from an unknown distribution ρ which is given by

where f ∗ ( x ) is the regression function and ϵ is a centered random noise. Suppose that the marginal distribution µ ( x ) of the radom variable x is supported in a non-empty bounded subset X of R d with C ∞ smooth boundary. The generalization error of an estimator ˆ f of f ∗ is given by excess risk

We introduce the following standard assumption on the noise(e.g., [21, 42]). It is clear that subGaussian noise satisfying this assumption.

Assumption 1 (Noise) . The noise term ϵ satisfies the following condition for some positive constant σ, L , and m ≥ 2 :

Notations Given a set of samples pairs { ( x i , y i ) } n i =1 , we denote X and Y to be vector ( x 1 , · · · , x n ) T and ( y 1 , · · · , y n ) T , respectively. In a similar manner, ( f ( x 1 ) , · · · , f ( x n )) T and ( f ( y 1 ) , · · · , f ( y n )) T are represented as f ( X ) and f ( Y ) , where f ( · ) : R d ↦→ R is an arbitrary given function. Regarding a kernel function k ( · , · ) : R d × R d ↦→ R , we use k ( x, X ) to denote the vector ( k ( x, x 1 ) , k ( x, x 2 ) , · · · , k ( x, x n )) and k ( X,X ) to denote the matrix [ k ( x i , x j )] n × n . For real number sequences such as { a n } and { b n } , we write a n = O ( b n ) (or a n = o ( b n ) ), if there exists absolute positive constant C such that | a n | ≤ C | b n | holds for any sufficiently large n (or | a n | / | b n | approaches zero). We also denote a n ≍ b n if there exists absolute positive constant c abd C such that c | b n | ≤ | a n | ≤ C | b n | holds for any sufficiently large n .

2.2 Reproducing kernel Hilbert space

Suppose that k is kernel function defined on the domain X satisfying that ∥ k ∥ ∞ ≤ κ 2 . Let H k be the reproducing Hilbert space associated with k which is the closure of linear span of { k ( x, · ) , x ∈ X}

under the inner product induced by ⟨ k ( x, · ) , k ( y, · ) ⟩ = k ( x, y ) . Given a distribution µ ( x ) on X , we can introduce an integral operator T k : L 2 ( X , µ ) → L 2 ( X , µ ) :

The celebrated Mercer's decomposition [12] asserts that

where { e i } i ∈ N and { λ 1 2 i e i } i ∈ N are the orthonormal basis of L 2 ( X , µ ) and H k respectively. It is well known that H k can be canonically embedded into L 2 ( X , µ ) .

If the eigenvalues λ i of k are polynomially decaying at rate β ( i.e, λ i ≍ i -β ), we can further introduce a concept of the relative smoothness of a function f ∈ L 2 ( X , µ ) . More precisely, let us recall the concept of real interpolation space [48] ( Please see more detailed information in the Appendix).

Real interpolation space The real interpolation space [ H k ] s is given by

with the inner product ⟨ ∑ i ∈ N a i λ s 2 i e i ( x ) , ∑ i ∈ N b j λ s 2 j e j ( x ) ⟩ [ H k ] s = ∑ i ∈ N a i b i for s ≥ 0 .

It is clear that [ H k ] s is a separable Hilbert space and is isometric to the l 2 space. With the definition above, we can see that [ H k ] 0 = L 2 ( X , µ ) and [ H k ] 1 = H k . Also, for any s 2 ≥ s 1 ≥ 0 , we know [ H k ] s 1 ⊆ [ H k ] s 2 with compact embedding. Let

which is often referred to the embedding index of an RKHS H k [21] . It is well known that α 0 ≥ 1 β and the equality holds for a large class of usual RKHSs if the eigenvalue decay rate is β . We further define the relative smoothness of a given function f :

Definition 2.1 (Relative smoothness) . Given a kernel k on X with respect to measure µ , the smoothness of a function f is defined as

where c i = ⟨ f, e i ⟩ L 2 ( X ,µ ) .

2.3 Kernel gradient flow

For a positive definite reproducing kernel k , the dynamic of kernel gradient flow (KGF) [22] is

where f GF t is the KGF predictor. In kernel gradient flow, the performance of kernel predictor depends on the relative smoothness of regression function. People often consider the case that α ( f, k ) ≥ 1 [10, 11] . When the smoothness satisfies α ( f, k ) < 1 , the regression function is said to be poorly smooth and belongs to the so-called misspecified spectral algorithm problem. We collect the related result in Zhang et al. [55] and apply it to our case, to derive the following proposition:

1

Proposition 2.2. Suppose the eigenvalue decay rate of k is β and the embedding index is β with respect to µ . Suppose the noise term ϵ satisfies Assumption 1. Let the dynamic (8) starts from f GF 0 = 0 . Also, suppose the regression function satisfies f ∗ ∈ [ H k ] s and ∥ f ∗ ∥ [ H k ] s ≤ R , for some s > 0 . Let γ ≤ s and 0 ≤ γ ≤ 1 . By choosing t ≍ n β sβ +1 , for any fixed δ ∈ (0 , 1) , when n is sufficient large, with probability at least 1 -δ , we have

where C is a positive constant.

3 Network and Neural Tangent Kernel

3.1 Network settings

Weconsider the fully-connected network with L hidden layers. As is commonly-used in deep learning, we consider the ReLU activation [44] defined by σ ( x ) := max( x, 0) . Denote f ( · ; θ ) : R d → R as the network output function, where θ representing the column vector that all parameters flattened into. We can write the recursive structure of network as following:

The parameter matrix for the l -th layer is denoted as W ( l ) . Their dimensions are of m l +1 × m l , where m l is the number of units in layer l and m l +1 is that of layer l +1 . Also, the bias term of the first layer is denoted as b (0) ∈ R m 1 × 1 . The setting of bias term is to make sure the positive definiteness of NTK [41]. We further assume that the number of units in each layer is at the same order while the width comes to infinity, as cm ≤ min( m 1 , · · · , m L +1 ) ≤ max( m 1 , · · · , m L +1 ) ≤ Cm where c, C are some absolute positive constants.

Standard initialization At initialization, the parameters are randomly set as i.i.d. standard normal variables:

Remark 3.1 (Mirrored initialization) . As to the mirrored initialization considered in [4, 35, 41], part of the network f (1) ( · ; θ (1) 0 ) undergoes standard initialization, while the other complicated corresponding part f (2) ( · ; θ (2) 0 ) holds the same structure as f (1) ( · ; θ (1) 0 ) , with parameters initialized to the same values as θ (2) 0 = θ (1) 0 . Lastly, the neural network output function is defined as f ( · ; θ 0 ) = √ 2 2 ( f (1) ( · ; θ (1) 0 ) -f (2) ( · ; θ (2) 0 ) ) . This setup ensures that f ( · ; θ 0 ) is constantly zero.

The network is trained under the mean square loss function. If we suppose { ( x i , y i ) } n i =1 be the training data, then the loss function is specified as

For notational simplicity, we denote by f NN t ( x ) = f ( x ; θ t ) . The training process for the network is performed by gradient flow, where the parameters are updated through the differential equation:

where ∂ θ f NN t ( X ) is a matrix with dimensions n × M , with M being the length of the parameter vector θ . This matrix represents the gradient of the network output f NN t ( X ) with respect to the parameters θ at time t . Incorporating the chain rule, we can formulate the gradient flow equation for the network function as follows:

3.2 Network at initialization

In order to state the properties of wide network with standard initialization, we need to introduce the concept of Gaussian process.

Gaussian process Gaussian process is a stochastic process for which every finite collection of random variables follows a multivariate Gaussian distribution. Let X be a Gaussian process with index t ∈ T . If the mean and covariance are given by the mean function m and the positive definite kernel k such that E [ X ( t )] = m ( t ) and Cov[ X ( t ) X ( t ′ )] = k ( t, t ′ ) , which holds for any t, t ′ ∈ T , then we say X ∼ GP ( m,k ) .

In standard initialization (10), the parameters of the neural network are i.i.d. samples from a standard normal distribution. If the network contains only one hidden-layer (that is, if L = 2 ), it is direct to prove that f NN 0 ( x ) converges to a centered Gaussian distribution by CLT, for any fixed point x ∈ X . As to the multi-layer network, prior research [25] also proved that such initialized network converges to a Gaussian process, as following:

Lemma 3.2 (Limit distribution of initialization) . As the network width m tend to infinity, the sequence of network stochastic process { f NN 0 } ∞ m =1 converges weakly in C ( X , R ) to a centered Gaussian process f GP . The covariance function is the so-called random feature kernel (RFK), which is denoted by K RFK ( x, x ′ ) as defined in (42) in Appendix C.1.

3.3 The kernel regime

As the gradient descent of neural network involves high non-linearity and non-convexity, it is difficult to study the training process. However, Jacot et al. [31] introduced the Neural Tangent Kernel (NTK) theory which provides a connection between network training and a class of kernel regression problems, when the network width comes to infinity. To demonstrate this, we first define a Neural Network Kernel (NNK):

Using this notation, we reformulate (13) in a kernel regression format:

NTK theory shows, if the network width m tends to infinity, then the random kernel K t m ( · , · ) will converge to a time-invariant kernel K NTK ( · , · ) : R d × R d → R , which is referred to as the NTK of network. The phenomenon is the so-called NTK regime [2, 31, 39]. The fixed kernel K NTK only depends on the structure of the neural network and the way of initialization. To get more knowledge of NTK, we present the explicit expression of NTK in Appendix C.1. In NTK theory, the dynamic of network (15) can be approximated by a kernel gradient flow equation:

which starts from Gaussian process f NTK 0 = f GP . In this way, if we aims to derive the generalization property of sufficiently wide network, we can achieve by considering the corresponding kernel gradient flow predictor. Such approximation is strictly ensured by uniform convergence of f NN t and f NTK t over all x ∈ X and all t ≥ 0 as m → ∞ , since we use L 2 excess risk to evaluate the generalization ability. Actually, we have the following theorem, whose proof is given in Appendix B.

Proposition 3.3 (Uniform convergence) . Given training sample pairs { ( x i , y i ) } n i =1 . For any δ ∈ (0 , 1) and ε > 0 , when network width m is large enough, we have

holds with probability at least 1 -δ .

In this theorem, we show the uniform convergence of network under standard initialization. Previous related studies [4, 35, 41, 54] always utilized delicately designed mirrored initialization (as shown in Remark 3.1) to avoid the analysis on the initial output function of network, since it will lead to the challenging problem that f NN t and f NTK t are both random, unlike that f NTK t is a fixed function in the case of mirrored initialization. However, as shown in Section 3.2, the initial output function is near a Gaussian process that can not be overlooked. To under the performance of neural networks commonly used in real world, it is necessary to analyzing the network initialization. To the best of our knowledge, we are the first to consider the initial output function of network in uniform convergence. This comprehensive result allows us to study the generalization error of network more precisely.

4 Impact of Initialization

4.1 Impact of standard initialization on the generalization error

The standard kernel gradient flow is always considered to start from zero, as in Proposition 2.2. Therefore, we need to do a transformation since the initial value of predictor f NTK t is actually f GP instead of zero. Firstly, we can yield a solution of (8) in matrix form:

where ϵ X is employed to represent the n × 1 column noise term vector Y -f ∗ ( X ) . We denote by f GF t the kernel gradient flow predictor under initial function f 0 and denote by ˜ f GF t the KGF predictor under initialization ˜ f GF 0 ≡ 0 . If we plug them into (17) and excess risk (2), respectively, we directly have the following theorem:

Proposition 4.1 (Impact of initialization in kernel gradient flow) . Denote ˜ f ∗ = f ∗ -f 0 as the biased regression function. For the KGF predictor f GF t and ˜ f GF t defined above, we have

The theorem establishes the equivalence of the generalization properties between the KGF predictor with initial value f 0 , regression function f ∗ and the KGF predictor with initial value zero, regression function f ∗ -f 0 . Back to the network case, combining uniform convergence result in Proposition 3.3, it suggests that, compared to mirrored initialization, the impact of standard initialization which has non-zero initial output function is equivalent to introducing a same-valued implicit bias to the regression function. This is a generalization of the main result in Zhang et al. [56], which only focused on case at t = ∞ . To summarize, Proposition 4.1 provides a convenient approach to quantify the impact of standard initialization in early-stopping neural networks.

4.2 Smoothness of Gaussian process

Building upon the analysis above, our focus now turns to illustrating the smoothness of the Gaussian process f GP , as it is the limit distribution of f NN 0 . Actually, we can derive the following theorem:

Theorem 4.2 (Smoothness of Gaussian Process) . Suppose that f GP is a Gaussian process with mean function 0 and covariance function K RFK . The following statements hold:

We furnish a comprehensive proof for Theorem 4.2 in the Appendix C.

Let us now turn our attention to the implications established by this theorem. Recall that Proposition 4.1 has shown that, in KGF, the existence of initialization function f GP is equivalent to adding a same-valued bias term to the regression function f ∗ . Consequently, the poor smoothness of initialization function causes the high smoothness assumption on the regression function meaningless. Regardless of how smooth we assume the regression function to be (e.g., α ( f ∗ , K NTK ) ≥ 2 ), the value of (relavtive) smoothness α ( f ∗ -f GP , K NTK ) will always be at most 3 d +1 . Namely, the biased regression function f ∗ -f GP is always poorly smooth. In this specific case, we could hardly expect the KGF predictor to have fine performance.

4.3 Upper bound

Now we are ready to provide the upper bound of generalization error of network. With the help of Proposition 2.2, Proposition 3.3, Proposition 4.1 and Theorem 4.2, we derive the following theorem:

Theorem 4.3 (Generalization error upper bound) . Assume that the regression function f ∗ ∈ [ H NT ] s for some s > 0 , and ∥ f ∗ ∥ [ H NT ] s ≤ R where R is a positive constant. Assume the marginal probability measure µ with density p ( x ) satisfies c ≤ p ( x ) ≤ C for some positive constant c and C .

For the case of s ≥ 3 d +1 , for any δ ∈ (0 , 1) and ε ∈ (0 , 3 d +3 ) , by choosing certain t = t ( n ) →∞ (as shown in Appendix), when n is sufficiently large and m is sufficiently large, with probability 1 -δ we have

where C ε is a positive constant related to ε .

For the case of 0 < s < 3 d +1 , for any δ ∈ (0 , 1) , by choosing t ≍ n d +1 s ( d +1)+ d , when n is sufficiently large and m is sufficiently large, with probability 1 -δ we have

where C s is a positive constant related to s .

The proof is provided in Appendix D. This result shows the generalization error upper bound for network with standard initialization and demonstrate its negative effect. Even if the goal function f ∗ is quite smooth, the generalization error upper bound n -3 d +3 remains to be a quite low rate, particularly considering that the dimension d of data is usually large in real world. It suggests that the network no longer generalizes well, even if we adopt the once useful early stopping strategy in Li et al. [41].

4.4 Lower bound

From the analysis above, we can see the poor generalization ability of network under standard initialization. Furthermore, in this section, we take spherical data as example and provide the lower bound of generalization error. Namely, we presume the input vectors x are distributed on the sphere S d with probability measure µ , which is a common assumption in NTK theory [6, 31, 36, 53]. We also slightly change the network structure. Compared to the network (9), we eliminate the bias term of the initial layer, as shown in (40) in Appendix. In this case, the NTK of new network is denoted by K NTK 0 , and the RKHS H NT 0 ( S d ) is abbreviated as H NT 0 , whose detailed properties is also given in Appendix C.1. Additionally, we make more assumption on the noise of data. We assume the noise term ϵ in (1) to have a constant second moment, as E [ | ϵ | 2 | x ] = σ 2 for x ∈ S d , a.e. . Under these conditions, with the help of method in Li et al. [40], we derive the theorem:

Theorem 4.4 (Generalization error lower bound) . We assume that the regression function f ∗ ∈ [ H NT 0 ] s for some s > 3 d +1 , and denote by ∥ f ∗ ∥ [ H NT 0 ] s ≤ R where R is a positive constant. Assume that µ is the uniform measure. For any δ ∈ (0 , 1) , when n is large enough and m is large enough, for any choice of t = t ( n ) →∞ , with probability at least 1 -δ we have

The proof is given in Appendix E. Through Theorem 4.4, we derive n -3 d +3 as the generalization lower bound of standardly random-initialized network in NTK theory, even if the regression function is quite smooth. The rate n -3 d +3 means model suffers notably from data that has large dimension: If d is relatively large, then this rate of convergence can be extremely slow. This is a manifestation of the curse of dimensionality. In fact, it contrasts with the fact that neural networks excel at highdimensional problems. This contradiction underscores the limitation of NTK theory for interpreting network performance.

5 Experiments

Our numerical experiments are conducted in two aspects to fully understand the impact of standard initialization. First, we show the performance of standard initialized network is indeed worse than the mirrored initialized case, on the aspect of learning rate. The phenomenon is in line with our theoretical analysis. Second, the smoothness of regression function of real data is significantly larger than 3 d +1 , which suggest the bad effect of non-zero intial output function of standard initialization will indeed destroy the performance of network if NTK theory holds. It demonstrates the drawback of NTK theory through contradiction.

5.1 Artificial data

In the first experiment, we employ artificial data to show the negative effect of standard initialization on the generalization error of network. The detailed settings are shown in Appendix F.

Learning rate of network under different initialization The experiments are conducted for both d = 5 and d = 10 , contrasting network performance subject to mirrored and standard initialization strategies. We choose a relatively smooth goal function to emphasize the impact of initialization. Specifically, we use m = 20 n , epoch = 10 n , and the gradient learning rate lr = 0 . 6 . The networks are made sufficiently wide to ensure the overparametrization assumption is met. Additionally, we implement the early-stopping strategy as mentioned in Theorem 4.3, that is, selecting the minimum loss across all epochs as the generalization error. Finally, we test the network's generalization error on different levels of sample size n , and plot the log value of the generalization error corresponding to log( n ) as shown in Figure 1. As we expected, the points in Figure 1 fits a linear trend. Moreover, the figure highlights the difference in learning rate under different initialization methods. This aligns with our theoretical results.

d=10

d=5

-1.6

Mirrored initialization, slope=-1.20

2.2

Mirrored initialization, slope=-1.09

Standard initialization,slope=-0.43

Standard initialization,slope=-0.50

-1.8

2.4

log1o(error)

log1o(eror)

2.0

-2.6

-2.2

-2.8

2.4

3.0

2.6

-3.2

2.8

2.9

3.0

3.1

3.2

2.80

2.85

2.90

2.953.003.053.103.15

log10(n)

5.2 Real data

In this subsection, we focus on datasets from the real world and estimate the smoothness of function. Although we could not know the goal function that the real data is generated from, there exists a way to estimate its smoothness [16]. We show the technical details in Appendix G.

Smoothness of goal function in real datasets We employed the MNIST, CIFAR-10 and FashionMNIST datasets[33, 38, 52]. In the experiments, we evaluate the smoothness of goal function of the datasets, with respect to the one-hidden layer NTK. The results are presented in Table 1. With the input dimension d = 784 , 3072 , 784 , we can compute that the smoothness of initialization function is equal to 3 d +1 ≈ 0 . However, the smoothness of goal function is far better than 3 d +1 , which implies that standard initialization will indeed destroy the generalization performance, under NTK theory. The contradiction between NTK theory and the real situation shows its limitation and once again confirms our conclusion.

6 Discussion

To summarize, this research focuses on the impact of standard random initialization on generalization property of fully-connected network in the NTK theory, which makes up the gap in this field. Many previous work [35, 41] verified the statistical optimality of neural network under delicately designed mirrored initialization, whose initial output function of network is zero. However, through our study, we pinpoint that if we consider the commonly-used standard initialization, the learning rate of network is notably slow when the dimension of data is slightly large, which fails to explain network's favorable performance in overcoming the curse of dimensionality. A direct implication of our work is the superiority of mirror initialization over standard initialization, which suggests a direction for future improvements. On a deeper level, although NTK theory can describe many properties of network, at least for the fully connected networks with Gaussian initialization discussed in this paper, we can explore better theoretical frameworks to characterize their generalization ability in the future.

Acknowledgments

Lin's research was supported in part by the National Natural Science Foundation of China (Grant 92370122, Grant 11971257).

References

Robert A Adams and John JF Fournier. Sobolev spaces . Elsevier, 2003.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

N. Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society , 68(3):337-404, 1950. URL http://dx.doi.org/10.2307/1990404 .

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems , 32, 2019.

Benedikt Bauer and Michael Kohler. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. 2019.

Alberto Bietti and Francis Bach. Deep equals shallow for relu networks in kernel regimes. arXiv preprint arXiv:2009.14397 , 2020.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems , 32, 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

William Elwood Byerly. An elemenatary treatise on Fourier's series, and spherical, cylindrical, and ellipsoidal harmonics, with applications to problems in mathematical physics . Dover Publicatiions, 1893.

Andrea Caponnetto. Optimal rates for regularization operators in learning theory. 2006.

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7:331-368, 2007.

Claudio Carmeli, Ernesto De Vito, and Alessandro Toigo. Reproducing kernel hilbert spaces and mercer theorem. arXiv preprint math/0504071 , 2005.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems , 32, 2019.

Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems , 22, 2009.

Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on learning theory , pages 698-728. PMLR, 2016.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems , 34:10131-10143, 2021.

George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989.

Eleonora Di Nezza, Giampiero Palatucci, and Enrico Valdinoci. Hitchhiker's guide to the fractional sobolev spaces. Bulletin des sciences mathématiques , 136(5):521-573, 2012.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2018.

Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. The Journal of Machine Learning Research , 21(1):8464-8501, 2020.

L Lo Gerfo, Lorenzo Rosasco, Francesca Odone, E De Vito, and Alessandro Verri. Spectral algorithms for supervised learning. Neural Computation , 20(7):1873-1897, 2008.

Nadine Große and Cornelia Schneider. Sobolev spaces on riemannian manifolds with bounded geometry: general coordinates and traces. Mathematische Nachrichten , 286(16):1586-1613, 2013.

Moritz Haas, David Holzmüller, Ulrike von Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. arXiv preprint arXiv:2305.14077 , 2023.

Boris Hanin. Random neural networks in the infinite width limit as gaussian processes. arXiv preprint arXiv:2107.01562 , 2021.

Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710.11278 , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359-366, 1989.

Tianyang Hu, Wenjia Wang, Cong Lin, and Guang Cheng. Regularization matters: A nonparametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics , pages 829-837. PMLR, 2021.

Simon Hubbert, Emilio Porcu, Chris Oates, Mark Girolami, et al. Sobolev spaces, kernels and discrepancies over hyperspheres. arXiv preprint arXiv:2211.09196 , 2022.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Vladimir Koltchinskii and Evarist Giné. Random matrix approximation of spectra of integral operators. Bernoulli , pages 113-167, 2000.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on r. arXiv preprint arXiv:2302.05933 , 2023.

Jianfa Lai, Zixiong Yu, Songtao Tian, and Qian Lin. Generalization ability of wide residual networks. arXiv preprint arXiv:2305.18506 , 2023.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems , 32:8572-8583, 2019.

Yicheng Li, Weiye Gan, Zuoqiang Shi, and Qian Lin. Generalization error curves for analytic spectral algorithms under power-law decay. arXiv preprint arXiv:2401.01599 , 2024.

Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains. Journal of Machine Learning Research , 25(82):1-47, 2024.

Junhong Lin and Volkan Cevher. Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. Journal of Machine Learning Research , 21(147): 1-63, 2020.

Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems , 30, 2017.

Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pages 807-814, 2010.

Yoshihiro Sawano et al. Theory of Besov spaces , volume 56. Springer, 2018.

Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. 2020.

Alex Smola, Zoltán Ovári, and Robert C Williamson. Regularization with dot-product kernels. Advances in neural information processing systems , 13, 2000.

Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and rkhss. Constructive Approximation , 35:363-417, 2012.

Robert S Strichartz. A guide to distribution theory and Fourier transforms . World Scientific Publishing Company, 2003.

Namjoon Suh, Hyunouk Ko, and Xiaoming Huo. A non-parametric regression viewpoint: Generalization of overparametrized deep relu network under noisy observations. In International Conference on Learning Representations , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, and Jeffrey Pennington. Precise learning curves and higher-order scalings for dot-product kernel regression. Advances in Neural Information Processing Systems , 35:4558-4570, 2022.

Jiaming Xu and Hanjing Zhu. Overparametrized multi-layer neural networks: Uniform concentration of neural tangent kernel and convergence of stochastic gradient descent. Journal of Machine Learning Research , 25(94):1-83, 2024.

Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms. arXiv preprint arXiv:2303.14942 , 2023.

Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo, and Zheng Ma. A type of generalization error induced by initialization in deep neural networks. In Mathematical and Scientific Machine Learning , pages 144-164. PMLR, 2020.

A Further notations

In appendix, we will provide many technical proofs. Before that, let us provide more notations. For two sets A and B with a mapping function ϕ : A → B , the notation ϕ ( A ) is used to denote the image set of A under ϕ . For two random variable sequences { u n } and { v n } , we denote by u n = o P ( v n ) (or u n = Ω P ( v n ) ) if the ratio u n /v n approaches zero (or u n ≥ cv n for some positive constant c ) in probability as n →∞ with respect to probability measure P . For two real number sequence { a n } and { b n } , we denote by a n = Ω( b n ) if there exists positive constant c and n 0 such that | a n | ≥ c | b n | holds for any n ≥ n 0 . For two sequences of real numbers { a n } and { b n } such that a n = Ω( b n ) (or a n = O ( b n ) ), we also denote by a n ≳ b n (or a n ≲ b n ) . If a n ≳ b n and a n ≲ b n , then we denote by a n ≍ b n .

B Proof of uniform convergence

In this section, we demonstrate the uniform convergence from f NN t to f NTK t .

B.1 Initialization

The following is a direct proposition based on Lemma H.2 and Lemma 3.2,

Proposition B.1. For the random network function sequence { f m 0 } with probability measures on ( C ( X , R ) , C ) , there exists { X m } and X GP defined on a new probability space (Ω ′ , F , P ) , on which we have

where X m and X GP has the same distribution as f m 0 and f GP , respectively.

Remark B.2. The separability of ( C ( X , R ) , C ) can be derived by the density of polynomials. Therefore, it satisfies the requirement of Lemma H.2. In the context of our study, our reliance is only on the distribution of { f m 0 } for each given value of m . Consequently, it is reasonable to reconstruct it in the new probability space. For convenience, we directly denote X m 0 as f m 0 (or f NN 0 ) and denote and X GP as f GP , respectively. In other words, we are considering the network function in a new probability space, even though this approach may result in a moderate abuse of notation.

B.2 Uniform convergence of network

Our aim is to give the uniform convergence between NTK regressor f NTK t and network function f NN t . Note that the NTK regressor is trained by NTK, and the network function is trained by NNK, which is denoted by K t m . Here we first show the uniform convergence between NNK and NTK as m comes to infinity.

Lemma B.3. For any δ ∈ (0 , 1) , suppose m is large enough, then with probability at least 1 -δ , we have

Proof. The proof is similar to that in Li et al. [41], while the difference is the way of initialization. So we only provide the sketch of proof. In Li et al. [41], the uniform convergence of NTK is proved through a standard ϵ -net argument, which is divided into point-wise convergence and continuity of both NTK and NNK. Namely, as the following decomposition:

where z, z ′ are the points in the ϵ -net which divides X .

Back to our case, in non-zero initialization, the structrue of NTK and NNK remain the same, as well as the continuity property. Consequently, the effect of initialization reflects on the point-wise convergence from K t m ( z, z ′ ) to K NTK ( z, z ′ ) , or more precisely, the NTK regime [2]. NTK regime requires that the residual decays to near zero and thereby the parameters will not deviate too far from their initial values in the training process, which holds under mirrored initialization. Standard

initialization lets the residual at time 0 be ∥ ∥ f NN 0 ( X ) -Y ∥ ∥ 2 , instead of ∥ Y ∥ 2 . Therefore, there is a slight risk that the residual is too large to decay to near zero during training. However, since

holds with high probability when m is large through Proposition B.1 and direct analysis on f GP , we can verify that the residual ∥ ∥ f NN t ( X ) -Y ∥ ∥ 2 is still not large enough to break the stable lazy regime. Namely, the control on parameter matrix that

still holds. In this way, we can finish the proof.

Then, we can derive the uniform convergence of network function.

Proof of Proposition 3.3. The proof is also similar to that of uniform convergence under mirrored initialization. Therefore, we only exhibit the sketch of different part. Define event A as

where C δ is some constant related to δ , such that event A holds with probability at least 1 -δ 2 when m is large enough. Such a constant C δ is ascertainable, as f NN 0 converges to f GP by Proposition B.1 and f GP is a Gaussian process with finite second moment. Define event B as

We have event B holds with probability at least 1 -δ 2 when m is large enough. Conditioned on event A and B , we do kernel gradient flow by K t m and K NTK on f NN 0 and f GP respectively. Let event C be

Conditioned on event A and B , we can prove that event C holds by Gronwall's inequality, as the same method in Lai et al. [35]. In this way, we can finish the proof.

After we get the uniform convergence of network function, we can obtain the proposition on the convergence of excess risk:

Proposition B.4. Suppose f ∗ ∈ L 2 ( X , µ ) . For any δ ∈ (0 , 1) and ε > 0 , when m is large enough , with probability at least 1 -δ , we have

Proof. Recall the dynamic equation of f NTK t , we have

Since the kernel function K NTK ( · , · ) is bounded, there exists some positive constant C , such that

The initial function of kernel gradient flow f NTK 0 = f GP follows a Gaussian process with mean 0 and covariance kernel function K RFK . By the boundness of K RFK , we can also bound f NTK 0 . That is, for any δ ∈ (0 , 1) , there exists a positive constant M δ such that with probability at least 1 -δ/ 2 ,

Denote λ 0 := λ min ( K NTK ( X,X ) ) . We have λ > 0 since K NTK is strictly positive definite [41]. Thus we have √ √

The excess risk

Since f ∗ ∈ L 2 ( X , µ ) where µ is probability measure, we also have f ∗ ∈ L 1 ( X , µ ) . Denote M f ∗ := ∥ f ∗ ∥ L 1 and ∆ := sup x ∈X ,t ≥ 0 | f NN t ( x ) -f NTK t | . We have

By Proposition 3.3, when m is large enough, with probability at least 1 -δ we have ∆ 2 ≤ ε/ (1 + C √ nλ -1 0 ( √ nM δ + ∥ Y ∥ 2 ) + M f ∗ ) . Thus the proposition is proved.

C Proof of the Theorem 4.2

Before the proof, first we introduce some basic properties of NTK and RFK, as well as some technical properties of Sobolev space. We say that two Hilbert space H 1 , H 2 are equivalent if they are equal as sets and share equivalent norm. If H 1 and H 2 are equivalent, we denote by H 1 ∼ = H 2 .

C.1 Basic properties of NTK and RFK

Dot-product kernel A reproducing kernel function k is dot-product if its value only depends on the dot-product of inputs. That is, there exists function κ such that

A dot-product kernel on sphere can be decomposed with spherical harmonic polynomials as the eigenfunction:

where spherical harmonic polynomials { Y n,l , l = 1 , · · · , a n } are also the orthonormal basis of L 2 ( S d , σ ) , with σ denoting the uniform measure on S d [47]. This is also its Mercer decomposition.

Now come back to our network case. We first define two dot-product kernels on S d ,

where u = ⟨ x, y ⟩ = x T y and

The definition of κ ( t ) 1 is given by the composition κ 1 ◦ κ 1 · · · ◦ κ 1 (a total of t compositions). The explicit expression indicates that K NTK 0 and K RFK 0 are dot-product kernels on S d .

K NTK 0 and K RFK 0 is the homogeneous NTK and RFK of a homogeneous fully-connected network f S defined on S d [6, 14], whose structural difference from (9) is the removal of the bias term in the first layer. Specifically, the network is structured as follows:

Homogeneous fully-connected network on sphere The network is constructed using the following recursive formula:

where the function σ is entrywise ReLU activation. The parameter matrix for the l -th layer is denoted as W ( l ) , whose dimensions are of m l +1 × m l , where m l is the number of units in layer l and m l +1 is that of layer l +1 for l ∈ { 0 , 1 , · · · , L -1 } . We also set m 0 to be equal to d +1 and m L +1 equal to 1 . The network is also random initialized as (10).

We can easily build a connection between our NTK and RFK for network (9) and the homogeneous kernels defined in (38). For x ∈ X , let ˜ x = ( x, 1) which means add 1 as the new last component of x . Define ϕ ( x ) := ( x, 1) ∥ ( x, 1) ∥ being an isomorphism from open set X to a subdomain of positive hemisphere shell S = ϕ ( X ) ⊂ S d + . Then we have

where f is network (9) and f S is network (40). Actually, we can thus verify that

We denote by H NT 0 and H RF 0 the RKHS on S d with respect to K NTK 0 and K RFK 0 . Their eigenvalue decay rates are well known:

Lemma C.1 (Bietti and Bach [6], Haas et al. [24]) . For K NTK 0 and K RFK 0 on S d with uniform measure σ , the decay rate of spherical harmonics coefficients satisfy

while the eigenvalues satisfy

Additionally, we list some further result on the eigenvalue decay rate of NTK and RFK provided by Li et al. [41], which will be used later:

Lemma C.2. Denote Ω as a non-empty subdomain of S d . For K NTK 0 and K RFK 0 , we have eigenvalue decay rate:

where σ is the uniform measure on S .

Lemma C.3. For K NTK and K RFK , we have eigenvalue decay rate:

where measure µ on bounded domain X ⊂ R d has density c ≤ p ( x ) ≤ C with respect to the Lebesgue measure.

C.2 Basic concepts of Sobolev space

Sobolev Space of integer power Let X be a open subset of R d . Let m ∈ N , 1 ≤ p ≤ + ∞ . Sobolev space W m,p ( X ) is defined as a set of function such that

where α is a vector with length n and D α f is the weak α -th partial derivative of f . In other words, the definition of W m,p is:

where 1 ≤ p ≤ ∞ . Conventionally, when the index p is equal to 2 , we denote W m,p by H m , since it is a Hilbert space. Further, if the index m > d 2 , the Sobolev space H m qualifies as a RKHS and thus embraces the properties of RKHS. In our work, we mainly utilize its property of interpolation as defined in (6). Consequently, we first introduce a generalized concept of real interpolation [1], as an expansion to the definition in (6).

Real interpolation For two Bananch spaces H 1 and H 2 , we use interpolation space to represent a space that lies in between them in some specific way. We introduce the commonly-used K-method to define real interpolation. Suppose 0 < s < 1 , q ≥ 1 . The space generated by their real interpolation H = ( H 1 , H 2 ) s,q is defined by following:

and

Based on the definition of real interpolation, we introduce some basic concepts about fractional power Sobolev space.

Sobolev space of fractional power Suppose X ∈ R d is a bounded domain with smooth boundary and denote Lebesgue measure by µ . We can define fractional power Sobolev space through real interpolation (we refer to [45] Chapter 4.2.2 for more details):

The fractional power Sobolev space H r ( X ) with r ≥ d 2 is also a RKHS [1]. Specifically, Steinwart and Scovel [48] reveals that for 0 < s < 1 ,

for RKHS H and the interpolation defined in (6). Therefore, the results above directly implies that

holds for any r ≥ d 2 and s > 0 .

Up to now, we have introduced the basic properties of Sobolev spaces on X , an open subset of R d . For Sobolev spaces defined on more intricate manifolds, such as hyperspheres, owing to the intricate property of Sobolev spaces, numerous equivalent definitions emerges [1, 18].

We now delineate a kind of definition that will facilitate our subsequent proofs, since we will consider RKHSs on S d , like H NT 0 and H RF 0 , which are the RKHSs associated with K NTK 0 and K RFK 0 , respectively. Such definition can form a linkage with the Sobolev spaces defined on S d and on domain X ⊂ R d , which is also utilized in Haas et al. [24]. Our exposition begins with the characterization of a manifold.

Trivilization Define a trivialization of a Riemannian manifold ( M,g ) with bounded geometry of dimension d , which consists three part. The first part is some locally finite open covering { U α } α ∈ I . The second part is the charts { κ α } α ∈ I which consists of smooth diffeomorphism κ α : V α ⊂ R d → U α . The third part is a partion of unity h α such that supp( h α ) ⊂ U α , ∑ α ∈ I h α = 1 and 0 ≤ h α ≤ 1 .

In our case, we write a trivialization of S d , which, is a manifold of dimension d . We write U 1 = { x d +1 < ϵ | x ∈ S d } and U 2 = { x d +1 > ϵ 2 | x ∈ S d } for a small fixed ϵ > 0 . Let ϕ 1 : U 1 → R d and ϕ 2 : U 2 → R d be stereographic projections with respect to x 1 = (0 , 0 , · · · , 1) and x 2 = (0 , 0 , · · · , -1) , respectively. Namely, they are

and

Finally, we can find C ∞ smooth functions h 1 and h 2 such that h 1 | S d + = 1 . For the simple trivialization above, we can directly verify that it meets the admissible trivialization condition (details see Große and Schneider [23]). Thus we can apply Theorem 14 of [23] to define the norm of Sobolev space on S d :

for distribution f ∈ D ′ ( S d ) [49]. It gives a kind of equivalent definition of Sobolev space on S d .

C.3 Relationship between dot-product kernel and Sobolev space

Previous work observed that, for dot-product kernels defined on sphere with polynomial eigenvalue decay rate, their RKHSs are equivalent to Sobolev spaces:

Lemma C.4 (Hubbert et al. [30] Section 3) . For a dot-product kernel k defined on S d and its RKHS H k , if the coefficients of spherical harmonic polynomials satisfies µ n ≍ n t for some t ≥ d , then there exists an equivalence between RKHS and Sobolev space:

Recall that K NTK 0 and K RFK 0 are both dot-product kernels with polynomial eigenvalue decay rate by Lemma C.1. Therefore, Lemma C.4 provides the equivalence between H NT 0 , H RF 0 and the corresponding Sobolev spaces on S d . We have the following proposition:

Proposition C.5. We have the following equivalence:

C.4 Interpolation of H NT 0 and H RF 0

In this subsection, we aims to provide the interpolation relationship between RKHSs associated with K NTK 0 and K RFK 0 , on a subdomain of S d . We remind that if we consider the case on S d , i.e. H NT 0 and H RF 0 , the conclusion is direct since they are both dot-product kernels and share the same orthogonal basis { Y n,l } as introduced in (37).

Suppose s ≥ d 2 and Ω be a subdomain of S d with C ∞ smooth boundary. With a little abuse of notation, we define H s (Ω) as the RKHS H s ( S d ) restricted to Ω in the way of Lemma H.3. For an injection φ : Ω → R d , we define H s ( φ (Ω)) ◦ φ := { f ◦ φ | f ∈ H s ( φ (Ω)) } with norm ∥ f ◦ φ ∥ = ∥ f ∥ H s ( φ (Ω)) . Recall that ϕ 1 is the stereographic projection defined in (52), now let us show the equivalence of H s ( S d + ) and H s ( ϕ 1 ( S d + )) ◦ ϕ 1 .

Lemma C.6 (Equivalence as sets) . Suppose s ≥ d 2 . At the aspect of sets, we have H s ( S d + ) = H s ( ϕ 1 ( S d + )) ◦ ϕ 1 .

Proof. For a f ∈ H s ( S d ) , we have an extension ∥ f ′ ∥ s S d < ∞ such f ′ | S d = f . Thus

which implies ( h 1 f ′ ) ◦ ϕ -1 1 ∈ H s ( R d ) . Then we have [( h 1 f ′ ) ◦ ϕ -1 1 ] | ϕ 1 ( S d + ) ∈ H s ( ϕ 1 ( S d + )) . Since f = f ′ | S d + and h 1 | S d + = 1 , we have f ◦ ϕ -1 1 ∈ H s ( ϕ 1 ( S d + )) .

In the converse direction, we assume f ∈ H s ( ϕ 1 ( S d + )) . Then we know there exists f ′ ∈ H s ( R d ) such that f ′ | ϕ 1 ( S d + ) = f . Now we want to show f ◦ ϕ 1 ∈ H s ( S d + ) . Define a ψ ∈ C ∞ ( R d ) such that ψ ( ϕ 1 ( S d + )) ≡ 1 and ψ (( ϕ 1 ( U 1 /U 2 )) c ) ≡ 0 . According to (54), we have

Thus we finish the proof.

Lemma C.7 (Equivalence as space) . Suppose s ≥ d 2 . At the aspect of spaces, we have H s ( S d + ) ∼ = H s ( ϕ 1 ( S d + )) ◦ ϕ 1 .

Proof. By Lemma C.6, we know H s ( S d + ) ∼ = H s ( ϕ 1 ( S d + )) ◦ ϕ 1 as sets. Since H s ( S d + ) and H s ( ϕ 1 ( S d + )) ◦ ϕ 1 are both RKHSs, we can finish the proof by closed graph theorem.

For notational simplicity, denote by H 1 = H s ( S d + ) and H 2 = H s ( ϕ 1 ( S d + )) ◦ ϕ 1 . Define the canonical map I : H 1 → H 2 as I : h ↦→ h . Let { h n } n ∈ N be a sequence such that there exists h ∈ H 1 and g ∈ H 2 where h n → h in H 1 and h n = Ih n → g in H 2 . It implies that h = g . Therefore, closed graph theorem shows that the linear operator I is bounded, which means that ∥ h ∥ H 1 ≤ C ∥ h ∥ H 2 holds for some positive constant C and any h ∈ H 1 . We can also prove ∥ h ∥ H 2 ≤ C ′ ∥ h ∥ H 1 for any h in the same way. Consequently, the lemma is proved.

Now we come back to our network case. Let S := ϕ ( X ) ⊂ S d + where X is the set from which data x is sampled, and ϕ is used in (42). Since the boundary of X is C ∞ smooth, we know that S is C ∞ smooth. If we combine Lemma C.4, Lemma C.7 and Proposition H.4, then we can directly show the following lemma:

Lemma C.8. Define X 1 = ϕ 1 ( S ) . For K NTK 0 and K RFK 0 defined on S , we have the following equivalence:

Now we can obtain the interpolation relationship between H RF 0 ( S ) and H NT 0 ( S ) .

Lemma C.9. Suppose s ≥ 0 . We have

Proof. Define X 1 = ϕ 1 ( S ) . Let σ be the uniform measure on S d . Recalling (51), we have the interpolation on X 1 with lebesgue measure denoted by µ 1 :

that is

Since f ↦→ f ◦ ϕ 1 is an isometric isomorphism, we have

Recall that X is bounded and thus X 1 = ϕ 1 ( ϕ ( X )) is bounded. Therefore, the Jacobian Jϕ -1 1 satisfies c ≤ | Jϕ -1 1 | ≤ C for some constant c and C . It is easy to verify that L 2 ( X 1 , µ 1 ) ◦ ϕ 1 = L 2 ( S, µ 1 ◦ ϕ 1 ) ∼ = L 2 ( S, σ ) . Finally, with Lemma C.8, Lemma H.5 and Lemma H.6, we have

with respect to the uniform measure σ on S .

C.5 Smoothness of Gaussian process

Lemma C.9 provides the interpolation relationship between H NT 0 ( S ) and H RF 0 ( S ) . By the kernel transformation relationship of NTK and RFK from R d and to S d as described in (42), we can also derive the interpolation relationship of H NT and H RF . It will help for us to derive the smoothness of f GP .

Lemma C.10 (Interpolation of RKHSs) . Suppose s > 0 . We have

with respect to measure µ on X which has Lebesgue density c ≤ p ( x ) ≤ C .

Proof. Define a function ρ ( x ) = ∥ ˜ x ∥ on X . Define measure ν on X such that the Radon-Nikodym derivative satisfies d ν d µ = ρ 2 . We consider measure ν ◦ ϕ on S as well as measure µ on X , and then define a map I : [ H NT 0 ( S )] s → [ H NT ( X )] s :

Now we prove I is an isometric isomorphism. We first show that for any eigen pair ( f, λ ) of ( K NTK 0 , S, ν ◦ ϕ ) , ( If, λ ) is also an eigen pair of ( K NTK , X , µ ) . Actually, for eigen pair ( f, λ ) we have

We perform a transformation of the integral domain,

Recalling the transformation between K NTK 0 and K NTK in (42), we have

These transformations are both reversible. Therefore, through the structure of real interpolation space as described in (6), we can see I is an isometric isomorphism . In the same way, there exist isometric isomorphism I ′ : [ H RF 0 ( S )] s ( d +1) d +3 → [ H RF ( X )] s ( d +1) d +3 :

Combined the result in Lemma C.9, the Lemma is proved.

Now we are ready to give the smoothness of Gaussian process f GP . We remind the reader that H NT and H RF are abbreviations used for denoting H NT ( X ) and H RF ( X ) , respectively.

Proof of Theorem 4.2. Let t = s ( d +1) d +3 to simplify the notation. By Lemma C.10, we have

Recalling the structure of interpolation space, we suppose [ H RF ] t can be written as

Recall that f GP represents a random function defined on (Ω , F , P ) , where each ω ∈ Ω corresponds to a path function f GP ω : X → R . We can express this in the orthonormal basis as f GP ω = ∑ i ∈ N a i ( ω ) λ t 2 t e i , where

Recall that as defined in Lemma 3.2, f GP has the distribution GP (0 , K RFK ) . From this, we can acquire the joined distribution for a i . Firstly, let us compute the covariance:

̸

The exchange of integration is accomplished by Fubini's theorem since K RFK is a bounded kernel function, and both e i and e j are L 2 integrable. Moreover, as f GP is a Gaussian process, we finally get a i ∼ N (0 , λ 1 -t i ) for i ∈ N , and a i , a j are independent for any i = j . Consequently, we can directly derive that

where { Z i } indicates a collection of independent and identically distributed standard Gaussian random variables. Finally, as Lemma C.3 establishes the eigenvalue decay rate as

it is direct to prove the theorem.

Part 1. When s < 3 d +1 , we have d +3 d · (1 -t ) > 1 and thus

Consequently we have P ( ∥ ∥ f GP ∥ ∥ 2 [ H NT ] s < ∞ ) = 1 .

Part 2. When s ≥ 3 d +1 , we ascertain that d +3 d · (1 -t ) ≤ 1 and consequently

Denote by X n = ∑ n i =1 λ 1 -t i Z 2 i . We then obtain

We can thus derive that

Given that ∥ ∥ f GP ∥ ∥ 2 [ H NT ] s ≥ X n for any n ∈ N + , we have

This completes the proof.

D Proof of Theorem 4.3

With the findings from Theorem 4.2, Proposition 4.1, and Proposition B.4, the influence of non-zero initialization could be interpreted in terms of a misspecified spectral algorithms problem. To apply Proposition 2.2, it only remains to determine the embedding index of H NT . Now, let's proceed to do so.

D.1 Embedding index of H NT

Recall that the Proposition 2.2 requires the embedding index of H NT on X under the probability measure µ . Fortunately, the embedding index of the Sobolev space has been previously established by Zhang et al. [55], which is helpful to simplify our proof.

Lemma D.1 (Zhang et al. [55] Section 4.2, Embedding index of Sobolev space) . Suppose r > d 2 . For a bounded open set X ⊂ R d and Lebesgue measure µ , the embedding index of H r ( X ) equals d 2 r .

Since we have established the relationship between H NT and the Sobolev space, we can easily get the embedding index through a similar way used in the proof of Lemma C.10.

Lemma D.2 (Embedding index of NTK) . Suppose that the density function p ( x ) of probability measure µ satisfies the condition c ≤ p ( x ) ≤ C , where c and C are positive constants. The embedding index of H NT ( X ) with respect to µ is concluded to be d d +1 .

We omit this proof as it can be carried out in the same manner as Lemma C.10. Here, we provide only the structure. First, the embedding index of H d +1 2 ( S ) is d d +1 ( Lemma D.1 and Lemma C.7). Second, the embedding index of H NT 0 ( S ) is d d +1 (Lemma C.4). Third, the embedding index of H NT ( X ) is d d +1 since I : f ↦→ ρ · ( f ◦ ϕ ) is isometric isomorphism both from H NT 0 ( S ) to H NT ( X ) and from L ∞ ( S, ν ′ ◦ ϕ ) to L ∞ ( X , µ ) , where measure ν ′ is defined on d ν ′ d µ = ρ (an argument similar to that in the proof of Lemma C.10).

D.2 Proof of Theorem 4.3

Proof of Theorem 4.3. Recall that Proposition 4.1 elucidates the impact of non-zero initialization. Namely, the generalization error of the kernel gradient flow with an initialization of f 0 and a regression function f ∗ , is consequently equivalent to that of kernel gradient flow with initialization at 0 and a regression function of f ∗ -f 0 . On the other hand, Proposition B.4 demonstrated the uniform convergence from the network function to the kernel gradient flow predictor as the network width m tends to infinity. Lemma D.2 verify the embbeding index condition in Proposition 2.2. Thus, we only need to verify the source condition that f GP -f ∗ fulfills and to incorporate it with Proposition 2.2 in order to derive the generalization error of the kernel gradient flow.

Now we start the proof. Since the proofs for the cases s ≥ 3 d +1 and 0 < s < 3 d +1 are exactly the same, we will only provide the proof for the former case here. Through Theorem 4.2, we know for any 0 < r < 3 d +1 , it follows that E ∥ ∥ f GP ∥ ∥ 2 [ H NT ] r = ∑ λ 1 -r i < ∞ . Let C t = E ∥ ∥ f GP ∥ ∥ [ H NT ] r . By the Markov inequality, for any δ ′ ∈ (0 , 1) , we have with probability exceeding 1 -δ ′ , that

Recall that the eigenvalue decay rate for K NTK is d +1 d as mentioned in Lemma C.2. Therefore, we have for any δ ∈ (0 , 1) and any ε ∈ (0 , 3 d +3 ) , there exists r < 3 d +1 such that rβ rβ +1 = 3 d +3 -ε (i.e., r = d 2 -6 d -3 d ( d +3) ε 3( d +1)+ ε ( d +1)( d +3) ). Denote by ˜ f ∗ = f ∗ -f GP and ˜ f NTK t be the kernel gradient flow predictor starts from initial value 0 . Through Proposition 2.2, We thus have

holds with probability at least 1 -2 δ ′ when t ≍ n β rβ +1 . Through Proposition 4.1, also we have

holds with probability at least 1 -2 δ ′ . Through uniform convergence in Proposition B.4, we have

with probability at least 1 -δ ′ when m is large enough. Therefore, with appropriate choice of δ ′ and C ′ , we can finish the proof.

E Proof of Theorem 4.4

In this section, we establish the generalization error rate lower bound in our problem. We incorporate a result delineated in [40], which systematically studies the learning rate of kernel regression. Prior to this, we take some preparatory work.

We assume k is a dot-product kernel on S d with eigenvalue decay rate β , with respect to the uniform measure. We notate the corresponding RKHS as H k . Then, we can verify that H k satisties to the definition of regular RKHS , as detailed in [40]. Subsequently, the main theorem in [40] can be applied under our proposed settings, since K NTK 0 is a dot-product kernel defined on S d . It engenders the following lemma.

Lemma E.1 (Generalization error lower bound) . Assume k is a dot-product kernel defined on S d , we have the interpolation space of its RKHS as [ H k ] s = { ∑ i ∈ N a i λ s 2 i e i | ∑ i ∈ N a 2 i < ∞ } where { e i } i ∈ N is the orthonormal basis of L 2 ( S d , σ ) and σ denotes the uniform measure. Decompose the regression function f ∗ over the series of basis:

We assume that f ∗ ∈ [ H ] t holds for any t < s for a given s > 0 . Also, we we assume that

We also assume that the noise term satisties E [ | ϵ | 2 | x ] = σ 2 holds for x ∈ S d ,a.e. Then, we define the main bias term in generalization error by

Fix the given input vectors of samples X . Consider the kernel gradient flow process detailed in (8) and let it start from 0 . For any choice of t = t ( n ) →∞ , we have

With the lemma above, now we are ready to prove Theorem 4.4.

Proof of Theorem 4.4. By Proposition 4.1, we know the initial output function introduce an implicit bias term to the regression function. And thus the original problem is same as to consider a standard kernel gradient flow problem start from initial output zero with regression function ˜ f ∗ = f ∗ -f GP . Recall that µ is the uniform measure. On sphere, the RKHSs of dot-product kernels K NTK 0 and K RFK 0 are equivalent to corresponding Sobolev spaces through Lemma C.4. More precisely, suppose that we have chosen an orthonormal basis { e i } i ∈ N consisting of spherical harmonic polynomials. Then we have

for any t ≥ 0 . Through Lemma C.2, we have the eigenvalue decay rate:

We denote by β 1 = d +1 d and β 2 = d +3 d . Similar to the proof of Theorem 4.2, we write the Kosambi-Karhunen-Loève expansion of ˜ f ∗ :

and define the variance term by where

Here a i is a sequence of independent standard Gaussian variables, and b i represents a sequence derived from the decomposition of f ∗ . With such decomposition, we can verify that (83) holds with probability 1 -δ ′ for any δ ′ ∈ (0 , 1) . Denote by g ( λ ) = ∑ i : λ>ω i | ˜ f i | 2 . Firstly, we have

We also have the variance

Where the second term is controlled by the source condition assumption on f ∗ . Therefore, we have

Define event A ( λ ) = {| g ( λ ) -E [ g ( λ ]) | ≤ E [ g ( λ )] / 2 } . For any δ ′ ∈ (0 , 1) , we choose a sequence ˜ λ j , such that ˜ λ j = C ′ j -2( d +1) 3 . Then we have

We can choose appropriate C ′ > 0 such that ∪ j ∈ N A ( ˜ λ j ) holds with probability at least 1 -δ ′ , we denote by event A . Conditioned on event A , for any ˜ λ j +1 ≤ λ ≤ ˜ λ j , we have

which shows that

Therefore, we finish the proof of (83).

Then, we turns to the calculation of generalization error lower bound. First, we plug in the decomposition and calculate the bias term R ( t ; ˜ f ∗ ) :

Recalling that the eigenvalue decay rate is denoted by β , it follows that

Also, the variance of R 2 ( t ; ˜ f ∗ ) follows that

Here we introduce the denotations:

We then have

As to V 2 , we first recall that the smoothness of f ∗ lead to the following inequality:

which implies that

Now we turn to the evaluation of V 2 :

It is worth noting that we use Cauchy's inequality to derive the upper bound above. With the control of V 0 and V 2 , we have

Consequently, by Chebyshev's inequality, we directly have

Since t = t ( n ) → + ∞ , we have where

f

(

x

) =

( ∑

d

j

=1

x

j

,

ε

∼ N

(0

, σ

In the same way, we also have the bound of variance term N ( t ) .

Finally, apply Lemma E.1 and Proposition 3.3. We derive that for any δ > 0 , as long as n is large enough and m is large enough, for any choice of t = t ( n ) →∞ , with probability at least 1 -δ we have

Thus the theorem is proved.

Remark E.2. In Proposition 3.3, we consider the situation that both input X and output Y of samples are fixed, while in the proof above we require that only X is fixed. However, the conclusion of Proposition 3.3 still holds when Y of samples is random. This is because the noise term has a finite second moment. Also, the change of domain from X to S d will not affect the uniform convergence result.

F Details in artificial data experiments

Fixing the dimension of data as d = 5 , 10 . We draw samples for variable x from the standard Gaussian distribution N (0 , I d ) , which are consequently standardized to lie on the surface of the unit hypersphere S d . The dependent variable y is formulated as:

)

and

σ

= 0

.

2

. The function

f

exhibits notable smoothness, since it can be linearly represented in terms of the first few spherical harmonic polynomials on S d [9] and the fact that K NTK 0 is a dot-product kernel. We consider fully-connected network with one singular hidden layer, choosing m = 20 ∗ n to ensure large enough width. Consistent to previous sections, we choose ReLU function as the non-linear activation and train the network using Gradient Descent for a sufficiently long time. We record the generalization error at each moment and define the moment of minimum generalization error as the final generalization error. This is done to align with the early stopping strategy mentioned in the Theorem 4.3.

G Details in real data experiments

In this subsection, we will provide the theoretical basis of the method which approximates the smoothness of the goal function of real dataset. Let X ⊂ R d be a bounded domain. Given a reproduce kernel k ( · , · ) on X and a probability measure µ . Denote the RKHS by H k = { ∑ i ∈ N a i λ 1 2 i e i ∣ ∣ ∣ ∑ i ∈ N a 2 i < ∞ } . We assume that there is a function f : X → R and a probability density µ on X . Suppose that the samples satisfies y = f ( x ) , then f has the decomposition:

)

2

2

The smoothness α f = α ( f, k ) depends on the coefficients c i : if we have θ i ≍ i -d c and λ i ≍ i -d λ , then we derive the smoothness: α f = 2 d c -1 d λ .

We consider n samples { ( x i , y i ) } n i =1 . The Gram matrix k ( X,X ) can be decomposed as

In this regard, we can utilize the eigenvalue of the empirical kernel matrix to estimate the eigenvalue of the kernel function, since previous work has shown the convergence of eigenvalue when n is large enough [32]. Through the decomposition Y = ϕc (i.e., c = 1 n ϕ T Y ) and the approximation c i ≈ θ i , we can roughly estimate the eigenvalue decay rate when n is large enough with respect to i :

In our experiments, we let n = 3000 . Namely, an arbitrary selection of 3000 samples was made from the each dataset. We did not use all samples in the datasets, because n = 3000 is already sufficient to calculate the decay rate of eigenvalues. We consider the NTK of a one-hidden-layer fully connected network as the kernel k . The results is shown in Figure 2, Figure 3 and Figure 4 for the three datasets, respectively. In each figure, the scatter plot shows the log value of the summed squares of each c k (for i ≤ k ≤ n as per equation (113)) against log 10 i on the x-axis. Also, the dashed line represents the corresponding least-square regression fitting using index i smaller than 2700 . Theoretically, the slope of the dashed line will be -α f d λ .

MNIST

-Decay rate=-0.40

0

M=

log10

-2

E-

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

log10(i)

Fashion-MNIST

1.5

Decay rate = -0.22

1.0

0.5

0160

F0°0

0.5 -

1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

log10(i)

CIFAR-10

1.5

--Decay rate =-0.09

1.0

0.5

M

log10

F00

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

log10(i)

H Technical Lemmas

In this section, we introduce a series of technical lemmas. These will be helpful in our proof, and many of the lemmas have been established by prior researchers.

Lemma H.1 (Change of measure [41] ) . For a positive definite kernel k defined on a compact set X , it has the same eigenvalue decay rate under two measure ν and σ :

if the Radon derivative p = d ν d σ exists and c ≤ p ≤ C holds for some positive constant c and C .

Lemma H.2 (Skorohod's Representation Theorem) . Suppose that a sequence of probability distribution { F n } converges weakly to F and F has a separable support. Then there exist random variables X n and X , defined on a new probability space (Ω ′ , F , P ) , such that the distribution of X n is F n , the distribution of X is P , and X n → X holds almost surely.

Lemma H.3 (Restriction of RKHS [3]) . Suppose H k is a RKHS defined on E with the norm ∥·∥ H k , then k | Ω restricted to a subset Ω ⊂ E is the reproducing kernel of space { f ′ = f | Ω | , f ∈ H k } with norm defined by

The following proposition is a direct proposition of Lemma H.3:

Proposition H.4 (Equivalence of RKHS under resriction) . Assume RKHSs H 1 ∼ = H 2 are defined on E . Write Ω as a subset of E . Then we have H 1 | Ω ∼ = H 2 | Ω .

The following two lemmas are common in real interpolation:

Lemma H.5 (Equivalence of interpolation spaces) . Suppose 0 < s < 1 . Denote L 2 = L 2 ( X , µ ) for abbreviation. If we have RKHSs H 1 ∼ = H 2 , then ( L 2 , H 1 ) s, 2 ∼ = ( L 2 , H 2 ) s, 2 .

Proof. To prove the lemma, we only need to prove that the embedding ( H 1 , L 2 ) s, 2 ↪ → ( L 2 , H 2 ) s, 2 and ( L 2 , H 2 ) s, 2 ↪ → ( L 2 , H 2 ) s, 2 are both bounded.

First we prove that ∥ ∥ ( L 2 , H 1 ) s, 2 ↪ → ( L 2 , H 2 ) s, 2 ∥ ∥ ≤ C 1 where C 1 is an absolute positive constant.

For any x ∈ L 2 + H 1 , define the K-functional

Since H 1 ∼ = H 2 , for any x ∈ H 1 , we have ∥ x ∥ H 2 ≤ C ∥ x ∥ H 1 . Thus we have

Then we have

Let C 1 = C 2 s , we have the canonical injection satisfies ∥ ∥ ( L 2 , H 1 ) s, 2 ↪ → ( L 2 , H 2 ) s, 2 ∥ ∥ op ≤ C 1 . Also, since H 1 ∼ = H 2 , for any x ∈ H 2 , we have ∥ x ∥ H 1 ≤ c ∥ x ∥ H 2 . We can prove ∥ ∥ ( L 2 , H 2 ) s, 2 ↪ → ( L 2 , H 1 ) s, 2 ∥ ∥ op ≤ C 2 in the same way. Then, we finish the proof.

Lemma H.6 (Equivalence of interpolation spaces) . Suppose 0 < s < 1 . Denote H be a RKHS and µ, ν be measures on set X . If we have L 2 ( X , µ ) ∼ = L 2 ( X , ν ) , then ( L 2 ( X , µ ) , H ) s, 2 ∼ = ( L 2 ( X , ν ) , H ) s, 2 .

Proof.

The proof in accomplished in the same way as Lemma H.5.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and the introduction, we have summarized the background of the NTK theory and the influence of initialization on neural networks under this background. We have also discussed the results, which reflects the contributions of this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Compared to the other sections, stronger assumptions were used in the lower bound section (data distributed on the sphere). Although this is common in kernel regression, we still reminded in the paper that this is a more strict assumption.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided assumptions in the main body of the paper, and have also given comprehensive theoretical background information and proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In our experiments, our model is simple and uses public datasets. The experimental results are easy to reproduce.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, the datasets we utilized are publicly accessible for everyone. We have comprehensively outlined the experimental parameters and model settings within the experimental section of our paper. Notwithstanding the low complexity of our model, we are more than willing to post our code on GitHub should there be a demand for it.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper has indeed provided all necessary training and testing details for understanding the results. We leveraged the several datasets which are publically accessible to everyone. Furthermore, we have comprehensively outlined all the experimental parameters and model settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In the artificial data experiment, we can directly see the generalization error decay rate in the figure. In the real data experiment, we do a least square regression to calculate the smoothness of a function.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, the computational resources needed are minimal, which means the experiments can be easily conducted on nearly any GPU without any additional requirements.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we conducted original theoretical research, which is in accordance with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is mainly a theoratical work focusing on learning theory, so this question is not applicable to our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is mainly a theoratical work focusing on learning theory, so this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have explicitly referenced the datasets in our paper, as MNIST, CIFAR-10 and Fashion-MNIST.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We did not utilize new assets in our work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing experiments or research related to human subjects, therefore this question is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve any study participants, so there is no need to discuss potential risks they might face. Therefore, there is also no need for an Institutional Review Board (IRB) approval. Hence, this question is not applicable to our paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors discuss the impact of random initialization of deep neural networks in the neural tangent kernel (NTK) regime.\nPrecisely, the authors prove that in the case of standard random initialization, the network function in the finite wide limit, converges to the NTK predictor uniformly.\nThen, by studying the behavior of the underlying Gaussian process, the authors establish upper and lower bounds for the generalization errors of DNNs in the NTK regime, in terms of the real interpolation space of the RKHS.\n\nSTRENGTHS:\nThe paper studies an important problem.\nThe paper is in general well written and not difficult to follow.\nThe numerical results, at least those with artificial data, look compelling.\n\nWEAKNESSES:\nIt is a bit difficult for me to evaluate this paper. \nThe technical results of Theorem 3.3 and 4.1 are not very surprising and their proofs also look more or less standard.\nThe result in Theorem 4.2 appear novel and interesting, but it is a bit difficult to position this in the existing literature of, kernel or DNN. I believe that the authors should make more efforts on that to make the message of this paper more clear.\n\nQUESTIONS:\n* Is Proposition 2.2 novel result? If yes, where is its proof?\n* it appears that the notations of expectation are not aligned in Assumption 1 and line 178.\n* The results of Theorem 3.3 and 4.1 are not very surprising and their proofs also look more or less standard\n* The result in Theorem 4.2 looks interesting, but essentially focuses on the Gaussian process with zero mean and random feature kernel of covariance function. Also, is this the FIRST time such results are established (e.g., in terms of real interpolation space of the RKHS)? How should we compare this result to other kernels/methods/models? It may be helpful to compare this behavior to the literature.\n\n---\nI thank the authors for their detailed reply and clarification. This helps me have a better grasp of this presented results, I will increase my score accordingly.\n\nI would like to ask the authors to carefully include the above discussions in a revised version of the paper, as also suggested by Reviewer 43yC.\n\nLIMITATIONS:\nI do not see any potential negative social impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to study the impact of standard random weight initialization - as opposed to mirror initialization - on NTK theory.  The key observation is that, for standard initialization, the operation of the network at initialization $f_0$ acts as a bias on the regression function $f^*$.  When analyzing the convergence of the excess risk in the kernel gradient flow model the convergence rate depends on the smoothness of the regression function, which is (effectively) $f_0+f^*$.  The paper demonstrates that, regardless of how smooth $f^*$ is, the smoothness of $f_0+f^*$ is bounded above by the smoothness of $f_0$ (theorem 4.2).  Non-zero initialization thus places upper and lower bounds on the generalization error predicted by KGF (theorem 4.3 and 4.4).  This is claimed to bring into focus the shortcomings of KGF in the NTK context for analyzing the convergence of neural networks even in the infinitely wide limit.\n\nSTRENGTHS:\nIt is obviously important to consider the role of standard random initialization on the training and convergence of neural networks.  The argument presented in the paper is clear and gives an interesting insight into the predictions of NTK and KGF for understanding generalization and convergence.  Mathematically the results are strong (as best as I can tell, having skimmed the supplementary).\n\nWEAKNESSES:\nPerhaps I am misreading the results, but it would seem to me that, far from demonstrating the downsides of NTK theory, you have in fact shown that NTK theory correctly predicts that standard initialization can degrade performance (in terms of the generalization error during training) compared to mirror initialization for the artificial data.  With regard the real data, all I can see is that you have shown that standard initialization will indeed decrease (effective) smoothness; however your predictions are asymptotic, so without a side-by-side comparison of standard and mirror initialization training (as per the artificial data) I don't see how this demonstrates anything at all.  It would certainly be interesting to see if the results for real data are the same as for artificial data as this would (presumably) indicate that mirror initialization is to be preferred where possible.\n\n(fwiw this is the core reason for the discrepancy between my evaluation of soundness and my recommendation.  Mathematically the paper seems sound, but the interpretation seems incorrect to me: far from showing the downsides of NTK theory, the results would appear to show its power.  If I'm wrong or the paper is suitably modified then I am happy to upgrade my recommendation).\n\nMinor point: please run this paper through a spell checker.\n\nQUESTIONS:\nMy main question is whether or not it would be feasible to run mirror-initialization vs standard-initialization experiments on the real-world dataset, or even a high-dimensional artificial dataset.  My guess would be that the results would replicate the Figure 1, which could have interesting practical implications.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the standard random mode of initialization of neural networks through the lens of the neural tangent kernel (NTK). This connection is made by showing that a randomly initialized neural network does indeed converge to the NTK uniformly during training thus allowing to analyze the generalization ability of a neural network with more relaxed assumptions on initialization. Furthermore, this paper finds that neural network performance under this regime comes at odds with the NTK theory which indicates that networks should perform poorly for high dimension inputs.\n\nSTRENGTHS:\nThe paper is sufficiently original in that it explores the more conventional initialization schemes used with neural nets and shows that, despite uniform convergence to the NTK, the kernel itself contradicts real world performance of the networks. This is a continuation of the theme in NTK theory where the infinite width regime fails to fully capture the finite's performance.\n\nThis is presented in a clear and thorough manner with rigorous proofs and straightforward, but convincing experiments. In regards to significance, again, this further builds on top of work that has \"poked holes\" in the practical usage of NTK theory and as such is a relatively significant work in that direction.\n\nWEAKNESSES:\nThere are a few typos/grammer mishaps that I caught in the paper:\n\nProposition 2.2 - \"...noise term $\\epsilon$ **satisfis**...\"\n\nLines 126-127 - \"...*let us reall the* concept...\"\n\nLine 233 - \"This is **an** generalization...\"\n\nLines 266-267 - \"...the network **on longer generalizes** well...\" (does not make sense)\n\nLine 314 - \"...datasets *from real world* and estimate the *smoothness of function*.\"\n\nQUESTIONS:\nLine 186 - Can you specify what space $C(\\mathcal{X}, \\mathbb{R})$ refers to?\n\nLines 193-194 - (Less of a question and more of a comment) I am assuming NNK is just your notation for the standard empirical NTK...\n\nLines 274-275 - Why is it necessary to remove the bias term for the derivation of the lower bound?\n\nLIMITATIONS:\nEverything is adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies various kernel theories of neural networks, both random feature and neural tangent kernels. The main approach is to use the decay rates of the target function and kernel eigenvalues to get generalization error rates. The paper's novel theoretical contribution is to show uniform convergence of the network's input-output function $f$ to that of a kernel regressor with NTK kernel. The setting is that the weights are scaled to be in the NTK regime and when both are trained under gradient flow. Previously known results are used to characterize the RKHS associated with the NTK and \"interpolations\" of this space (which is useful if the target function isn't actually in the RKHS). It is argued that, since the network initialization with random weights is not smooth, that the network overall will have a slow error rate. These theoretical arguments are backed up with experiments on synthetic data comparing the standard initialization, that was studied theoretically, with a \"mirrored\" one. Also, the smoothness of the MNIST, CIFAR-10, and Fashion-MNIST classification problems is estimated and found to be quite different from the basic bound based on input dimensionality.\n\nSTRENGTHS:\nThe work here brings together a bunch of connected kernel theory and neural networks results in a (relatively) readable treatement. The uniform convergence of the network's output to a Gaussian process is good to know. The overall goal of understanding the effect of initialization is certainly important, as is highlighting the weaknesses of kernel frameworks in understanding neural networks.\n\nWEAKNESSES:\n* In the introduction and experimental sections, the standard and mirror initializations are contrasted. From the introduction, I was expecting to see theoretical results that differentiated between the mirror and standard initializations. The authors could be clearer about what their contributions are and more clearly state what is known about the mirror initialization. \n* As someone who is fairly familiar with kernel theory of neural networks, I still found it hard to identify which theorems and other results presented by the authors here are actually novel and which are applications of known results to their setting. For instance, isn't the decay rate of $1/(d+3)$ classical and known from Sobolev theory? I would suggest that the authors clarify this. \n* The bulk of the paper is really just setting the stage for your results. I would have liked to see more explanation of how to interpret your results and discussion of how they fit into the greater understanding of these networks/kernels. I would also have liked more detail on the experiments within the main text.\n* The synthetic experiments (Fig 1) only cover less than an order-of-magnitude in $n$. They would be much more convincing if they covered more. However, I can imagine you'd run into issues with double-descent and other factors that your asymptotic theory could have trouble with.\n\nQUESTIONS:\n* Please clarify the definition of the interpolation space (Eq 6 and around it). Your notation for the space of functions is unclear to me. Some explanation should be given. It is unusual to see a function written as a series $\\sum_i a_i \\lambda_i^{s/2} e_i(x)$\n* Fix the formatting issue in line 104\n* Prop 2.2 \"satisfis\" typo\n* Do you think it would be more consistent to use the NTK superscript everywhere you treat it rather than f^NTK and K^NT ?\n* Line 267 \"network on longer\" typo\n* Does the mirror initialization lead to a significantly different error bound than yours, Thm 4.3?\n* Can you include the method of estimating the smoothness of the various classification problems (Table 1) in the main text? Looking at the appendix, I see it depends on an empirical kernel/Gram matrix, but under what kernel is not clear. NTK?\n* You report various least-square fits on log-log plots. You should be clear about how these were performed. Linear fits in log-space are known to have problems versus least-square fitting with power-laws. See the work by Clauset et al.\n* In your notation section, you use standard $O, \\Omega, o, \\omega$ symbols but do not define the frequent $\\asymp$ symbol. This should be explained.\n\nLIMITATIONS:\nThe work here is very focused on the NTK theory of a multi-layer perceptron/fully-connected network. Strong claims are made about the weakness of such theories. However, different architectures lead to different kernels with different spectral properties and generalization performance. For instance, convolutional architectures lead to convolutional kernels which are known to work better in image classification and other spatially structured settings. It is also known that initialization with weights that are not Gaussian white noise can lead to better performance (see the work on \"structured random features\" and \"rainbow networks\").\n\nI did not check the appendix proofs, but I read all of the main text results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}