Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

1 INTRODUCTION . In the past few years , one of the most seminal discoveries in the theory of neural networks is the neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) . The gradient flow on a normally initialized , fully connected neural network with a linear output layer in the infinite-width limit turns out to be equivalent to kernel regression with respect to the NTK ( This statement does not necessarily hold for a non-linear output layer , because the NTK is non-constant ( Liu et al. , 2020 ) ) . Through the NTK , theoretical tools from kernel methods were introduced to the study of deep overparametrized neural networks . Theoretical results were thereby established regarding the convergence ( Allen-Zhu et al. , 2019 ; Du et al. , 2019b ; a ; Zou et al. , 2020 ) , generalization ( Cao & Gu , 2019 ; Arora et al. , 2019b ) , and loss landscape ( Kuditipudi et al. , 2019 ) of overparametrized neural networks in the NTK regime . While NTK has proved to be a powerful theoretical tool , a recent work ( Geifman et al. , 2020 ) posed an important question whether the NTK is significantly different from our repertoire of standard kernels . Prior work provided empirical evidence that supports a negative answer . For example , Belkin et al . ( 2018 ) showed experimentally that the Laplace kernel and neural networks had similar performance in fitting random labels . In the task of speech enhancement , exponential power kernels Kγ , σexp ( x , y ) = e−‖x−y‖ γ/σ , which include the Laplace kernel as a special case , outperform deep neural networks with even shorter training time ( Hui et al. , 2019 ) . The experiments in ( Geifman et al. , 2020 ) also exhibited similar performance of the Laplace kernel and the NTK . The expressive power of a positive definite kernel can be characterized by its associated reproducing kernel Hilbert space ( RKHS ) ( Saitoh & Sawano , 2016 ) . The work ( Geifman et al. , 2020 ) considered the RKHS of the kernels restricted to the sphere Sd−1 , { x ∈ Rd | ‖x‖2 = 1 } and presented a partial answer to the question by showing the following subset inclusion relation HGauss ( Sd−1 ) ( HLap ( Sd−1 ) = HN1 ( Sd−1 ) ⊆ HNk ( Sd−1 ) , where the four spaces denote the RKHS associated with the Gaussian kernel , Laplace kernel , the NTK of two-layer and ( k+ 1 ) -layer ( k ≥ 1 ) fully connected neural networks , respectively . All four kernels are restricted to Sd−1 . However , the relation between HLap ( Sd−1 ) and HNk ( Sd−1 ) remains open in ( Geifman et al. , 2020 ) . We make a final conclusion on this problem and show that the RKHS of the Laplace kernel and the NTK with any number of layers have the same set of functions , when they are both restricted to Sd−1 . In other words , we prove the following theorem . Theorem 1 . Let HLap ( Sd−1 ) and HNk ( Sd−1 ) be the RKHS associated with the Laplace kernel KLap ( x , y ) = e −c‖x−y‖ ( c > 0 ) and the neural tangent kernel of a ( k + 1 ) -layer fully connected ReLU network . Both kernels are restricted to the sphere Sd−1 . Then the two spaces include the same set of functions : HLap ( Sd−1 ) = HNk ( Sd−1 ) , ∀k ≥ 1 . Our second result is that the exponential power kernel with a smaller power ( making the kernel less smooth ) leads to a larger RKHS , both when it is restricted to the sphere Sd−1 and when it is defined on the entire Rd . Theorem 2 . LetHKγ , σexp ( S d−1 ) andHKγ , σexp ( R d ) be the RKHS associated with the exponential power kernel Kγ , σexp ( x , y ) = exp ( −‖x−y‖ γ σ ) ( γ , σ > 0 ) when it is restricted to the unit sphere Sd−1 and defined on the entire Rd , respectively . Then we have the following RKHS inclusions : ( 1 ) If 0 < γ1 < γ2 < 2 , HKγ2 , σ2exp ( S d−1 ) ⊆ HKγ1 , σ1exp ( S d−1 ) . ( 2 ) If 0 < γ1 < γ2 < 2 are rational , HKγ2 , σ2exp ( R d ) ⊆ HKγ1 , σ1exp ( R d ) . If it is restricted to the unit sphere , the RKHS of the exponential power kernel with γ < 1 is even larger than that of NTK . This result partially explains the observation in ( Hui et al. , 2019 ) that the best performance is attained by a highly non-smooth exponential power kernel with γ < 1 . Geifman et al . ( 2020 ) applied the exponential power kernel and the NTK to classification and regression tasks on the UCI dataset and other large scale datasets . Their experiment results also showed that the exponential power kernel slightly outperforms the NTK . 1.1 FURTHER RELATED WORK . Minh et al . ( 2006 ) showed the complete spectrum of the polynomial and Gaussian kernels on Sd−1 . They also gave a recursive relation for the eigenvalues of the polynomial kernel on the hypercube { −1 , 1 } d. Prior to the NTK ( Jacot et al. , 2018 ) , Cho & Saul ( 2009 ) presented a pioneering study on kernel methods for neural networks . Bach ( 2017 ) studied the eigenvalues of positively homogeneous activation functions of the form σα ( u ) = max { u , 0 } α ( e.g. , the ReLU activation when α = 1 ) in their Mercer decomposition with Gegenbauer polynomials . Using the results in ( Bach , 2017 ) , Bietti & Mairal ( 2019 ) analyzed the two-layer NTK and its RKHS in order to investigate the inductive bias in the NTK regime . They studied the Mercer decomposition of two-layer NTK with ReLU activation on Sd−1 and characterized the corresponding RKHS by showing the asymptotic decay rate of the eigenvalues in the Mercer decomposition with Gegenbauer polynomials . In their derivation of a more concise expression of the ReLU NTK , they used the calculation of ( Cho & Saul , 2009 ) on arc-cosine kernels of degree 0 and 1 . Cao et al . ( 2019 ) improved the eigenvalue bound for the k-th eigenvalue derived in ( Bietti & Mairal , 2019 ) when d k. Geifman et al . ( 2020 ) used the results in ( Bietti & Mairal , 2019 ) and considered the two-layer ReLU NTK with bias β initialized with zero , rather than initialized with a normal distribution ( Jacot et al. , 2018 ) . However , neither ( Bietti & Mairal , 2019 ) nor ( Geifman et al. , 2020 ) went beyond two layers when they tried to characterize the RKHS of the ReLU NTK . This line of work ( Bach , 2017 ; Bietti & Mairal , 2019 ; Geifman et al. , 2020 ) is closely related to the Mercer decomposition with spherical harmonics . Interested readers are referred to ( Atkinson & Han , 2012 ) for spherical harmonics on the unit sphere . The concurrent work ( Bietti & Bach , 2021 ) analyzed the eigenvalues of the ReLU NTK . Arora et al . ( 2019a ) presented a dynamic programming algorithm that computes convolutional NTK with ReLU activation . Yang & Salman ( 2019 ) analyzed the spectra of the conjugate kernel ( CK ) and NTK on the boolean cube . Fan & Wang ( 2020 ) studied the spectrum of the gram matrix of training samples under the CK and NTK and showed that their eigenvalue distributions converge to a deterministic limit . The limit depends on the eigenvalue distribution of the training samples . 2 PRELIMINARIES . Let C denote the set of all complex numbers and write i , √ −1 . For z ∈ C , write < z , =z , arg z ∈ ( −π , π ] for its real part , imaginary part , and argument , respectively . Let H+ , { z ∈ C | =z > 0 } denote the upper half-plane and H− , { z ∈ C | =z < 0 } denote the lower half-plane . Write Bz ( r ) for the open ball { w ∈ C | |z − w| < r } and B̄z ( r ) for the closed ball { w ∈ C | |z − w| ≤ r } . Suppose that f ( z ) has a power series representation f ( z ) = ∑ n≥0 anz n around 0 . Denote [ zn ] f ( z ) , an to be the coefficient of the n-th order term . For two sequences { an } and { bn } , write an ∼ bn if limn→∞ anbn = 1 . Similarly , for two functions f ( z ) and g ( z ) , write f ( z ) ∼ g ( z ) as z → z0 if limz→z0 f ( z ) g ( z ) = 1 . We also use big-O and little-o notation to characterize asymptotics . Write L { f ( t ) } ( s ) , ∫∞ 0 f ( t ) e−stdt for the Laplace transform of a function f ( t ) . The inverse Laplace transform of F ( s ) is denoted by L −1 { F ( s ) } ( t ) . 2.1 POSITIVE DEFINITE KERNELS . For any positive definite kernel functionK ( x , y ) defined for x , y ∈ E , denoteHK ( E ) its associated reproducing kernel Hilbert space ( RKHS ) . For any two positive definite kernel functions K1 and K2 , we write K1 4 K2 if K2−K1 is a positive definite kernel . For a complete review of results on kernels and RKHS , please see ( Saitoh & Sawano , 2016 ) . We will study positive definite zonal kernels on the sphere Sd−1 = { x ∈ Rd | ‖x‖ = 1 } . For a zonal kernel K ( x , y ) , there exists a real function K̃ : [ −1 , 1 ] → R such that K ( x , y ) = K̃ ( u ) , where u = x > y . We abuse the notation and use K ( u ) to denote K̃ ( u ) , i.e. , K ( u ) here is real function on [ −1 , 1 ] . In the sequel , we introduce two instances of the positive definite kernel that this paper will investigate . Laplace Kernel The Laplace kernel KLap ( x , y ) = e−c‖x−y‖ with c > 0 restricted to the sphere Sd−1 is given by KLap ( x , y ) = e−c √ 2 ( 1−x > y ) = e−c̃ √ 1−u , KLap ( u ) , where by our convention u = x > y and c̃ , √ 2c > 0 . We denote its associated RKHS byHLap . Exponential Power Kernel The exponential power kernel ( Hui et al. , 2019 ) with γ > 0 and σ > 0 is given by Kγ , σexp ( x , y ) = exp ( −‖x−y‖ γ σ ) . If x and y are restricted to the sphere Sd−1 , we have Kγ , σexp ( x , y ) = exp ( − ( 2 ( 1−x > y ) ) γ/2 σ ) . Neural Tangent Kernel Given the input x ∈ Rd ( we define d0 , d ) and parameter θ , this paper considers the following network model with ( k + 1 ) layers fθ ( x ) = w > √ 2 dk σ ( Wk √ 2 dk−1 σ ( . . . √ 2 d2 σ ( W2 √ 2 d1 σ ( W1x+ βb1 ) + βb2 ) . . . ) + βbk ) + βbk+1 , ( 1 ) where the parameter θ encodes Wl ∈ Rdl×dl−1 , bl ∈ Rdl ( l = 1 , . . . , k ) , w ∈ Rdk , and bk+1 ∈ R. The weight matrices W1 , . . . , Wk , w are initialized with N ( 0 , I ) and the biases b1 , . . . , bk+1 are initialized with zero , where N ( 0 , I ) is the multivariate standard normal distribution . The activation function is chosen to be the ReLU function σ ( x ) , max { x , 0 } . Geifman et al . ( 2020 ) and Bietti & Mairal ( 2019 ) presented the following recursive relations of the NTK Nk ( x , y ) of the above ReLU network ( 1 ) : Σk ( x , y ) = √ Σk−1 ( x , x ) Σk−1 ( y , y ) κ1 ( Σk−1 ( x , y ) √ Σk−1 ( x , x ) Σk−1 ( y , y ) ) Nk ( x , y ) = Σk ( x , y ) +Nk−1 ( x , y ) κ0 ( Σk−1 ( x , y ) √ Σk−1 ( x , x ) Σk−1 ( y , y ) ) + β2 , ( 2 ) where κ0 and κ1 are the arc-cosine kernels of degree 0 and 1 ( Cho & Saul , 2009 ) given by κ0 ( u ) = 1 π ( π − arccos ( u ) ) , κ1 ( u ) = 1 π ( u · ( π − arccos ( u ) ) + √ 1− u2 ) . The initial conditions are N0 ( x , y ) = u+ β 2 , Σ0 ( x , y ) = u , ( 3 ) where u = x > y by our convention . The NTKs defined in ( Bietti & Mairal , 2019 ) and ( Geifman et al. , 2020 ) are slightly different . There is no bias term β2 in ( Bietti & Mairal , 2019 ) , while the bias term appears in ( Geifman et al. , 2020 ) . We adopt the more general setup with the bias term . Lemma 3 ( Proof in Appendix A.1 ) . Σk ( x , x ) = 1 for any x ∈ Sd−1 and k ≥ 0 . Lemma 3 simplifies ( 2 ) and gives Σk ( u ) = κ ( k ) 1 ( u ) , Nk ( u ) = κ ( k ) 1 ( u ) +Nk−1 ( u ) κ0 ( κ ( k−1 ) 1 ( u ) ) + β 2 , ( 4 ) where κ ( k ) 1 ( u ) , κ1 ( κ1 ( · · ·κ1 ( κ1︸ ︷︷ ︸ k ( u ) ) · · · ) ) is the k-th iterate of κ1 ( u ) . For example , κ ( 0 ) 1 ( u ) = u , κ ( 1 ) 1 ( u ) = κ1 ( u ) and κ ( 2 ) 1 ( u ) = κ1 ( κ1 ( u ) ) . We present a detailed derivation of ( 4 ) in Appendix A.2 .

Summary: This paper proves that the reproducing kernel Hilbert spaces of a deep neural tangent kernel and the Laplace kernel have the same set of functions when they restricted to the sphere $S^{d-1}$, which improves the results established in Geifman et al., 2020. Moreover, the paper proves that more non-smooth of the exponential power kernel leads to a larger RKHS with restriction on the sphere $S^{d-1}$ and the entire $R^d$. Furthermore, the authors conduct numerical experiments to verify the asymptotics of the Maclaurin coefficients of the Laplace kernel and NTKs kernel. In summary, the paper is well-written and organized logically. The proof of theoretical results of this paper seems to be correct and reasonable, resulting from the full details of the proof provided in the appendices.