Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

1 INTRODUCTION . In the past few years , one of the most seminal discoveries in the theory of neural networks is the neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) . The gradient flow on a normally initialized , fully connected neural network with a linear output layer in the infinite-width limit turns out to be equivalent to kernel regression with respect to the NTK ( This statement does not necessarily hold for a non-linear output layer , because the NTK is non-constant ( Liu et al. , 2020 ) ) . Through the NTK , theoretical tools from kernel methods were introduced to the study of deep overparametrized neural networks . Theoretical results were thereby established regarding the convergence ( Allen-Zhu et al. , 2019 ; Du et al. , 2019b ; a ; Zou et al. , 2020 ) , generalization ( Cao & Gu , 2019 ; Arora et al. , 2019b ) , and loss landscape ( Kuditipudi et al. , 2019 ) of overparametrized neural networks in the NTK regime . While NTK has proved to be a powerful theoretical tool , a recent work ( Geifman et al. , 2020 ) posed an important question whether the NTK is significantly different from our repertoire of standard kernels . Prior work provided empirical evidence that supports a negative answer . For example , Belkin et al . ( 2018 ) showed experimentally that the Laplace kernel and neural networks had similar performance in fitting random labels . In the task of speech enhancement , exponential power kernels Kγ , σexp ( x , y ) = e−‖x−y‖ γ/σ , which include the Laplace kernel as a special case , outperform deep neural networks with even shorter training time ( Hui et al. , 2019 ) . The experiments in ( Geifman et al. , 2020 ) also exhibited similar performance of the Laplace kernel and the NTK . The expressive power of a positive definite kernel can be characterized by its associated reproducing kernel Hilbert space ( RKHS ) ( Saitoh & Sawano , 2016 ) . The work ( Geifman et al. , 2020 ) considered the RKHS of the kernels restricted to the sphere Sd−1 , { x ∈ Rd | ‖x‖2 = 1 } and presented a partial answer to the question by showing the following subset inclusion relation HGauss ( Sd−1 ) ( HLap ( Sd−1 ) = HN1 ( Sd−1 ) ⊆ HNk ( Sd−1 ) , where the four spaces denote the RKHS associated with the Gaussian kernel , Laplace kernel , the NTK of two-layer and ( k+ 1 ) -layer ( k ≥ 1 ) fully connected neural networks , respectively . All four kernels are restricted to Sd−1 . However , the relation between HLap ( Sd−1 ) and HNk ( Sd−1 ) remains open in ( Geifman et al. , 2020 ) . We make a final conclusion on this problem and show that the RKHS of the Laplace kernel and the NTK with any number of layers have the same set of functions , when they are both restricted to Sd−1 . In other words , we prove the following theorem . Theorem 1 . Let HLap ( Sd−1 ) and HNk ( Sd−1 ) be the RKHS associated with the Laplace kernel KLap ( x , y ) = e −c‖x−y‖ ( c > 0 ) and the neural tangent kernel of a ( k + 1 ) -layer fully connected ReLU network . Both kernels are restricted to the sphere Sd−1 . Then the two spaces include the same set of functions : HLap ( Sd−1 ) = HNk ( Sd−1 ) , ∀k ≥ 1 . Our second result is that the exponential power kernel with a smaller power ( making the kernel less smooth ) leads to a larger RKHS , both when it is restricted to the sphere Sd−1 and when it is defined on the entire Rd . Theorem 2 . LetHKγ , σexp ( S d−1 ) andHKγ , σexp ( R d ) be the RKHS associated with the exponential power kernel Kγ , σexp ( x , y ) = exp ( −‖x−y‖ γ σ ) ( γ , σ > 0 ) when it is restricted to the unit sphere Sd−1 and defined on the entire Rd , respectively . Then we have the following RKHS inclusions : ( 1 ) If 0 < γ1 < γ2 < 2 , HKγ2 , σ2exp ( S d−1 ) ⊆ HKγ1 , σ1exp ( S d−1 ) . ( 2 ) If 0 < γ1 < γ2 < 2 are rational , HKγ2 , σ2exp ( R d ) ⊆ HKγ1 , σ1exp ( R d ) . If it is restricted to the unit sphere , the RKHS of the exponential power kernel with γ < 1 is even larger than that of NTK . This result partially explains the observation in ( Hui et al. , 2019 ) that the best performance is attained by a highly non-smooth exponential power kernel with γ < 1 . Geifman et al . ( 2020 ) applied the exponential power kernel and the NTK to classification and regression tasks on the UCI dataset and other large scale datasets . Their experiment results also showed that the exponential power kernel slightly outperforms the NTK . 1.1 FURTHER RELATED WORK . Minh et al . ( 2006 ) showed the complete spectrum of the polynomial and Gaussian kernels on Sd−1 . They also gave a recursive relation for the eigenvalues of the polynomial kernel on the hypercube { −1 , 1 } d. Prior to the NTK ( Jacot et al. , 2018 ) , Cho & Saul ( 2009 ) presented a pioneering study on kernel methods for neural networks . Bach ( 2017 ) studied the eigenvalues of positively homogeneous activation functions of the form σα ( u ) = max { u , 0 } α ( e.g. , the ReLU activation when α = 1 ) in their Mercer decomposition with Gegenbauer polynomials . Using the results in ( Bach , 2017 ) , Bietti & Mairal ( 2019 ) analyzed the two-layer NTK and its RKHS in order to investigate the inductive bias in the NTK regime . They studied the Mercer decomposition of two-layer NTK with ReLU activation on Sd−1 and characterized the corresponding RKHS by showing the asymptotic decay rate of the eigenvalues in the Mercer decomposition with Gegenbauer polynomials . In their derivation of a more concise expression of the ReLU NTK , they used the calculation of ( Cho & Saul , 2009 ) on arc-cosine kernels of degree 0 and 1 . Cao et al . ( 2019 ) improved the eigenvalue bound for the k-th eigenvalue derived in ( Bietti & Mairal , 2019 ) when d k. Geifman et al . ( 2020 ) used the results in ( Bietti & Mairal , 2019 ) and considered the two-layer ReLU NTK with bias β initialized with zero , rather than initialized with a normal distribution ( Jacot et al. , 2018 ) . However , neither ( Bietti & Mairal , 2019 ) nor ( Geifman et al. , 2020 ) went beyond two layers when they tried to characterize the RKHS of the ReLU NTK . This line of work ( Bach , 2017 ; Bietti & Mairal , 2019 ; Geifman et al. , 2020 ) is closely related to the Mercer decomposition with spherical harmonics . Interested readers are referred to ( Atkinson & Han , 2012 ) for spherical harmonics on the unit sphere . The concurrent work ( Bietti & Bach , 2021 ) analyzed the eigenvalues of the ReLU NTK . Arora et al . ( 2019a ) presented a dynamic programming algorithm that computes convolutional NTK with ReLU activation . Yang & Salman ( 2019 ) analyzed the spectra of the conjugate kernel ( CK ) and NTK on the boolean cube . Fan & Wang ( 2020 ) studied the spectrum of the gram matrix of training samples under the CK and NTK and showed that their eigenvalue distributions converge to a deterministic limit . The limit depends on the eigenvalue distribution of the training samples . 2 PRELIMINARIES . Let C denote the set of all complex numbers and write i , √ −1 . For z ∈ C , write < z , =z , arg z ∈ ( −π , π ] for its real part , imaginary part , and argument , respectively . Let H+ , { z ∈ C | =z > 0 } denote the upper half-plane and H− , { z ∈ C | =z < 0 } denote the lower half-plane . Write Bz ( r ) for the open ball { w ∈ C | |z − w| < r } and B̄z ( r ) for the closed ball { w ∈ C | |z − w| ≤ r } . Suppose that f ( z ) has a power series representation f ( z ) = ∑ n≥0 anz n around 0 . Denote [ zn ] f ( z ) , an to be the coefficient of the n-th order term . For two sequences { an } and { bn } , write an ∼ bn if limn→∞ anbn = 1 . Similarly , for two functions f ( z ) and g ( z ) , write f ( z ) ∼ g ( z ) as z → z0 if limz→z0 f ( z ) g ( z ) = 1 . We also use big-O and little-o notation to characterize asymptotics . Write L { f ( t ) } ( s ) , ∫∞ 0 f ( t ) e−stdt for the Laplace transform of a function f ( t ) . The inverse Laplace transform of F ( s ) is denoted by L −1 { F ( s ) } ( t ) . 2.1 POSITIVE DEFINITE KERNELS . For any positive definite kernel functionK ( x , y ) defined for x , y ∈ E , denoteHK ( E ) its associated reproducing kernel Hilbert space ( RKHS ) . For any two positive definite kernel functions K1 and K2 , we write K1 4 K2 if K2−K1 is a positive definite kernel . For a complete review of results on kernels and RKHS , please see ( Saitoh & Sawano , 2016 ) . We will study positive definite zonal kernels on the sphere Sd−1 = { x ∈ Rd | ‖x‖ = 1 } . For a zonal kernel K ( x , y ) , there exists a real function K̃ : [ −1 , 1 ] → R such that K ( x , y ) = K̃ ( u ) , where u = x > y . We abuse the notation and use K ( u ) to denote K̃ ( u ) , i.e. , K ( u ) here is real function on [ −1 , 1 ] . In the sequel , we introduce two instances of the positive definite kernel that this paper will investigate . Laplace Kernel The Laplace kernel KLap ( x , y ) = e−c‖x−y‖ with c > 0 restricted to the sphere Sd−1 is given by KLap ( x , y ) = e−c √ 2 ( 1−x > y ) = e−c̃ √ 1−u , KLap ( u ) , where by our convention u = x > y and c̃ , √ 2c > 0 . We denote its associated RKHS byHLap . Exponential Power Kernel The exponential power kernel ( Hui et al. , 2019 ) with γ > 0 and σ > 0 is given by Kγ , σexp ( x , y ) = exp ( −‖x−y‖ γ σ ) . If x and y are restricted to the sphere Sd−1 , we have Kγ , σexp ( x , y ) = exp ( − ( 2 ( 1−x > y ) ) γ/2 σ ) . Neural Tangent Kernel Given the input x ∈ Rd ( we define d0 , d ) and parameter θ , this paper considers the following network model with ( k + 1 ) layers fθ ( x ) = w > √ 2 dk σ ( Wk √ 2 dk−1 σ ( . . . √ 2 d2 σ ( W2 √ 2 d1 σ ( W1x+ βb1 ) + βb2 ) . . . ) + βbk ) + βbk+1 , ( 1 ) where the parameter θ encodes Wl ∈ Rdl×dl−1 , bl ∈ Rdl ( l = 1 , . . . , k ) , w ∈ Rdk , and bk+1 ∈ R. The weight matrices W1 , . . . , Wk , w are initialized with N ( 0 , I ) and the biases b1 , . . . , bk+1 are initialized with zero , where N ( 0 , I ) is the multivariate standard normal distribution . The activation function is chosen to be the ReLU function σ ( x ) , max { x , 0 } . Geifman et al . ( 2020 ) and Bietti & Mairal ( 2019 ) presented the following recursive relations of the NTK Nk ( x , y ) of the above ReLU network ( 1 ) : Σk ( x , y ) = √ Σk−1 ( x , x ) Σk−1 ( y , y ) κ1 ( Σk−1 ( x , y ) √ Σk−1 ( x , x ) Σk−1 ( y , y ) ) Nk ( x , y ) = Σk ( x , y ) +Nk−1 ( x , y ) κ0 ( Σk−1 ( x , y ) √ Σk−1 ( x , x ) Σk−1 ( y , y ) ) + β2 , ( 2 ) where κ0 and κ1 are the arc-cosine kernels of degree 0 and 1 ( Cho & Saul , 2009 ) given by κ0 ( u ) = 1 π ( π − arccos ( u ) ) , κ1 ( u ) = 1 π ( u · ( π − arccos ( u ) ) + √ 1− u2 ) . The initial conditions are N0 ( x , y ) = u+ β 2 , Σ0 ( x , y ) = u , ( 3 ) where u = x > y by our convention . The NTKs defined in ( Bietti & Mairal , 2019 ) and ( Geifman et al. , 2020 ) are slightly different . There is no bias term β2 in ( Bietti & Mairal , 2019 ) , while the bias term appears in ( Geifman et al. , 2020 ) . We adopt the more general setup with the bias term . Lemma 3 ( Proof in Appendix A.1 ) . Σk ( x , x ) = 1 for any x ∈ Sd−1 and k ≥ 0 . Lemma 3 simplifies ( 2 ) and gives Σk ( u ) = κ ( k ) 1 ( u ) , Nk ( u ) = κ ( k ) 1 ( u ) +Nk−1 ( u ) κ0 ( κ ( k−1 ) 1 ( u ) ) + β 2 , ( 4 ) where κ ( k ) 1 ( u ) , κ1 ( κ1 ( · · ·κ1 ( κ1︸ ︷︷ ︸ k ( u ) ) · · · ) ) is the k-th iterate of κ1 ( u ) . For example , κ ( 0 ) 1 ( u ) = u , κ ( 1 ) 1 ( u ) = κ1 ( u ) and κ ( 2 ) 1 ( u ) = κ1 ( κ1 ( u ) ) . We present a detailed derivation of ( 4 ) in Appendix A.2 .

Summary: This paper uses singularity analysis developed in the context of analytic combinatorics to study the relationship between the reproducing kernel Hilbert spaces of the NTK in a deep fully connected ReLU network, the Laplace kernel, and exponential power kernels. The main results are when these kernels are restricted to the unit sphere. In particular, the authors show that, as vector spaces, the RKHS on the unit sphere of the NTK for a ReLU network of any fixed depth are the same and in fact coincides with that of the Laplace kernel. The authors also compare the RKHS for exponential power kernels, demonstrating that larger powers lead to smaller Hilbert spaces.