Deep Equals Shallow for ReLU Networks in Kernel Regimes

1 Introduction . The question of which functions can be well approximated by neural networks is crucial for understanding when these models are successful , and has always been at the heart of the theoretical study of neural networks ( e.g. , Hornik et al. , 1989 ; Pinkus , 1999 ) . While early works have mostly focused on shallow networks with only two layers , more recent works have shown benefits of deep networks for approximating certain classes of functions ( Eldan & Shamir , 2016 ; Mhaskar & Poggio , 2016 ; Telgarsky , 2016 ; Daniely , 2017 ; Yarotsky , 2017 ; Schmidt-Hieber et al. , 2020 ) . Unfortunately , many of these approaches rely on constructions that are not currently known to be learnable using efficient algorithms . A separate line of work has considered over-parameterized networks with random neurons ( Neal , 1996 ) , which also display universal approximation properties while additionally providing efficient algorithms based on kernel methods or their approximations such as random features ( Rahimi & Recht , 2007 ; Bach , 2017b ) . Many recent results on gradient-based optimization of certain over-parameterized networks have been shown to be equivalent to kernel methods with an architecture-specific kernel called the neural tangent kernel ( NTK ) and thus also fall in this category ( e.g. , Jacot et al. , 2018 ; Li & Liang , 2018 ; Allen-Zhu et al. , 2019b ; Du et al. , 2019a ; b ; Zou et al. , 2019 ) . This regime has been coined lazy ( Chizat et al. , 2019 ) , as it does not capture the common phenomenon where weights move significantly away from random initialization and thus may not provide a satisfying model for learning adaptive representations , in contrast to other settings such as the mean field or active regime , which captures complex training dynamics where weights may move in a non-trivial manner and adapt to the data ( e.g. , Chizat & Bach , 2018 ; Mei et al. , 2018 ) . Nevertheless , ∗Work done while at Inria . †Center for Data Science , New York University . New York , USA . ‡Inria - Département d ’ Informatique de l ’ École Normale Supérieure . PSL Research University . Paris , France . one benefit compared to the mean field regime is that the kernel approach easily extends to deep architectures , leading to compositional kernels similar to the ones of Cho & Saul ( 2009 ) ; Daniely et al . ( 2016 ) . Our goal in this paper is to study the role of depth in determining approximation properties for such kernels , with a focus on fully-connected deep ReLU networks . Our approximation results rely on the study of eigenvalue decays of integral operators associated to the obtained dot-product kernels on the sphere , which are diagonalized in the basis of spherical harmonics . This provides a characterization of the functions in the corresponding reproducing kernel Hilbert space ( RKHS ) in terms of their smoothness , and leads to convergence rates for non-parametric regression when the data are uniformly distributed on the sphere . We show that for ReLU networks , the eigenvalue decays for the corresponding deep kernels remain the same regardless of the depth of the network . Our key result is that the decay for a certain class of kernels is characterized by a property related to differentiability of the kernel function around the point where the two inputs are aligned . In particular , the property is preserved when adding layers with ReLU activations , showing that depth plays essentially no role for such networks in kernel regimes . This highlights the limitations of the kernel regime for understanding the power of depth in fully-connected networks , and calls for new models of deep networks beyond kernels ( see , e.g. , Allen-Zhu & Li , 2020 ; Chen et al. , 2020 , for recent works in this direction ) . We also provide applications of our result to other kernels and architectures , and illustrate our results with numerical experiments on synthetic and real datasets . Related work . Kernels for deep learning were originally derived by Neal ( 1996 ) for shallow networks , and later for deep networks ( Cho & Saul , 2009 ; Daniely et al. , 2016 ; Lee et al. , 2018 ; Matthews et al. , 2018 ) . Smola et al . ( 2001 ) ; Minh et al . ( 2006 ) study regularization properties of dot-product kernels on the sphere using spherical harmonics , and Bach ( 2017a ) derives eigenvalue decays for such dot-product kernels arising from shallow networks with positively homogeneous activations including the ReLU . Extensions to shallow NTK or Laplace kernels are studied by Basri et al . ( 2019 ) ; Bietti & Mairal ( 2019b ) ; Geifman et al . ( 2020 ) . The observation that depth does not change the decay of the NTK was previously made by Basri et al . ( 2020 ) empirically , and Geifman et al . ( 2020 ) provide a lower bound on the eigenvalues for deep networks ; our work makes this observation rigorous by providing tight asymptotic decays . Spectral properties of wide neural networks were also considered in ( Cao et al. , 2019 ; Fan & Wang , 2020 ; Ghorbani et al. , 2019 ; Xie et al. , 2017 ; Yang & Salman , 2019 ) . Azevedo & Menegatto ( 2014 ) ; Scetbon & Harchaoui ( 2020 ) also study eigenvalue decays for dot-product kernels but focus on kernels with geometric decays , while our main focus is on polynomial decays . Additional works on over-parameterized or infinite-width networks in lazy regimes include ( Allen-Zhu et al. , 2019a ; b ; Arora et al. , 2019a ; b ; Brand et al. , 2020 ; Lee et al. , 2020 ; Song & Yang , 2019 ) . Concurrently to our work , Chen & Xu ( 2021 ) also studied the RKHS of the NTK for deep ReLU networks , showing that it is the same as for the Laplace kernel on the sphere . They achieve this by studying asymptotic decays of Taylor coefficients of the kernel function at zero using complex-analytic extensions of the kernel functions , and leveraging this to obtain both inclusions between the two RKHSs . In contrast , we obtain precise descriptions of the RKHS and regularization properties in the basis of spherical harmonics for various dot-product kernels through spectral decompositions of integral operators , using ( real ) asymptotic expansions of the kernel function around endpoints . The equality between the RKHS of the deep NTK and Laplace kernel then easily follows from our results by the fact that the two kernels have the same spectral decay . 2 Review of Approximation with Dot-Product Kernels . In this section , we provide a brief review of the kernels that arise from neural networks and their approximation properties . 2.1 Kernels for wide neural networks . Wide neural networks with random weights or weights close to random initialization naturally lead to certain dot-product kernels that depend on the architecture and activation function , which we now present , with a focus on fully-connected architectures . Random feature kernels . We first consider a two-layer ( shallow ) network of the form f ( x ) = 1√ m ∑m j=1 vjσ ( w > j x ) , for some activation function σ . When wj ∼ N ( 0 , I ) ∈ Rd are fixed and only vj ∈ R are trained with ` 2 regularization , this corresponds to using a random feature approximation Rahimi & Recht ( 2007 ) of the kernel k ( x , x′ ) = Ew∼N ( 0 , I ) [ σ ( w > x ) σ ( w > x′ ) ] . ( 1 ) If x , x′ are on the sphere , then by spherical symmetry of the Gaussian distribution , one may show that k is invariant to unitary transformations and takes the form k ( x , x′ ) = κ ( x > x′ ) for a certain function κ . More precisely , if σ ( u ) = ∑ i≥0 aihi ( u ) is the decomposition of σ in the basis of Hermite polynomials hi , which are orthogonal w.r.t . the Gaussian measure , then we have ( Daniely et al. , 2016 ) : κ ( u ) = ∑ i≥0 a2iu i . ( 2 ) Conversely , given a kernel function of the form above with κ ( u ) = ∑ i≥0 biu i with bi ≥ 0 , one may construct corresponding activations using Hermite polynomials by taking σ ( u ) = ∑ i aihi ( u ) , ai ∈ { ± √ bi } . ( 3 ) In the case where σ is s-positively homogeneous , such as the ReLU σ ( u ) = max ( u , 0 ) ( with s = 1 ) , or more generally σs ( u ) = max ( u , 0 ) s , then the kernel ( 1 ) takes the form k ( x , x′ ) = ‖x‖s‖x′‖sκ ( x > x′‖x‖‖x′‖ ) for any x , x′ . This leads to RKHS functions of the form f ( x ) = ‖x‖sg ( x‖x‖ ) , with g in the RKHS of the kernel restricted to the sphere ( Bietti & Mairal , 2019b , Prop . 8 ) . In particular , for the step and ReLU activations σ0 and σ1 , the functions κ are given by the following arc-cosine kernels ( Cho & Saul , 2009 ) :1 κ0 ( u ) = 1 π ( π − arccos ( u ) ) , κ1 ( u ) = 1 π ( u · ( π − arccos ( u ) ) + √ 1− u2 ) . ( 4 ) Note that given a kernel function κ , the corresponding activations ( 3 ) will generally not be homogeneous , thus the inputs to a random network with such activations need to lie on the sphere ( or be appropriately normalized ) in order to yield the kernel κ . Extension to deep networks . When considering a deep network with more than two layers and fixed random weights before the last layer , the connection to random features is less direct since the features are correlated through intermediate layers . Nevertheless , when the hidden layers are wide enough , one still approaches a kernel obtained by letting the widths go to infinity ( see , e.g. , Daniely et al. , 2016 ; Lee et al. , 2018 ; Matthews et al. , 2018 ) , which takes a similar form to the multi-layer kernels of Cho & Saul ( 2009 ) : kL ( x , x′ ) = κL ( x > x′ ) : = κ ◦ · · · ◦ κ︸ ︷︷ ︸ L−1 times ( x > x′ ) , for x , x′ on the sphere , where κ is obtained as described above for a given activation σ , and L is the number of layers . We still refer to this kernel as the random features ( RF ) kernel in this paper , noting that it is sometimes known as the “ conjugate kernel ” or NNGP kernel ( for neural network Gaussian process ) . It is usually good to normalize κ such that κ ( 1 ) = 1 , so that we also have κL ( 1 ) = 1 , avoiding exploding or vanishing behavior for deep networks . In practice , this corresponds to using an activation-dependent scaling in the random weight initialization , which is commonly used by practitioners ( He et al. , 2015 ) . 1Here we assume a scaling √ 2/m instead of √ 1/m in the definition of f , which yields κ ( 1 ) = 1 , a useful normalization for deep networks , as explained below . Neural tangent kernels . When intermediate layers are trained along with the last layer using gradient methods , the resulting problem is non-convex and the statistical properties of such approaches are not well understood in general , particularly for deep networks . However , in a specific over-parameterized regime , it may be shown that gradient descent can reach a global minimum while keeping weights very close to random initialization . More precisely , for a network f ( x ; θ ) parameterized by θ with large width m , the model remains close to its linearization around random initialization θ0 throughout training , that is , f ( x ; θ ) ≈ f ( x ; θ0 ) + 〈θ− θ0 , ∇θf ( x ; θ0 ) 〉 . This is also known as the lazy training regime ( Chizat et al. , 2019 ) . Learning is then equivalent to a kernel method with another architecture-specific kernel known as the neural tangent kernel ( NTK , Jacot et al. , 2018 ) , given by kNTK ( x , x′ ) = lim m→∞ 〈∇f ( x ; θ0 ) , ∇f ( x′ ; θ0 ) 〉 . ( 5 ) For a simple two-layer network with activation σ , it is then given by kNTK ( x , x′ ) = ( x > x′ ) Ew [ σ′ ( w > x ) σ′ ( w > x′ ) ] + Ew [ σ ( w > x ) σ ( w > x′ ) ] . ( 6 ) For a ReLU network with L layers with inputs on the sphere , taking appropriate limits on the widths , one can show ( Jacot et al. , 2018 ) : kNTK ( x , x′ ) = κLNTK ( x > x′ ) , with κ1NTK ( u ) = κ1 ( u ) = u and for ` = 2 , . . . , L , κ ` ( u ) = κ1 ( κ ` −1 ( u ) ) κ ` NTK ( u ) = κ ` −1NTK ( u ) κ0 ( κ ` −1 ( u ) ) + κ ` ( u ) , ( 7 ) where κ0 and κ1 are given in ( 4 ) .

Summary: This paper analyzed the expressive power of kernels by studying the reproducing kernel Hilbert space (RKHS) associated with the kernels. Specifically, the authors analyzed the eigenvalue decays in terms of the power series expansions of the kernel function around some points, which is related to the RKHS of the kernel. This analysis can be used to recover some previous results. Besides, using this analysis, the authors have shown several interesting results, including that NTK (which corresponds to fully-connected ReLU networks with infinite width, small learning rate, and proper initialization) with any depth has the same RKHS. The main result also has other corollaries about other kinds of kernels, e.g., Laplace kernel and infinitely differentiable kernels. Experiments were done to validate the theoretical results on synthetic datasets and MNIST/Fashion-MNIST.