Deep Equals Shallow for ReLU Networks in Kernel Regimes

1 Introduction . The question of which functions can be well approximated by neural networks is crucial for understanding when these models are successful , and has always been at the heart of the theoretical study of neural networks ( e.g. , Hornik et al. , 1989 ; Pinkus , 1999 ) . While early works have mostly focused on shallow networks with only two layers , more recent works have shown benefits of deep networks for approximating certain classes of functions ( Eldan & Shamir , 2016 ; Mhaskar & Poggio , 2016 ; Telgarsky , 2016 ; Daniely , 2017 ; Yarotsky , 2017 ; Schmidt-Hieber et al. , 2020 ) . Unfortunately , many of these approaches rely on constructions that are not currently known to be learnable using efficient algorithms . A separate line of work has considered over-parameterized networks with random neurons ( Neal , 1996 ) , which also display universal approximation properties while additionally providing efficient algorithms based on kernel methods or their approximations such as random features ( Rahimi & Recht , 2007 ; Bach , 2017b ) . Many recent results on gradient-based optimization of certain over-parameterized networks have been shown to be equivalent to kernel methods with an architecture-specific kernel called the neural tangent kernel ( NTK ) and thus also fall in this category ( e.g. , Jacot et al. , 2018 ; Li & Liang , 2018 ; Allen-Zhu et al. , 2019b ; Du et al. , 2019a ; b ; Zou et al. , 2019 ) . This regime has been coined lazy ( Chizat et al. , 2019 ) , as it does not capture the common phenomenon where weights move significantly away from random initialization and thus may not provide a satisfying model for learning adaptive representations , in contrast to other settings such as the mean field or active regime , which captures complex training dynamics where weights may move in a non-trivial manner and adapt to the data ( e.g. , Chizat & Bach , 2018 ; Mei et al. , 2018 ) . Nevertheless , ∗Work done while at Inria . †Center for Data Science , New York University . New York , USA . ‡Inria - Département d ’ Informatique de l ’ École Normale Supérieure . PSL Research University . Paris , France . one benefit compared to the mean field regime is that the kernel approach easily extends to deep architectures , leading to compositional kernels similar to the ones of Cho & Saul ( 2009 ) ; Daniely et al . ( 2016 ) . Our goal in this paper is to study the role of depth in determining approximation properties for such kernels , with a focus on fully-connected deep ReLU networks . Our approximation results rely on the study of eigenvalue decays of integral operators associated to the obtained dot-product kernels on the sphere , which are diagonalized in the basis of spherical harmonics . This provides a characterization of the functions in the corresponding reproducing kernel Hilbert space ( RKHS ) in terms of their smoothness , and leads to convergence rates for non-parametric regression when the data are uniformly distributed on the sphere . We show that for ReLU networks , the eigenvalue decays for the corresponding deep kernels remain the same regardless of the depth of the network . Our key result is that the decay for a certain class of kernels is characterized by a property related to differentiability of the kernel function around the point where the two inputs are aligned . In particular , the property is preserved when adding layers with ReLU activations , showing that depth plays essentially no role for such networks in kernel regimes . This highlights the limitations of the kernel regime for understanding the power of depth in fully-connected networks , and calls for new models of deep networks beyond kernels ( see , e.g. , Allen-Zhu & Li , 2020 ; Chen et al. , 2020 , for recent works in this direction ) . We also provide applications of our result to other kernels and architectures , and illustrate our results with numerical experiments on synthetic and real datasets . Related work . Kernels for deep learning were originally derived by Neal ( 1996 ) for shallow networks , and later for deep networks ( Cho & Saul , 2009 ; Daniely et al. , 2016 ; Lee et al. , 2018 ; Matthews et al. , 2018 ) . Smola et al . ( 2001 ) ; Minh et al . ( 2006 ) study regularization properties of dot-product kernels on the sphere using spherical harmonics , and Bach ( 2017a ) derives eigenvalue decays for such dot-product kernels arising from shallow networks with positively homogeneous activations including the ReLU . Extensions to shallow NTK or Laplace kernels are studied by Basri et al . ( 2019 ) ; Bietti & Mairal ( 2019b ) ; Geifman et al . ( 2020 ) . The observation that depth does not change the decay of the NTK was previously made by Basri et al . ( 2020 ) empirically , and Geifman et al . ( 2020 ) provide a lower bound on the eigenvalues for deep networks ; our work makes this observation rigorous by providing tight asymptotic decays . Spectral properties of wide neural networks were also considered in ( Cao et al. , 2019 ; Fan & Wang , 2020 ; Ghorbani et al. , 2019 ; Xie et al. , 2017 ; Yang & Salman , 2019 ) . Azevedo & Menegatto ( 2014 ) ; Scetbon & Harchaoui ( 2020 ) also study eigenvalue decays for dot-product kernels but focus on kernels with geometric decays , while our main focus is on polynomial decays . Additional works on over-parameterized or infinite-width networks in lazy regimes include ( Allen-Zhu et al. , 2019a ; b ; Arora et al. , 2019a ; b ; Brand et al. , 2020 ; Lee et al. , 2020 ; Song & Yang , 2019 ) . Concurrently to our work , Chen & Xu ( 2021 ) also studied the RKHS of the NTK for deep ReLU networks , showing that it is the same as for the Laplace kernel on the sphere . They achieve this by studying asymptotic decays of Taylor coefficients of the kernel function at zero using complex-analytic extensions of the kernel functions , and leveraging this to obtain both inclusions between the two RKHSs . In contrast , we obtain precise descriptions of the RKHS and regularization properties in the basis of spherical harmonics for various dot-product kernels through spectral decompositions of integral operators , using ( real ) asymptotic expansions of the kernel function around endpoints . The equality between the RKHS of the deep NTK and Laplace kernel then easily follows from our results by the fact that the two kernels have the same spectral decay . 2 Review of Approximation with Dot-Product Kernels . In this section , we provide a brief review of the kernels that arise from neural networks and their approximation properties . 2.1 Kernels for wide neural networks . Wide neural networks with random weights or weights close to random initialization naturally lead to certain dot-product kernels that depend on the architecture and activation function , which we now present , with a focus on fully-connected architectures . Random feature kernels . We first consider a two-layer ( shallow ) network of the form f ( x ) = 1√ m ∑m j=1 vjσ ( w > j x ) , for some activation function σ . When wj ∼ N ( 0 , I ) ∈ Rd are fixed and only vj ∈ R are trained with ` 2 regularization , this corresponds to using a random feature approximation Rahimi & Recht ( 2007 ) of the kernel k ( x , x′ ) = Ew∼N ( 0 , I ) [ σ ( w > x ) σ ( w > x′ ) ] . ( 1 ) If x , x′ are on the sphere , then by spherical symmetry of the Gaussian distribution , one may show that k is invariant to unitary transformations and takes the form k ( x , x′ ) = κ ( x > x′ ) for a certain function κ . More precisely , if σ ( u ) = ∑ i≥0 aihi ( u ) is the decomposition of σ in the basis of Hermite polynomials hi , which are orthogonal w.r.t . the Gaussian measure , then we have ( Daniely et al. , 2016 ) : κ ( u ) = ∑ i≥0 a2iu i . ( 2 ) Conversely , given a kernel function of the form above with κ ( u ) = ∑ i≥0 biu i with bi ≥ 0 , one may construct corresponding activations using Hermite polynomials by taking σ ( u ) = ∑ i aihi ( u ) , ai ∈ { ± √ bi } . ( 3 ) In the case where σ is s-positively homogeneous , such as the ReLU σ ( u ) = max ( u , 0 ) ( with s = 1 ) , or more generally σs ( u ) = max ( u , 0 ) s , then the kernel ( 1 ) takes the form k ( x , x′ ) = ‖x‖s‖x′‖sκ ( x > x′‖x‖‖x′‖ ) for any x , x′ . This leads to RKHS functions of the form f ( x ) = ‖x‖sg ( x‖x‖ ) , with g in the RKHS of the kernel restricted to the sphere ( Bietti & Mairal , 2019b , Prop . 8 ) . In particular , for the step and ReLU activations σ0 and σ1 , the functions κ are given by the following arc-cosine kernels ( Cho & Saul , 2009 ) :1 κ0 ( u ) = 1 π ( π − arccos ( u ) ) , κ1 ( u ) = 1 π ( u · ( π − arccos ( u ) ) + √ 1− u2 ) . ( 4 ) Note that given a kernel function κ , the corresponding activations ( 3 ) will generally not be homogeneous , thus the inputs to a random network with such activations need to lie on the sphere ( or be appropriately normalized ) in order to yield the kernel κ . Extension to deep networks . When considering a deep network with more than two layers and fixed random weights before the last layer , the connection to random features is less direct since the features are correlated through intermediate layers . Nevertheless , when the hidden layers are wide enough , one still approaches a kernel obtained by letting the widths go to infinity ( see , e.g. , Daniely et al. , 2016 ; Lee et al. , 2018 ; Matthews et al. , 2018 ) , which takes a similar form to the multi-layer kernels of Cho & Saul ( 2009 ) : kL ( x , x′ ) = κL ( x > x′ ) : = κ ◦ · · · ◦ κ︸ ︷︷ ︸ L−1 times ( x > x′ ) , for x , x′ on the sphere , where κ is obtained as described above for a given activation σ , and L is the number of layers . We still refer to this kernel as the random features ( RF ) kernel in this paper , noting that it is sometimes known as the “ conjugate kernel ” or NNGP kernel ( for neural network Gaussian process ) . It is usually good to normalize κ such that κ ( 1 ) = 1 , so that we also have κL ( 1 ) = 1 , avoiding exploding or vanishing behavior for deep networks . In practice , this corresponds to using an activation-dependent scaling in the random weight initialization , which is commonly used by practitioners ( He et al. , 2015 ) . 1Here we assume a scaling √ 2/m instead of √ 1/m in the definition of f , which yields κ ( 1 ) = 1 , a useful normalization for deep networks , as explained below . Neural tangent kernels . When intermediate layers are trained along with the last layer using gradient methods , the resulting problem is non-convex and the statistical properties of such approaches are not well understood in general , particularly for deep networks . However , in a specific over-parameterized regime , it may be shown that gradient descent can reach a global minimum while keeping weights very close to random initialization . More precisely , for a network f ( x ; θ ) parameterized by θ with large width m , the model remains close to its linearization around random initialization θ0 throughout training , that is , f ( x ; θ ) ≈ f ( x ; θ0 ) + 〈θ− θ0 , ∇θf ( x ; θ0 ) 〉 . This is also known as the lazy training regime ( Chizat et al. , 2019 ) . Learning is then equivalent to a kernel method with another architecture-specific kernel known as the neural tangent kernel ( NTK , Jacot et al. , 2018 ) , given by kNTK ( x , x′ ) = lim m→∞ 〈∇f ( x ; θ0 ) , ∇f ( x′ ; θ0 ) 〉 . ( 5 ) For a simple two-layer network with activation σ , it is then given by kNTK ( x , x′ ) = ( x > x′ ) Ew [ σ′ ( w > x ) σ′ ( w > x′ ) ] + Ew [ σ ( w > x ) σ ( w > x′ ) ] . ( 6 ) For a ReLU network with L layers with inputs on the sphere , taking appropriate limits on the widths , one can show ( Jacot et al. , 2018 ) : kNTK ( x , x′ ) = κLNTK ( x > x′ ) , with κ1NTK ( u ) = κ1 ( u ) = u and for ` = 2 , . . . , L , κ ` ( u ) = κ1 ( κ ` −1 ( u ) ) κ ` NTK ( u ) = κ ` −1NTK ( u ) κ0 ( κ ` −1 ( u ) ) + κ ` ( u ) , ( 7 ) where κ0 and κ1 are given in ( 4 ) .

Summary: Recently, there are a large number of deep learning theory papers related to the property of neural tangent kernel. This paper shows that for ReLU, the kernels derived from deep fully-connected networks have the same approx. properties as their shallow two-layer counterpart. This highlights the limitation of the kernel framework for understanding the benefits of deep networks from such perspective.