Uniform Generalization Bounds for Overparameterized Neural Networks

1 INTRODUCTION . Neural networks have shown an unprecedented success in the hands of practitioners in several fields . Examples of successful applications include classification ( Krizhevsky et al. , 2012 ; LeCun et al. , 2015 ) and generative modeling ( Goodfellow et al. , 2014 ; Brown et al. , 2020 ) . This practical efficiency has accelerated developing analytical tools for understanding the properties of neural networks . A long standing dilemma in the analysis of neural networks is their favorable generalization error despite typically being extremely overparameterized . It is well known that the classical statistical learning methods for bounding the generalization error , such as Vapnik–Chervonenkis ( VC ) dimension ( Bartlett , 1998 ; Anthony & Bartlett , 2009 ) and Rademacher complexity ( Bartlett & Mendelson , 2002 ; Neyshabur et al. , 2015 ) , often result in vacuous bounds in the case of neural networks ( Dziugaite & Roy , 2017 ) . Motivated with the problem mentioned above , we investigate the generalization error in overparameterized neural networks . In particular , let Dn = { ( xi , yi ) } ni=1 be a possibly noisy dataset collected from a true model f : yi = f ( xi ) + i , where i are well behaved zero mean noise terms . Let f̂n be the learnt model using a gradient based training of an overparameterized neural network on Dn . We are interested in bounding |f ( x ) − f̂n ( x ) | uniformly at all test points x belonging to a particular domain X . A recent line of work has shown that gradient based learning of certain overparameterized neural networks can be analyzed using a projection on the tangent space at random initialization . This leads to the equivalence of the solution of the neural network model to a kernel ridge regression , that is a linear regression in the possibly infinite dimensional feature space of the corresponding kernel . The particular kernel arising from this approach is called the neural tangent ( NT ) kernel ( Jacot et al. , 2018 ) . This equivalence is based on the observation that , during training , the parameters of the neural network remain within a close vicinity of the random initialization . This regime is sometimes referred to as lazy training ( Chizat et al. , 2019 ) . 1.1 CONTRIBUTIONS . Here , we summarize our contributions . We propose the maximal information gain ( MIG ) , denoted by γk ( n ) , of the NT kernel k , as a measure of complexity of the learning problem for overparameterized neural networks in kernel regimes . The MIG is closely related to the effective dimension of the kernel ( for the formal definition of the MIG and its connection to the effective dimension of the kernel , see Section 4 ) . We provide kernel specific bounds on the MIG of the NT kernel . Using these bounds , for networks with infinite number of parameters , we analytically prove novel generalization error bounds decaying with respect to the dataset size n. Furthermore , our results capture the dependence of the decay rate of the error bound on the smoothness properties of the activation functions . Our analysis crucially uses Mercer ’ s theorem and the spectral decomposition of the NT kernel in the basis of spherical harmonics ( which are special functions on the surface of the hypersphere ; see Section 3 ) . We derive bounds on the decay rate of the Mercer eigenvalues ( referred to hereafter as eigendecay ) based on the differentiability of the activation functions . These eigendecays are obtained by careful characterization of a recent result relating the eigendecay to the asymptotic endpoint expansions of the kernel , given in ( Bietti & Bach , 2020 ) . In particular , we consider s − 1 times differentiable activation functions as ( u ) = ( max ( 0 , u ) ) s , s ∈ N. We show that , with these activation functions and a d dimensional hyperspherical support , the eigenvalues λi of the corresponding NT kernel decay at the rate λi ∼ i− d+2s−2 d−1 1 ( up to certain parity constraints on i in special cases ; see Section 2 for details ) . We then use the eigendecay of the NT kernel to establish novel bounds on its MIG , which are of the form γk ( n ) = Õ ( n d−1 d+2s−2 ) . Then , we show this implies high probability error bounds of the form supx∈X |f ( x ) − f̂n ( x ) | = Õ ( n −2s+1 2d+4s−4 ) , when f belongs to the corresponding reproducing kernel Hilbert space ( RKHS ) , and Dn is distributed over X in a sufficiently informative way . In particular , we consider a Dn that is collected sequentially in a way that maximally reduces the uncertainty in the model . That results in a quasi-uniform dataset over the entire domain X ( see Section 5.2 for details ) . A summary of our results on MIG and error bounds is reported in Table 1 . As a byproduct of our results , we show that the RKHS of the NT kernel with activation function as ( . ) is equivalent to the RKHS of a Matérn kernel with smoothness parameter ν = s − 12 . This result was already known for a special case . In particular , Geifman et al . ( 2020 ) ; Chen & Xu ( 2021 ) recently showed the equivalence between the RKHS of the NT kernel with ReLU activation function , a1 ( . ) , and that of the Matérn kernel with ν = 12 ( also known as the Laplace kernel ) . Our contribution includes generalizing this result to the equivalence between the RKHS of the NT kernel under various activation functions and that of a Matérn kernel with the corresponding smoothness ( see Section 3.2 ) . We note that the members of the RKHS of Matérn family of kernels can uniformly approximate almost all continuous functions ( Srinivas et al. , 2010 ) . Therefore , our results suggest that the NT kernels induce a very general class of models . As an intermediate step , we also consider the simpler random feature ( RF ) model which is the result of partial training of the neural network . Also referred to as conjugate kernel or neural networks Gaussian process ( NNGP ) ( Cho & Saul , 2009 ; Daniely et al. , 2016 ; Lee et al. , 2018 ; de G. Matthews et al. , 2018 ) , this model corresponds to the case where only the last layer of the neural network is trained ( see Section 2 for details ) . We note that the decay rate of the error bound is faster in the case of RF kernel with the same activation function . Also , the decay rate of the error bound is faster when s is larger , for both NT and RF kernels . In interpreting these results , we should however take into account the RKHS of the corresponding kernels . The RKHS of the RF kernel is smaller than the RKHS of the NT kernel . Also , when s is larger , the RKHS is smaller for both NT and RF kernels . In Section 6 , we provide experimental results on the error rate for synthetic functions belonging to the RKHS of the NT kernel , which are consistent with the analytically predicted behaviors . Our bounds on MIG may be of independent interest . In Section 7 , we comment on the implications of our results for the regret bounds in reinforcement learning ( RL ) and bandit problems , which use overparameterized neural network models ( e.g. , Yang et al. , 2020 ; Gu et al. , 2021 ) . 2 OVERPARAMETERIZED NEURAL NETWORKS IN KERNEL REGIMES . In this section , we briefly overview the equivalence of overparameterized neural network models to kernel methods . It has been recently shown that , in overparameterized regimes , gradient based training of a neural network reaches a global minimum , where the weights remain within a close vicinity of the random initialization ( Jacot et al. , 2018 ) . In particular , let f ( x , W ) denote a neural network with a large width m parameterized by W . The model can be approximated with its linear projection on the tangent space at random initialization W0 , as f ( x , W ) ≈ f ( x , W0 ) + 〈W −W0 , ∇W f ( x , W0 ) 〉 . The approximation error can be bounded by the second order term ξm‖W −W0‖2 , and shown to be diminishing as m grows , where ξm is the spectral norm of the Hessian matrix of f ( Liu et al. , 2020 ) . The approximation becomes an equality when the width of the hidden layers approaches infinity . Known as lazy training regime ( Chizat et al. , 2019 ) , this model is equivalent to the NT kernel ( Jacot et al. , 2018 ) , given by k ( x , x′ ) = lim m→∞ 〈∇W f ( x , W0 ) , ∇W f ( x′ , W0 ) 〉 . 2.1 NEURAL KERNELS FOR A 2 LAYER NETWORK . Consider the following 2 layer neural network with width m and a proper normalization f ( x , W ) = c√ m m∑ j=1 vja ( w > j x ) , ( 1 ) where a ( . ) is the activation function , c is a constant , x ∈ X is a d dimensional input to the network , wj ∈ Rd are the weights in the first layer , and vj ∈ R are the weights in the second layer . The weights wj are initialized randomly according to N ( 0d , Id ) and vj are initialized randomly according to N ( 0 , 1 ) . Then , the NT kernel of this network can be expressed as ( Jacot et al. , 2018 ) kNT ( x , x ′ ) = c2 ( x > x′ ) Ew∼N ( 0d , Id ) [ a ′ ( w > x ) a′ ( w > x′ ) ] + c2Ew∼N ( 0d , Id ) [ a ( w > x ) a ( w > x′ ) ] . ( 2 ) When wj are fixed at initialization , and only vj are trained with ` 2 regularization , the model is equivalent to a certain Gaussian process ( GP ) ( Neal , 2012 ) that is often referred to as a random feature ( RF ) model . The RF kernel is given as ( Rahimi et al. , 2007 ) k ( x , x′ ) = c2Ew∼N ( 0d , Id ) [ a ( w > j x ) a ( w > x′ ) ] . ( 3 ) Note that the RF kernel is the same as the second term in the expression of the NT kernel given in equation 2 . 2.2 ROTATIONAL INVARIANCE . When the input domain X is the hypersphere Sd−1 , by the spherical symmetry of the Gaussian distribution , the neural kernels are invariant to unitary transformations2 and can be expressed as 2Rotationally invariant kernels are also sometimes referred to as dot product or zonal kernels . a function of x > x′ . We thus use the notations κNT , s ( x > x′ ) and κs ( x > x′ ) to refer to the NT and RF kernels , as defined in equation 2 and equation 3 , respectively . In this notation , s specifies the smoothness of the activation function , as ( u ) = ( max ( 0 , u ) ) s. The particular form of the neural kernels depends on s. For example , for the special case of s = 1 , which corresponds to the ReLU activation function , the following closed form expressions can be derived from the expressions given in equation 2 and equation 3. κNT,1 ( u ) = u π ( π − arccos ( u ) ) + κ1 ( u ) , κ1 ( u ) = 1 π ( u ( π − arccos ( u ) ) + √ 1− u2 ) . In deriving the expressions above , the constant c in equations 2 and 3 is set to √ 2 . In general , we choose c2 = 2 ( 2s−1 ) ! ! , that normalizes the maximum value of the RF kernel to 1 : κs ( 1 ) = 1 , ∀s ≥ 1 .

Summary: The main contribution of this paper is to characterise the eigenvalue decay of the Neural Tangent Kernels associated with fully-connected neural networks evaluated on the hypersphere when the activation functions of the form $Relu^{s}$ where $s$ is a parameter that regulates the smoothness of the activation functions. The results generalise existing results for the particular case $s=1$, i.e. the case of ReLU activations [1,2]. The idea of the proof is to rely on a recent result from [1] which bounds the eigenvalye decay of an arbitrary rotationally invariant kernel based on smoothness and end-point asymptotic properties of the kernel written as a function from $[-1,1]$  to $\mathbb{R}$ where $[-1,1]$ corresponds to the inner product between the two arguments of the kernel function. The computation of the  asymptotics relies on an elegant use of Stein's Lemma as well as the recurrent relation between the NTK and RFK (random feature kernel) of various layers. Results for the eigenvalue decay of the RFK are also provided. Similarly to other works, once eigenvalue decay is obtained, various results can be obtained which further characterise the kernel and the learning problem: (1) it is shown that the kernel is equivalent to a Matérn family of kernels (which itself is equivalent to a sobolev space approach), (2) data-dependent generalisation bounds are shown where the distance between the empirical estimate and the groudn truth is bounded by a term that depends on the relationship between the test point and the training points (3) generalization bounds in $L^\infty$ norm with some strong assumption on the training points (they essentially have to be chosen in a "perfect" way). Applications to reinforcement learning are hinted at in a discussion at the end.      / ================== References ==================  [1] Alberto Bietti and Francis Bach. Deep equals shallow for ReLU networks in kernel regimes. ICLR 2021.   [2] Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the Laplace and neural tangent kernels. NeurIPS 2020.