Uniform Generalization Bounds for Overparameterized Neural Networks

1 INTRODUCTION . Neural networks have shown an unprecedented success in the hands of practitioners in several fields . Examples of successful applications include classification ( Krizhevsky et al. , 2012 ; LeCun et al. , 2015 ) and generative modeling ( Goodfellow et al. , 2014 ; Brown et al. , 2020 ) . This practical efficiency has accelerated developing analytical tools for understanding the properties of neural networks . A long standing dilemma in the analysis of neural networks is their favorable generalization error despite typically being extremely overparameterized . It is well known that the classical statistical learning methods for bounding the generalization error , such as Vapnik–Chervonenkis ( VC ) dimension ( Bartlett , 1998 ; Anthony & Bartlett , 2009 ) and Rademacher complexity ( Bartlett & Mendelson , 2002 ; Neyshabur et al. , 2015 ) , often result in vacuous bounds in the case of neural networks ( Dziugaite & Roy , 2017 ) . Motivated with the problem mentioned above , we investigate the generalization error in overparameterized neural networks . In particular , let Dn = { ( xi , yi ) } ni=1 be a possibly noisy dataset collected from a true model f : yi = f ( xi ) + i , where i are well behaved zero mean noise terms . Let f̂n be the learnt model using a gradient based training of an overparameterized neural network on Dn . We are interested in bounding |f ( x ) − f̂n ( x ) | uniformly at all test points x belonging to a particular domain X . A recent line of work has shown that gradient based learning of certain overparameterized neural networks can be analyzed using a projection on the tangent space at random initialization . This leads to the equivalence of the solution of the neural network model to a kernel ridge regression , that is a linear regression in the possibly infinite dimensional feature space of the corresponding kernel . The particular kernel arising from this approach is called the neural tangent ( NT ) kernel ( Jacot et al. , 2018 ) . This equivalence is based on the observation that , during training , the parameters of the neural network remain within a close vicinity of the random initialization . This regime is sometimes referred to as lazy training ( Chizat et al. , 2019 ) . 1.1 CONTRIBUTIONS . Here , we summarize our contributions . We propose the maximal information gain ( MIG ) , denoted by γk ( n ) , of the NT kernel k , as a measure of complexity of the learning problem for overparameterized neural networks in kernel regimes . The MIG is closely related to the effective dimension of the kernel ( for the formal definition of the MIG and its connection to the effective dimension of the kernel , see Section 4 ) . We provide kernel specific bounds on the MIG of the NT kernel . Using these bounds , for networks with infinite number of parameters , we analytically prove novel generalization error bounds decaying with respect to the dataset size n. Furthermore , our results capture the dependence of the decay rate of the error bound on the smoothness properties of the activation functions . Our analysis crucially uses Mercer ’ s theorem and the spectral decomposition of the NT kernel in the basis of spherical harmonics ( which are special functions on the surface of the hypersphere ; see Section 3 ) . We derive bounds on the decay rate of the Mercer eigenvalues ( referred to hereafter as eigendecay ) based on the differentiability of the activation functions . These eigendecays are obtained by careful characterization of a recent result relating the eigendecay to the asymptotic endpoint expansions of the kernel , given in ( Bietti & Bach , 2020 ) . In particular , we consider s − 1 times differentiable activation functions as ( u ) = ( max ( 0 , u ) ) s , s ∈ N. We show that , with these activation functions and a d dimensional hyperspherical support , the eigenvalues λi of the corresponding NT kernel decay at the rate λi ∼ i− d+2s−2 d−1 1 ( up to certain parity constraints on i in special cases ; see Section 2 for details ) . We then use the eigendecay of the NT kernel to establish novel bounds on its MIG , which are of the form γk ( n ) = Õ ( n d−1 d+2s−2 ) . Then , we show this implies high probability error bounds of the form supx∈X |f ( x ) − f̂n ( x ) | = Õ ( n −2s+1 2d+4s−4 ) , when f belongs to the corresponding reproducing kernel Hilbert space ( RKHS ) , and Dn is distributed over X in a sufficiently informative way . In particular , we consider a Dn that is collected sequentially in a way that maximally reduces the uncertainty in the model . That results in a quasi-uniform dataset over the entire domain X ( see Section 5.2 for details ) . A summary of our results on MIG and error bounds is reported in Table 1 . As a byproduct of our results , we show that the RKHS of the NT kernel with activation function as ( . ) is equivalent to the RKHS of a Matérn kernel with smoothness parameter ν = s − 12 . This result was already known for a special case . In particular , Geifman et al . ( 2020 ) ; Chen & Xu ( 2021 ) recently showed the equivalence between the RKHS of the NT kernel with ReLU activation function , a1 ( . ) , and that of the Matérn kernel with ν = 12 ( also known as the Laplace kernel ) . Our contribution includes generalizing this result to the equivalence between the RKHS of the NT kernel under various activation functions and that of a Matérn kernel with the corresponding smoothness ( see Section 3.2 ) . We note that the members of the RKHS of Matérn family of kernels can uniformly approximate almost all continuous functions ( Srinivas et al. , 2010 ) . Therefore , our results suggest that the NT kernels induce a very general class of models . As an intermediate step , we also consider the simpler random feature ( RF ) model which is the result of partial training of the neural network . Also referred to as conjugate kernel or neural networks Gaussian process ( NNGP ) ( Cho & Saul , 2009 ; Daniely et al. , 2016 ; Lee et al. , 2018 ; de G. Matthews et al. , 2018 ) , this model corresponds to the case where only the last layer of the neural network is trained ( see Section 2 for details ) . We note that the decay rate of the error bound is faster in the case of RF kernel with the same activation function . Also , the decay rate of the error bound is faster when s is larger , for both NT and RF kernels . In interpreting these results , we should however take into account the RKHS of the corresponding kernels . The RKHS of the RF kernel is smaller than the RKHS of the NT kernel . Also , when s is larger , the RKHS is smaller for both NT and RF kernels . In Section 6 , we provide experimental results on the error rate for synthetic functions belonging to the RKHS of the NT kernel , which are consistent with the analytically predicted behaviors . Our bounds on MIG may be of independent interest . In Section 7 , we comment on the implications of our results for the regret bounds in reinforcement learning ( RL ) and bandit problems , which use overparameterized neural network models ( e.g. , Yang et al. , 2020 ; Gu et al. , 2021 ) . 2 OVERPARAMETERIZED NEURAL NETWORKS IN KERNEL REGIMES . In this section , we briefly overview the equivalence of overparameterized neural network models to kernel methods . It has been recently shown that , in overparameterized regimes , gradient based training of a neural network reaches a global minimum , where the weights remain within a close vicinity of the random initialization ( Jacot et al. , 2018 ) . In particular , let f ( x , W ) denote a neural network with a large width m parameterized by W . The model can be approximated with its linear projection on the tangent space at random initialization W0 , as f ( x , W ) ≈ f ( x , W0 ) + 〈W −W0 , ∇W f ( x , W0 ) 〉 . The approximation error can be bounded by the second order term ξm‖W −W0‖2 , and shown to be diminishing as m grows , where ξm is the spectral norm of the Hessian matrix of f ( Liu et al. , 2020 ) . The approximation becomes an equality when the width of the hidden layers approaches infinity . Known as lazy training regime ( Chizat et al. , 2019 ) , this model is equivalent to the NT kernel ( Jacot et al. , 2018 ) , given by k ( x , x′ ) = lim m→∞ 〈∇W f ( x , W0 ) , ∇W f ( x′ , W0 ) 〉 . 2.1 NEURAL KERNELS FOR A 2 LAYER NETWORK . Consider the following 2 layer neural network with width m and a proper normalization f ( x , W ) = c√ m m∑ j=1 vja ( w > j x ) , ( 1 ) where a ( . ) is the activation function , c is a constant , x ∈ X is a d dimensional input to the network , wj ∈ Rd are the weights in the first layer , and vj ∈ R are the weights in the second layer . The weights wj are initialized randomly according to N ( 0d , Id ) and vj are initialized randomly according to N ( 0 , 1 ) . Then , the NT kernel of this network can be expressed as ( Jacot et al. , 2018 ) kNT ( x , x ′ ) = c2 ( x > x′ ) Ew∼N ( 0d , Id ) [ a ′ ( w > x ) a′ ( w > x′ ) ] + c2Ew∼N ( 0d , Id ) [ a ( w > x ) a ( w > x′ ) ] . ( 2 ) When wj are fixed at initialization , and only vj are trained with ` 2 regularization , the model is equivalent to a certain Gaussian process ( GP ) ( Neal , 2012 ) that is often referred to as a random feature ( RF ) model . The RF kernel is given as ( Rahimi et al. , 2007 ) k ( x , x′ ) = c2Ew∼N ( 0d , Id ) [ a ( w > j x ) a ( w > x′ ) ] . ( 3 ) Note that the RF kernel is the same as the second term in the expression of the NT kernel given in equation 2 . 2.2 ROTATIONAL INVARIANCE . When the input domain X is the hypersphere Sd−1 , by the spherical symmetry of the Gaussian distribution , the neural kernels are invariant to unitary transformations2 and can be expressed as 2Rotationally invariant kernels are also sometimes referred to as dot product or zonal kernels . a function of x > x′ . We thus use the notations κNT , s ( x > x′ ) and κs ( x > x′ ) to refer to the NT and RF kernels , as defined in equation 2 and equation 3 , respectively . In this notation , s specifies the smoothness of the activation function , as ( u ) = ( max ( 0 , u ) ) s. The particular form of the neural kernels depends on s. For example , for the special case of s = 1 , which corresponds to the ReLU activation function , the following closed form expressions can be derived from the expressions given in equation 2 and equation 3. κNT,1 ( u ) = u π ( π − arccos ( u ) ) + κ1 ( u ) , κ1 ( u ) = 1 π ( u ( π − arccos ( u ) ) + √ 1− u2 ) . In deriving the expressions above , the constant c in equations 2 and 3 is set to √ 2 . In general , we choose c2 = 2 ( 2s−1 ) ! ! , that normalizes the maximum value of the RF kernel to 1 : κs ( 1 ) = 1 , ∀s ≥ 1 .

Summary: The authors consider neural tangent kernels (NTKs) kappa_{NT, s}^l of the neural networks of depth l associated with ReLU a and its powers a_s with a positive integer s. Theorem 1 gives relations between the RKHSs generated by the NTKs and those generated by Matern kernels. Theorem 2 gives bounds for the maximal information gain of the NTKs. Theorem 3 provides some uniform generalization bounds for kernel ridge regression with the NTKs.