Neural tangent kernel eigenvalues accurately predict generalization

1 INTRODUCTION . Understanding and predicting a machine learning model ’ s generalization to unseen data is a central goal of machine learning theory . For a given class of model , one would ideally want a simple picture of a model ’ s inductive bias , identifying the set of functions on which a given model will generalize well and those on which it will generalize poorly and making quantitative predictions of key measures of the quality of generalization . In this paper , we derive such a theory for ridgeless kernel regression with any kernel and , using the neural tangent kernel ( NTK ) equivalence between kernel regression and infinitely wide deep neural networks , shed light on the inductive bias and generalization of wide deep neural networks . Our main contributions are as follows : • We prove a basic conservation law describing the inductive bias of kernel regression ( Theorem 1 ) . This law states that any kernel has a fixed budget of a quantity we call “ learnability ” that it must allocate to an orthogonal basis of functions , and this budget is equal to the size of the training set . As a consequence of this conservation law , we prove that for every kernel , there exists a target function on which kernel regression generalizes worse than chance ( Corollary 1 ) , and we provide a recipe to construct such functions . • We derive a new theory of generalization for kernel regression , culminating in analytical expressions for all first- and second-order statistics of the learned function ( Equations 11 and 13 ) . This theory extends the spectral picture of kernel regression revealed by Bordelon et al . ( 2020 ) and Canatar et al . ( 2021 ) . We conclude from our theory that , in realistic settings , most kernel eigenfunctions have the counterintuitive property that MSE increases as examples are added to a small training set ( Section 2.7 ) . • We empirically verify all our results on synthetic datasets using both exact kernel regression and deep networks of width 500 trained with gradient descent . We find that our conservation law and analytical expressions for generalization performance hold to an excellent approximation even for wide finite networks , and we observe worse-than-chance generalization and increasing MSE as predicted . We find that our theory ’ s core predictions remain fairly accurate even down to width 20 for depth-four networks , suggesting it is a promising starting point for understanding generalization in practical architectures . 1.1 RELATED WORK . The study of the generalization of kernel regression via spectral analysis began in the literature on Gaussian process inference , for which kernel regression gives the mean of the posterior function . In a limited teacher-student setting in which teacher and student were described by the same Gaussian process , Sollich ( 1999 ) and Vivarelli & Opper ( 1999 ) studied expected MSE as a function of training set size , and Sollich ( 2001 ) extended their results to the setting in which the Gaussian processes ’ eigenvalues can differ , but no similar results from this era described the fully general setting . The discovery of the equivalence between wide neural networks and kernel regression via the NTK ( Jacot et al. , 2018 ; Lee et al. , 2019 ) sparked a resurgence of interest in the generalization of kernel regression ( Belkin et al. , 2018a ; 2019b ; a ; Liang & Rakhlin , 2020 ; Bietti & Mairal , 2019 ) . Further noting that both kernel regression and deep learning can generalize well despite perfectly interpolating their training data , Belkin et al . ( 2018b ) argued that “ to understand deep learning we need to understand kernel learning. ” Arora et al . ( 2019a ) derived a data-dependent generalization bound for a wide two-layer architecture involving its infinite-width NTK . Though this bound is a significant advance over VC-dimension-based bounds inapplicable to overparameterized models , it only applies to a specific architecture and can be many times greater than the true loss , while our results predict true loss within small error bars and apply to any infinite-width architecture ( and in fact to any incarnation of kernel regression ) . In pioneering work , Bordelon et al . ( 2020 ) and Canatar et al . ( 2021 ) extended classic results on the generalization of kernel regression to the fully general case and , using the NTK , confirmed that their approximate expressions for expected MSE agree even with wide finite deep neural networks . Their results reveal a simple picture of neural network generalization : as samples are added to the training set , the network generalizes well on a larger and larger subspace of input functions . The natural basis for this subspace of learnable functions is the eigenbasis of the NTK , and its eigenfunctions are learned in descending order of their eigenvalues . The results of the present paper corroborate and extend this picture of the generalization of kernel regression . Our work chiefly differs from theirs in that ( a ) our conservation law is new , ( b ) we derive and test expressions for generic first- and second-order statistics of the learned function , not just MSE , and ( c ) our derivation is quite different ( and , we believe , easier to understand ) even when it arrives at the same results . Our work is related to the well-known observation that neural networks have a “ spectral bias ” towards representing slowly-varying functions ( Valle-Perez et al. , 2018 ; Yang & Salman , 2019 ) ; in particular , the kernel spectrum picture clarifies that this bias is a consequence of the fact that high NTK eigenmodes are typically slowly-varying . We also note a body of work studying the related phenomenon that slowly-varying functions are learned first during training ( Rahaman et al. , 2019 ; Xu et al. , 2019b ; a ; Xu , 2018 ; Cao et al. , 2019 ; Su & Yang , 2019 ) . 2 THEORY . 2.1 A REVIEW OF KERNEL REGRESSION . Consider the task of learning an m-element function f : X → Rm given a set of n unique training points D = { xi } ni=1 ⊆ X and their corresponding function values f ( D ) ∈ Rn×m . To simplify our analysis , we will let the domain X be discrete with size M ≡ |X | and assume the n training points are uniformly sampled from X . This choice of discrete domain will permit us to use matrices and vectors instead of operators and functions in our derivations . By taking M →∞ , our results easily extend to problems with continuous domain : for example , if M →∞ and we allow the points in X to approach a density equal to some desired measure over Rd , we recover the standard setting where the data are sampled nonuniformly from Rd . We will use f̂ to denote the function learned by a neural network trained on this dataset . Remarkably , for an infinite-width neural network optimized via gradient descent to zero training MSE loss , this learned function is given by f̂ ( x ) = K ( x , D ) K ( D , D ) −1f ( D ) , ( 1 ) where K : X × X → R is the network ’ s “ neural tangent kernel ” ( NTK ) ( Jacot et al. , 2018 ; Lee et al. , 2019 ) , K ( D , D ) is the “ kernel matrix ” defined by K ( D , D ) ij = K ( xi , xj ) , and K ( x , D ) is a row vector with components K ( x , D ) i = K ( x , xi ) .1 We give a brief introduction to the NTK in Appendix C. Due to its similarity to the normal equation of linear regression , Equation 1 is often called “ kernel regression. ” 2 Equation 1 holds exactly in the infinite-width limit of fully-connected networks ( Lee et al. , 2019 ) , convolutional networks ( Arora et al. , 2019b ) , transformers ( Hron et al. , 2020 ) , and more ( Yang , 2019 ) . Moreover , several empirical studies have shown it to be a good approximation for networks of even modest width ( Lee et al. , 2019 ; 2020 ) . Our approach will be to study the generalization behavior of Equation 1 , conjecture that our results also apply to finite networks , and finally provide strong support for our conjecture with experiments . Examining Equation 1 , one finds that the m indices of f can each be treated separately : the learned f̂ is equivalent to simply performing kernel regression with each of the m indices as a scalar target function and then vectorizing the results . For simplicity , then , we hereafter assume m = 1 . The extension to m > 1 is straightforward . 2.2 FIGURES OF MERIT OF f̂ We will study three measures of the quality of the learned function f̂ . All three will be defined in terms of the inner product over X : for two functions g , h : X → R , their inner product is 〈g , h〉 ≡ 1M ∑ x∈X g ( x ) h ( x ) . The first measure of quality is mean-squared error ( MSE ) . For a particular dataset D , MSE is given by E ( D ) ( f ) ≡ 〈f − f̂ , f − f̂〉 . Of more interest will be the expected MSE over all datasets of size n , given by E ( f ) ≡ ED [ E ( D ) ( f ) ] . We note that the inner product is taken over all X , including D , even though f̂ ( x ) = f ( x ) for x ∈ D for kernel regression . In maximizing the similarity of f̂ to f , we typically wish to minimize its similarity to all functions orthogonal to f . The second measure examines the coefficient in f̂ of one such orthogonal function to f . Letting g : X → R be a function such that 〈f , g〉 = 0 , we consider the mean and variance of the quantity 〈f̂ , g〉 . We will derive accurate predictions for this metric of generalization . Lastly , we introduce a figure of merit quantifying the alignment of f and f̂ , which we call “ learnability. ” It is given by L ( D ) ( f ) ≡ 〈f , f̂〉 〈f , f〉 , L ( f ) ≡ ED [ L ( D ) ( f ) ] , ( 2 ) where L ( D ) ( f ) is the dataset-dependent learnability of the function f ( “ D-learnability ” ) and L ( f ) is its expectation over random data ( “ learnability ” ) . Though at first glance these two seem like odd figures of merit , we will soon show that they have many desirable properties when f̂ is given by Equation 1 : unlike MSE , both are bounded in [ 0 , 1 ] ( Lemma 1d ) , always change monotonically as new data points are added ( Lemma 1e ) , are invariant to rescalings of f , and obey a simple conservation law ( Theorem 1 ) . Furthermore , expanding the inner product in the definition of E and noting that E ( f ) ≥ 〈f , f〉 ( 1 − L ( f ) ) 2 , one can see that low MSE is impossible without high learnability . We will ultimately derive an accurate approximation for learnability that is substantially simpler than any known approximation for MSE . 1Naively , Equation 1 is only the expected learned function , and the true learned function will include a fluctuation term reflecting the random initialization . However , by storing a copy of the parameters at t = 0 and redefining f̂t ( x ) : = f̂t ( x ) − f̂0 ( x ) throughout optimization and at test time , this term becomes zero , and so we neglect it in our theory and use this trick in our experiments . 2Interestingly , exact Bayesian inference for infinite-width neural networks yields predictions of the same form as Equation 1 , with K being the “ neural network Gaussian process ” ( NNGP ) kernel instead of the NTK ( Lee et al. , 2018 ) . We will proceed treating K as a network ’ s NTK , but our theory and exact results ( including our “ no-free-lunch ” theorem ) apply equally well to any incarnation of kernel regression , including RBF kernel regression and linear regression .

Summary: The paper examines the eigenvalues of a neural network’s “Neural Tangent Kernel” to analyze its generalization performance in the infinite-width regime. It conjectures that the same results will also apply in the finite width regime as well. By analyzing kernel regression and by defining a measure as the “learnability” of a given target function, the paper proves a “no-free-lunch” theorem which implies that improving a network’s or kernel's generalization for a given target function must worsen its generalization performance for its "orthogonal functions". The paper then analytically predicts two phenomena: worse than chance generalization for hard functions and non-monotonic error curves in a small data regime. It also provides some simulations to corroborate the analytic results.