Neural tangent kernel eigenvalues accurately predict generalization

1 INTRODUCTION . Understanding and predicting a machine learning model ’ s generalization to unseen data is a central goal of machine learning theory . For a given class of model , one would ideally want a simple picture of a model ’ s inductive bias , identifying the set of functions on which a given model will generalize well and those on which it will generalize poorly and making quantitative predictions of key measures of the quality of generalization . In this paper , we derive such a theory for ridgeless kernel regression with any kernel and , using the neural tangent kernel ( NTK ) equivalence between kernel regression and infinitely wide deep neural networks , shed light on the inductive bias and generalization of wide deep neural networks . Our main contributions are as follows : • We prove a basic conservation law describing the inductive bias of kernel regression ( Theorem 1 ) . This law states that any kernel has a fixed budget of a quantity we call “ learnability ” that it must allocate to an orthogonal basis of functions , and this budget is equal to the size of the training set . As a consequence of this conservation law , we prove that for every kernel , there exists a target function on which kernel regression generalizes worse than chance ( Corollary 1 ) , and we provide a recipe to construct such functions . • We derive a new theory of generalization for kernel regression , culminating in analytical expressions for all first- and second-order statistics of the learned function ( Equations 11 and 13 ) . This theory extends the spectral picture of kernel regression revealed by Bordelon et al . ( 2020 ) and Canatar et al . ( 2021 ) . We conclude from our theory that , in realistic settings , most kernel eigenfunctions have the counterintuitive property that MSE increases as examples are added to a small training set ( Section 2.7 ) . • We empirically verify all our results on synthetic datasets using both exact kernel regression and deep networks of width 500 trained with gradient descent . We find that our conservation law and analytical expressions for generalization performance hold to an excellent approximation even for wide finite networks , and we observe worse-than-chance generalization and increasing MSE as predicted . We find that our theory ’ s core predictions remain fairly accurate even down to width 20 for depth-four networks , suggesting it is a promising starting point for understanding generalization in practical architectures . 1.1 RELATED WORK . The study of the generalization of kernel regression via spectral analysis began in the literature on Gaussian process inference , for which kernel regression gives the mean of the posterior function . In a limited teacher-student setting in which teacher and student were described by the same Gaussian process , Sollich ( 1999 ) and Vivarelli & Opper ( 1999 ) studied expected MSE as a function of training set size , and Sollich ( 2001 ) extended their results to the setting in which the Gaussian processes ’ eigenvalues can differ , but no similar results from this era described the fully general setting . The discovery of the equivalence between wide neural networks and kernel regression via the NTK ( Jacot et al. , 2018 ; Lee et al. , 2019 ) sparked a resurgence of interest in the generalization of kernel regression ( Belkin et al. , 2018a ; 2019b ; a ; Liang & Rakhlin , 2020 ; Bietti & Mairal , 2019 ) . Further noting that both kernel regression and deep learning can generalize well despite perfectly interpolating their training data , Belkin et al . ( 2018b ) argued that “ to understand deep learning we need to understand kernel learning. ” Arora et al . ( 2019a ) derived a data-dependent generalization bound for a wide two-layer architecture involving its infinite-width NTK . Though this bound is a significant advance over VC-dimension-based bounds inapplicable to overparameterized models , it only applies to a specific architecture and can be many times greater than the true loss , while our results predict true loss within small error bars and apply to any infinite-width architecture ( and in fact to any incarnation of kernel regression ) . In pioneering work , Bordelon et al . ( 2020 ) and Canatar et al . ( 2021 ) extended classic results on the generalization of kernel regression to the fully general case and , using the NTK , confirmed that their approximate expressions for expected MSE agree even with wide finite deep neural networks . Their results reveal a simple picture of neural network generalization : as samples are added to the training set , the network generalizes well on a larger and larger subspace of input functions . The natural basis for this subspace of learnable functions is the eigenbasis of the NTK , and its eigenfunctions are learned in descending order of their eigenvalues . The results of the present paper corroborate and extend this picture of the generalization of kernel regression . Our work chiefly differs from theirs in that ( a ) our conservation law is new , ( b ) we derive and test expressions for generic first- and second-order statistics of the learned function , not just MSE , and ( c ) our derivation is quite different ( and , we believe , easier to understand ) even when it arrives at the same results . Our work is related to the well-known observation that neural networks have a “ spectral bias ” towards representing slowly-varying functions ( Valle-Perez et al. , 2018 ; Yang & Salman , 2019 ) ; in particular , the kernel spectrum picture clarifies that this bias is a consequence of the fact that high NTK eigenmodes are typically slowly-varying . We also note a body of work studying the related phenomenon that slowly-varying functions are learned first during training ( Rahaman et al. , 2019 ; Xu et al. , 2019b ; a ; Xu , 2018 ; Cao et al. , 2019 ; Su & Yang , 2019 ) . 2 THEORY . 2.1 A REVIEW OF KERNEL REGRESSION . Consider the task of learning an m-element function f : X → Rm given a set of n unique training points D = { xi } ni=1 ⊆ X and their corresponding function values f ( D ) ∈ Rn×m . To simplify our analysis , we will let the domain X be discrete with size M ≡ |X | and assume the n training points are uniformly sampled from X . This choice of discrete domain will permit us to use matrices and vectors instead of operators and functions in our derivations . By taking M →∞ , our results easily extend to problems with continuous domain : for example , if M →∞ and we allow the points in X to approach a density equal to some desired measure over Rd , we recover the standard setting where the data are sampled nonuniformly from Rd . We will use f̂ to denote the function learned by a neural network trained on this dataset . Remarkably , for an infinite-width neural network optimized via gradient descent to zero training MSE loss , this learned function is given by f̂ ( x ) = K ( x , D ) K ( D , D ) −1f ( D ) , ( 1 ) where K : X × X → R is the network ’ s “ neural tangent kernel ” ( NTK ) ( Jacot et al. , 2018 ; Lee et al. , 2019 ) , K ( D , D ) is the “ kernel matrix ” defined by K ( D , D ) ij = K ( xi , xj ) , and K ( x , D ) is a row vector with components K ( x , D ) i = K ( x , xi ) .1 We give a brief introduction to the NTK in Appendix C. Due to its similarity to the normal equation of linear regression , Equation 1 is often called “ kernel regression. ” 2 Equation 1 holds exactly in the infinite-width limit of fully-connected networks ( Lee et al. , 2019 ) , convolutional networks ( Arora et al. , 2019b ) , transformers ( Hron et al. , 2020 ) , and more ( Yang , 2019 ) . Moreover , several empirical studies have shown it to be a good approximation for networks of even modest width ( Lee et al. , 2019 ; 2020 ) . Our approach will be to study the generalization behavior of Equation 1 , conjecture that our results also apply to finite networks , and finally provide strong support for our conjecture with experiments . Examining Equation 1 , one finds that the m indices of f can each be treated separately : the learned f̂ is equivalent to simply performing kernel regression with each of the m indices as a scalar target function and then vectorizing the results . For simplicity , then , we hereafter assume m = 1 . The extension to m > 1 is straightforward . 2.2 FIGURES OF MERIT OF f̂ We will study three measures of the quality of the learned function f̂ . All three will be defined in terms of the inner product over X : for two functions g , h : X → R , their inner product is 〈g , h〉 ≡ 1M ∑ x∈X g ( x ) h ( x ) . The first measure of quality is mean-squared error ( MSE ) . For a particular dataset D , MSE is given by E ( D ) ( f ) ≡ 〈f − f̂ , f − f̂〉 . Of more interest will be the expected MSE over all datasets of size n , given by E ( f ) ≡ ED [ E ( D ) ( f ) ] . We note that the inner product is taken over all X , including D , even though f̂ ( x ) = f ( x ) for x ∈ D for kernel regression . In maximizing the similarity of f̂ to f , we typically wish to minimize its similarity to all functions orthogonal to f . The second measure examines the coefficient in f̂ of one such orthogonal function to f . Letting g : X → R be a function such that 〈f , g〉 = 0 , we consider the mean and variance of the quantity 〈f̂ , g〉 . We will derive accurate predictions for this metric of generalization . Lastly , we introduce a figure of merit quantifying the alignment of f and f̂ , which we call “ learnability. ” It is given by L ( D ) ( f ) ≡ 〈f , f̂〉 〈f , f〉 , L ( f ) ≡ ED [ L ( D ) ( f ) ] , ( 2 ) where L ( D ) ( f ) is the dataset-dependent learnability of the function f ( “ D-learnability ” ) and L ( f ) is its expectation over random data ( “ learnability ” ) . Though at first glance these two seem like odd figures of merit , we will soon show that they have many desirable properties when f̂ is given by Equation 1 : unlike MSE , both are bounded in [ 0 , 1 ] ( Lemma 1d ) , always change monotonically as new data points are added ( Lemma 1e ) , are invariant to rescalings of f , and obey a simple conservation law ( Theorem 1 ) . Furthermore , expanding the inner product in the definition of E and noting that E ( f ) ≥ 〈f , f〉 ( 1 − L ( f ) ) 2 , one can see that low MSE is impossible without high learnability . We will ultimately derive an accurate approximation for learnability that is substantially simpler than any known approximation for MSE . 1Naively , Equation 1 is only the expected learned function , and the true learned function will include a fluctuation term reflecting the random initialization . However , by storing a copy of the parameters at t = 0 and redefining f̂t ( x ) : = f̂t ( x ) − f̂0 ( x ) throughout optimization and at test time , this term becomes zero , and so we neglect it in our theory and use this trick in our experiments . 2Interestingly , exact Bayesian inference for infinite-width neural networks yields predictions of the same form as Equation 1 , with K being the “ neural network Gaussian process ” ( NNGP ) kernel instead of the NTK ( Lee et al. , 2018 ) . We will proceed treating K as a network ’ s NTK , but our theory and exact results ( including our “ no-free-lunch ” theorem ) apply equally well to any incarnation of kernel regression , including RBF kernel regression and linear regression .

Summary: This paper provides a novel theoretical account of generalization for kernel regression. To do so, the authors study a matrix built out of the kernel eigensystem evaluated on the training set that they call the "learning transfer matrix," and which relates the decomposition of the true function in the eigenbasis to the decomposition of the learned function in the eigenbasis. In other words, this matrix characterizes the kernel regression solution in the eigenbasis. By making a number of approximations, they are able to find a closed-form expression for this matrix and study its first and second order statistics. A main conclusion from this analysis is that functions are more learnable by kernel regression if they have more weight in higher eigenvalue modes.  Moreover, the authors also introduce a new metric, which they call "learnability," that they use to prove a new no-free-lunch theorem whose content is that when averaging over a complete basis of functions the learnability is independent of the kernel. This means that the choice of kernel should be tailored to the details of the function being learned in order for kernel regression to succeed. A related result is that there are some functions for which the kernel regression solution generalizes worse than simply outputting "0" on data outside the training set, i.e. for which the solution fails to generalize at all.