Learning Curves for Deep Neural Networks: A field theory perspective

1 INTRODUCTION . Several pleasant features underlay the success of deep learning : The scarcity of bad minima encountered in their optimization ( Draxler et al. , 2018 ; Choromanska et al. , 2014 ) , their ability to generalize well despite being heavily over-parameterized ( Neyshabur et al. , 2018 ; 2014 ) and expressive ( Zhang et al. , 2016 ) , and their ability to generate internal representations which generalize across different domains and tasks ( Yosinski et al. , 2014 ; Sermanet et al. , 2013 ) . Due to the complexity of DNNs our current understanding of these features is still largely empirical . Notwithstanding , progress has been made recently in the highly over-parametrized regime ( Daniely et al. , 2016 ; Jacot et al. , 2018 ) due to the fact that the networks ’ parameters , in all non-linear layers , change in a minor yet important manner during training . This facilitated the derivation of various bounds ( Allen-Zhu et al. , 2018 ; Cao & Gu , 2019b ; a ) on generalization and , more relevant for this work , the following correspondence with GPs : Considering finite-depth DNNs which are much wider than the dataset-size , trained with MSE loss , no weight decay , and at vanishing learning rate ( the NTK-regime ) one finds that the initialization-averaged predictions are the same as those of Gaussian Processes Regression ( GPR ) with a kernel known as the NTK . Several subsequent works corroborated these results empirically ( Lee et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019 ) and extended them ( Arora et al. , 2019 ) . For fully-connected DNNs , the NTK-regime ( and GPs associated with DNNs in general ( Lee et al. , 2018 ; Novak et al. , 2018 ) ) seems to faithfully capture the generalization power of DNNs trained with MSE loss ( Lee et al. , 2019 ) . One of the most detailed objects quantifying generalization are learning-curves : graphs of how the test error diminishes with the number of datapoints ( N ) . There are currently no analytical predictions or bounds we are aware of for DNN learning-curves which are tight even just in terms of their scaling with N , let alone tight in an absolute sense . In contrast , the theory of GPR is rich with analytical tools which have yielded in the past high accuracy predictions for learning curves . One of the most transparent ones is the equivalence kernel ( EK ) ( Rasmussen & Williams , 2005 ) which will be introduced in section 3 . In short , EK gives an intuitive functional prediction of the expected GPR predictor of a fixed target function when averaged over datasets of size N , and consequently of the learning curves . Clearly such a detailed understanding of generalization in DNNs is desirable . However , several technical issues prohibit the application of the EK and related results ( Rasmussen & Williams , 2005 ; Malzahn & Opper , 2001 ) to DNNs trained in the NTK-regime . First , the NTK-regime corresponds to noiseless GPR where the DNN and corresponding GP both fit the training dataset exactly , whereas EK assumes Gaussian measurement noise on the training labels . In the case of noiseless GPR , EK and related approximations break down and predict perfect generalization with very small datasets . Second , the eigenfunctions ( features ) and eigenvalues of the NTK are needed so that the EK can be interpreted . Third , as EK results can be misleading , it is important to estimate the validity range of these approaches and , in a related manner , derive sub-leading corrections . In this work we make the following contributions : I We establish that noiseless GP inference using NTK can be well approximated by noisy GP inference on a related set of kernels we dub the renormalized NTKs . II We obtain closed expression for the leading and sub-leading asymptotics of learning curves for any fixed target function ( fixed-teacher learning curves ) using an extension of the field-theory formalism of Malzahn & Opper ( 2001 ) for GPR . III For uniform datasets , where the input is distributed uniformly on a hypersphere , these expression simplify considerably and , together with the features and eigenvalues we obtain for the renormalized NTKs , lead to clear relations between deep fully-connected networks and polynomial regression . IV Again for uniform datasets , we provide analytical predictions for learning curves along with estimates for their range of validity . We believe our learning curves estimates stand-out in terms of accuracy as they get to within 3 % of experimental generalization values . We emphasize that our predictions can be applied to fully-connected DNNs of any depth , trained in the NTK regime , and that our approach provides a systematic way of making further accuracy improvements . 2 PRIOR WORKS . Learning curves for GPs have been analyzed using a variety of techniques ( see Rasmussen & Williams ( 2005 ) for a review ) most of which focus on a GP-teacher averaged case where the target/teacher is drawn from the same GP used for inference ( matched priors ) and is furthermore averaged over . Fixedteacher or fixed-target learning curves have been analyzed using a similar grand-canonical/Poissonaveraged approach ( Malzahn & Opper , 2001 ) as our , however , the treatment of the resulting partition function was variational whereas we take a perturbation-theory approach . In addition previous cited results for MSE-loss breakdown in the noiseless limit ( Malzahn & Opper , 2001 ) . To the best of our knowledge , noiseless GPs learning-curves have been analyzed analytically only in the teacheraveraged case and in the following settings : For matched priors , exact results are known for one dimensional data ( Williams & Vivarelli , 2000 ; Rasmussen & Williams , 2005 ) and two dimensional data with some limitations of how one samples the inputs ( in the context of optimal design ) ( Ritter , 2007 ; 1996 ) . In addition Micchelli & Wahba ( 1979 ) derived a lower bound on generalization . For noiseless inference with partially mismatched-priors ( matching features , mismatching eigenvalues ) and at large input dimension the teacher and dataset averaging involved in obtained learning curves has been performed analytically and the resulting matrix traces analyzed numerically Sollich ( 2001 ) . Notably none of these cited results apply in any straightforward manner in the NTK-regime . Considering kernel eigenvalues , explicit expression for the features and eigenvalues of dot-product kernels ( K = K ( x ·x′ ) ) were given in Azevedo & Menegatto ( 2015 ) . The fact that the l-th eigenvalue of such kernels scales as d−l ( d being the input dimension ) , which we used in our derivation of the bound , has been noticed in Sollich ( 2001 ) . Kernels with a trimmed spectrum where the spectrum is trimmed after the first r ’ s leading eigenvalues , has previously been suggested as a way of reducing the computational cost of GP inference ( Ferrari-Trecate et al. , 1998 ) . In contrast we trim the Taylor expansion of the kernel function rather than the spectrum ( which has a very different effect ) and show that an effective observation noise compensates for our trimming/renormalization procedure . Several interesting recent works give bounds on generalization ( Allen-Zhu et al. , 2018 ; Cao & Gu , 2019b ; a ) which show O ( 1/ √ N ) asymptotic decay of the learning-curve ( at best ) . In contrast our predictions are typically well below this bound . 3 FIELD THEORY FORMULATION OF GP LEARNING-CURVES . 3.1 GPR DEFINITION AND PROBLEM STATEMENT . We begin with standard definitions of GPs and Bayesian Inference on GPs . A GP is defined as a stochastic process of which any finite subset of random variables follow a multivariate normal distribution . In a similar fashion to multivariate normal variables , GPs are also determined by their first and second moments . The first is typically taken to be zero , and second is known as the covariance function or the kernel Kxx′ = E [ f ( x ) f ( x′ ) ] , where E [ · ] here denotes expectation under the GP distribution . The main appeal of GPs is that Bayesian Inference with GP priors is tractable . In GPR we use the posterior mean as the predictor g ? , and it is given by : g ? ( x ? ) = N∑ n , m=1 Kx ? , xn [ K ( D ) + σ 2I ] −1nmgm ( 1 ) where x ? is a new datapoint , gm are the training targets , xn are the training data-points , [ K ( D ) ] nm = Kxn , xm is the covariance-matrix ( the covariance-function projected on the training dataset D ) , and σ2 is the variance of the assumed Gaussian noise of the labels , which also acts as a regulator of the prediction . Some intuition for this formula can be gained by verifying that in the noiseless case ( σ2 = 0 ) the prediction at some training point x ? = xq coincides with that point ’ s label g ? = gq . The quantity of interest in this paper will be the dataset averaged generalization error which we define now . Throughout this paper we will assume that both train and test points are i.i.d . random variables drawn from a probability measure µ ( x ) . With this in mind we define the generalization error of a prediction g ? or MSE loss as ‖g ( x ? ) − g ? ( x ? ) ‖2 = ∫ dµx ? ( g ( x ? ) − g ? ( x ? ) ) 2 ( 2 ) Note that g ? is itself a function of N draws from µ which make up the training set DN . The dataset averaged generalization error is ( 2 ) averaged over the ensemble of all possible N sized training sets . We denote this average as 〈·〉DN , so for instance the dataset averaged generalization error would be 〈‖g ? − g‖2〉DN . We see that in order to calculate learning-curves , one needs to average quantities like g ? and g ? 2 . These averages turn out to be difficult to handle analytically , so to facilitate their computation we adopt the approach of Malzahn & Opper ( 2001 ) and instead consider a related quantity given by the Poisson averaging of the former one 〈 ... 〉η = e−η ∞∑ n=0 ηn n ! 〈 ... 〉Dn ( 3 ) where ... can be any quantity , in particular g ? and g ? 2 . Borrowing jargon from physics we refer to the original data ensemble as the canonical ensemble and to the above as the grand-canonical ensemble . Taking η = N , means we are essentially averaging over values of N in an √ N vicinity of N . This means that as far as the leading asymptotic behavior is concerned , one can safely exchange N and η as the differences would be sub-leading . In App . A we compare learning curves as a function of N and η and show that they match very well . Equation ( 1 ) determines the predictions , and therefore the learning-curves , but it is not very convenient for analytic exploration of the expected predictions . This fact is due to the ( potentially very ) large matrix inversion involved , and the additional averaging over DN required . Nonetheless there are some results for the expected prediction g ? , the most famous of which is the equivalence kernel ( EK ) result for the prediction of a fixed target function as a function of the training set size N ( Rasmussen & Williams , 2005 ) : g ? EK , N ( x ) = ∑ n λn λn + σ2/N gnφn ( x ) ( 4 ) Where λn and φn ( x ) here are the eigenvalues and eigenfunctions of the kernel w.r.t the input probability measure ( µ ) and g ( x ) = ∑ n gnφn ( x ) is the target . One notices immediately that the prediction breaks down completely in the noiseless case where ( 4 ) implies perfect estimation of the target with just one datapoint . To gain some intuition as to why having σ2 = 0 hinders predictions of 〈g ? 〉DN one can view it as a hard constraint ( f ( xn ) = g ( xn ) ) , and hard constraints are typically less tractable than soft ones . Indeed this can also be seen as a motivation for considering the above Grand-canonical dataset ensemble . In a related view , finite σ2 can be seen as a form of averaging which smooths and regulates analytical expressions making them more tractable . Another limitation of the EK result is that ( to the best of our knowledge ) there is no systematic way to extend it in orders of 1/N and get a more detailed picture of GPR generalization .

Summary: This paper explores how tools from perturbative field theory can be used to shed light on properties of the generalization error of Gaussian process/kernel regression, particularly on how the error depends on the number of samples N. For uniform data on the sphere, a controlled expansion is obtained in terms of the eigendecomposition of the kernel. Although the expansion breaks when the noise term goes to zero, a renormalized kernel is introduced for which an accurate perturbative expansion is possible. A variety of empirical results confirm the theoretical analysis.