Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm

1 INTRODUCTION . In supervised learning machines learn from a finite collection of n training data , and their generalization error is then evaluated on unseen data drawn from the same distribution . How many data are needed to learn a task is characterized by the learning curve relating generalization error to n. In various cases , the generalization error decays as a power law n−β , with an exponent β that depends on both the data and the algorithm . In ( Hestness et al. , 2017 ) β is reported for state-of-the-art ( SOTA ) deep neural networks for various tasks : in for neural-machine translation β ≈ 0.3–0.36 ( for fixed model size ) or β ≈ 0.13 ( for best-fit models at any n ) ; language modeling shows β ≈ 0.06–0.09 ; in speech recognition β ≈ 0.3 ; SOTA models for image classification ( on ImageNet ) have exponents β ≈ 0.3–0.5 . Currently there is no available theory of deep learning to rationalize these observations . Recently it was shown that for a proper initialization of the weights , deep learning in the infinite-width limit ( Jacot et al. , 2018 ) converges to kernel learning . Moreover , it is nowadays part of the lore that there exist kernels whose performance is nearly comparable to deep networks ( Bruna and Mallat , 2013 ; Arora et al. , 2019 ) , at least for some tasks . It is thus of great interest to understand the learning curves of kernels . For regression , if the target function being learned is simply assumed to be Lipschitz , then the best guarantee is β = 1/d ( Luxburg and Bousquet , 2004 ; Bach , 2017 ) where d is the data dimension . Thus for large d , β is very small : learning is completely inefficient , a phenomenon referred to as the curse of dimensionality . As a result , various works on kernel regression make the much stronger assumption that the training points are sampled from a target function that belongs to the reproducing kernel Hilbert space ( RKHS ) of the kernel ( see for example ( Smola et al. , 1998 ) ) . With this assumption β does not depend on d ( for instance in ( Rudi and Rosasco , 2017 ) β = 1/2 is guaranteed ) . Yet , RKHS is a very strong assumption which requires the smoothness of the target function to increase with d ( Bach , 2017 ) ( see more on this point below ) , which may not be realistic in large dimensions . In this work we compute β empirically for kernel methods applied on MNIST and CIFAR10 datasets . We find βMNIST ≈ 0.4 and βCIFAR10 ≈ 0.1 respectively . Quite remarkably , we observe essentially the same exponents for regression and classification tasks , using either a Gaussian or a Laplace kernel . Thus the exponents are not as small as 1/d ( d = 784 for MNIST , d = 3072 for CIFAR10 ) , but neither are they 1/2 as one would expect under the RKHS assumption . These facts call for frameworks in which assumptions on the smoothness of the data can be intermediary between Lipschitz and RKHS . Here we propose such a framework for regression , in which the target function is assumed to be a Gaussian random field of zero mean with translation-invariant isotropic covariance KT ( x ) . The data can equivalently be thought as being synthesized by a “ Teacher ” kernelKT ( x ) . Learning is performed with a “ Student ” kernel KS ( x ) that minimizes the mean-square error . In general KT ( x ) 6= KS ( x ) . In this set-up learning is very similar to a technique referred to as kriging , or Gaussian process regression , originally developed in the geostatistics community ( Matheron , 1963 ; Stein , 1999b ) . To quantify learning , we first perform numerical experiments for data points distributed uniformly at random on a hypersphere of varying dimension d , focusing on a Laplace kernel for the Student , and considering a Laplace or Gaussian kernel for the Teacher . We observe that in both cases β ( d ) is a decreasing function . To derive β ( d ) we consider the simplified situation where the Gaussian random field is sampled at training points lying on a regular lattice . Building on the kriging literature ( Stein , 1999b ) , we show that β is controlled by the high-frequency scaling of both the Teacher and Student kernels : assuming that the Fourier transforms of the kernels decay as K̃T ( w ) = cT ||w||−αT + o ( ||w||−αT ) and K̃S ( w ) = cS ||w||−αS + o ( ||w||−αS ) , we obtain β = 1 d min ( αT − d , 2αS ) . ( 1 ) Importantly ( i ) Eq . ( 1 ) leads to a prediction for β ( d ) that accurately matches our numerical study for random training data points , leading to the conjecture that Eq . ( 1 ) holds in that case as well . We offer the following interpretation : ultimately , kernel methods are performing a local interpolation whose quality depends on the distance δ ( n ) between adjacent data points . δ ( n ) is asymptotically similar for random data or data sitting on a lattice . ( ii ) If the kernel KS is not too sensitive to high-frequencies , then learning is optimal as far as scaling is concerned and β = ( αT − d ) /d . We will argue that the smoothness index s ≡ [ ( αT − d ) /2 ] characterizes the number of derivatives of the target function that are continuous . We thus recover the curse of dimensionality : s needs to be of order d to have non-vanishing β in large dimensions . Point ( ii ) leads to an apparent paradox : β is significant for MNIST and CIFAR10 , for which d is a priori very large , leading to a smoothness value s in the hundreds in both cases , which appears unrealistic . The paradox is resolved by considering that real datasets actually live on lower-dimensional manifolds . As far as kernel learning is concerned , our findings support that the correct definition of dimension should be based on how the nearest-neighbors distance δ ( n ) scales with n : δ ( n ) ∼ n−1/deff . Direct measurements of δ ( n ) support that MNIST and CIFAR10 live on manifolds of lower dimensions deffMNIST ≈ 15 and deffCIFAR10 ≈ 35 . Considering the effective dimensions that we find , the observed values for β would be obtained for Gaussian fields of smoothness sMNIST ≈ 3 and sCIFAR10 ≈ 1 , values that appear intuitively more reasonable . More generally this analogy with Gaussian fields allows one to associate a smoothness index s to any dataset once β and deff are measured , which may turn out to be a useful characterization of data complexity in the future . 2 RELATED WORKS . Our set-up of Teacher-Student learning with kernels is also referred to as kriging , or Gaussian process regression , and it was originally developed in the geostatistics community ( Matheron , 1963 ) . In Section 5 we present a theorem that allows one to know the rate at which the test error decreases as we increase the number of training points , assumed to lie on a high-dimensional regular lattice . Similar results have been previously derived in the kriging literature ( Stein , 1999b ) when sampling occurs on the regular lattice with the exception of the origin , where the inference is made . Here we propose an alternative derivation that some readers might find simpler . We also study a slightly different problem : instead of computing the test error when the inference is carried on at the origin , we compute the average error for a test point that lie at an arbitrary point , sampled uniformly at random and not necessarily on the lattice . In what follows we show , via extensive numerical simulations , that such predictions are accurate even when the training points do not lie on a regular lattice , but are taken at random on a hypersphere . An exact proof of our result in such a general setting is difficult and can not be found even in the kriging literature . To our knowledge the results that get closer to the point are those discussed in ( Stein , 1999a ) , where the author studies one-dimensional processes where the training data are not necessarily evenly spaced . In this work the effective dimension of the data plays an import role , as it controls how the distance between nearest neighbors scales with the dataset size . Of course , there exists a vast literature ( Grassberger and Procaccia , 1983 ; Costa and Hero , 2004 ; Hein and Audibert , 2005 ; Levina and Bickel , 2005 ; Rozza et al. , 2012 ; Facco et al. , 2017 ; Allegra et al. , 2019 ) devoted to the study of effective dimensions , where other definitions are analyzed . The effective dimensions that we find are compatible with those obtained with more refined methods . 3 LEARNING CURVE FOR KERNEL METHODS APPLIED TO REAL DATA . In what follows we apply kernel methods to the MNIST and CIFAR10 datasets , each consisting of a set of images ( xµ ) n µ=1 . We simplify the problem by considering only two classes whose label Z ( xµ ) = ±1 correspond to odd and even numbers for MNIST , and to two groups of 5 classes in CIFAR10 . The goal is to infer the value of the label ẐS ( x ) of an image x that does not belong to the dataset . The S subscript reminds us that inference is performed using a positive definite kernel KS . We perform inference in both a regression and a classification setting . The following algorithms and associated results can be found in ( Scholkopf and Smola , 2001 ) . Regression . Learning corresponds to minimizing a mean-square error : min n∑ µ=1 [ ẐS ( xµ ) − Z ( xµ ) ] 2 . ( 2 ) For algorithms seeking solutions of the form ẐS ( x ) = ∑ µ aµKS ( xµ , x ) ≡ a · kS ( x ) by minimizing the man-square loss over the vector a , one obtains : ẐS ( x ) = kS ( x ) ·K−1S Z , ( 3 ) where the vector Z contains all the labels in the training set , Z ≡ ( Z ( xµ ) ) nµ=1 , and KS , µν ≡ KS ( xµ , xν ) is the Gram matrix . The Gram matrix is always invertible if the kernel KS is positive definite . The generalization error is then evaluated as the expected mean-square error on unseen data , estimated by averaging over a test set composed of ntest unseen data points : MSE = 1 ntest ntest∑ µ=1 [ ẐS ( xµ ) − Z ( xµ ) ] 2 . ( 4 ) Classification . We perform kernel classification via the algorithm soft-margin SVM . The details can be found in Appendix A . After learning from the training data with a student kernel KS , performance is evaluated via the generalization error . It is estimated as the fraction of correctly predicted labels for data points belonging to a test set with ntest elements . In Fig . 1 we present the learning curves for ( binary ) MNIST and CIFAR10 , for regression and classification . Learning is performed both with a Gaussian kernel K ( x ) ∝ exp ( −||x||2/ ( 2σ2 ) ) and a Laplace one K ( x ) ∝ exp ( −||x||/σ ) . Remarkably , the power laws in the two tasks are essentially identical ( although the estimated exponent appears to be slightly larger , in absolute value , for classification ) . Moreover , the two kernels display a very similar behavior , compatible with the same exponent : about −0.4 for MNIST and −0.1 for CIFAR10 . The presented data are for σ = 1000 ; in Appendix B we show that the same behaviour is observed for different values .

Summary: This paper studies, empirically and theoretically, the learning rates of (shift-invariant) kernel learners in a misspecified setting. In the well-specified setting, the rate of kernel learners is at least $n^{-1/2}$, and in a misspecified setting assuming only Lipschitz targets, the rate is $n^{-1/d}$. Neither seems to match the experimental rate on MNIST and CIFAR-10; this paper proposes a theoretical model that can more-or-less match the experimental rate with essentially-reasonable assumptions.