Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm

1 INTRODUCTION . In supervised learning machines learn from a finite collection of n training data , and their generalization error is then evaluated on unseen data drawn from the same distribution . How many data are needed to learn a task is characterized by the learning curve relating generalization error to n. In various cases , the generalization error decays as a power law n−β , with an exponent β that depends on both the data and the algorithm . In ( Hestness et al. , 2017 ) β is reported for state-of-the-art ( SOTA ) deep neural networks for various tasks : in for neural-machine translation β ≈ 0.3–0.36 ( for fixed model size ) or β ≈ 0.13 ( for best-fit models at any n ) ; language modeling shows β ≈ 0.06–0.09 ; in speech recognition β ≈ 0.3 ; SOTA models for image classification ( on ImageNet ) have exponents β ≈ 0.3–0.5 . Currently there is no available theory of deep learning to rationalize these observations . Recently it was shown that for a proper initialization of the weights , deep learning in the infinite-width limit ( Jacot et al. , 2018 ) converges to kernel learning . Moreover , it is nowadays part of the lore that there exist kernels whose performance is nearly comparable to deep networks ( Bruna and Mallat , 2013 ; Arora et al. , 2019 ) , at least for some tasks . It is thus of great interest to understand the learning curves of kernels . For regression , if the target function being learned is simply assumed to be Lipschitz , then the best guarantee is β = 1/d ( Luxburg and Bousquet , 2004 ; Bach , 2017 ) where d is the data dimension . Thus for large d , β is very small : learning is completely inefficient , a phenomenon referred to as the curse of dimensionality . As a result , various works on kernel regression make the much stronger assumption that the training points are sampled from a target function that belongs to the reproducing kernel Hilbert space ( RKHS ) of the kernel ( see for example ( Smola et al. , 1998 ) ) . With this assumption β does not depend on d ( for instance in ( Rudi and Rosasco , 2017 ) β = 1/2 is guaranteed ) . Yet , RKHS is a very strong assumption which requires the smoothness of the target function to increase with d ( Bach , 2017 ) ( see more on this point below ) , which may not be realistic in large dimensions . In this work we compute β empirically for kernel methods applied on MNIST and CIFAR10 datasets . We find βMNIST ≈ 0.4 and βCIFAR10 ≈ 0.1 respectively . Quite remarkably , we observe essentially the same exponents for regression and classification tasks , using either a Gaussian or a Laplace kernel . Thus the exponents are not as small as 1/d ( d = 784 for MNIST , d = 3072 for CIFAR10 ) , but neither are they 1/2 as one would expect under the RKHS assumption . These facts call for frameworks in which assumptions on the smoothness of the data can be intermediary between Lipschitz and RKHS . Here we propose such a framework for regression , in which the target function is assumed to be a Gaussian random field of zero mean with translation-invariant isotropic covariance KT ( x ) . The data can equivalently be thought as being synthesized by a “ Teacher ” kernelKT ( x ) . Learning is performed with a “ Student ” kernel KS ( x ) that minimizes the mean-square error . In general KT ( x ) 6= KS ( x ) . In this set-up learning is very similar to a technique referred to as kriging , or Gaussian process regression , originally developed in the geostatistics community ( Matheron , 1963 ; Stein , 1999b ) . To quantify learning , we first perform numerical experiments for data points distributed uniformly at random on a hypersphere of varying dimension d , focusing on a Laplace kernel for the Student , and considering a Laplace or Gaussian kernel for the Teacher . We observe that in both cases β ( d ) is a decreasing function . To derive β ( d ) we consider the simplified situation where the Gaussian random field is sampled at training points lying on a regular lattice . Building on the kriging literature ( Stein , 1999b ) , we show that β is controlled by the high-frequency scaling of both the Teacher and Student kernels : assuming that the Fourier transforms of the kernels decay as K̃T ( w ) = cT ||w||−αT + o ( ||w||−αT ) and K̃S ( w ) = cS ||w||−αS + o ( ||w||−αS ) , we obtain β = 1 d min ( αT − d , 2αS ) . ( 1 ) Importantly ( i ) Eq . ( 1 ) leads to a prediction for β ( d ) that accurately matches our numerical study for random training data points , leading to the conjecture that Eq . ( 1 ) holds in that case as well . We offer the following interpretation : ultimately , kernel methods are performing a local interpolation whose quality depends on the distance δ ( n ) between adjacent data points . δ ( n ) is asymptotically similar for random data or data sitting on a lattice . ( ii ) If the kernel KS is not too sensitive to high-frequencies , then learning is optimal as far as scaling is concerned and β = ( αT − d ) /d . We will argue that the smoothness index s ≡ [ ( αT − d ) /2 ] characterizes the number of derivatives of the target function that are continuous . We thus recover the curse of dimensionality : s needs to be of order d to have non-vanishing β in large dimensions . Point ( ii ) leads to an apparent paradox : β is significant for MNIST and CIFAR10 , for which d is a priori very large , leading to a smoothness value s in the hundreds in both cases , which appears unrealistic . The paradox is resolved by considering that real datasets actually live on lower-dimensional manifolds . As far as kernel learning is concerned , our findings support that the correct definition of dimension should be based on how the nearest-neighbors distance δ ( n ) scales with n : δ ( n ) ∼ n−1/deff . Direct measurements of δ ( n ) support that MNIST and CIFAR10 live on manifolds of lower dimensions deffMNIST ≈ 15 and deffCIFAR10 ≈ 35 . Considering the effective dimensions that we find , the observed values for β would be obtained for Gaussian fields of smoothness sMNIST ≈ 3 and sCIFAR10 ≈ 1 , values that appear intuitively more reasonable . More generally this analogy with Gaussian fields allows one to associate a smoothness index s to any dataset once β and deff are measured , which may turn out to be a useful characterization of data complexity in the future . 2 RELATED WORKS . Our set-up of Teacher-Student learning with kernels is also referred to as kriging , or Gaussian process regression , and it was originally developed in the geostatistics community ( Matheron , 1963 ) . In Section 5 we present a theorem that allows one to know the rate at which the test error decreases as we increase the number of training points , assumed to lie on a high-dimensional regular lattice . Similar results have been previously derived in the kriging literature ( Stein , 1999b ) when sampling occurs on the regular lattice with the exception of the origin , where the inference is made . Here we propose an alternative derivation that some readers might find simpler . We also study a slightly different problem : instead of computing the test error when the inference is carried on at the origin , we compute the average error for a test point that lie at an arbitrary point , sampled uniformly at random and not necessarily on the lattice . In what follows we show , via extensive numerical simulations , that such predictions are accurate even when the training points do not lie on a regular lattice , but are taken at random on a hypersphere . An exact proof of our result in such a general setting is difficult and can not be found even in the kriging literature . To our knowledge the results that get closer to the point are those discussed in ( Stein , 1999a ) , where the author studies one-dimensional processes where the training data are not necessarily evenly spaced . In this work the effective dimension of the data plays an import role , as it controls how the distance between nearest neighbors scales with the dataset size . Of course , there exists a vast literature ( Grassberger and Procaccia , 1983 ; Costa and Hero , 2004 ; Hein and Audibert , 2005 ; Levina and Bickel , 2005 ; Rozza et al. , 2012 ; Facco et al. , 2017 ; Allegra et al. , 2019 ) devoted to the study of effective dimensions , where other definitions are analyzed . The effective dimensions that we find are compatible with those obtained with more refined methods . 3 LEARNING CURVE FOR KERNEL METHODS APPLIED TO REAL DATA . In what follows we apply kernel methods to the MNIST and CIFAR10 datasets , each consisting of a set of images ( xµ ) n µ=1 . We simplify the problem by considering only two classes whose label Z ( xµ ) = ±1 correspond to odd and even numbers for MNIST , and to two groups of 5 classes in CIFAR10 . The goal is to infer the value of the label ẐS ( x ) of an image x that does not belong to the dataset . The S subscript reminds us that inference is performed using a positive definite kernel KS . We perform inference in both a regression and a classification setting . The following algorithms and associated results can be found in ( Scholkopf and Smola , 2001 ) . Regression . Learning corresponds to minimizing a mean-square error : min n∑ µ=1 [ ẐS ( xµ ) − Z ( xµ ) ] 2 . ( 2 ) For algorithms seeking solutions of the form ẐS ( x ) = ∑ µ aµKS ( xµ , x ) ≡ a · kS ( x ) by minimizing the man-square loss over the vector a , one obtains : ẐS ( x ) = kS ( x ) ·K−1S Z , ( 3 ) where the vector Z contains all the labels in the training set , Z ≡ ( Z ( xµ ) ) nµ=1 , and KS , µν ≡ KS ( xµ , xν ) is the Gram matrix . The Gram matrix is always invertible if the kernel KS is positive definite . The generalization error is then evaluated as the expected mean-square error on unseen data , estimated by averaging over a test set composed of ntest unseen data points : MSE = 1 ntest ntest∑ µ=1 [ ẐS ( xµ ) − Z ( xµ ) ] 2 . ( 4 ) Classification . We perform kernel classification via the algorithm soft-margin SVM . The details can be found in Appendix A . After learning from the training data with a student kernel KS , performance is evaluated via the generalization error . It is estimated as the fraction of correctly predicted labels for data points belonging to a test set with ntest elements . In Fig . 1 we present the learning curves for ( binary ) MNIST and CIFAR10 , for regression and classification . Learning is performed both with a Gaussian kernel K ( x ) ∝ exp ( −||x||2/ ( 2σ2 ) ) and a Laplace one K ( x ) ∝ exp ( −||x||/σ ) . Remarkably , the power laws in the two tasks are essentially identical ( although the estimated exponent appears to be slightly larger , in absolute value , for classification ) . Moreover , the two kernels display a very similar behavior , compatible with the same exponent : about −0.4 for MNIST and −0.1 for CIFAR10 . The presented data are for σ = 1000 ; in Appendix B we show that the same behaviour is observed for different values .

Summary: This paper experimentally investigates how fast the generalization error decreases when some specific kernel functions are used in real datasets. This paper conducted numerical experiments on several datasets to investigate the decreasing rate of the generalization error, and the rate is determined for such datasets. This decreasing rate is theoretically analyzed by using the approximation theory of RKHS in the teacher-student setting. It is shown that the rate is determined with the smoothness and effective dimensionality of input. Then, the smoothness of the teacher function is also derived through this analysis.