Locality defeats the curse of dimensionality in convolutional teacher-student scenarios

1 Introduction . Deep Convolutional Neural Networks ( CNNs ) are widely recognised as the engine of the latest successes of deep learning methods , yet such a success is surprising . Indeed , any supervised learning model suffers in principle from the curse of dimensionality : under minimal assumptions on the function to be learnt , achieving a fixed target generalisation error requires a number of training samples P which grows exponentially with the dimensionality d of input data [ 1 ] , i.e . ( P ) ∼ P−1/d . Nonetheless , empirical evidence shows that the curse of dimensionality is beaten in practice [ 2 , 3 , 4 ] , with ( P ) ∼ P−β , β 1/d . ( 1 ) CNNs , in particular , achieve excellent performances on high-dimensional tasks such as image classification on ImageNet with state-of-the-art architectures , for which β ≈ [ 0.3 , 0.5 ] [ 2 ] . Natural data must then possess additional structures that make them learnable . A classical idea [ 5 ] ascribes the success of recognition systems to the compositionality of data , i.e . the fact that objects are made of features , themselves made of sub-features [ 6 , 7 , 8 ] . In this view , the locality of CNNs plays a key role for their performance , as supported by empirical observations [ 9 ] . Yet , there is no clear ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . analytical understanding of the relationship between the compositionality of the data and learning curves . In order to study this relationship quantitively , we introduce a teacher-student framework for kernel regression , where the function to be learnt takes one of the following two forms : fLC ( x ) = ∑ i∈P gi ( xi ) , f CN ( x ) = ∑ i∈P g ( xi ) . ( 2 ) Here , x is a d-dimensional input and xi denotes the i-th t-dimensional patch of x , xi = ( xi , . . . , xi+t−1 ) . i ranges in a subset P of { 1 , . . . , d } . The gi ’ s and g are random functions of t variables whose smoothness is controlled by some exponent αt . Such functions model the local nature of certain datasets and can be generated , for example , by randomly-initialised one-hidden-layer neural networks : fLC corresponds to a locally connected network ( LCN ) [ 10 , 11 ] , in which the input is split into lower-dimensional patches before being processed , whereas a network enforcing invariance with respect to shifts of the input patches via weight sharing can be described by fCN . In such cases t would be the filter size of the network . Our goal is to compute the asymptotic decay of the error of a student kernel performing regression on such data , and to relate the corresponding exponent β to the locality of the target function . The student kernel corresponds to a prior on the true function of the form described by Eq . ( 2 ) , except that the filter size s and its prior αs on the smoothness of the g functions can differ from those of the target function . Such students include overparametrised one-hidden-layer neural networks operating in the lazy training regime [ 12 , 13 , 14 , 15 , 16 ] . 1.1 Our contributions . We consider a teacher-student framework for kernel regression , where the target function has one of the forms in Eq . ( 2 ) , where the gi ’ s and g are Gaussian random fields of given covariance . Target functions are characterised by the dimensionality t of the g functions—the filter size—and a smoothness exponent αt , such that αt > 2n implies that typical target functions are at least n times differentiable . Kernel regression is performed by local or convolutional student kernels , having filter size s and a prior on the target smoothness characterised by another exponent αs > 0 . Our main contributions follow : ◦ We use recent results based on the replica method of statistical physics on the generalisation error of kernel methods [ 17 , 18 , 19 ] to estimate the exponent β . We find that β = αt/s if t ≤ s and αt ≤ 2 ( αs + s ) . This approach is non-rigorous , but it can be proven if data are sampled on a lattice [ 4 ] and corresponds to a provable lower bound on the error when teacher and student are equal [ 20 ] . ◦ In particular , we find the same exponent for students with a prior on the shift invariance of the target function and students without this prior , implying that the curse of dimensionality is beaten due to locality and not shift invariance . ◦ We confirm systematically our predictions by performing kernel ridgeless regression numerically for various t , s and embedding dimension d. ◦ We use the recent framework of [ 21 ] and a natural Gaussian universality assumption to prove a rigorous estimate of β in the case where the ridge decreases with the size of the training set . The estimate of β depends again on s and not on d , demonstrating that the curse of dimensionality can indeed be beaten by using local filters on such compositional data . 1.2 Related work . Several recent works study the role of the compositional structure of data [ 6 , 22 , 23 ] . When such structure is hierarchical , deep convolutional networks can be much more expressive than shallow ones [ 6 , 24 , 7 ] . Concerning training , [ 25 ] shows that both convolutional and locally-connected networks can achieve a target generalisation error in polynomial time , whereas fully-connected networks can not , for a class of functions which depend only on s consecutive bits of the d-dimensional input , with s=O ( log d ) . In [ 8 ] the effects of the architecture ’ s locality are studied from a kernel perspective , using a class of deep convolutional kernels introduced in [ 26 , 27 ] and characterising their Reproducing Kernel Hilbert Space ( RKHS ) . In general , belonging to the RKHS ensures favourable bounds on performance and , for isotropic kernels , is a constraint on the function smoothness that becomes stringent in large d. For local functions , the corresponding constraint on smoothness is governed by the filter size s and not d [ 8 ] . Lastly , a recent work shows that weight sharing , in the absence of locality , leads to a mild improvement of the generalisation error of shift-invariant kernels [ 28 ] . By contrast , our work focuses on computing non-trivial training curve exponents in a setup where the locality and shift-invariance priors of the kernel can differ from those of the class of functions being learnt . In our setup , the latter are in general not in the RKHS of the kernel2 . Technically , our result that the size of the student filter s controls the learning curve ( and not that of the teacher t ) relates to the fact that kernels are not able to detect data anisotropy ( the fact that the function depends only on a subset of the coordinates ) in worst-case settings [ 30 ] nor in the typical case for Gaussian fields [ 31 ] . 2 Setup . Kernel ridge regression Kernel ridge regression is a method to learn a target function f∗ : Rd → R from P observations { ( xµ , f∗ ( xµ ) ) } Pµ=1 , where the inputs xµ are i.i.d . random variables distributed according to a certain measure p ( ddx ) on Rd . Let K be a positive-definite kernel and H the corresponding Reproducing Kernel Hilbert Space ( RKHS ) . The kernel ridge regression estimator f of the target function f∗ is defined as f = argmin f∈H { 1 P P∑ µ=1 ( f ( xµ ) − f∗ ( xµ ) ) 2 + λ ‖f‖2H } , ( 3 ) where ‖ · ‖H denotes the RKHS norm and λ is the ridge parameter . The limit λ→ 0+ is known as the ridgeless case and corresponds to the solution with minimum RKHS norm that interpolates the P observations . Eq . ( 3 ) is a convex optimisation problem , having the unique solution f ( x ) = 1 P P∑ µ , ν=1 K ( x , xµ ) ( ( 1 P KP + λIP ) −1 ) µ , ν f∗ ( xν ) , ( 4 ) where KP is the Gram matrix defined as ( KP ) µν = K ( xµ , xν ) , and IP denotes the P -dimensional identity matrix . Our goal is to compute the generalisation error , which we define as the expectation of the mean squared error over the data distribution p ( ddx ) , averaged over an ensemble of target functions f∗ , i.e ( P ) = Ex , f∗ [ ( f ( x ) − f∗ ( x ) ) 2 ] . ( 5 ) The error depends on the number of samples P through the predictor of Eq . ( 4 ) and we refer to the graph of ( P ) as learning curve . Statistical mechanics of generalisation in kernel regression The theoretical understanding of generalisation is still an open problem . A few recent works [ 17 , 21 , 18 ] relate the generalisation error to the decomposition of the target function in the eigenbasis of the kernel . A positive-definite kernel K can indeed be written , by Mercer ’ s theorem , in terms of its eigenvalues { λρ } and eigenfunctions { φρ } : K ( x , y ) = ∞∑ ρ=1 λρφρ ( x ) φρ ( y ) , ∫ p ( ddy ) K ( x , y ) φρ ( y ) = λρφρ ( x ) . ( 6 ) In [ 17 , 21 , 18 ] it is shown that , when the target function can be written in terms of the kernel eigenbasis , f∗ ( x ) = ∑ ρ cρφρ ( x ) , ( 7 ) the error can also be cast as a sum of modal contributions , = ∑ ρ ρ . The details of the general formulation are summarised in Appendix A . Here we present an intuitive limiting case , obtained in the ridgeless limit λ→ 0+ , when λρ ∼ ρ−a for large ρ , and E [ |cρ|2 ] ∼ ρ−b with 2a > b− 1 , that is ( P ) ∼ ∑ ρ > P E [ |cρ|2 ] ≡ B ( P ) , ( 8 ) 2A Gaussian field of covariance K is never in the RKHS of the kernel K , see e.g . [ 29 ] . with ∼ denoting asymptotic equivalence for large P . Eq . ( 8 ) indicates that , given P examples , the generalisation error can be estimated as the tail sum of the power in the target function past the first P modes of the kernel , which we denote as B ( P ) . Although the general modal decomposition can not be proven rigorously in the ridgeless limit [ 21 , 19 ] , additional results are available when the target functions are Gaussian random fields with covariance specified by a teacher kernel : ◦ Eq . ( 8 ) can be proven rigorously [ 4 ] if teacher and student are isotropic kernels and the input points xµ are sampled on the lattice Zd , i.e . all the elements of each input sequence are integer multiples of an arbitrary unit ; ◦ If teacher and student coincide then E [ |cρ|2 ] equals the ρ-th eigenvalue λρ and ( see e.g . [ 20 ] ) ( P ) ≥ B ( P ) , i.e . the estimate of Eq . ( 8 ) is a lower bound .

Summary: In this paper, the authors study the generalization error of kernel regression in a teacher-student scenario where the kernels are obtained as NTK of either one-hidden layer locally-connected NNs or convolutional NNs. They consider the teacher as a Gaussian random field of covariance the same kernel as the student kernel but with different parameters, and proceed to characterize the learning curve exponents. Based on a decomposition of the two kernels in orthonormal basis functions, they use recent (non-rigorous) statistical physics predictions to deduce an exponent that can be shown rigorously to be an upper bound in some cases. This exponent does not depend on the ambient dimension d, only on the patch size, and is the same for the two kernels. The authors deduce that locality is the important property that breaks the curse of dimensionality and notice that adding the invariance by translation only lower the prefactor.  Finally, using a Gaussian universality assumption and a recent framework, they prove a lower bound on the learning curve exponent.  They further check their predictions in numerical experiments on synthetic data.