Locality defeats the curse of dimensionality in convolutional teacher-student scenarios

1 Introduction . Deep Convolutional Neural Networks ( CNNs ) are widely recognised as the engine of the latest successes of deep learning methods , yet such a success is surprising . Indeed , any supervised learning model suffers in principle from the curse of dimensionality : under minimal assumptions on the function to be learnt , achieving a fixed target generalisation error requires a number of training samples P which grows exponentially with the dimensionality d of input data [ 1 ] , i.e . ( P ) ∼ P−1/d . Nonetheless , empirical evidence shows that the curse of dimensionality is beaten in practice [ 2 , 3 , 4 ] , with ( P ) ∼ P−β , β 1/d . ( 1 ) CNNs , in particular , achieve excellent performances on high-dimensional tasks such as image classification on ImageNet with state-of-the-art architectures , for which β ≈ [ 0.3 , 0.5 ] [ 2 ] . Natural data must then possess additional structures that make them learnable . A classical idea [ 5 ] ascribes the success of recognition systems to the compositionality of data , i.e . the fact that objects are made of features , themselves made of sub-features [ 6 , 7 , 8 ] . In this view , the locality of CNNs plays a key role for their performance , as supported by empirical observations [ 9 ] . Yet , there is no clear ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . analytical understanding of the relationship between the compositionality of the data and learning curves . In order to study this relationship quantitively , we introduce a teacher-student framework for kernel regression , where the function to be learnt takes one of the following two forms : fLC ( x ) = ∑ i∈P gi ( xi ) , f CN ( x ) = ∑ i∈P g ( xi ) . ( 2 ) Here , x is a d-dimensional input and xi denotes the i-th t-dimensional patch of x , xi = ( xi , . . . , xi+t−1 ) . i ranges in a subset P of { 1 , . . . , d } . The gi ’ s and g are random functions of t variables whose smoothness is controlled by some exponent αt . Such functions model the local nature of certain datasets and can be generated , for example , by randomly-initialised one-hidden-layer neural networks : fLC corresponds to a locally connected network ( LCN ) [ 10 , 11 ] , in which the input is split into lower-dimensional patches before being processed , whereas a network enforcing invariance with respect to shifts of the input patches via weight sharing can be described by fCN . In such cases t would be the filter size of the network . Our goal is to compute the asymptotic decay of the error of a student kernel performing regression on such data , and to relate the corresponding exponent β to the locality of the target function . The student kernel corresponds to a prior on the true function of the form described by Eq . ( 2 ) , except that the filter size s and its prior αs on the smoothness of the g functions can differ from those of the target function . Such students include overparametrised one-hidden-layer neural networks operating in the lazy training regime [ 12 , 13 , 14 , 15 , 16 ] . 1.1 Our contributions . We consider a teacher-student framework for kernel regression , where the target function has one of the forms in Eq . ( 2 ) , where the gi ’ s and g are Gaussian random fields of given covariance . Target functions are characterised by the dimensionality t of the g functions—the filter size—and a smoothness exponent αt , such that αt > 2n implies that typical target functions are at least n times differentiable . Kernel regression is performed by local or convolutional student kernels , having filter size s and a prior on the target smoothness characterised by another exponent αs > 0 . Our main contributions follow : ◦ We use recent results based on the replica method of statistical physics on the generalisation error of kernel methods [ 17 , 18 , 19 ] to estimate the exponent β . We find that β = αt/s if t ≤ s and αt ≤ 2 ( αs + s ) . This approach is non-rigorous , but it can be proven if data are sampled on a lattice [ 4 ] and corresponds to a provable lower bound on the error when teacher and student are equal [ 20 ] . ◦ In particular , we find the same exponent for students with a prior on the shift invariance of the target function and students without this prior , implying that the curse of dimensionality is beaten due to locality and not shift invariance . ◦ We confirm systematically our predictions by performing kernel ridgeless regression numerically for various t , s and embedding dimension d. ◦ We use the recent framework of [ 21 ] and a natural Gaussian universality assumption to prove a rigorous estimate of β in the case where the ridge decreases with the size of the training set . The estimate of β depends again on s and not on d , demonstrating that the curse of dimensionality can indeed be beaten by using local filters on such compositional data . 1.2 Related work . Several recent works study the role of the compositional structure of data [ 6 , 22 , 23 ] . When such structure is hierarchical , deep convolutional networks can be much more expressive than shallow ones [ 6 , 24 , 7 ] . Concerning training , [ 25 ] shows that both convolutional and locally-connected networks can achieve a target generalisation error in polynomial time , whereas fully-connected networks can not , for a class of functions which depend only on s consecutive bits of the d-dimensional input , with s=O ( log d ) . In [ 8 ] the effects of the architecture ’ s locality are studied from a kernel perspective , using a class of deep convolutional kernels introduced in [ 26 , 27 ] and characterising their Reproducing Kernel Hilbert Space ( RKHS ) . In general , belonging to the RKHS ensures favourable bounds on performance and , for isotropic kernels , is a constraint on the function smoothness that becomes stringent in large d. For local functions , the corresponding constraint on smoothness is governed by the filter size s and not d [ 8 ] . Lastly , a recent work shows that weight sharing , in the absence of locality , leads to a mild improvement of the generalisation error of shift-invariant kernels [ 28 ] . By contrast , our work focuses on computing non-trivial training curve exponents in a setup where the locality and shift-invariance priors of the kernel can differ from those of the class of functions being learnt . In our setup , the latter are in general not in the RKHS of the kernel2 . Technically , our result that the size of the student filter s controls the learning curve ( and not that of the teacher t ) relates to the fact that kernels are not able to detect data anisotropy ( the fact that the function depends only on a subset of the coordinates ) in worst-case settings [ 30 ] nor in the typical case for Gaussian fields [ 31 ] . 2 Setup . Kernel ridge regression Kernel ridge regression is a method to learn a target function f∗ : Rd → R from P observations { ( xµ , f∗ ( xµ ) ) } Pµ=1 , where the inputs xµ are i.i.d . random variables distributed according to a certain measure p ( ddx ) on Rd . Let K be a positive-definite kernel and H the corresponding Reproducing Kernel Hilbert Space ( RKHS ) . The kernel ridge regression estimator f of the target function f∗ is defined as f = argmin f∈H { 1 P P∑ µ=1 ( f ( xµ ) − f∗ ( xµ ) ) 2 + λ ‖f‖2H } , ( 3 ) where ‖ · ‖H denotes the RKHS norm and λ is the ridge parameter . The limit λ→ 0+ is known as the ridgeless case and corresponds to the solution with minimum RKHS norm that interpolates the P observations . Eq . ( 3 ) is a convex optimisation problem , having the unique solution f ( x ) = 1 P P∑ µ , ν=1 K ( x , xµ ) ( ( 1 P KP + λIP ) −1 ) µ , ν f∗ ( xν ) , ( 4 ) where KP is the Gram matrix defined as ( KP ) µν = K ( xµ , xν ) , and IP denotes the P -dimensional identity matrix . Our goal is to compute the generalisation error , which we define as the expectation of the mean squared error over the data distribution p ( ddx ) , averaged over an ensemble of target functions f∗ , i.e ( P ) = Ex , f∗ [ ( f ( x ) − f∗ ( x ) ) 2 ] . ( 5 ) The error depends on the number of samples P through the predictor of Eq . ( 4 ) and we refer to the graph of ( P ) as learning curve . Statistical mechanics of generalisation in kernel regression The theoretical understanding of generalisation is still an open problem . A few recent works [ 17 , 21 , 18 ] relate the generalisation error to the decomposition of the target function in the eigenbasis of the kernel . A positive-definite kernel K can indeed be written , by Mercer ’ s theorem , in terms of its eigenvalues { λρ } and eigenfunctions { φρ } : K ( x , y ) = ∞∑ ρ=1 λρφρ ( x ) φρ ( y ) , ∫ p ( ddy ) K ( x , y ) φρ ( y ) = λρφρ ( x ) . ( 6 ) In [ 17 , 21 , 18 ] it is shown that , when the target function can be written in terms of the kernel eigenbasis , f∗ ( x ) = ∑ ρ cρφρ ( x ) , ( 7 ) the error can also be cast as a sum of modal contributions , = ∑ ρ ρ . The details of the general formulation are summarised in Appendix A . Here we present an intuitive limiting case , obtained in the ridgeless limit λ→ 0+ , when λρ ∼ ρ−a for large ρ , and E [ |cρ|2 ] ∼ ρ−b with 2a > b− 1 , that is ( P ) ∼ ∑ ρ > P E [ |cρ|2 ] ≡ B ( P ) , ( 8 ) 2A Gaussian field of covariance K is never in the RKHS of the kernel K , see e.g . [ 29 ] . with ∼ denoting asymptotic equivalence for large P . Eq . ( 8 ) indicates that , given P examples , the generalisation error can be estimated as the tail sum of the power in the target function past the first P modes of the kernel , which we denote as B ( P ) . Although the general modal decomposition can not be proven rigorously in the ridgeless limit [ 21 , 19 ] , additional results are available when the target functions are Gaussian random fields with covariance specified by a teacher kernel : ◦ Eq . ( 8 ) can be proven rigorously [ 4 ] if teacher and student are isotropic kernels and the input points xµ are sampled on the lattice Zd , i.e . all the elements of each input sequence are integer multiples of an arbitrary unit ; ◦ If teacher and student coincide then E [ |cρ|2 ] equals the ρ-th eigenvalue λρ and ( see e.g . [ 20 ] ) ( P ) ≥ B ( P ) , i.e . the estimate of Eq . ( 8 ) is a lower bound .

Summary: # Post Rebuttal:  I partly accept the claims of the authors in the rebuttal and have decided to raise my score a little bit to reflect that. I still believe the paper would benefit greatly from clarifications. In particular, I advise the authors to incorporate the changes mentioned in their response to Remarks 1-2 in the rebuttal, as these strengthen the paper in my opinion. As for Remarks 3-4, I disagree with the authors' claim; my point that convolutional neural networks do not provide any benefit over other simpler hypothesis classes in the regime of lazy training still stands and I strongly encourage the authors to omit any reference to neural networks as motivation for studying the problem. I find this reference to be too artificial and somewhat forced.  Lastly, I would like to wish the authors the best of luck in the future.    # Original Review:  The paper studies the expected generalization error when learning in a teacher-student setting where the teacher is a Gaussian random field and the student is a kernel exhibiting a certain convolutional structure. It is shown that under strong assumptions the expected generalization error decays at a rate which does not depend on the input dimension but rather on the filter size of the student. Additionally, experiments are performed to corroborate the theoretical findings in the paper.