Approximation and Learning with Deep Convolutional Models: a Kernel Perspective

1 INTRODUCTION . Deep convolutional models have been at the heart of the recent successes of deep learning in problems where the data consists of high-dimensional signals , such as image classification or speech recognition . Convolution and pooling operations have notably contributed to the practical success of these models , yet our theoretical understanding of how they enable efficient learning is still limited . One key difficulty for understanding such models is the curse of dimensionality : due to the highdimensionality of the input data , it is hopeless to learn arbitrary functions from samples . For instance , classical non-parametric regression techniques for learning generic target functions typically require either low dimension or very high degrees of smoothness in order to obtain good generalization ( e.g. , Wainwright , 2019 ) , which makes them impractical for dealing with high-dimensional signals . Thus , further assumptions on the target function are needed to make the problem tractable , and we seek assumptions that make convolutions a useful modeling tool . Various works have studied approximation benefits of depth with models that resemble deep convolutional architectures ( Cohen & Shashua , 2017 ; Mhaskar & Poggio , 2016 ; Schmidt-Hieber et al. , 2020 ) . Nevertheless , while such function classes may provide improved statistical efficiency in theory , it is unclear if there exist efficient algorithms to learn such models , and hence , whether they might correspond to what convolutional networks learn in practice . To overcome this issue , we consider instead function classes based on kernel methods ( Schölkopf & Smola , 2001 ; Wahba , 1990 ) , which are known to be learnable with efficient ( polynomial-time ) algorithms , such as kernel ridge regression or gradient descent . We consider “ deep ” structured kernels known as convolutional kernels , which yield good empirical performance on standard computer vision benchmarks ( Li et al. , 2019 ; Mairal , 2016 ; Mairal et al. , 2014 ; Shankar et al. , 2020 ) , and are related to over-parameterized convolutional networks ( CNNs ) in so-called “ kernel regimes ” ( Arora et al. , 2019 ; Bietti & Mairal , 2019b ; Daniely et al. , 2016 ; GarrigaAlonso et al. , 2019 ; Jacot et al. , 2018 ; Novak et al. , 2019 ; Yang , 2019 ) . Such regimes may be seen as providing a first-order description of what common deep models trained with gradient methods may learn . Studying the corresponding function spaces ( reproducing kernel Hilbert spaces , or RKHS ) may then provide insight into the benefits of various architectural choices . For fully-connected architectures , such kernels are rotation-invariant , and the corresponding RKHSs are well understood in terms of regularity properties on the sphere ( Bach , 2017a ; Smola et al. , 2001 ) , but do not show any major differences between deep and shallow kernels ( Bietti & Bach , 2021 ; Chen & Xu , 2021 ; Geifman et al. , 2020 ) . In contrast , in this work we show that even in the kernel setting , multiple layers of convolution and pooling operations can be crucial for efficient learning of functions with specific structures that are well-suited for natural signals . Our work paves the way for further studies of the inductive bias of optimization algorithms on deep convolutional networks beyond kernel regimes , for instance by incorporating adaptivity to low-dimensional structure ( Bach , 2017a ; Chizat & Bach , 2020 ; Wei et al. , 2019 ) or hierarchical learning ( Allen-Zhu & Li , 2020 ) . We make the following contributions : • We revisit convolutional kernel networks ( Mairal , 2016 ) , finding that simple two or three layers models with Gaussian pooling and polynomial kernels of degree 2-4 at higher layers provide competitive performance with state-of-the-art convolutional kernels such as Myrtle kernels ( Shankar et al. , 2020 ) on Cifar10 . • For such kernels , we provide an exact description of the RKHS functions and their norm , illustrating representation benefits of multiple convolutional and pooling layers for capturing additive and interaction models on patches with certain spatial regularities among interaction terms . • We provide generalization bounds that illustrate the benefits of architectural choices such as pooling and patches for learning additive interaction models with spatial invariance in the interaction terms , namely , improvements in sample complexity by polynomial factors in the size of the input signal . Related work . Convolutional kernel networks were introduced by Mairal et al . ( 2014 ) ; Mairal ( 2016 ) . Empirically , they used kernel approximations to improve computational efficiency , while we evaluate the exact kernels in order to assess their best performance , as in ( Arora et al. , 2019 ; Li et al. , 2019 ; Shankar et al. , 2020 ) . Bietti & Mairal ( 2019a ; b ) show invariance and stability properties of its RKHS functions , and provide upper bounds on the RKHS norm for some specific functions ( see also Zhang et al. , 2017 ) ; in contrast , we provide exact characterizations of the RKHS norm , and study generalization benefits of certain architectures . Scetbon & Harchaoui ( 2020 ) study statistical properties of simple convolutional kernels without pooling , while we focus on the role of architecture choices with an emphasis on pooling . Cohen & Shashua ( 2016 ; 2017 ) ; Mhaskar & Poggio ( 2016 ) ; Poggio et al . ( 2017 ) study expressivity and approximation with models that resemble CNNs , showing benefits thanks to hierarchy or local interactions , but such models are not known to be learnable with tractable algorithms , while we focus on ( tractable ) kernels . Regularization properties of convolutional models were also considered in ( Gunasekar et al. , 2018 ; Heckel & Soltanolkotabi , 2020 ) , but in different regimes or architectures than ours . Li et al . ( 2021 ) ; Malach & Shalev-Shwartz ( 2021 ) study benefits of convolutional networks with efficient algorithms , but do not study the gains of pooling . Du et al . ( 2018 ) study sample complexity of learning CNNs , focusing on parametric rather than non-parametric models . Mei et al . ( 2021 ) study statistical benefits of global pooling for learning invariant functions , but only consider one layer with full-size patches . Concurrently to our work , Favero et al . ( 2021 ) study benefits of local patches , but focus on one-layer models . 2 DEEP CONVOLUTIONAL KERNELS . In this section , we recall the construction of multi-layer convolutional kernels on discrete signals , following most closely the convolutional kernel network ( CKN ) architectures studied by Mairal ( 2016 ) ; Bietti & Mairal ( 2019a ) . These architectures rely crucially on pooling layers , typically with Gaussian filters , which make them empirically effective even with just two convolutional layers . These kernels define function spaces that will be the main focus of our theoretical study of approximation and generalization in the next sections . In particular , when learning a target function of the form f∗ ( x ) = ∑ i fi ( x ) , we will show that they are able to efficiently exploit two useful properties of f ∗ : ( locality ) each fi may depend on only one or a few small localized patches of the signal ; ( invariance ) many different terms fi may involve the same function applied to different input patches . We provide further background and motivation in Appendix A . For simplicity , we will focus on discrete 1D input signals , though one may easily extend our results to 2D or higher-dimensional signals . We will assume periodic signals in order to avoid difficulties with border effects , or alternatively , a cyclic domain Ω = Z/|Ω|Z . A convolutional kernel of depth L may then be defined for input signals x , x′ ∈ L2 ( Ω , Rp ) by KL ( x , x′ ) = 〈Ψ ( x ) , Ψ ( x′ ) 〉 , through the explicit feature map Ψ ( x ) = ALMLPL · · ·A1M1P1x . ( 1 ) Here , P ` , M ` and A ` are linear or non-linear operators corresponding to patch extraction , kernel mapping and pooling , respectively , and are described below . They operate on feature maps in L2 ( Ω ` ) ( with Ω0 =Ω ) with values in different Hilbert spaces , starting fromH0 : = Rp , and are defined below . An illustration of this construction is given in Figure 1 . Patch extraction . Given a patch shape S ` ⊂ Ω ` −1 , such as S ` = [ −1 , 0 , 1 ] for one-dimensional patches of size 3 , the operator P ` is defined for x ∈ L2 ( Ω ` −1 , H ` −1 ) by1 P ` x [ u ] = ( x [ u+ v ] ) v∈S ` ∈ H |S ` | ` −1 . Kernel mapping . The operators M ` perform a nonlinear embedding of patches into a new Hilbert space using dot-product kernels . We consider homogeneous dot-product kernels given for z , z′ ∈ H|S ` | ` −1 by k ` ( z , z ′ ) = ‖z‖‖z′‖κ ` ( 〈z , z′〉 ‖z‖‖z′‖ ) = 〈ϕ ` ( z ) , ϕ ` ( z′ ) 〉H ` , ( 2 ) where ϕ ` : H|S ` | ` −1 → H ` is a feature map for the kernel . The kernel functions take the form κ ` ( u ) =∑ j≥0 bju j with bj ≥ 0 . This includes the exponential kernel κ ( u ) = eα ( u−1 ) ( i.e. , Gaussian kernel on the sphere ) and the arc-cosine kernel arising from random ReLU features ( Cho & Saul , 2009 ) , for which our construction is equivalent to that of the conjugate or NNGP kernel for an infinite-width random ReLU network with the same architecture . The operator M ` is then defined pointwise by M ` x [ u ] = ϕ ` ( x [ u ] ) . ( 3 ) At the first layer on image patches , these kernels lead to functional spaces consisting of homogeneous functions with varying degrees of smoothness on the sphere , depending on the properties of the kernel ( Bach , 2017a ; Smola et al. , 2001 ) . At higher layers , our theoretical analysis will also consider simple polynomial kernels such as k ` ( z , z′ ) = ( 〈z , z′〉 ) r , in which case the feature map may be explicitly written in terms of tensor products . For instance , r = 2 gives ϕ ` ( z ) = z ⊗ z and H ` = ( H|S ` | ` −1 ) ⊗2 = ( H ` −1 ⊗ H ` −1 ) |S ` |×|S ` | . See Appendix A.2 for more background on dot-product kernels , their tensor products , and their regularization properties . Pooling . Finally , the pooling operators A ` perform local averaging through convolution with a filter h ` [ u ] , which we may consider to be symmetric ( h ` [ −u ] = : h̄ ` [ u ] =h ` [ u ] ) . In practice , the pooling operation is often followed by downsampling by a factor s ` , in which case the new signal A ` x is defined on a new domain Ω ` with |Ω ` | = |Ω ` −1|/s ` , and we may write for x ∈ L2 ( Ω ` −1 ) and u ∈ Ω ` , A ` x [ u ] = ∑ v∈Ω ` −1 h ` [ s ` u− v ] x [ v ] . ( 4 ) Our experiments consider Gaussian pooling filters with a size and bandwidth proportional to the downsampling factor s ` , following Mairal ( 2016 ) , namely , size 2s ` + 1 and bandwidth √ 2s ` . In Section 3 , we will often assume no downsampling for simplicity , in which case we may see the filter bandwidth as increasing with the layers . Links with other convolutional kernels . We note that our construction closely resembles kernels derived from infinitely wide convolutional networks , known as conjugate or NNGP kernels ( GarrigaAlonso et al. , 2019 ; Novak et al. , 2019 ) , and is also related to convolutional neural tangent kernels ( Arora et al. , 2019 ; Bietti & Mairal , 2019b ; Yang , 2019 ) . The Myrtle family of kernels ( Shankar et al. , 2020 ) also resembles our models , but they use small average pooling filters instead of Gaussian filters , which leads to deeper architectures due to smaller receptive fields .

Summary: In this paper, the authors study some simple convolutional kernels. They show that two or three layers convolutional kernels with polynomial kernels on the higher level and Gaussian pooling provide similar performance as the much more complicated state-of-the-art convolutional kernels (e.g., Myrtle kernel). Motivated by these good performance, they proceed to characterize the RKHS of these kernels, and describe how extra layers and pooling allows to capture interaction between patches with more or less spatial dependency. They then use the RKHS norm and some standard bound on the generalization error to show how choosing an architecture adapted to the target function can improve the statistical efficiency.