Learning with convolution and pooling operations in kernel methods

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have become essential elements of the deep learning toolbox , achieving state-of-the-art performance in many computer vision tasks ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) . CNNs are constructed by stacking convolution and pooling layers , which were shown to be paramount to their empirical success ( LeCun et al. , 2015 ) . A widely accepted hypothesis to explain their favorable properties is that these architectures successfully encode useful properties of natural images : locality and compositionality of the data , stability by local deformations , and translation invariance . While some theoretical progress has been made in studying the approximation and generalization benefits brought by convolution and pooling operations ( Cohen & Shashua , 2016a ; b ; Bietti , 2021 ) , our mathematical understanding of the interaction between network architecture , image distribution , and efficient learning remains limited . Consider x ∈ Rd an input signal , which we can think of as a grayscale pixel representation of an image . For mathematical convenience , we will consider one-dimensional images with cyclic convention xd+i : = xi , and denote x ( k ) = ( xk , xk+1 , . . . , xk+q−1 ) the k-th patch of the signal x , k ∈ [ d ] , with patch size q ≤ d. Most of our results can be extended to two-dimensional images . We further consider a simple convolutional neural network composed of a single convolution layer followed by local average pooling and downsampling . The network first computes the nonlinear convolution of N filters w1 , . . . , wN ∈ Rq with the image patches x ( k ) . The outputs of the convolution operation σ ( 〈wi , x ( k ) 〉 ) are then averaged locally over segments of length ω ( local average pooling ) . This pooling operation is followed by downsampling which extracts one out of every ∆ output coordinates ( for simplicity , ∆ is assumed to be a divisor of d ) . Finally , the results are combined linearly using coefficients ( aik ) i∈ [ N ] , k∈ [ d/∆ ] : fCNN ( x ; a , Θ ) = √ ∆ Nωd ∑ i∈ [ N ] ∑ k∈ [ d/∆ ] aik ∑ s∈ [ ω ] σ ( 〈wi , x ( k∆+s ) 〉 ) . ( CNN-AP-DS ) Note that pooling and downsampling operations are often tied together in the literature . However in this work we will treat these two operations separately . In the formula above , different values for q , ω , ∆ lead to different architectures with vastly different behaviors . For example , when q = ∆ = d and ω = 1 , we recover a two-layer fully-connected neural network fFC ( x ; a , Θ ) = N−1/2 ∑ i∈ [ N ] aiσ ( 〈wi , x〉 ) which has the universal approximation property at large N . When ω = ∆ = 1 and q < d , the network is “ locally connected ” fLC ( x ; a , Θ ) = N −1/2∑ i∈ [ N ] , k∈ [ d ] aikσ ( 〈wi , x ( k ) 〉 ) , and not a universal approximator anymore : however , fLC vastly outperforms fFC in some cases ( Li et al. , 2020 ) . For ω > 1 , local pooling enables learning functions that are locally invariant by translations more efficiently than without pooling . For ω = d ( global pooling ) , the network only fits functions fully invariant by cyclic translations . The aim of this paper is to formalize and quantify the trade-off between the target function class and the statistical efficiency brought by these different architectures . As a concrete first step in this direction , we consider kernel models that are naturally associated with the convolutional neural networks ( CNN-AP-DS ) through the neural tangent kernel perspective ( Daniely et al. , 2016 ; Jacot et al. , 2018 ) . Kernel methods have the advantage of 1 ) being tractable—leaving the computational issue of learning CNNs aside ; 2 ) having well-understood approximation and generalization properties , which depends on the eigendecomposition of the kernel and the alignment between the target function and associated RKHS ( Caponnetto & De Vito , 2007 ; Wainwright , 2019 ) ( see Appendices B and C for background ) . While kernel models only describe neural networks in the lazy training regime ( Chizat et al. , 2019 ; Du et al. , 2019b ; a ; Allen-Zhu et al. , 2019 ; Zou et al. , 2018 ) and miss important properties of deep learning , such as feature learning , we see that architecture choice is already crucial to enable efficient learning of ‘ image-like ’ functions in the fixed-feature regime . Neural tangent kernels are obtained by linearizing the associated neural networks . Here we consider the tangent kernel associated to the network fCNN ( c.f . Appendix A.2 for a detailed derivation ) : HCKω , ∆ ( x , y ) = ∆ dω ∑ k∈ [ d/∆ ] ∑ s , s′∈ [ ω ] h ( 〈x ( k∆+s ) , y ( k∆+s′ ) 〉/q ) , ( CK-AP-DS ) where h : R → R is related to the activation function σ in ( CNN-AP-DS ) . As a linearization of CNNs , the kernel ( CK-AP-DS ) inherits some of the favorable properties of convolution , pooling , and downsampling operations . Indeed , a line of work ( Mairal et al. , 2014 ; Mairal , 2016 ; Arora et al. , 2019 ; Li et al. , 2019 ; Shankar et al. , 2020 ) showed that , though performing slightly worse than CNNs , such ( hierarchical ) convolutional kernels have empirically outperformed the former state-of-the-art kernels . For instance , these kernels achieved test accuracy around 87 % − 90 % on CIFAR-10 , against 79.6 % for the best former unsupervised feature-extraction method ( Coates et al. , 2011 ) ( currently , the state-of-the-art CNNs can achieve test accuracy 99 % ) . In this paper , we will further consider a stylized setting with input signal distribution x ∼ Unif ( Qd ) ( uniform distribution over Qd : = { −1 , +1 } d the discrete hypercube in d dimensions ) . This simple choice allows for a complete characterization of the eigendecomposition of HCKω , ∆ , thanks to all patches having same marginal distribution x ( k ) ∼ Unif ( Qq ) . We will be particularly interested in four specific choices of ( q , ω , ∆ ) in ( CK-AP-DS ) : HFC ( x , y ) = h ( 〈x , y〉/d ) , ( FC ) HCK ( x , y ) = 1 d ∑ k∈ [ d ] h ( 〈x ( k ) , y ( k ) 〉/q ) , ( CK ) HCKω ( x , y ) = 1 dω ∑ k∈ [ d ] ∑ s , s′∈ [ ω ] h ( 〈x ( k+s ) , y ( k+s′ ) 〉/q ) , ( CK-AP ) HCKGP ( x , y ) = 1 d ∑ k , k′∈ [ d ] h ( 〈x ( k ) , y ( k′ ) 〉/q ) . ( CK-GP ) These kernels are respectively the neural tangent kernels of a fully-connected network fFC ( FC ) , a convolutional network fLC ( CK ) , a convolutional network followed by local average pooling ( CKAP ) and a convolutional network followed by global pooling ( CK-GP ) . We will further be interested in ( CK-GP ) with patch size q = d , which we denote HFCGP : this corresponds to a convolutional kernel with full-size patches q = d , followed by global pooling . In this paper , we first characterize the reproducing kernel Hilbert space ( RKHS ) of these convolutional kernels , and then investigate their generalization properties in the regression setup . More specifically , assume { ( xi , yi ) } i≤n are n i.i.d . samples with xi ∼ Unif ( Qd ) and yi = f ? ( xi ) + εi . Here f ? ∈ L2 ( Qd ) and ( εi ) i≤n are independent errors with mean zero and variance bounded by σ2ε . We will focus on the generalization error of kernel ridge regression ( KRR ) ( see Appendix B.1 for general kernel methods ) . In particular , given a kernel function H : Qd × Qd → R and a regularization parameter λ ≥ 0 , the KRR estimator is the solution of the tractable convex problem f̂λ = arg min f∈H { ∑ i∈ [ n ] ( yi − f ( xi ) ) 2 + λ‖f‖2H } , ( KRR ) whereH is the RKHS associated toH with RKHS norm ‖ ·‖H . We denote the test error with square loss by R ( f ? , f̂λ ) = Ex { ( f ? ( x ) − f̂λ ( x ) ) 2 } . We will sometimes consider the expected test error Eε { R ( f ? , f̂λ ) } , where expectation is taken with respect to noise ε = ( εi ) i≤n in the training data . The generalization properties of the kernelsHFC andHFCGP were recently studied in Mei et al . ( 2021b ) ; Bietti et al . ( 2021 ) . In particular , they showed that global pooling ( kernel HFCGP ) leads to a gain of a factor d in sample complexity when fitting cyclic invariant functions , but still suffers from the curse of dimensionality ( HFCGP only fits very smooth functions in high-dimension ) . More precisely , Mei et al . ( 2021b ) considered the high-dimensional framework of Mei et al . ( 2021a ) and showed the following : KRR with HFC requires n ≈ d ` samples to fit degree- ` cyclic polynomials , while KRR with HFCGP only needs n ≈ d ` −1 . To enable milder dependence on the dimension d , further structural assumptions on the kernel and the target function should be considered ( for instance , in this paper , we use the kernel HCK and consider ‘ local ’ functions ) . 1.1 SUMMARY OF MAIN RESULTS . Our contributions are two-fold . First , we describe the RKHS associated to the convolutional kernel ( CK-AP-DS ) in the stylized setting x ∼ Unif ( Qd ) , which provides a fully explicit illustration of the roles of convolution , pooling and downsampling operations in learning specific classes of functions . Second , based on the eigendecomposition of the kernels , we provide generalization bounds on kernel ridge regression both in the fixed and high-dimension settings . In particular , we quantify the gain in sample complexity achieved by different architectures when learning target functions with corresponding structures . Let us define the q-local function class L2 ( Qd , Locq ) and the cyclic q-local function class L2 ( Qd , CycLocq ) ( subspace of L 2 ( Qd , Locq ) consisting of cyclic-invariant functions ) as follow : L2 ( Qd , Locq ) = { f ∈ L2 ( Qd ) : ∃ { gk } k∈ [ d ] ⊆ L2 ( Qq ) , f ( x ) = ∑ k∈ [ d ] gk ( x ( k ) ) } , ( LOC ) L2 ( Qd , CycLocq ) = { f ∈ L2 ( Qd ) : ∃g ∈ L2 ( Qq ) , f ( x ) = ∑ k∈ [ d ] g ( x ( k ) ) } . ( CYC-LOC ) We summarize below some of the insights that follow from our results . One-layer convolutional layer . The RKHS of HCK is constituted of q-local functions L2 ( Qd , Locq ) . Furthermore , the decay of the eigenvalues of HCK is controlled by a q-dimensional kernel , instead of a d-dimensional kernel for HFC . In particular , this implies that the test error of kernel ridge regression decays at rate n−O ( 1/q ) instead of n−O ( 1/d ) , when the target function is in L2 ( Qd , Locq ) . Hence , when q d , kernel methods with convolutional kernel HCK breaks the curse of dimensionality . Average pooling . The RKHS of HCKω is still constituted of q-local functions f ∈ L2 ( Qd , Locq ) , but penalizes differently the frequency components fj ( x ) by reweighting their eigenspaces by a factor κj , where fj ( x ) = ∑ k∈ [ d ] ρ k j f ( tk · x ) with ρj = e 2iπj d and tk · x = ( xk+1 , . . . , xd , x1 , . . . , xk ) denotes the k-shift . As ω increases , local pooling penalizes more and more heavily the high-frequency components ( κj 1 ) , while making lowfrequency components statistically easier to learn ( κj 1 ) . For global pooling ω = d , HCKGP only learns cyclic local functions L 2 ( Qd , CycLocq ) and enjoy a factor d gain in statistical complexity compared to HCK . Downsampling . When ∆ ≤ ω , downsampling after average pooling leaves the low-frequency eigenspaces of HCKω stable . In particular , the downsampling operation does not modify the statistical complexity of learning low-frequency functions in one-layer kernels , while being potentially beneficial in further layers in deep convolutional kernels . In Table 1 , we report our high-dimensional predictions for the sample complexity of learning q-local cyclic-invariant polynomials of degree ` . These results are proven within the framework of Mei et al . ( 2021a ) ( see Appendix C for background ) . The rest of the paper is organized as follows . We discuss related work in Section 1.2 . In Section 2 , we present our main results on convolutional kernels and describe precisely the roles of convolution , pooling and downsampling operations . We briefly mention the multilayer case in Section 2.4 . Finally , we present a numerical simulation on synthetic data in Section 3 and conclude in Section 4 . Some details and discussions are deferred to Appendix A .

Summary: This work proposes to study CKNs (defined via random features) which are defined via convolutions on “patches” (e.g., operators that writes (x,y)->\sum_k h(<x_k,y_k>) where $x_k,y_k$ are some patches of $x,y$), and to exploit the fact that those types of kernels are actually related to convolutions on this space, and that one can decompose them on an orthogonal basis of “zonal-like” Polynomials, h(<x,y>)=\sum_n P_n(<x,y>) where $\{P_n\}$ is an orthogonal basis s.t. P_n(<x,y>)=\sum_k Y_k(x)Y_k(y) are some “harmonics-like” functions. Then, one diagonalizes every kernel in this common basis, and all the next bounds exploit this linear structure and a fast decay of the eigen-values, which allows to derive Generalization errors based on this linear structure with a low-dimension. Overall, this is a standard framework (Bach, 2021, 2017) which has been well explained and cited across the text. The link with lazy training is done and the description of the space of L^2 patches(I prefer to phrase it this way rather than L^2 hypercube, I think it’s makes more clear what the authors try to achieve) is well-done, while very similar to L^2(S^{d-1}).