Learning with convolution and pooling operations in kernel methods

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have become essential elements of the deep learning toolbox , achieving state-of-the-art performance in many computer vision tasks ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) . CNNs are constructed by stacking convolution and pooling layers , which were shown to be paramount to their empirical success ( LeCun et al. , 2015 ) . A widely accepted hypothesis to explain their favorable properties is that these architectures successfully encode useful properties of natural images : locality and compositionality of the data , stability by local deformations , and translation invariance . While some theoretical progress has been made in studying the approximation and generalization benefits brought by convolution and pooling operations ( Cohen & Shashua , 2016a ; b ; Bietti , 2021 ) , our mathematical understanding of the interaction between network architecture , image distribution , and efficient learning remains limited . Consider x ∈ Rd an input signal , which we can think of as a grayscale pixel representation of an image . For mathematical convenience , we will consider one-dimensional images with cyclic convention xd+i : = xi , and denote x ( k ) = ( xk , xk+1 , . . . , xk+q−1 ) the k-th patch of the signal x , k ∈ [ d ] , with patch size q ≤ d. Most of our results can be extended to two-dimensional images . We further consider a simple convolutional neural network composed of a single convolution layer followed by local average pooling and downsampling . The network first computes the nonlinear convolution of N filters w1 , . . . , wN ∈ Rq with the image patches x ( k ) . The outputs of the convolution operation σ ( 〈wi , x ( k ) 〉 ) are then averaged locally over segments of length ω ( local average pooling ) . This pooling operation is followed by downsampling which extracts one out of every ∆ output coordinates ( for simplicity , ∆ is assumed to be a divisor of d ) . Finally , the results are combined linearly using coefficients ( aik ) i∈ [ N ] , k∈ [ d/∆ ] : fCNN ( x ; a , Θ ) = √ ∆ Nωd ∑ i∈ [ N ] ∑ k∈ [ d/∆ ] aik ∑ s∈ [ ω ] σ ( 〈wi , x ( k∆+s ) 〉 ) . ( CNN-AP-DS ) Note that pooling and downsampling operations are often tied together in the literature . However in this work we will treat these two operations separately . In the formula above , different values for q , ω , ∆ lead to different architectures with vastly different behaviors . For example , when q = ∆ = d and ω = 1 , we recover a two-layer fully-connected neural network fFC ( x ; a , Θ ) = N−1/2 ∑ i∈ [ N ] aiσ ( 〈wi , x〉 ) which has the universal approximation property at large N . When ω = ∆ = 1 and q < d , the network is “ locally connected ” fLC ( x ; a , Θ ) = N −1/2∑ i∈ [ N ] , k∈ [ d ] aikσ ( 〈wi , x ( k ) 〉 ) , and not a universal approximator anymore : however , fLC vastly outperforms fFC in some cases ( Li et al. , 2020 ) . For ω > 1 , local pooling enables learning functions that are locally invariant by translations more efficiently than without pooling . For ω = d ( global pooling ) , the network only fits functions fully invariant by cyclic translations . The aim of this paper is to formalize and quantify the trade-off between the target function class and the statistical efficiency brought by these different architectures . As a concrete first step in this direction , we consider kernel models that are naturally associated with the convolutional neural networks ( CNN-AP-DS ) through the neural tangent kernel perspective ( Daniely et al. , 2016 ; Jacot et al. , 2018 ) . Kernel methods have the advantage of 1 ) being tractable—leaving the computational issue of learning CNNs aside ; 2 ) having well-understood approximation and generalization properties , which depends on the eigendecomposition of the kernel and the alignment between the target function and associated RKHS ( Caponnetto & De Vito , 2007 ; Wainwright , 2019 ) ( see Appendices B and C for background ) . While kernel models only describe neural networks in the lazy training regime ( Chizat et al. , 2019 ; Du et al. , 2019b ; a ; Allen-Zhu et al. , 2019 ; Zou et al. , 2018 ) and miss important properties of deep learning , such as feature learning , we see that architecture choice is already crucial to enable efficient learning of ‘ image-like ’ functions in the fixed-feature regime . Neural tangent kernels are obtained by linearizing the associated neural networks . Here we consider the tangent kernel associated to the network fCNN ( c.f . Appendix A.2 for a detailed derivation ) : HCKω , ∆ ( x , y ) = ∆ dω ∑ k∈ [ d/∆ ] ∑ s , s′∈ [ ω ] h ( 〈x ( k∆+s ) , y ( k∆+s′ ) 〉/q ) , ( CK-AP-DS ) where h : R → R is related to the activation function σ in ( CNN-AP-DS ) . As a linearization of CNNs , the kernel ( CK-AP-DS ) inherits some of the favorable properties of convolution , pooling , and downsampling operations . Indeed , a line of work ( Mairal et al. , 2014 ; Mairal , 2016 ; Arora et al. , 2019 ; Li et al. , 2019 ; Shankar et al. , 2020 ) showed that , though performing slightly worse than CNNs , such ( hierarchical ) convolutional kernels have empirically outperformed the former state-of-the-art kernels . For instance , these kernels achieved test accuracy around 87 % − 90 % on CIFAR-10 , against 79.6 % for the best former unsupervised feature-extraction method ( Coates et al. , 2011 ) ( currently , the state-of-the-art CNNs can achieve test accuracy 99 % ) . In this paper , we will further consider a stylized setting with input signal distribution x ∼ Unif ( Qd ) ( uniform distribution over Qd : = { −1 , +1 } d the discrete hypercube in d dimensions ) . This simple choice allows for a complete characterization of the eigendecomposition of HCKω , ∆ , thanks to all patches having same marginal distribution x ( k ) ∼ Unif ( Qq ) . We will be particularly interested in four specific choices of ( q , ω , ∆ ) in ( CK-AP-DS ) : HFC ( x , y ) = h ( 〈x , y〉/d ) , ( FC ) HCK ( x , y ) = 1 d ∑ k∈ [ d ] h ( 〈x ( k ) , y ( k ) 〉/q ) , ( CK ) HCKω ( x , y ) = 1 dω ∑ k∈ [ d ] ∑ s , s′∈ [ ω ] h ( 〈x ( k+s ) , y ( k+s′ ) 〉/q ) , ( CK-AP ) HCKGP ( x , y ) = 1 d ∑ k , k′∈ [ d ] h ( 〈x ( k ) , y ( k′ ) 〉/q ) . ( CK-GP ) These kernels are respectively the neural tangent kernels of a fully-connected network fFC ( FC ) , a convolutional network fLC ( CK ) , a convolutional network followed by local average pooling ( CKAP ) and a convolutional network followed by global pooling ( CK-GP ) . We will further be interested in ( CK-GP ) with patch size q = d , which we denote HFCGP : this corresponds to a convolutional kernel with full-size patches q = d , followed by global pooling . In this paper , we first characterize the reproducing kernel Hilbert space ( RKHS ) of these convolutional kernels , and then investigate their generalization properties in the regression setup . More specifically , assume { ( xi , yi ) } i≤n are n i.i.d . samples with xi ∼ Unif ( Qd ) and yi = f ? ( xi ) + εi . Here f ? ∈ L2 ( Qd ) and ( εi ) i≤n are independent errors with mean zero and variance bounded by σ2ε . We will focus on the generalization error of kernel ridge regression ( KRR ) ( see Appendix B.1 for general kernel methods ) . In particular , given a kernel function H : Qd × Qd → R and a regularization parameter λ ≥ 0 , the KRR estimator is the solution of the tractable convex problem f̂λ = arg min f∈H { ∑ i∈ [ n ] ( yi − f ( xi ) ) 2 + λ‖f‖2H } , ( KRR ) whereH is the RKHS associated toH with RKHS norm ‖ ·‖H . We denote the test error with square loss by R ( f ? , f̂λ ) = Ex { ( f ? ( x ) − f̂λ ( x ) ) 2 } . We will sometimes consider the expected test error Eε { R ( f ? , f̂λ ) } , where expectation is taken with respect to noise ε = ( εi ) i≤n in the training data . The generalization properties of the kernelsHFC andHFCGP were recently studied in Mei et al . ( 2021b ) ; Bietti et al . ( 2021 ) . In particular , they showed that global pooling ( kernel HFCGP ) leads to a gain of a factor d in sample complexity when fitting cyclic invariant functions , but still suffers from the curse of dimensionality ( HFCGP only fits very smooth functions in high-dimension ) . More precisely , Mei et al . ( 2021b ) considered the high-dimensional framework of Mei et al . ( 2021a ) and showed the following : KRR with HFC requires n ≈ d ` samples to fit degree- ` cyclic polynomials , while KRR with HFCGP only needs n ≈ d ` −1 . To enable milder dependence on the dimension d , further structural assumptions on the kernel and the target function should be considered ( for instance , in this paper , we use the kernel HCK and consider ‘ local ’ functions ) . 1.1 SUMMARY OF MAIN RESULTS . Our contributions are two-fold . First , we describe the RKHS associated to the convolutional kernel ( CK-AP-DS ) in the stylized setting x ∼ Unif ( Qd ) , which provides a fully explicit illustration of the roles of convolution , pooling and downsampling operations in learning specific classes of functions . Second , based on the eigendecomposition of the kernels , we provide generalization bounds on kernel ridge regression both in the fixed and high-dimension settings . In particular , we quantify the gain in sample complexity achieved by different architectures when learning target functions with corresponding structures . Let us define the q-local function class L2 ( Qd , Locq ) and the cyclic q-local function class L2 ( Qd , CycLocq ) ( subspace of L 2 ( Qd , Locq ) consisting of cyclic-invariant functions ) as follow : L2 ( Qd , Locq ) = { f ∈ L2 ( Qd ) : ∃ { gk } k∈ [ d ] ⊆ L2 ( Qq ) , f ( x ) = ∑ k∈ [ d ] gk ( x ( k ) ) } , ( LOC ) L2 ( Qd , CycLocq ) = { f ∈ L2 ( Qd ) : ∃g ∈ L2 ( Qq ) , f ( x ) = ∑ k∈ [ d ] g ( x ( k ) ) } . ( CYC-LOC ) We summarize below some of the insights that follow from our results . One-layer convolutional layer . The RKHS of HCK is constituted of q-local functions L2 ( Qd , Locq ) . Furthermore , the decay of the eigenvalues of HCK is controlled by a q-dimensional kernel , instead of a d-dimensional kernel for HFC . In particular , this implies that the test error of kernel ridge regression decays at rate n−O ( 1/q ) instead of n−O ( 1/d ) , when the target function is in L2 ( Qd , Locq ) . Hence , when q d , kernel methods with convolutional kernel HCK breaks the curse of dimensionality . Average pooling . The RKHS of HCKω is still constituted of q-local functions f ∈ L2 ( Qd , Locq ) , but penalizes differently the frequency components fj ( x ) by reweighting their eigenspaces by a factor κj , where fj ( x ) = ∑ k∈ [ d ] ρ k j f ( tk · x ) with ρj = e 2iπj d and tk · x = ( xk+1 , . . . , xd , x1 , . . . , xk ) denotes the k-shift . As ω increases , local pooling penalizes more and more heavily the high-frequency components ( κj 1 ) , while making lowfrequency components statistically easier to learn ( κj 1 ) . For global pooling ω = d , HCKGP only learns cyclic local functions L 2 ( Qd , CycLocq ) and enjoy a factor d gain in statistical complexity compared to HCK . Downsampling . When ∆ ≤ ω , downsampling after average pooling leaves the low-frequency eigenspaces of HCKω stable . In particular , the downsampling operation does not modify the statistical complexity of learning low-frequency functions in one-layer kernels , while being potentially beneficial in further layers in deep convolutional kernels . In Table 1 , we report our high-dimensional predictions for the sample complexity of learning q-local cyclic-invariant polynomials of degree ` . These results are proven within the framework of Mei et al . ( 2021a ) ( see Appendix C for background ) . The rest of the paper is organized as follows . We discuss related work in Section 1.2 . In Section 2 , we present our main results on convolutional kernels and describe precisely the roles of convolution , pooling and downsampling operations . We briefly mention the multilayer case in Section 2.4 . Finally , we present a numerical simulation on synthetic data in Section 3 and conclude in Section 4 . Some details and discussions are deferred to Appendix A .

Summary: The paper considers the Convolutional Neural Tangent Kernel (CNTK) of depth one L=1 for the inputs that are distributed over the hypercube {-1,+1}^d. Under this setting, the paper presents the eigendecomposition of the kernel function. The paper also investigates the generalization properties of kernel regression using the depth one CNTK and bounds the risk for regression.