Enhanced Convolutional Neural Tangent Kernels

Recent research shows that for training with ` 2 loss , convolutional neural networks ( CNNs ) whose width ( number of channels in convolutional layers ) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel ( CNN-GP ) ( Novak et al. , 2019 ) if only the last layer is trained , and correspond to regression with respect to the Convolutional Neural Tangent Kernel ( CNTK ) if all layers are trained . An exact algorithm to compute CNTK ( Arora et al. , 2019 ) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7 % of that of the corresponding CNN architecture ( best figure being around 78 % ) which is interesting performance for a fixed kernel . Here we show how to significantly enhance the performance of these kernels using two ideas . ( 1 ) Modifying the kernel using a new operation called Local Average Pooling ( LAP ) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts . Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression . This idea is inspired by Global Average Pooling ( GAP ) , which we show for CNN-GP and CNTK is equivalent to full translation data augmentation . ( 2 ) Representing the input image using a pre-processing technique proposed by Coates et al . ( 2011 ) , which uses a single convolutional layer composed of random image patches . On CIFAR-10 , the resulting kernel , CNN-GP with LAP and horizontal flip data augmentation , achieves 89 % accuracy , matching the performance of AlexNet ( Krizhevsky et al. , 2012 ) , and outperforms the best previous classifier that is not a trained neural network ( Mairal , 2016 ) . Similar improvements are obtained for Fashion-MNIST . 1 INTRODUCTION . Recent research shows that for training with ` 2 loss , convolutional neural networks ( CNNs ) whose width ( number of channels in convolutional layers ) goes to infinity , correspond to regression with respect to the CNN Gaussian Process kernel ( CNN-GP ) if only the last layer is trained ( Novak et al. , 2019 ; Garriga-Alonso et al. , 2019 ) , and correspond to regression with respect to the Convolutional Neural Tangent Kernel ( CNTK ) if all layers are trained ( Jacot et al. , 2018 ; Allen-Zhu et al. , 2018 ; Du et al. , 2019b ; Arora et al. , 2019 ) . Novak et al . ( 2019 ) ; Garriga-Alonso et al . ( 2019 ) also implemented CNN-GP and tested its empirical performance . An efficient exact algorithm was given ( Arora et al. , 2019 ) to compute CNTK for CNN architectures , as well as those that include a Global Average Pooling ( GAP ) layer ( defined below ) . This is a fixed kernel that inherits some benefits of CNNs , including exploitation of locality via convolution , as well as multiple layers of processing . For CIFAR-10 , incorporating GAP into the kernel improves classification accuracy by up to 10 % compared to pure convolutional CNTK . While this performance is encouraging for a fixed kernel , the best accuracy is still under 78 % , which is disappointing even compared to AlexNet . One hope for improving the accuracy further is to somehow capture modern innovations such as batch normalization , data augmentation , residual layers , etc . in CNTK . The current paper shows how to incorporate simple data augmentation . Specifically , the idea of creating new training images from existing images using pixel translation and flips , while assuming that these operations should not change the label . Since deep learning uses stochastic gradient descent ( SGD ) , it is trivial to do such data augmentation on the fly . However , it ’ s unclear how to efficiently incorporate data augmentation in kernel regression , since training time is quadratic in the number of training images . 1 Thus somehow data augmentation has to be incorporated into the computation of the kernel itself . The main observation here is that the above-mentioned algorithm for computing CNTK involves a dynamic programming whose recursion depth is equal to the depth of the corresponding finite CNN . It is possible to impose symmetry constraints at any desired layer during this computation . In this viewpoint , it can be shown that prediction using CNTK/CNN-GP with GAP is equivalent to prediction using CNTK/CNN-GP without GAP but with full translation data augmentation with wraparound at the boundary . The translation invariance property implicitly assumed in data augmentation is exactly equivalent to an imposed symmetry constraint in the computation of the CNTK which in turn is derived from the pooling layer in the CNN . See Section 4 for more details . Thus GAP corresponds to full translation data augmentation scheme , but in practice such data augmentation creates unrealistic images ( cf . Figure 2 ) and training on them can harm performance . However , the idea of incorporating symmetry in the dynamic programming leads to a variant we call Local Average Pooling ( LAP ) . This implicitly is like data augmentation where image labels are assumed to be invariant to small translation , say by a few pixels . Interestingly , LAP corresponds to a average pooling layer for CNNs , named box filtering ( Szeliski , 2010 ) . Experimentally , we find LAP significantly enhances the performance as discussed below . • In extensive experiments on CIFAR-10 and Fashion-MNIST , we find LAP consistently improves performance of CNN-GP and CNTK . In particular , we find CNN-GP with LAP achieves 81 % on CIFAR-10 dataset , outperforming the best previous kernel predictor by 3 % . • When using the technique proposed by Coates et al . ( 2011 ) , which uses randomly sampled patches from training data as filters to do pre-processing,2 CNN-GP with LAP and horizontal flip data augmentation achieves 89 % accuracy on CIFAR-10 , matching the performance of AlexNet ( Krizhevsky et al. , 2012 ) and is the strongest classifier that is not a trained neural network.3 • We also test performance of CNNs with an extra layer corresponding to LAP and observe that it improves the performance on certain architectures . 2 RELATED WORK . Data augmentation has long been known to improve the performance of neural network and kernel methods ( Sietsma & Dow , 1991 ; Schölkopf et al. , 1996 ) . Theoretical study of data augmentation dates back to Chapelle et al . ( 2001 ) . Recently , Dao et al . ( 2018 ) proposed a theoretical framework for understanding data augmentation and showed data augmentation with a kernel classifier can have feature averaging and variance regularization effects . More recently , Chen et al . ( 2019 ) quantitatively shows in certain settings , data augmentation provably improves the classifier performance . For more comprehensive discussion on data augmentation and its properties , we refer readers to Dao et al . ( 2018 ) ; Chen et al . ( 2019 ) and references therein . CNN-GP and CNTK correspond to infinitely wide CNN with different training strategies ( only training the top layer or training all layers jointly ) . The correspondence between infinite neural networks and kernel machines was first noted by Neal ( 1996 ) . More recently , this was extended to deep and convolutional neural networks ( Lee et al. , 2018 ; Matthews et al. , 2018 ; Novak et al. , 2019 ; Garriga-Alonso et al. , 2019 ) . These kernels correspond to infinitely wide neural networks where only the last layer is trained . A recent line of work studied overparameterized neural networks where all layers are trained ( Allen-Zhu et al. , 2018 ; Du et al. , 2019b ; 2018 ; Li & Liang , 2018 ; Zou et al. , 2018 ) . Their proofs imply the gradient kernel is close to a fixed kernel which only depends the training data and neural network architecture . These kernels thus correspond to infinitely wide neural networks where are all layers are trained . Jacot et al . ( 2018 ) named this kernel , neural tangent kernel ( NTK ) . Arora et al . ( 2019 ) formally proved infinitely wide neural net predictor trained by 1For CIFAR 10 , the bottleneck of using CNN-GP and CNTK is not solving least square but constructing kernel . The time complexity is O ( p2n2 ) where p is the number of pixels in each image and n is number of data point . 2See Section 6.2 for the precise procedure . 3https : //benchmarks.ai/cifar-10 gradient descent is equivalent to NTK predictor . Recently , NTKs induced by various neural network architectures are derived and shown to achieve strong empirical performance ( Arora et al. , 2019 ; Yang , 2019 ; Du et al. , 2019a ) . Global Average Pooling ( GAP ) was first proposed in Lin et al . ( 2013 ) and is common in modern CNN design ( Springenberg et al. , 2014 ; He et al. , 2016 ; Huang et al. , 2017 ) . However , current theoretical understanding on GAP is still rather limited . It has been conjectured in Lin et al . ( 2013 ) that GAP reduces the number of parameters in the last fully-connected layer and thus avoids overfitting , and that GAP is more robust to spatial translations of the input since it sums out the spatial information . In this work , we study GAP from the CNN-GP and CNTK perspective , and draw an interesting connection between GAP and data augmentation . Here we are interested in methods that are not trained neural networks . If the features are predefined before seeing the data , Oyallon & Mallat ( 2015 ) proposed the scattering network which achieves 82 % classification accuracy on CIFAR-10 . If one uses unsupervised learning methods to extract features , the method proposed in Coates et al . ( 2011 ) is one of the best-performing approaches on CIFAR-10 preceding modern CNNs . To our knowledge , the best result via unsupervised learning method in this line is by Mairal ( 2016 ) , who used the convolutional kernel network to achieve 86 % accuracy on CIFAR-10 . In this work we combine CNTK with LAP and the idea in Coates et al . ( 2011 ) to achieve the best performance for classifiers that are not trained neural networks . 3 PRELIMINARIES . 3.1 NOTATION . We use bold-faced letters for vectors , matrices and tensors . For a vector a , let [ a ] i be its i-th entry ; for a matrix A , let [ A ] i , j be its ( i , j ) -th entry ; for an order 4 tensor T , let [ T ] ij , i′j′ be its ( i , j , i′ , j′ ) -th entry . For a symmetric tensor , wet let tr ( T ) = ∑ i , j Tij , ij . For an order d tensor T ∈ RC1×C2× ... ×Cd and an integer α ∈ [ Cd ] , we use T ( α ) ∈ RC1×C2× ... ×Cd−1 to denote the order d− 1 tensor formed by fixing the coordinate of the last dimension of T to be α . 3.2 CNN , CNN-GP AND CNTK In this section we give formal definitions of CNN , CNN-GP and CNTK that we study in this paper . Throughout the paper , we let P be the width and Q be the height of the image . We use q ∈ Z+ to denote the filter size . In practice , q = 1 , 3 , 5 or 7 . Padding Schemes . In the definition of CNN , CNTK and CNN-GP , we may use different padding schemes . Let x ∈ RP×Q be an matrix . For a given index pair ( i , j ) with i ≤ 0 , i ≥ P + 1 , j ≤ 0 or j ≥ Q+ 1 , different padding schemes define different value for [ x ] i , j . For circular padding , we define [ x ] i , j to be [ x ] i mod P , j mod Q . For zero padding , we simply define [ x ] i , j to be 0 . Note the difference between circular padding and zero padding occurs only on the boundary of images . We will prove our theoretical results for the circular padding scheme to avoid boundary effects . CNN . Now we describe CNN with and without GAP . For any input image x , after L intermediate layers , we obtain x ( L ) ∈ RP×Q×C ( L ) where C ( L ) is the number of channels of the last layer . See Section A for the definition of x ( L ) . For the output , there are two choices : with and without GAP . • Without GAP : the final output is defined as f ( θ , x ) = ∑C ( L ) α=1 〈 W ( L+1 ) ( α ) , x ( L ) ( α ) 〉 where x ( L ) ( α ) ∈ RP×Q , andW ( L+1 ) ( α ) ∈ R P×Q is the weight of the last fully-connected layer . • With GAP : the final output is defined as f ( θ , x ) = 1PQ ∑C ( L ) α=1 W ( L+1 ) ( α ) · ∑ ( i , j ) ∈ [ P ] × [ Q ] [ x ( L ) ( α ) ] i , j whereW ( L+1 ) ( α ) ∈ R is the weight of the last fully-connected layer . CNN-GP and CNTK . Now we describe CNN-GP and CNTK . Let x , x′ be two input images . We denote the L-th layer ’ s CNN-GP kernel as Σ ( L ) ( x , x′ ) ∈ R [ P ] × [ Q ] × [ P ] × [ Q ] and the L-th layer ’ s CNTK kernel as Θ ( L ) ( x , x′ ) ∈ R [ P ] × [ Q ] × [ P ] × [ Q ] . See Section A for the precise definitions of Σ ( L ) ( x , x′ ) and Θ ( L ) ( x , x′ ) . For the output kernel value , again , there are two choices , without GAP ( equivalent to using a fully-connected layer ) or with GAP . • Without GAP : the output of CNN-GP is ΣFC ( x , x′ ) = tr ( Σ ( L ) ( x , x′ ) ) and the output of CNTK is ΘFC ( x , x′ ) = tr ( Θ ( L ) ( x , x′ ) ) . • With GAP : the output of CNN-GP is ΣGAP ( x , x′ ) = 1 P 2Q2 ∑ i , j , i′ , j′∈ [ P ] × [ Q ] × [ P ] × [ Q ] [ Σ ( L ) ( x , x′ ) ] i , j , i′ , j′ , and the output of CNTK is ΘGAP ( x , x ′ ) = 1P 2Q2 ∑ i , j , i′ , j′∈ [ P ] × [ Q ] × [ P ] × [ Q ] [ Θ ( L ) ( x , x′ ) ] i , j , i′ , j′ . Kernel Prediction . Lastly , we recall the formula for kernel regression . For simplicity , throughout the paper , we will assume all kernels are invertible . Given a kernel K ( x , x′ ) and a dataset ( X , y ) with data { ( xi , yi ) } Ni=1 , define KX ∈ RN×N where [ KX ] i , j = K ( xi , xj ) . The prediction for unseen data x′ is ∑N i=1 αiK ( x ′ , xi ) , where α = K−1X y .

Summary: This paper shows that there is a one-to-one correspondence between pixel-shift based data augmentation and average pooling operations in CNN-NNGP/NTK based ridge regression. Interestingly, the authors show that standard average pooling + flatten can lead to a better performance than simple global average pooling. This paper further shows that using the data pre-processing step proposed in (Coates et al., 2011) can boost performance of CNN-NNGP/NTK based ridge regression by ~7% which allowed the authors to achieve classification accuracy in high 80s which is AFAIK SOTA on CIFAR-10 when not using learned representations.