Rethinking Compressed Convolution Neural Network from a Statistical Perspective

1 INTRODUCTION . The introduction of AlexNet ( Krizhevsky et al. , 2012 ) spurred a line of research in 2D CNNs , which progressively achieve high levels of accuracy in the domain of image recognition ( Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) . The current stateof-the-art CNNs leave little room to achieve significant improvement on accuracy in learning stillimages , and attention has hence been diverted towards two directions . The first is to deploy deep CNNs on mobile devices by removing redundancy from the over-parametrized network , and some representative models include MobileNetV1 & V2 ( Howard et al. , 2017 ; Sandler et al. , 2018 ) . The second direction is to utilize CNNs to learn from higher-order inputs , for instance , video clips ( Tran et al. , 2018 ; Hara et al. , 2017 ) or electronic health records ( Cheng et al. , 2016 ; Suo et al. , 2017 ) . This area has not yet seen a widely-accepted state-of-the-art network . High-order kernel tensors are usually required to account for the multiway dependence of the input . This notoriously leads to heavy computational burden , as the number of parameters to be trained grows exponentially with the dimension of inputs . Subsequently , model compression becomes the critical juncture to guarantee the successful training and deployment of tensor CNNs . Tensor methods for compressing CNNs . Denil et al . ( 2013 ) showed that there is huge redundancy in network weights such that the entire network can be approximately recovered with a small fraction of parameters . Tensor decomposition recently has been widely used to compress the weights in a CNN network ( Lebedev et al. , 2015 ; Kim et al. , 2016 ; Kossaifi et al. , 2020b ; Hayashi et al. , 2019 ) . Specifically , the weights at each layer are first summarized into a tensor , and then tensor decomposition , CP or Tucker decomposition , can be applied to reduce the number of parameters . Different tensor decomposition to convolution layers will lead to a variety of compressed CNN block designs . For instance , the bottleneck block in ResNet ( He et al. , 2016 ) corresponds to the convolution kernel with a special Tucker low-rank structure , and the depthwise separable block in MobileNetV1 ( Howard et al. , 2017 ) and the inverted residual block in MobileNetV2 ( Sandler et al. , 2018 ) correspond to the convolution kernel with special CP forms . All the above are for 2D CNNs , and Kossaifi et al . ( 2020b ) and Su et al . ( 2018 ) considered tensor decomposition to factorize convolution kernels for higher-order tensor inputs . Tensor decomposition can also be applied to fully-connected layers since they may introduce a large number of parameters ( Kossaifi et al. , 2017 ; 2020a ) ; see also the discussions in Section 5 . Moreover , Kossaifi et al . ( 2019 ) summarized all weights of a network into one single high-order tensor , and then directly imposed a low-rank structure to achieve full network compression . While the idea is highly motivating , the proposed structure of the high-order tensor is heuristic and can be further improved ; see the discussions in Section 2.4 . Parameter efficiency of the above proposed architectures was heuristically justified by methods , such as FLOPs counting , naive parameter counting and/or empirical running time . However , there is still lack of a theoretical study to understand the mechanism of how tensor decomposition can compress CNNs . This paper attempts to fill this gap from statistical perspectives . Sample Complexity Analysis . Du et al . ( 2018a ) first characterized the statistical sample complexity of a CNN ; see also Wang et al . ( 2019 ) for compact autoregressive nets . Specifically , consider a CNN model , y = FCNN ( x , W ) + ξ , where y and x are output and input , respectively , W contains all weights and ξ is an additive error . Given the trained and true underlying networks FCNN ( x , Ŵ ) and FCNN ( x , W ∗ ) , the root-mean-square prediction error is defined as E ( Ŵ ) = √ Ex|FCNN ( x , Ŵ ) − FCNN ( x , W∗ ) |2 , ( 1 ) where Ŵ and W∗ are trained and true underlying weights , respectively , and Ex is the expectation on x . The sample complexity analysis is to investigate how many samples are needed to guarantee a given tolerance on the prediction error . It can also be used to detect the model redundancy . Consider two nested CNNs , where F1 is more compressed than F2 . Given the same true underlying networks , when the prediction errors from trained F1 and F2 are comparable , we then can argue that F2 has redundant weights comparing with F1 . As a result , conducting sample complexity analysis to CNNs with higher order inputs will shed light on the compressing mechanism of popular compressed CNNs via tensor decomposition . The study in Du et al . ( 2018a ) is limited to 1-dimensional convolution with a single kernel , followed by weighted summation , and its theoretical analysis can not be generalized to CNNs with compressed layers . In comparison , our paper presents a more realistic modeling of CNN by introducing a general N -dimensional convolution with multiple kernels , followed by an average pooling layer and a fully-connected layer . The convolution kernel and fully-connected weights are in tensor forms , and this allows us to explicitly model compressed CNNs via imposing low-rank assumption on weight tensors . Moreover , we used an alternative technical tool , and a sharper upper bound on the sample complexity can be obtained . Our paper makes three main contributions : 1 . We formulate CNNs with high-order inputs into statistical models , and show that they have an explicit “ Tucker-like ” form . 2 . The sample complexity analysis can then be conducted to CNNs as well as compressed CNNs via tensor decomposition , with weak conditions allowing for time-dependent inputs like video data . 3 . From theoretical analysis , we draw an interesting finding that forcing low dimensionality on output channels may introduce unnecessary parameter redundancy to a compressed network . 1.1 COMPARISON WITH OTHER EXISTING WORKS . Deep neural networks are usually over-parametrized , yet empirically , they can generalize well . It is an important topic in the literature to theoretically study the generalization ability of deep neural networks , including deep CNNs ( Li et al. , 2020 ; Arora et al. , 2018 ) . The generalization error , defined as the difference between test and training errors , is commonly used to evaluate such ability , and many techniques have been developed to control its bound ; see , for example , the VC dimension ( Vapnik , 2013 ) , the Rademacher complexity & covering number ( Bartlett & Mendelson , 2002 ) , the norm-based capacity control ( Neyshabur et al. , 2017 ; Golowich et al. , 2018 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2015 ) and low-rank compression based methods ( Li et al. , 2020 ; Zhou & Feng , 2018 ; Arora et al. , 2018 ) . These works use a model-agnostic framework , and hence relies heavily on explicit regularization such as weight decay , dropout or data augumentation , as well as algorithmbased implicit regularization to remove the redundancy in the network . We , however , attempt to theoretically explain how much compressibility is achieved in a compressed network architecture . Specifically , we make a comparison between a CNN and its compressed version , and makes theoretically-supported modification to the latter to further increase efficiency . Subsequently , our analysis requires an explicit formulation for the network architecture , which is provided in Section 2 , and the prediction error at ( 1 ) is adopted as our evaluation criteria . We notice that Li et al . ( 2020 ) also proposes to use the CP Layers to compress the weights in each convolution layer . But their study is still model-agnostic , since the ranks of the underlying CP Layers depend on the trained weights . In details , their proposed approach uses regularization assumptions on the weights and hence , their derived theoretical bound is influenced by training and not suitable to analyze the network design exclusively . Other existing works , that aim to provide theoretical understanding for the neural networks , include the understanding of parameter recovery with gradient-based algorithms for deep neural networks ( Zhong et al. , 2017b ; Fu et al. , 2020 ; Goel et al. , 2018 ; Zhong et al. , 2017a ) ; the development of other provably efficient algorithms ( Cao & Gu , 2019 ; Du & Goel , 2018 ) ; and the investigation of convergence in an over-parameterized regime ( Allen-Zhu et al. , 2019 ; Li & Liang , 2018 ; Du et al. , 2018b ) . Our work differs greatly from these works in both target and methodology . We do not consider computational complexity or algorithm convergence . Instead , we focus on the statistical sample complexity to depict the mechanism of compressed block designs for CNNs . 2 FORMULATING CNNS WITH HIGHER-ORDER INPUTS . 2.1 NOTATION . Tensor notations . We follow the notations in Kolda & Bader ( 2009 ) to denote vectors by lowercase boldface letters , e.g . a ; matrices by capital boldface letters , e.g . A ; tensors of order 3 or higher by Euler script boldface letters , e.g . A . For an N th-order tensor A ∈ Rl1×···×lN , denote its elements by A ( i1 , i2 , . . . , iN ) and the n-mode unfolding by A ( n ) , where the columns of A ( n ) are the n-mode vectors of A , for 1 ≤ n ≤ N . And A ( : , · · · , in , · · · , : ) denotes the subtensor of A , holding only the nth index fixed . The vectorization operation is denoted by vec ( · ) . The inner product of two tensors A , B ∈ Rl1×···×lN is defined as 〈A , B〉 =∑ i1 · · · ∑ iN A ( i1 , . . . , iN ) B ( i1 , . . . , iN ) , and the Frobenius norm is ‖A‖F = √ 〈A , A〉 . The mode-n multiplication ×n of a tensor A ∈ Rl1×···×lN and a matrix B ∈ Rpn×ln is defined as ( A×n B ) ( i1 , . . . , jn , . . . , iN ) = ∑ln in=1 A ( i1 , . . . , in , . . . , iN ) B ( jn , in ) , for 1 ≤ n ≤ N , respectively . The mode-n multiplication ×̄N of a tensor A ∈ Rl1×···×lN and a vector b ∈ Rln is defined as ( A×̄nb ) ( i1 , . . . , in−1 , in+1 , . . . , iN ) = ∑ln in=1 A ( i1 , . . . , in , . . . , iN ) b ( in ) , for 1 ≤ n ≤ N . The symbol “ ⊗ ” is the Kronecker product and “ ◦ ” is the outer product . We extend the definition of Khatri-Rao product to tensors : given tensors A ∈ Rl1×l2×···lN×K and B ∈ Rp1×p2×···pN×K , their Khatri-Rao product is a tensor of size Rl1p1×l2p2×···lNpN×K , denoted by C = A B , where C ( : , · · · , k ) = A ( : , · · · , k ) ⊗B ( : , · · · , k ) , 1 ≤ k ≤ K. CP decomposition . The Canonical Polyadic ( CP ) decomposition ( Kolda & Bader , 2009 ) factorizes the tensor A ∈ Rl1×···×lN into a sum of rank-1 tensors , i.e . A = ∑R r=1 αrh ( 1 ) r ◦ h ( 2 ) r ◦ · · · ◦ h ( N ) r , where h ( j ) r is a unit norm vector of size Rlj for all 1 ≤ j ≤ N . The CP rank is the number of rank-1 tensors R. Tucker decomposition . The Tucker ranks of an N th-order tensor A ∈ Rl1×···×lN are defined as the matrix ranks of the unfoldings of A along all modes . If the Tucker ranks of A are ( R1 , . . . , RN ) , then there exist a core tensor G ∈ RR1×···×RN and matrices H ( i ) ∈ Rli×Ri , such that A = G×1 H ( 1 ) ×2 H ( 2 ) · · · ×N H ( N ) , known as Tucker decomposition ( Tucker , 1966 ) ,

Summary: This paper provides theoretical analysis of the estimating power of CNN (3 and 5 layers). By formulating the problem using tensors, the authors showed that the estimating error of the learned CNN weights with respect to the true weights is of the order $\sqrt{d/n}$ where $d$ measures model complexity and $n$ is the training sample size. In addition, the authors considered low rank approximation to the convolution tensor through CP and Tucker decompositions, and they derived convergence result for the CNN weights in this case. The authors then applied these results to analyze different block designs through numerical experiments and ablation studies.