NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES

1 INTRODUCTION . Despite decades of research , the design of neural networks is still an empirical process . Practitioners make design choices , such as the number of layers , type of layers , number of filters per layer , etc. , based on intuition or brute-force search . Nevertheless , the performance of these algorithms , together with the advances of GPU devices , have led to a growing popularity of these techniques in both academia and industry . Recent advances are unveiling some properties of neural networks . For example , there is a consensus that depth can accelerate learning , and that wider layers help optimization ( Arora et al. , 2018 ; Livni et al. , 2014 ; Cohen et al. , 2018 ) . However , in practical applications , the size of these networks is often a limiting factor when deploying on devices with constrained storage , memory , and computation resources . Another known neural network property is that the responses of a layer exhibit considerable correlation ( Denil et al. , 2013 ) , inspiring the idea of learning decorrelated filters ( Cogswell et al. , 2016 ; Rodríguez et al. , 2017 ) . These algorithms propose a modified loss function to encourage decorrelation during training and show that accuracy improves with decorrelated filters . However , such algorithms focus on training and do not address network compression . Our hypothesis is that layers that exhibit high correlation in filter responses could learn equally well using a smaller number of filters . Principal Filter Analysis ( PFA ) draws from the recent findings that it is easier to start with an overparametrized network and it then exploits intra-layer correlation for guiding network compression . PFA analyzes a trained network and is agnostic to the training methodology and the loss function . Inference is performed on a dataset , and the correlation within the responses of each layer is used to provide a compression recipe . A new smaller architecture based on this recipe can then be retrained . We propose two closed-form algorithms based on spectral energy analysis for suggesting the number of filters to remove in a layer : PFA-En uses Principal Component Analysis ( PCA ) ( Hotelling , 1933 ) to allow a user to specify the proportion of the energy in the original response that should be preserved in each layer ; PFA-KL is a heuristic that leads to a parameter-free approach that uses Kullback-Leibler ( KL ) divergence ( Kullback & Leibler , 1951 ) to identify the number of redundant filters . Based on the new suggested number of filters per layer identified by PFA , we remove those that are maximally correlated with other filters and fine-tune the network by retraining . Both PFA algorithms are straightforward to implement and , as shown in Sec . 4 , they achieve better compression and , in most cases , better accuracy than the state of the art on several datasets and architectures . In Sec . 4 we also show how PFA can be used to perform simulations compression and domain adaptation . 2 RELATED WORK . The field of network compression encompasses a wide range of techniques that can be grouped into the following families : quantization , knowledge distillation , tensor factorization and network pruning . Quantization algorithms compress networks by reducing the number of bits used to represent each weight ( Wu et al. , 2016 ; Han et al. , 2016a ; Rastegari et al. , 2016 ; He & Cheng , 2018 ; Wu et al. , 2018 ) . Knowledge distillation ( Hinton et al. , 2014 ) and model compression ( Buciluǎ et al. , 2006 ) aim to create a simpler model that mimics the output of a more complex model . Variations on this concept include Ba & Caurana ( 2014 ) ; Romero et al . ( 2015 ) ; Chen et al . ( 2016 ) . Tensor factorization algorithms exploit the redundancy present in convolution layers by replacing the original tensors with a sequence of smaller or sparser counterparts that produce similar responses ( Denton et al. , 2014 ; Lebedev et al. , 2014 ; Jaderberg et al. , 2014 ; Cheng et al. , 2016 ; Masana et al. , 2017 ; Wen et al. , 2017 ; Yu et al. , 2017 ; Alvarez & Salzmann , 2017 ; Peng et al. , 2018 ) . Network pruning is a family of techniques that compress networks by iteratively removing connections based on the salience of their weights . Early work , like Optimal Brain Damage ( LeCun et al. , 1990 ) and Optimal Brain Surgeon ( Hassibi et al. , 1993 ) , targeted fully connected networks . Recent work can be divided into two sub-families : sparse pruning ( Han et al. , 2016b ; Srinivas & Babu , 2015 ; Wen et al. , 2016 ; Tung & Mori , 2018 ; Aghasi et al. , 2017 ; Carreira-Perpinan & Idelbayev , 2018 ; Zhang et al. , 2018 ; Dai et al. , 2018 ) , where individual neurons are removed , and structured pruning ( Li et al. , 2017 ; He et al. , 2017 ; Luo et al. , 2017 ; Molchanov et al. , 2017 ; Yu et al. , 2018 ) , where entire filters are removed . PFA falls within the family of structured network pruning . Some of these techniques , e.g. , Li et al . ( 2017 ) , require user defined parameters that are hard to choose and whose effect on the footprint is difficult to predict ( see Sec . 4.6 for a more detailed discussion ) . Others also require modification of the loss function , e.g. , Liu et al . ( 2017 ) . In contrast , PFA-En has only one intuitive parameter , which is the proportion of the response energy to be preserved at each layer , and PFA-KL is parameter-free . Furthermore , instead of learning the saliency of the filters during training by modifying the loss function , PFA estimates it after training without requiring knowledge of the training details . This makes PFA applicable to any trained network , without the need to know its loss function . Within the structured pruning family , there are approaches based on the singular value decomposition ( SVD ) ( Xue et al. , 2013 ; Nakkiran et al. , 2015 ; Prabhavalkar et al. , 2016 ) , where a new set of filters are obtained by projecting the original weights onto a lower dimensional space . PFA differs from these methods by the fact that SVD is performed on the responses of the layers rather than on the filter weights , and no projection is done . This is particularly relevant for domain adaption applications , where a trained network is specialized for a different task . Techniques that make compression decisions based on weights , rather than the responses , can not take into account the specificity of the task . As shown in Sec . 4.7 , PFA derives different architectures from the same initial model when analyzed for different tasks . Some methods , e.g. , Li et al . ( 2017 ) ; Masana et al . ( 2017 ) ; Peng et al . ( 2018 ) , also reason on the layer responses . These techniques aim to find a smaller set of filters that , in different ways , minimize the reconstruction error of the feature maps or the response output . Note that PFA uses the spectral energy of the filters ’ responses only to decide how many filters should be preserved . Similarly , the subsequent filter selection process does not take the reconstruction error into account . PFA is also philosophically and practically different : PFA uses the concept of correlation within the responses to identify redundancy within a layer . In practice this means that PFA can compress all layers in one run , while the majority of the techniques that use responses need to operate on one layer at the time . Finally , PFA is orthogonal to the quantization , tensor factorization and distillation methods , and could be used as a complementary strategy to further compress neural networks . 3 PRINCIPAL FILTER ANALYSIS . In this section , PFA-En and PFA-KL algorithms are described in detail . Both algorithms share the idea of exploiting correlations between responses in convolutional layers and neurons in fully connected layers to obtain a principled recommendation for network compression . 3.1 DEFINITIONS . PFA is inherently data driven and thus takes advantage of a dataset { Xi } ∈ RM×I where Xi is the ith input data sample , M is the number of samples in the dataset , and I is the input dimensionality . Typically , this dataset is the data used to train the network , but it can also be a representative set that covers the distribution of inputs likely to be encountered . Without loss of generality , we assume that the input data are images : { Xi } ∈ RM×H×W×C , where H is the image height , W is the image width and C is the number channels . Let T [ ` ] i ∈ R1×H [ ` ] ×W [ ` ] ×C [ ` ] be the output tensor produced by a given layer ` of a network on the ith input sample . Any operation in the network is considered a layer ( e.g. , batch normalization , ReLU , etc. ) . In this work we analyze the output of convolutional and fully connected layers . However , PFA can be used to analyze the output of any layer in the network . Recurrent neural networks could also be analyzed , but are out of the scope for this paper . For a convolutional layer ` , let W [ ` ] ∈ Rf [ ` ] h ×f [ ` ] w ×C [ ` −1 ] ×C [ ` ] be the set of C [ ` ] trainable filters with a kernel of size f [ ` ] h × f [ ` ] w × C [ ` −1 ] . Therefore , we can formally express T [ ` ] i produced by the convolutional layer as T [ ` ] i = W [ ` ] ∗ T [ ` −1 ] i with T [ 0 ] i = Xi , where ∗ denotes the convolution operator . We omit the bias term to improve readability . We define the response vector a [ ` ] i ∈ RC [ ` ] of a given layer ` with respect to an input Xi to be the spatially max-pooled and flattened tensor T [ ` ] i ( i.e. , max-pooling over the dimensions H [ ` ] and W [ ` ] ) . For fully-connected layers , W [ ` ] ∈ RC [ ` ] , with C [ ` ] being the number of neurons in layer ` . The output tensor is T [ ` ] i = W [ ` ] T [ ` −1 ] i , and since no pooling is required , the response vector is a [ ` ] i = T [ ` ] i ∈ RC [ ` ] . Let A [ ` ] = [ a [ ` ] 1 , . . . , a [ ` ] M ] > ∈ RM×C [ ` ] be the matrix of responses of layer ` given a dataset with M samples . Lastly , let λ [ ` ] ∈ RC [ ` ] be the distribution of eigenvalues of the covariance matrix of A [ ` ] sorted in descending order and normalized to sum to 1 . In the following sections we present two algorithms that exploit the distribution λ [ ` ] to guide network compression .

Summary: The paper proposes to prune convolutional networks by analyzing the observed correlation between the filters of a same layer as expressed by the eigenvalue spectrum of their covariance matrix. The authors propose two strategies to decide of a compression level, one based on an eigenvalue threshold, the other one based on a heuristic based on the KL divergence between the observed eigenvalue distribution and the uniform one. This is a bit bizarre but does not require searching a parameter. Once one has decided the number of filters to keep, one can either retrain the network from scratch, or iteratively remove the most correlated filters, which, unsurprisingly, work better.