NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES

1 INTRODUCTION . Despite decades of research , the design of neural networks is still an empirical process . Practitioners make design choices , such as the number of layers , type of layers , number of filters per layer , etc. , based on intuition or brute-force search . Nevertheless , the performance of these algorithms , together with the advances of GPU devices , have led to a growing popularity of these techniques in both academia and industry . Recent advances are unveiling some properties of neural networks . For example , there is a consensus that depth can accelerate learning , and that wider layers help optimization ( Arora et al. , 2018 ; Livni et al. , 2014 ; Cohen et al. , 2018 ) . However , in practical applications , the size of these networks is often a limiting factor when deploying on devices with constrained storage , memory , and computation resources . Another known neural network property is that the responses of a layer exhibit considerable correlation ( Denil et al. , 2013 ) , inspiring the idea of learning decorrelated filters ( Cogswell et al. , 2016 ; Rodríguez et al. , 2017 ) . These algorithms propose a modified loss function to encourage decorrelation during training and show that accuracy improves with decorrelated filters . However , such algorithms focus on training and do not address network compression . Our hypothesis is that layers that exhibit high correlation in filter responses could learn equally well using a smaller number of filters . Principal Filter Analysis ( PFA ) draws from the recent findings that it is easier to start with an overparametrized network and it then exploits intra-layer correlation for guiding network compression . PFA analyzes a trained network and is agnostic to the training methodology and the loss function . Inference is performed on a dataset , and the correlation within the responses of each layer is used to provide a compression recipe . A new smaller architecture based on this recipe can then be retrained . We propose two closed-form algorithms based on spectral energy analysis for suggesting the number of filters to remove in a layer : PFA-En uses Principal Component Analysis ( PCA ) ( Hotelling , 1933 ) to allow a user to specify the proportion of the energy in the original response that should be preserved in each layer ; PFA-KL is a heuristic that leads to a parameter-free approach that uses Kullback-Leibler ( KL ) divergence ( Kullback & Leibler , 1951 ) to identify the number of redundant filters . Based on the new suggested number of filters per layer identified by PFA , we remove those that are maximally correlated with other filters and fine-tune the network by retraining . Both PFA algorithms are straightforward to implement and , as shown in Sec . 4 , they achieve better compression and , in most cases , better accuracy than the state of the art on several datasets and architectures . In Sec . 4 we also show how PFA can be used to perform simulations compression and domain adaptation . 2 RELATED WORK . The field of network compression encompasses a wide range of techniques that can be grouped into the following families : quantization , knowledge distillation , tensor factorization and network pruning . Quantization algorithms compress networks by reducing the number of bits used to represent each weight ( Wu et al. , 2016 ; Han et al. , 2016a ; Rastegari et al. , 2016 ; He & Cheng , 2018 ; Wu et al. , 2018 ) . Knowledge distillation ( Hinton et al. , 2014 ) and model compression ( Buciluǎ et al. , 2006 ) aim to create a simpler model that mimics the output of a more complex model . Variations on this concept include Ba & Caurana ( 2014 ) ; Romero et al . ( 2015 ) ; Chen et al . ( 2016 ) . Tensor factorization algorithms exploit the redundancy present in convolution layers by replacing the original tensors with a sequence of smaller or sparser counterparts that produce similar responses ( Denton et al. , 2014 ; Lebedev et al. , 2014 ; Jaderberg et al. , 2014 ; Cheng et al. , 2016 ; Masana et al. , 2017 ; Wen et al. , 2017 ; Yu et al. , 2017 ; Alvarez & Salzmann , 2017 ; Peng et al. , 2018 ) . Network pruning is a family of techniques that compress networks by iteratively removing connections based on the salience of their weights . Early work , like Optimal Brain Damage ( LeCun et al. , 1990 ) and Optimal Brain Surgeon ( Hassibi et al. , 1993 ) , targeted fully connected networks . Recent work can be divided into two sub-families : sparse pruning ( Han et al. , 2016b ; Srinivas & Babu , 2015 ; Wen et al. , 2016 ; Tung & Mori , 2018 ; Aghasi et al. , 2017 ; Carreira-Perpinan & Idelbayev , 2018 ; Zhang et al. , 2018 ; Dai et al. , 2018 ) , where individual neurons are removed , and structured pruning ( Li et al. , 2017 ; He et al. , 2017 ; Luo et al. , 2017 ; Molchanov et al. , 2017 ; Yu et al. , 2018 ) , where entire filters are removed . PFA falls within the family of structured network pruning . Some of these techniques , e.g. , Li et al . ( 2017 ) , require user defined parameters that are hard to choose and whose effect on the footprint is difficult to predict ( see Sec . 4.6 for a more detailed discussion ) . Others also require modification of the loss function , e.g. , Liu et al . ( 2017 ) . In contrast , PFA-En has only one intuitive parameter , which is the proportion of the response energy to be preserved at each layer , and PFA-KL is parameter-free . Furthermore , instead of learning the saliency of the filters during training by modifying the loss function , PFA estimates it after training without requiring knowledge of the training details . This makes PFA applicable to any trained network , without the need to know its loss function . Within the structured pruning family , there are approaches based on the singular value decomposition ( SVD ) ( Xue et al. , 2013 ; Nakkiran et al. , 2015 ; Prabhavalkar et al. , 2016 ) , where a new set of filters are obtained by projecting the original weights onto a lower dimensional space . PFA differs from these methods by the fact that SVD is performed on the responses of the layers rather than on the filter weights , and no projection is done . This is particularly relevant for domain adaption applications , where a trained network is specialized for a different task . Techniques that make compression decisions based on weights , rather than the responses , can not take into account the specificity of the task . As shown in Sec . 4.7 , PFA derives different architectures from the same initial model when analyzed for different tasks . Some methods , e.g. , Li et al . ( 2017 ) ; Masana et al . ( 2017 ) ; Peng et al . ( 2018 ) , also reason on the layer responses . These techniques aim to find a smaller set of filters that , in different ways , minimize the reconstruction error of the feature maps or the response output . Note that PFA uses the spectral energy of the filters ’ responses only to decide how many filters should be preserved . Similarly , the subsequent filter selection process does not take the reconstruction error into account . PFA is also philosophically and practically different : PFA uses the concept of correlation within the responses to identify redundancy within a layer . In practice this means that PFA can compress all layers in one run , while the majority of the techniques that use responses need to operate on one layer at the time . Finally , PFA is orthogonal to the quantization , tensor factorization and distillation methods , and could be used as a complementary strategy to further compress neural networks . 3 PRINCIPAL FILTER ANALYSIS . In this section , PFA-En and PFA-KL algorithms are described in detail . Both algorithms share the idea of exploiting correlations between responses in convolutional layers and neurons in fully connected layers to obtain a principled recommendation for network compression . 3.1 DEFINITIONS . PFA is inherently data driven and thus takes advantage of a dataset { Xi } ∈ RM×I where Xi is the ith input data sample , M is the number of samples in the dataset , and I is the input dimensionality . Typically , this dataset is the data used to train the network , but it can also be a representative set that covers the distribution of inputs likely to be encountered . Without loss of generality , we assume that the input data are images : { Xi } ∈ RM×H×W×C , where H is the image height , W is the image width and C is the number channels . Let T [ ` ] i ∈ R1×H [ ` ] ×W [ ` ] ×C [ ` ] be the output tensor produced by a given layer ` of a network on the ith input sample . Any operation in the network is considered a layer ( e.g. , batch normalization , ReLU , etc. ) . In this work we analyze the output of convolutional and fully connected layers . However , PFA can be used to analyze the output of any layer in the network . Recurrent neural networks could also be analyzed , but are out of the scope for this paper . For a convolutional layer ` , let W [ ` ] ∈ Rf [ ` ] h ×f [ ` ] w ×C [ ` −1 ] ×C [ ` ] be the set of C [ ` ] trainable filters with a kernel of size f [ ` ] h × f [ ` ] w × C [ ` −1 ] . Therefore , we can formally express T [ ` ] i produced by the convolutional layer as T [ ` ] i = W [ ` ] ∗ T [ ` −1 ] i with T [ 0 ] i = Xi , where ∗ denotes the convolution operator . We omit the bias term to improve readability . We define the response vector a [ ` ] i ∈ RC [ ` ] of a given layer ` with respect to an input Xi to be the spatially max-pooled and flattened tensor T [ ` ] i ( i.e. , max-pooling over the dimensions H [ ` ] and W [ ` ] ) . For fully-connected layers , W [ ` ] ∈ RC [ ` ] , with C [ ` ] being the number of neurons in layer ` . The output tensor is T [ ` ] i = W [ ` ] T [ ` −1 ] i , and since no pooling is required , the response vector is a [ ` ] i = T [ ` ] i ∈ RC [ ` ] . Let A [ ` ] = [ a [ ` ] 1 , . . . , a [ ` ] M ] > ∈ RM×C [ ` ] be the matrix of responses of layer ` given a dataset with M samples . Lastly , let λ [ ` ] ∈ RC [ ` ] be the distribution of eigenvalues of the covariance matrix of A [ ` ] sorted in descending order and normalized to sum to 1 . In the following sections we present two algorithms that exploit the distribution λ [ ` ] to guide network compression .

Summary: This paper proposes a compression method based on spectral analysis. The basic idea is to analyse correlation between responses of difference layers and select those that are more relevant discarding the others. That, in principle (as mentioned in the paper) differs from other compression methods based on compressing the weights independently of the data being used. Therefore, in theory (nothing shown), different task would provide different outputs while similar works would compress in the same manner.