Data-Independent Neural Pruning via Coresets

1 INTRODUCTION . Neural networks today are the most popular and effective instrument of machine learning with numerous applications in different domains . Since Krizhevsky et al . ( 2012 ) used a model with 60M parameters to win the ImageNet competition in 2012 , network architectures have been growing wider and deeper . The vast overparametrization of neural networks offers better convergence ( AllenZhu et al. , 2019 ) and better generalization ( Neyshabur et al. , 2018 ) . The downside of the overparametrization is its high memory and computational costs , which prevent the use of these networks in small devices , e.g. , smartphones . Fortunately , it was observed that a trained network could be reduced to smaller sizes without much accuracy loss . Following this observation , many approaches to compress existing models have been proposed ( see Gale et al . ( 2019 ) for a recent review on network sparsification , and Mozer & Smolensky ( 1989 ) ; Srivastava et al . ( 2014 ) ; Yu et al . ( 2018 ) ; He et al . ( 2017 ) for neural pruning ) . Although a variety of model compression heuristics have been successfully applied to different neural network models , such as Jacob et al . ( 2018 ) ; Han et al . ( 2015 ) ; Alvarez & Salzmann ( 2017 ) , these approaches generally lack strong provable guarantees on the trade-off between the compression rate and the approximation error . The absence of worst-case performance analysis can potentially be a glaring problem depending on the application . Moreover , data-dependent methods for model compression ( e.g. , Mozer & Smolensky ( 1989 ) ; Srivastava et al . ( 2014 ) ; Hu et al . ( 2016 ) ; Yu et al . ( 2018 ) ; Baykal et al . ( 2018 ) ) rely on the statistics presented in a data set . Hence , these methods are vulnerable to adversarial attacks ( Szegedy et al. , 2014 ) , which design inputs that do not follow these statistics . Ideally , a network compression framework should 1 ) provide provable guarantees on the tradeoff between the compression rate and the approximation error , 2 ) be data independent , 3 ) provide high compression rate , and 4 ) be computationally efficient . To address these goals , we propose an efficient framework with provable guarantees for neural pruning , which is based on the existing theory of coresets such as ( Braverman et al. , 2016 ) . Coresets decrease massive inputs to smaller instances while maintaining a good provable approximation of the original set with respect to a given function . Our main idea is to treat neurons of a neural network as inputs in a coreset framework . Specifically , we reduce the number of neurons in layer i by constructing a coreset of neurons in this layer that provably approximates the output of neurons in layer i + 1 and discarding the rest . The coreset algorithm provides us with the choice of neurons in layer i and with the new weights connecting these neurons to layer i+1 . The coreset algorithm is applied layer-wise from the bottom to the top of the network . The size of the coreset , and consequently the number of remaining neurons in layer i , is provably related to the approximation error of the output for every neuron in layer i + 1 . Thus , we can theoretically derive the trade-off between the compression rate and the approximation error of any layer in the neural network . The coreset approximation of neurons provably holds for any input ; thus our compression is data-independent . Similar to our approach , Baykal et al . ( 2018 ) used coresets for model compression . However , their coresets are data-dependent ; therefore , they can not guarantee robustness over inputs . Moreover , they construct coresets of weights , while our approach constructs coresets of neurons . Neural pruning reduces the size of the weight tensors , while keeping the network dense . Hence the implementation of the pruned network requires no additional effort . Implementing networks with sparse weights ( which is the result of weight pruning ) is harder and in many cases does not result in actual computational savings . Our empirical results on LeNet-300-100 for MNIST ( LeCun et al. , 1998 ) and VGG-16 ( Simonyan & Zisserman , 2014 ) for CIFAR-10 ( Krizhevsky , 2009 ) demonstrate that our framework based on coresets of neurons outperforms sampling-based coresets by improving compression without sacrificing the accuracy . Finally , our construction is very fast ; it took about 56 sec . to compress each dense layer in the VGG-16 network using the platform specified in the experimental section . Our Contributions : We propose an efficient , data-independent neural pruning algorithm with a provable trade-off between the compression rate and the output approximation error . This is the first framework to perform neural pruning via coresets . We provide theoretical compression rates for some of the most popular neural activation functions summarized in Table 2 . 2 RELATED WORK . 2.1 CORESETS . Our compression algorithm is based on a data summarization approach known as coresets . Over the past decade , coreset constructions have been recognized for high achievements in data reduction in a variety of applications , including k-means , SVD , regression , low-rank approximation , PageRank , convex hull , and SVM ; see details in Phillips ( 2016 ) . Many of the non-deterministic coreset based methods rely on the sensitivity framework , in which elements of the input are sampled according to their sensitivity ( Langberg & Schulman , 2010 ; Braverman et al. , 2016 ; Tolochinsky & Feldman , 2018 ) , which is used as a measure of their importance . The sampled elements are usually reweighted afterwards . 2.2 MODEL COMPRESSION . State-of-the-art neural networks are often overparameterized , which causes a significant redundancy of weights . To reduce both computation time and memory requirements of trained networks , many approaches aim at removing this redundancy by model compression . Weight Pruning : Weight pruning was considered as far back as 1990 ( LeCun et al. , 1990 ) , but has recently seen more study ( Lebedev & Lempitsky , 2016 ; Dong et al. , 2017 ) . One of the most popular approaches is pruning via sparsity . Sparsity can be enforced by L1 regularization to push weights towards zero during training ( Hu et al. , 2016 ) . However , it was observed ( Han et al. , 2015 ) that after fine-tuning of the pruned network , L2 regularized network outperformed L1 , as there is no benefit to pushing values towards zero compared to pruning unimportant ( small weight ) connections . The approach in Denton et al . ( 2014 ) exploits the linearity of the neural network by finding a lowrank approximation of the weights and keeping the accuracy within 1 % of the uncompressed model . Jacob et al . ( 2018 ) performs quantization of the neural network ’ s weights and suggests a new training procedure to preserve the model accuracy after the quantization . These methods showed high compression rates , e.g. , the compression rate of AlexNet can reach 35x with the combination of pruning , quantization , and Huffman coding ( Han et al. , 2016 ) . Nevertheless , strong provable worst-case analysis is noticeably absent for most weight pruning methods . Neural pruning : Weight pruning leads to an irregular network structure , which needs a special treatment to deal with sparse representations , making it hard to achieve actual computational savings . On the other hand , neural pruning ( Hu et al. , 2016 ) and filter pruning in CNNs ( e.g , Zhuang et al . ( 2018 ) ; Li et al . ( 2017 ) ; Liu et al . ( 2017 ) simply reduce the size of the tensors . The method in Hu et al . ( 2016 ) first identifies weak neurons by analyzing their activiations on a large validation dataset . Then those weak neurons are pruned and the network is retrained . The processes are repeated several times . Zhuang et al . ( 2018 ) introduces channel pruning based on the contribution to the discriminative power . These methods are data-dependent ; thus they can not provide guarantees of approximation error for any future input . Li et al . ( 2017 ) measures the importance of channels by calculating the sum of absolute values of weights . Other channel pruning methods either impose channel-wise sparsity in training , followed by pruning channels with small scaling factors , and fine-tuning ( e.g , Liu et al . ( 2017 ) ) or perform channel pruning by minimizing the reconstruction error of feature maps between the pruned and pre-trained model ( e.g. , He et al . ( 2017 ) . ) These methods lack provable guarantees on the trade-offs between their accuracy and compression . Coreset-Based Model Compression Similar to our work , the approach in Baykal et al . ( 2018 ) uses corests for model compression . However , they construct coresets of weights , while we construct coresets of neurons . Their approach computes the importance of each weight , which is termed sensitivity , using a subset from the validation set . The coreset is chosen for the specific distribution ( of data ) so consequently , the compressed model is data-dependent . In our construction , the input of the neural network is assumed to be an arbitrary vector in Rd and the sensitivity of a neuron is computed for every input in Rd . This means that we create a data-independent coreset ; its size is independent of the properties of the specific data at hand , and the compression provably approximates any future test sample . Dubey et al . ( 2018 ) builds upon k-means coresets by adding a sparsity constraint . The weighting of the filters in the coreset is obtained based on their activation magnitudes over the training set . The compression pipeline also includes a pre-processing step that follows a simple heuristic that eliminates filters based on the mean of their activation norms over the training set . This construction is obviously data-dependent and it uses corsets as an alternative mechanism for low-rank approximation of filters . 3 METHOD . We propose an algorithm for compressing layer i and we apply it to all layers from the bottom to the top of the network . We first give an intuitive description of the algorithm . We then formalize it and provide a theoretical analysis of the proposed construction . 3.1 DATA-INDEPENDENT CORESET FOR NEURAL PRUNING . Let aij = φ ( p T j x ) be the jth neuron in layer i , in which pj denotes its weights , and x denotes an arbitrary input in Rd ( see Figure 1 , top ) . We first consider a single neuron in layer i + 1 . The linear part of this neuron is z = ∑|P | j=1 w ( pj ) φ ( p T j x ) . We would like to approximate z by z̃ =∑ l∈J∗ u ( pl ) φ ( p T l x ) where J ∗ ⊂ { 1 , ... , |P | } is a small subset , and we want this approximation to be bounded by a multiplicative factor that holds for any x ∈ Rd . Unfortunately , our result in Theorem 6 shows that this idealized goal is impossible . However , we show in Theorem 7 and Corollary 8 that we can construct a small coreset C , such that |z − z̃| ≤ ε for any input x ∈ Rd . Algorithm 1 summarizes the coreset construction for a single neuron with an activation function φ , ( our results for common neural activation functions are summarized in Table 2 ) . Algorithm 2 and Corollary 9 show the construction of a single coreset with possibly different weights for all neurons in layer i+ 1 ( see Figure 1 , bottom ) .

Summary: The authors proposed a model compressing method based on coreset framework. The goal of the paper is to reduce the number of neurons. The basic idea is sampling the neurons on each layer with probability equal to the neuron's max share among all the outputs to the next layer, and updating the weights associated with the remained neurons. Another main contribution is the authors provided theoretical analysis to guarantee the accuracy vs compression trade-off for all possible inputs.