Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds

1 INTRODUCTION . Within the past decade , large-scale neural networks have demonstrated unprecedented empirical success in high-impact applications such as object classification , speech recognition , computer vision , and natural language processing . However , with the ever-increasing size of state-of-the-art neural networks , the resulting storage requirements and performance of these models are becoming increasingly prohibitive in terms of both time and space . Recently proposed architectures for neural networks , such as those in Krizhevsky et al . ( 2012 ) ; Long et al . ( 2015 ) ; Badrinarayanan et al . ( 2015 ) , contain millions of parameters , rendering them prohibitive to deploy on platforms that are resource-constrained , e.g. , embedded devices , mobile phones , or small scale robotic platforms . In this work , we consider the problem of sparsifying the parameters of a trained fully-connected neural network in a principled way so that the output of the compressed neural network is approximately preserved . We introduce a neural network compression approach based on identifying and removing weighted edges with low relative importance via coresets , small weighted subsets of the original set that approximate the pertinent cost function . Our compression algorithm hinges on extensions of the traditional sensitivity-based coresets framework ( Langberg & Schulman , 2010 ; Braverman et al. , 2016 ) , and to the best of our knowledge , is the first to apply coresets to parameter downsizing . In this regard , our work aims to simultaneously introduce a practical algorithm for compressing neural network parameters with provable guarantees and close the research gap in prior coresets work , which has predominantly focused on compressing input data points . In particular , this paper contributes the following : 1 . A coreset approach to compressing problem-specific parameters based on a novel , empirical notion of sensitivity that extends state-of-the-art coreset constructions . 2 . An efficient neural network compression algorithm , CoreNet , based on our extended coreset approach that sparsifies the parameters via importance sampling of weighted edges . 3 . Extensions of the CoreNet method , CoreNet+ and CoreNet++ , that improve upon the edge sampling approach by additionally performing neuron pruning and amplification . †Computer Science and Artificial Intelligence Laboratory , Massachusetts Institute of Technology , emails : { baykal , lucasl , igilitschenski , rus } @ mit.edu ‡Robotics and Big Data Laboratory , University of Haifa , email : dannyf.post @ gmail.com * These authors contributed equally to this work 4 . Analytical results establishing guarantees on the approximation accuracy , size , and generalization of the compressed neural network . 5 . Evaluations on real-world data sets that demonstrate the practical effectiveness of our algorithm in compressing neural network parameters and validate our theoretical results . 2 RELATED WORK . Our work builds upon the following prior work in coresets and compression approaches . Coresets Coreset constructions were originally introduced in the context of computational geometry ( Agarwal et al. , 2005 ) and subsequently generalized for applications to other problems via an importance sampling-based , sensitivity framework ( Langberg & Schulman , 2010 ; Braverman et al. , 2016 ) . Coresets have been used successfully to accelerate various machine learning algorithms such as k-means clustering ( Feldman & Langberg , 2011 ; Braverman et al. , 2016 ) , graphical model training ( Molina et al. , 2018 ) , and logistic regression ( Huggins et al. , 2016 ) ( see the surveys of Bachem et al . ( 2017 ) and Munteanu & Schwiegelshohn ( 2018 ) for a complete list ) . In contrast to prior work , we generate coresets for reducing the number of parameters – rather than data points – via a novel construction scheme based on an efficiently-computable notion of sensitivity . Low-rank Approximations and Weight-sharing Denil et al . ( 2013 ) were among the first to empirically demonstrate the existence of significant parameter redundancy in deep neural networks . A predominant class of compression approaches consists of using low-rank matrix decompositions , such as Singular Value Decomposition ( SVD ) ( Denton et al. , 2014 ) , to approximate the weight matrices with their low-rank counterparts . Similar works entail the use of low-rank tensor decomposition approaches applicable both during and after training ( Jaderberg et al. , 2014 ; Kim et al. , 2015 ; Tai et al. , 2015 ; Ioannou et al. , 2015 ; Alvarez & Salzmann , 2017 ; Yu et al. , 2017 ) . Another class of approaches uses feature hashing and weight sharing ( Weinberger et al. , 2009 ; Shi et al. , 2009 ; Chen et al. , 2015b ; a ; Ullrich et al. , 2017 ) . Building upon the idea of weight-sharing , quantization ( Gong et al. , 2014 ; Wu et al. , 2016 ; Zhou et al. , 2017 ) or regular structure of weight matrices was used to reduce the effective number of parameters ( Zhao et al. , 2017 ; Sindhwani et al. , 2015 ; Cheng et al. , 2015 ; Choromanska et al. , 2016 ; Wen et al. , 2016 ) . Despite their practical effectiveness in compressing neural networks , these works generally lack performance guarantees on the quality of their approximations and/or the size of the resulting compressed network . Weight Pruning Similar to our proposed method , weight pruning ( LeCun et al. , 1990 ) hinges on the idea that only a few dominant weights within a layer are required to approximately preserve the output . Approaches of this flavor have been investigated by Lebedev & Lempitsky ( 2016 ) ; Dong et al . ( 2017 ) , e.g. , by embedding sparsity as a constraint ( Iandola et al. , 2016 ; Aghasi et al. , 2017 ; Lin et al. , 2017 ) . Another related approach is that of Han et al . ( 2015 ) , which considers a combination of weight pruning and weight sharing methods . Nevertheless , prior work in weight pruning lacks rigorous theoretical analysis of the effect that the discarded weights can have on the compressed network . To the best of our knowledge , our work is the first to introduce a practical , sampling-based weight pruning algorithm with provable guarantees . Generalization The generalization properties of neural networks have been extensively investigated in various contexts ( Dziugaite & Roy , 2017 ; Neyshabur et al. , 2017a ; Bartlett et al. , 2017 ) . However , as was pointed out by Neyshabur et al . ( 2017b ) , current approaches to obtaining non-vacuous generalization bounds do not fully or accurately capture the empirical success of state-of-the-art neural network architectures . Recently , Arora et al . ( 2018 ) and Zhou et al . ( 2018 ) highlighted the close connection between compressibility and generalization of neural networks . Arora et al . ( 2018 ) presented a compression method based on the Johnson-Lindenstrauss ( JL ) Lemma ( Johnson & Lindenstrauss , 1984 ) and proved generalization bounds based on succinct reparameterizations of the original neural network . Building upon the work of Arora et al . ( 2018 ) , we extend our theoretical compression results to establish novel generalization bounds for fully-connected neural networks . Unlike the method of Arora et al . ( 2018 ) , which exhibits guarantees of the compressed network ’ s performance only on the set of training points , our method ’ s guarantees hold ( probabilistically ) for any random point drawn from the distribution . In addition , we establish that our method can ε-approximate the neural network output neuron-wise , which is stronger than the norm-based guarantee of Arora et al . ( 2018 ) . In contrast to prior work , this paper addresses the problem of compressing a fully-connected neural network while provably preserving the network ’ s output . Unlike previous theoretically-grounded compression approaches – which provide guarantees in terms of the normed difference – , our method provides the stronger entry-wise approximation guarantee , even for points outside of the available data set . As our empirical results show , ensuring that the output of the compressed network entry-wise approximates that of the original network is critical to retaining high classification accuracy . Overall , our compression approach remedies the shortcomings of prior approaches in that it ( i ) exhibits favorable theoretical properties , ( ii ) is computationally efficient , e.g. , does not require retraining of the neural network , ( iii ) is easy to implement , and ( iv ) can be used in conjunction with other compression approaches – such as quantization or Huffman coding – to obtain further improved compression rates . 3 PROBLEM DEFINITION . 3.1 FULLY-CONNECTED NEURAL NETWORKS . A feedforward fully-connected neural network withL ∈ N+ layers and parameters θ defines a mapping fθ : X → Y for a given input x ∈ X ⊆ Rd to an output y ∈ Y ⊆ Rk as follows . Let η ` ∈ N+ denote the number of neurons in layer ` ∈ [ L ] , where [ L ] = { 1 , . . . , L } denotes the index set , and where η1 = d and ηL = k. Further , let η = ∑L ` =2 η ` and η∗ = max ` ∈ { 2 , ... , L } η ` . For layers ` ∈ { 2 , . . . , L } , let W ` ∈ Rη ` ×η ` −1 be the weight matrix for layer ` with entries denoted by w ` ij , rows denoted by w ` i ∈ R1×η ` −1 , and θ = ( W 2 , . . . , WL ) . For notational simplicity , we assume that the bias is embedded in the weight matrix . Then for an input vector x ∈ Rd , let a1 = x and z ` = W ` a ` −1 ∈ Rη ` , ∀ ` ∈ { 2 , . . . , L } , where a ` −1 = φ ( z ` −1 ) ∈ Rη ` −1 denotes the activation . We consider the activation function to be the Rectified Linear Unit ( ReLU ) function , i.e. , φ ( · ) = max { · , 0 } ( entry-wise , if the input is a vector ) . The output of the network for an input x is fθ ( x ) = zL , and in particular , for classification tasks the prediction is argmaxi∈ [ k ] fθ ( x ) i = argmaxi∈ [ k ] z L i . 3.2 NEURAL NETWORK CORESET PROBLEM . Consider the setting where a neural network fθ ( · ) has been trained on a training set of independent and identically distributed ( i.i.d . ) samples from a joint distribution on X × Y , yielding parameters θ = ( W 2 , . . . , WL ) . We further denote the input points of a validation data set as P = { xi } ni=1 ⊆ X and the marginal distribution over the input space X as D. We define the size of the parameter tuple θ , nnz ( θ ) , to be the sum of the number of non-zero entries in the weight matrices W 2 , . . . , WL . For any given ε , δ ∈ ( 0 , 1 ) , our overarching goal is to generate a reparameterization θ̂ , yielding the neural network fθ̂ ( · ) , using a randomized algorithm , such that nnz ( θ̂ ) nnz ( θ ) , and the neural network output fθ ( x ) , x ∼ D can be approximated up to 1 ± ε multiplicative error with probability greater than 1 − δ . We define the 1 ± ε multiplicative error between two k-dimensional vectors a , b ∈ Rk as the following entry-wise bound : a ∈ ( 1± ε ) b ⇔ ai ∈ ( 1± ε ) bi ∀i ∈ [ k ] , and formalize the definition of an ( ε , δ ) -coreset as follows . Definition 1 ( ( ε , δ ) -coreset ) . Given user-specified ε , δ ∈ ( 0 , 1 ) , a set of parameters θ̂ = ( Ŵ 2 , . . . , ŴL ) is an ( ε , δ ) -coreset for the network parameterized by θ if for x ∼ D , it holds that P̂ θ , x ( fθ̂ ( x ) ∈ ( 1± ε ) fθ ( x ) ) ≥ 1− δ , where Pθ̂ , x denotes a probability measure with respect to a random data point x and the output θ̂ generated by a randomized compression scheme .

Summary: Given an additively decomposable function F(X, Q) = sum_over_x_in_X cost(x, Q), one can approximate it using either random sampling of x in X (unbiased, possibly high variance), or using importance sampling and replace the sum_over_x with a sum_over_coreset importance_of_a_point * cost(x, Q) which if properly defined can be both unbiased and have low variance [1]. In this work the authors consider the weighted sum of activations as F and suggest that for each neuron we can subsample the incoming edges. To construct the importance sampling strategy the authors adapt the classic notion of sensitivity from the coreset literature. Then, one has to carefully balance the approximation quality from one layer to the next and essentially union bound the results over all layers and all sampled points. The performed analysis is sound (up to my knowledge).