SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY

1 INTRODUCTION . Despite the success of deep neural networks in machine learning , they are often found to be highly overparametrized making them computationally expensive with excessive memory requirements . Pruning such large networks with minimal loss in performance is appealing for real-time applications , especially on resource-limited devices . In addition , compressed neural networks utilize the model capacity efficiently , and this interpretation can be used to derive better generalization bounds for neural networks ( Arora et al . ( 2018 ) ) . In network pruning , given a large reference neural network , the goal is to learn a much smaller subnetwork that mimics the performance of the reference network . The majority of existing methods in the literature attempt to find a subset of weights from the pretrained reference network either based on a saliency criterion ( Mozer & Smolensky ( 1989 ) ; LeCun et al . ( 1990 ) ; Han et al . ( 2015 ) ) or utilizing sparsity enforcing penalties ( Chauvin ( 1989 ) ; Carreira-Perpiñán & Idelbayev ( 2018 ) ) . Unfortunately , since pruning is included as a part of an iterative optimization procedure , all these methods require many expensive prune – retrain cycles and heuristic design choices with additional hyperparameters , making them non-trivial to extend to new architectures and tasks . In this work , we introduce a saliency criterion that identifies connections in the network that are important to the given task in a data-dependent way before training . Specifically , we discover important connections based on their influence on the loss function at a variance scaling initialization , which we call connection sensitivity . Given the desired sparsity level , redundant connections are pruned once prior to training ( i.e. , single-shot ) , and then the sparse pruned network is trained in the standard way . Our approach has several attractive properties : • Simplicity . Since the network is pruned once prior to training , there is no need for pretraining and complex pruning schedules . Our method has no additional hyperparameters and once pruned , training of the sparse network is performed in the standard way . • Versatility . Since our saliency criterion chooses structurally important connections , it is robust to architecture variations . Therefore our method can be applied to various architectures including convolutional , residual and recurrent networks with no modifications . i • Interpretability . Our method determines important connections with a mini-batch of data at single-shot . By varying this mini-batch used for pruning , our method enables us to verify that the retained connections are indeed essential for the given task . We evaluate our method on MNIST , CIFAR-10 , and Tiny-ImageNet classification datasets with widely varying architectures . Despite being the simplest , our method obtains extremely sparse networks with virtually the same accuracy as the existing baselines across all tested architectures . Furthermore , we investigate the relevance of the retained connections as well as the effect of the network initialization and the dataset on the saliency score . 2 RELATED WORK . Classical methods . Essentially , early works in network pruning can be categorized into two groups ( Reed ( 1993 ) ) : 1 ) those that utilize sparsity enforcing penalties ; and 2 ) methods that prune the network based on some saliency criterion . The methods from the former category ( Chauvin ( 1989 ) ; Weigend et al . ( 1991 ) ; Ishikawa ( 1996 ) ) augment the loss function with some sparsity enforcing penalty terms ( e.g. , L0 or L1 norm ) , so that back-propagation effectively penalizes the magnitude of the weights during training . Then weights below a certain threshold may be removed . On the other hand , classical saliency criteria include the sensitivity of the loss with respect to the neurons ( Mozer & Smolensky ( 1989 ) ) or the weights ( Karnin ( 1990 ) ) and Hessian of the loss with respect to the weights ( LeCun et al . ( 1990 ) ; Hassibi et al . ( 1993 ) ) . Since these criteria are heavily dependent on the scale of the weights and are designed to be incorporated within the learning process , these methods are prohibitively slow requiring many iterations of pruning and learning steps . Our approach identifies redundant weights from an architectural point of view and prunes them once at the beginning before training . Modern advances . In recent years , the increased space and time complexities as well as the risk of overfitting in deep neural networks prompted a surge of further investigation in network pruning . While Hessian based approaches employ the diagonal approximation due to its computational simplicity , impressive results ( i.e. , extreme sparsity without loss in accuracy ) are achieved using magnitude of the weights as the criterion ( Han et al . ( 2015 ) ) . This made them the de facto standard method for network pruning and led to various implementations ( Guo et al . ( 2016 ) ; Carreira-Perpiñán & Idelbayev ( 2018 ) ) . The magnitude criterion is also extended to recurrent neural networks ( Narang et al . ( 2017 ) ) , yet with heavily tuned hyperparameter setting . Unlike our approach , the main drawbacks of magnitude based approaches are the reliance on pretraining and the expensive prune – retrain cycles . Furthermore , since pruning and learning steps are intertwined , they often require highly heuristic design choices which make them non-trivial to be extended to new architectures and different tasks . Meanwhile , Bayesian methods are also applied to network pruning ( Ullrich et al . ( 2017 ) ; Molchanov et al . ( 2017a ) ) where the former extends the soft weight sharing in Nowlan & Hinton ( 1992 ) to obtain a sparse and compressed network , and the latter uses variational inference to learn the dropout rate which can then be used to prune the network . Unlike the above methods , our approach is simple and easily adaptable to any given architecture or task without modifying the pruning procedure . Network compression in general . Apart from weight pruning , there are approaches focused on structured simplification such as pruning filters ( Li et al . ( 2017 ) ; Molchanov et al . ( 2017b ) ) , structured sparsity with regularizers ( Wen et al . ( 2016 ) ) , low-rank approximation ( Jaderberg et al . ( 2014 ) ) , matrix and tensor factorization ( Novikov et al . ( 2015 ) ) , and sparsification using expander graphs ( Prabhu et al . ( 2018 ) ) or Erdős-Rényi random graph ( Mocanu et al . ( 2018 ) ) . In addition , there is a large body of work on compressing the representation of weights . A non-exhaustive list includes quantization ( Gong et al . ( 2014 ) ) , reduced precision ( Gupta et al . ( 2015 ) ) and binary weights ( Hubara et al . ( 2016 ) ) . In this work , we focus on weight pruning that is free from structural constraints and amenable to further compression schemes . 3 NEURAL NETWORK PRUNING . The main hypothesis behind the neural network pruning literature is that neural networks are usually overparametrized , and comparable performance can be obtained by a much smaller network ( Reed ( 1993 ) ) while improving generalization ( Arora et al . ( 2018 ) ) . To this end , the objective is to learn ii a sparse network while maintaining the accuracy of the standard reference network . Let us first formulate neural network pruning as an optimization problem . Given a dataset D = { ( xi , yi ) } ni=1 , and a desired sparsity level κ ( i.e. , the number of non-zero weights ) neural network pruning can be written as the following constrained optimization problem : min w L ( w ; D ) = min w 1 n n∑ i=1 ` ( w ; ( xi , yi ) ) , ( 1 ) s.t . w ∈ Rm , ‖w‖0 ≤ κ . Here , ` ( · ) is the standard loss function ( e.g. , cross-entropy loss ) , w is the set of parameters of the neural network , m is the total number of parameters and ‖ · ‖0 is the standard L0 norm . The conventional approach to optimize the above problem is by adding sparsity enforcing penalty terms ( Chauvin ( 1989 ) ; Weigend et al . ( 1991 ) ; Ishikawa ( 1996 ) ) . Recently , Carreira-Perpiñán & Idelbayev ( 2018 ) attempts to minimize the above constrained optimization problem using the stochastic version of projected gradient descent ( where the projection is accomplished by pruning ) . However , these methods often turn out to be inferior to saliency based methods in terms of resulting sparsity and require heavily tuned hyperparameter settings to obtain comparable results . On the other hand , saliency based methods treat the above problem as selectively removing redundant parameters ( or connections ) in the neural network . In order to do so , one has to come up with a good criterion to identify such redundant connections . Popular criteria include magnitude of the weights , i.e. , weights below a certain threshold are redundant ( Han et al . ( 2015 ) ; Guo et al . ( 2016 ) ) and Hessian of the loss with respect to the weights , i.e. , the higher the value of Hessian , the higher the importance of the parameters ( LeCun et al . ( 1990 ) ; Hassibi et al . ( 1993 ) ) , defined as follows : sj = |wj | , for magnitude based w2jHjj 2 or w2j 2H−1jj for Hessian based . ( 2 ) Here , for connection j , sj is the saliency score , wj is the weight , and Hjj is the value of the Hessian matrix , where the Hessian H = ∂2L/∂w2 ∈ Rm×m . Considering Hessian based methods , the Hessian matrix is neither diagonal nor positive definite in general , approximate at best , and intractable to compute for large networks . Despite being popular , both of these criteria depend on the scale of the weights and in turn require pretraining and are very sensitive to the architectural choices . For instance , different normalization layers affect the scale of the weights in a different way , and this would non-trivially affect the saliency score . Furthermore , pruning and the optimization steps are alternated many times throughout training , resulting in highly expensive prune – retrain cycles . Such an exorbitant requirement hinders the use of pruning methods in large-scale applications and raises questions about the credibility of the existing pruning criteria . In this work , we design a criterion which directly measures the connection importance in a datadependent manner . This alleviates the dependency on the weights and enables us to prune the network once at the beginning , and then the training can be performed on the sparse pruned network . Therefore , our method eliminates the need for the expensive prune – retrain cycles , and in theory , it can be an order of magnitude faster than the standard neural network training as it can be implemented using software libraries that support sparse matrix computations . 4 SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY . Given a neural network and a dataset , our goal is to design a method that can selectively prune redundant connections for the given task in a data-dependent way even before training . To this end , we first introduce a criterion to identify important connections and then discuss its benefits .

Summary: This work introduces SNIP, a simple way to prune neural network weights before training according to a specific criterion. SNIP identifies prunable weights by the normalised gradient of the loss w.r.t. an implicit multiplicative factor “c” on the weights, denoted as the “sensitivity”. Essentially, this criterion takes two factors into account when determining the relevance of each weight; the scale of the gradient and the scale of the actual weight. The authors then rank the weights according to their sensitivity and remove the ones that are not in the top-k. They then proceed to train the surviving weights as normal on the task at hand. In experiments they show that this method can offer competitive results while being much simpler to implement than other methods in the literature.