SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY

1 INTRODUCTION . Despite the success of deep neural networks in machine learning , they are often found to be highly overparametrized making them computationally expensive with excessive memory requirements . Pruning such large networks with minimal loss in performance is appealing for real-time applications , especially on resource-limited devices . In addition , compressed neural networks utilize the model capacity efficiently , and this interpretation can be used to derive better generalization bounds for neural networks ( Arora et al . ( 2018 ) ) . In network pruning , given a large reference neural network , the goal is to learn a much smaller subnetwork that mimics the performance of the reference network . The majority of existing methods in the literature attempt to find a subset of weights from the pretrained reference network either based on a saliency criterion ( Mozer & Smolensky ( 1989 ) ; LeCun et al . ( 1990 ) ; Han et al . ( 2015 ) ) or utilizing sparsity enforcing penalties ( Chauvin ( 1989 ) ; Carreira-Perpiñán & Idelbayev ( 2018 ) ) . Unfortunately , since pruning is included as a part of an iterative optimization procedure , all these methods require many expensive prune – retrain cycles and heuristic design choices with additional hyperparameters , making them non-trivial to extend to new architectures and tasks . In this work , we introduce a saliency criterion that identifies connections in the network that are important to the given task in a data-dependent way before training . Specifically , we discover important connections based on their influence on the loss function at a variance scaling initialization , which we call connection sensitivity . Given the desired sparsity level , redundant connections are pruned once prior to training ( i.e. , single-shot ) , and then the sparse pruned network is trained in the standard way . Our approach has several attractive properties : • Simplicity . Since the network is pruned once prior to training , there is no need for pretraining and complex pruning schedules . Our method has no additional hyperparameters and once pruned , training of the sparse network is performed in the standard way . • Versatility . Since our saliency criterion chooses structurally important connections , it is robust to architecture variations . Therefore our method can be applied to various architectures including convolutional , residual and recurrent networks with no modifications . i • Interpretability . Our method determines important connections with a mini-batch of data at single-shot . By varying this mini-batch used for pruning , our method enables us to verify that the retained connections are indeed essential for the given task . We evaluate our method on MNIST , CIFAR-10 , and Tiny-ImageNet classification datasets with widely varying architectures . Despite being the simplest , our method obtains extremely sparse networks with virtually the same accuracy as the existing baselines across all tested architectures . Furthermore , we investigate the relevance of the retained connections as well as the effect of the network initialization and the dataset on the saliency score . 2 RELATED WORK . Classical methods . Essentially , early works in network pruning can be categorized into two groups ( Reed ( 1993 ) ) : 1 ) those that utilize sparsity enforcing penalties ; and 2 ) methods that prune the network based on some saliency criterion . The methods from the former category ( Chauvin ( 1989 ) ; Weigend et al . ( 1991 ) ; Ishikawa ( 1996 ) ) augment the loss function with some sparsity enforcing penalty terms ( e.g. , L0 or L1 norm ) , so that back-propagation effectively penalizes the magnitude of the weights during training . Then weights below a certain threshold may be removed . On the other hand , classical saliency criteria include the sensitivity of the loss with respect to the neurons ( Mozer & Smolensky ( 1989 ) ) or the weights ( Karnin ( 1990 ) ) and Hessian of the loss with respect to the weights ( LeCun et al . ( 1990 ) ; Hassibi et al . ( 1993 ) ) . Since these criteria are heavily dependent on the scale of the weights and are designed to be incorporated within the learning process , these methods are prohibitively slow requiring many iterations of pruning and learning steps . Our approach identifies redundant weights from an architectural point of view and prunes them once at the beginning before training . Modern advances . In recent years , the increased space and time complexities as well as the risk of overfitting in deep neural networks prompted a surge of further investigation in network pruning . While Hessian based approaches employ the diagonal approximation due to its computational simplicity , impressive results ( i.e. , extreme sparsity without loss in accuracy ) are achieved using magnitude of the weights as the criterion ( Han et al . ( 2015 ) ) . This made them the de facto standard method for network pruning and led to various implementations ( Guo et al . ( 2016 ) ; Carreira-Perpiñán & Idelbayev ( 2018 ) ) . The magnitude criterion is also extended to recurrent neural networks ( Narang et al . ( 2017 ) ) , yet with heavily tuned hyperparameter setting . Unlike our approach , the main drawbacks of magnitude based approaches are the reliance on pretraining and the expensive prune – retrain cycles . Furthermore , since pruning and learning steps are intertwined , they often require highly heuristic design choices which make them non-trivial to be extended to new architectures and different tasks . Meanwhile , Bayesian methods are also applied to network pruning ( Ullrich et al . ( 2017 ) ; Molchanov et al . ( 2017a ) ) where the former extends the soft weight sharing in Nowlan & Hinton ( 1992 ) to obtain a sparse and compressed network , and the latter uses variational inference to learn the dropout rate which can then be used to prune the network . Unlike the above methods , our approach is simple and easily adaptable to any given architecture or task without modifying the pruning procedure . Network compression in general . Apart from weight pruning , there are approaches focused on structured simplification such as pruning filters ( Li et al . ( 2017 ) ; Molchanov et al . ( 2017b ) ) , structured sparsity with regularizers ( Wen et al . ( 2016 ) ) , low-rank approximation ( Jaderberg et al . ( 2014 ) ) , matrix and tensor factorization ( Novikov et al . ( 2015 ) ) , and sparsification using expander graphs ( Prabhu et al . ( 2018 ) ) or Erdős-Rényi random graph ( Mocanu et al . ( 2018 ) ) . In addition , there is a large body of work on compressing the representation of weights . A non-exhaustive list includes quantization ( Gong et al . ( 2014 ) ) , reduced precision ( Gupta et al . ( 2015 ) ) and binary weights ( Hubara et al . ( 2016 ) ) . In this work , we focus on weight pruning that is free from structural constraints and amenable to further compression schemes . 3 NEURAL NETWORK PRUNING . The main hypothesis behind the neural network pruning literature is that neural networks are usually overparametrized , and comparable performance can be obtained by a much smaller network ( Reed ( 1993 ) ) while improving generalization ( Arora et al . ( 2018 ) ) . To this end , the objective is to learn ii a sparse network while maintaining the accuracy of the standard reference network . Let us first formulate neural network pruning as an optimization problem . Given a dataset D = { ( xi , yi ) } ni=1 , and a desired sparsity level κ ( i.e. , the number of non-zero weights ) neural network pruning can be written as the following constrained optimization problem : min w L ( w ; D ) = min w 1 n n∑ i=1 ` ( w ; ( xi , yi ) ) , ( 1 ) s.t . w ∈ Rm , ‖w‖0 ≤ κ . Here , ` ( · ) is the standard loss function ( e.g. , cross-entropy loss ) , w is the set of parameters of the neural network , m is the total number of parameters and ‖ · ‖0 is the standard L0 norm . The conventional approach to optimize the above problem is by adding sparsity enforcing penalty terms ( Chauvin ( 1989 ) ; Weigend et al . ( 1991 ) ; Ishikawa ( 1996 ) ) . Recently , Carreira-Perpiñán & Idelbayev ( 2018 ) attempts to minimize the above constrained optimization problem using the stochastic version of projected gradient descent ( where the projection is accomplished by pruning ) . However , these methods often turn out to be inferior to saliency based methods in terms of resulting sparsity and require heavily tuned hyperparameter settings to obtain comparable results . On the other hand , saliency based methods treat the above problem as selectively removing redundant parameters ( or connections ) in the neural network . In order to do so , one has to come up with a good criterion to identify such redundant connections . Popular criteria include magnitude of the weights , i.e. , weights below a certain threshold are redundant ( Han et al . ( 2015 ) ; Guo et al . ( 2016 ) ) and Hessian of the loss with respect to the weights , i.e. , the higher the value of Hessian , the higher the importance of the parameters ( LeCun et al . ( 1990 ) ; Hassibi et al . ( 1993 ) ) , defined as follows : sj = |wj | , for magnitude based w2jHjj 2 or w2j 2H−1jj for Hessian based . ( 2 ) Here , for connection j , sj is the saliency score , wj is the weight , and Hjj is the value of the Hessian matrix , where the Hessian H = ∂2L/∂w2 ∈ Rm×m . Considering Hessian based methods , the Hessian matrix is neither diagonal nor positive definite in general , approximate at best , and intractable to compute for large networks . Despite being popular , both of these criteria depend on the scale of the weights and in turn require pretraining and are very sensitive to the architectural choices . For instance , different normalization layers affect the scale of the weights in a different way , and this would non-trivially affect the saliency score . Furthermore , pruning and the optimization steps are alternated many times throughout training , resulting in highly expensive prune – retrain cycles . Such an exorbitant requirement hinders the use of pruning methods in large-scale applications and raises questions about the credibility of the existing pruning criteria . In this work , we design a criterion which directly measures the connection importance in a datadependent manner . This alleviates the dependency on the weights and enables us to prune the network once at the beginning , and then the training can be performed on the sparse pruned network . Therefore , our method eliminates the need for the expensive prune – retrain cycles , and in theory , it can be an order of magnitude faster than the standard neural network training as it can be implemented using software libraries that support sparse matrix computations . 4 SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY . Given a neural network and a dataset , our goal is to design a method that can selectively prune redundant connections for the given task in a data-dependent way even before training . To this end , we first introduce a criterion to identify important connections and then discuss its benefits .

Summary: The paper focuses on pruning neural networks. They propose to identify the nodes to be pruned even before training the whole network (conventionally, it is done as a separate step after the nn was trained and involves a number of iterations of retraining pruned nn). This initial step that identifies the connections to be pruned works off a mini-batch of data.