Rethinking the Value of Network Pruning

1 INTRODUCTION . Over-parameterization is a widely-recognized property of deep neural networks ( Denton et al. , 2014 ; Ba & Caruana , 2014 ) , which leads to high computational cost and high memory footprint for inference . As a remedy , network pruning ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; Molchanov et al. , 2016 ; Li et al. , 2017 ) has been identified as an effective technique to improve the efficiency of deep networks for applications with limited computational budget . A typical procedure of network pruning consists of three stages : 1 ) train a large , over-parameterized model ( sometimes there are pretrained models available ) , 2 ) prune the trained large model according to a certain criterion , and 3 ) fine-tune the pruned model to regain the lost performance . Generally , there are two common beliefs behind this pruning procedure . First , it is believed that starting with training a large , over-parameterized network is important ( Luo et al. , 2017 ; CarreiraPerpinán & Idelbayev , 2018 ) , as it provides a highperformance model ( due to stronger representation & optimization power ) from which one can safely remove a set of redundant parameters without sig- nificantly hurting the accuracy . Therefore , this is usually believed , and reported to be superior to directly training a smaller network from scratch ( Li et al. , 2017 ; Luo et al. , 2017 ; He et al. , 2017b ; Yu et al. , 2018 ) – a commonly used baseline approach . Second , both the pruned architecture and its associated weights are believed to be essential for obtaining the final efficient model ( Han et al. , ∗Equal contribution . †Work done while visiting UC Berkeley . 2015 ) . Thus most existing pruning techniques choose to fine-tune a pruned model instead of training it from scratch . The preserved weights after pruning are usually considered to be critical , as how to accurately select the set of important weights is a very active research topic in the literature ( Molchanov et al. , 2016 ; Li et al. , 2017 ; Luo et al. , 2017 ; He et al. , 2017b ; Liu et al. , 2017 ; Suau et al. , 2018 ) . In this work , we show that both of the beliefs mentioned above are not necessarily true for structured pruning methods , which prune at the levels of convolution channels or larger . Based on an extensive empirical evaluation of state-of-the-art pruning algorithms on multiple datasets with multiple network architectures , we make two surprising observations . First , for structured pruning methods with predefined target network architectures ( Figure 2 ) , directly training the small target model from random initialization can achieve the same , if not better , performance , as the model obtained from the three-stage pipeline . In this case , starting with a large model is not necessary and one could instead directly train the target model from scratch . Second , for structured pruning methods with autodiscovered target networks , training the pruned model from scratch can also achieve comparable or even better performance than fine-tuning . This observation shows that for these pruning methods , what matters more may be the obtained architecture , instead of the preserved weights , despite training the large model is needed to find that target architecture . Interestingly , for a unstructured pruning method ( Han et al. , 2015 ) that prunes individual parameters , we found that training from scratch can mostly achieve comparable accuracy with pruning and fine-tuning on smaller-scale datasets , but fails to do so on the large-scale ImageNet benchmark . Note that in some cases , if a pretrained large model is already available , pruning and fine-tuning from it can save the training time required to obtain the efficient model . The contradiction between some of our results and those reported in the literature might be explained by less carefully chosen hyper-parameters , data augmentation schemes and unfair computation budget for evaluating baseline approaches . Our results advocate a rethinking of existing structured network pruning algorithms . It seems that the over-parameterization during the first-stage training is not as beneficial as previously thought . Also , inheriting weights from a large model is not necessarily optimal , and might trap the pruned model into a bad local minimum , even if the weights are considered “ important ” by the pruning criterion . Instead , our results suggest that the value of automatic structured pruning algorithms sometimes lie in identifying efficient structures and performing implicit architecture search , rather than selecting “ important ” weights . For most structured pruning methods which prune channels/filters , this corresponds to searching the number of channels in each layer . In section 5 , we discuss this viewpoint through carefully designed experiments , and show the patterns in the pruned model could provide design guidelines for efficient architectures . The rest of the paper is organized as follows : in Section 2 , we introduce background and some related works on network pruning ; in Section 3 , we describe our methodology for training the pruned model from scratch ; in Section 4 we experiment on various pruning methods and show our main results for both pruning methods with predefined or automatically discovered target architectures ; in Section 5 , we discuss the value of automatic pruning methods in searching efficient network architectures ; in Section 6 we discuss some implications and conclude the paper . 2 BACKGROUND . Recent success of deep convolutional networks ( LeCun et al. , 1998 ; Deng et al. , 2009 ; Girshick et al. , 2014 ; Long et al. , 2015 ; He et al. , 2016 ; 2017a ) has been coupled with increased requirement of computation resources . In particular , the model size , memory footprint , the number of computation operations ( FLOPs ) and power usage are major aspects inhibiting the use of deep neural networks in some resource-constrained settings . Those large models can be infeasible to store , and run in real time on embedded systems . To address this issue , many methods have been proposed such as low-rank approximation of weights ( Denton et al. , 2014 ; Lebedev et al. , 2014 ) , weight quantization ( Courbariaux et al. , 2016 ; Rastegari et al. , 2016 ) , knowledge distillation ( Hinton et al. , 2014 ; Romero et al. , 2015 ) and network pruning ( Han et al. , 2015 ; Li et al. , 2017 ) , among which network pruning has gained notable attention due to their competitive performance and compatibility . One major branch of network pruning methods is individual weight pruning , and it dates back to Optimal Brain Damage ( LeCun et al. , 1990 ) and Optimal Brain Surgeon ( Hassibi & Stork , 1993 ) , which prune weights based on Hessian of the loss function . More recently , Han et al . ( 2015 ) proposes to prune network weights with small magnitude , and this technique is further incorporated into the “ Deep Compression ” pipeline ( Han et al. , 2016b ) to obtain highly compressed models . Srinivas & Babu ( 2015 ) proposes a data-free algorithm to remove redundant neurons iteratively . Molchanov et al . ( 2017 ) uses Variatonal Dropout ( P. Kingma et al. , 2015 ) to prune redundant weights . Louizos et al . ( 2018 ) learns sparse networks through L0-norm regularization based on stochastic gate . However , one drawback of these unstructured pruning methods is that the resulting weight matrices are sparse , which can not lead to compression and speedup without dedicated hardware/libraries ( Han et al. , 2016a ) . In contrast , structured pruning methods prune at the level of channels or even layers . Since the original convolution structure is still preserved , no dedicated hardware/libraries are required to realize the benefits . Among structured pruning methods , channel pruning is the most popular , since it operates at the most fine-grained level while still fitting in conventional deep learning frameworks . Some heuristic methods include pruning channels based on their corresponding filter weight norm ( Li et al. , 2017 ) and average percentage of zeros in the output ( Hu et al. , 2016 ) . Group sparsity is also widely used to smooth the pruning process after training ( Wen et al. , 2016 ; Alvarez & Salzmann , 2016 ; Lebedev & Lempitsky , 2016 ; Zhou et al. , 2016 ) . Liu et al . ( 2017 ) and Ye et al . ( 2018 ) impose sparsity constraints on channel-wise scaling factors during training , whose magnitudes are then used for channel pruning . Huang & Wang ( 2018 ) uses a similar technique to prune coarser structures such as residual blocks . He et al . ( 2017b ) and Luo et al . ( 2017 ) minimizes next layer ’ s feature reconstruction error to determine which channels to keep . Similarly , Yu et al . ( 2018 ) optimizes the reconstruction error of the final response layer and propagates a “ importance score ” for each channel . Molchanov et al . ( 2016 ) uses Taylor expansion to approximate each channel ’ s influence over the final loss and prune accordingly . Suau et al . ( 2018 ) analyzes the intrinsic correlation within each layer and prune redundant channels . Chin et al . ( 2018 ) proposes a layer-wise compensate filter pruning algorithm to improve commonly-adopted heuristic pruning metrics . He et al . ( 2018a ) proposes to allow pruned filters to recover during the training process . Lin et al . ( 2017 ) ; Wang et al . ( 2017 ) prune certain structures in the network based on the current input . Our work is also related to some recent studies on the characteristics of pruning algorithms . Mittal et al . ( 2018 ) shows that random channel pruning ( Anwar & Sung , 2016 ) can perform on par with a variety of more sophisticated pruning criteria , demonstrating the plasticity of network models . In the context of unstructured pruning , The Lottery Ticket Hypothesis ( Frankle & Carbin , 2019 ) conjectures that certain connections together with their randomly initialized weights , can enable a comparable accuracy with the original network when trained in isolation . We provide comparisons between Frankle & Carbin ( 2019 ) and this work in Appendix A. Zhu & Gupta ( 2018 ) shows that training a small-dense model can not achieve the same accuracy as a pruned large-sparse model with identical memory footprint . In this work , we reveal a different and rather surprising characteristic of structured network pruning methods : fine-tuning the pruned model with inherited weights is not better than training it from scratch ; the resulting pruned architectures are more likely to be what brings the benefit . 3 METHODOLOGY . In this section , we describe in detail our methodology for training a small target model from scratch . Target Pruned Architectures . We first divide network pruning methods into two categories . In a pruning pipeline , the target pruned model ’ s architecture can be determined by either a human ( i.e. , predefined ) or the pruning algorithm ( i.e. , automatic ) ( see Figure 2 ) . When a human predefines the target architecture , a common criterion is the ratio of channels to prune in each layer . For example , we may want to prune 50 % channels in each layer of VGG . In this case , no matter which specific channels are pruned , the pruned target architecture remains the same , because the pruning algorithm only locally prunes the least important 50 % channels in each layer . In practice , the ratio in each layer is usually selected through empirical studies or heuristics . Examples of predefined structured pruning include Li et al . ( 2017 ) , Luo et al . ( 2017 ) , He et al . ( 2017b ) and He et al . ( 2018a ) When the target architecture is automatically determined by a pruning algorithm , it is usually based on a pruning criterion that globally compares the importance of structures ( e.g. , channels ) across layers . Examples of automatic structured pruning include Liu et al . ( 2017 ) , Huang & Wang ( 2018 ) , Molchanov et al . ( 2016 ) and Suau et al . ( 2018 ) . Unstructured pruning ( Han et al. , 2015 ; Molchanov et al. , 2017 ; Louizos et al. , 2018 ) also falls in the category of automatic methods , where the positions of pruned weights are determined by the training process and the pruning algorithm , and it is usually not possible to predefine the positions of zeros before training starts . Datasets , Network Architectures and Pruning Methods . In the network pruning literature , CIFAR-10 , CIFAR-100 ( Krizhevsky , 2009 ) , and ImageNet ( Deng et al. , 2009 ) datasets are the de-facto benchmarks , while VGG ( Simonyan & Zisserman , 2015 ) , ResNet ( He et al. , 2016 ) and DenseNet ( Huang et al. , 2017 ) are the common network architectures . We evaluate four predefined pruning methods , Li et al . ( 2017 ) , Luo et al . ( 2017 ) , He et al . ( 2017b ) , He et al . ( 2018a ) , two automatic structured pruning methods , Liu et al . ( 2017 ) , Huang & Wang ( 2018 ) , and one unstructured pruning method ( Han et al. , 2015 ) . For the first six methods , we evaluate using the same ( target model , dataset ) pairs as presented in the original paper to keep our results comparable . For the last one ( Han et al. , 2015 ) , we use the aforementioned architectures instead , since the ones in the original paper are no longer state-of-the-art . On CIFAR datasets , we run each experiment with 5 random seeds , and report the mean and standard deviation of the accuracy . Training Budget . One crucial question is how long we should train the small pruned model from scratch . Naively training for the same number of epochs as we train the large model might be unfair , since the small pruned model requires significantly less computation for one epoch . Alternatively , we could compute the floating point operations ( FLOPs ) for both the pruned and large models , and choose the number of training epoch for the pruned model that would lead to the same amount of computation as training the large model . Note that it is not clear how to train the models to “ full convergence ” given the stepwise decaying learning rate schedule commonly used in the CIFAR/ImageNet classification tasks . In our experiments , we use Scratch-E to denote training the small pruned models for the same epochs , and Scratch-B to denote training for the same amount of computation budget ( on ImageNet , if the pruned model saves more than 2× FLOPs , we just double the number of epochs for training Scratch-B , which amounts to less computation budget than large model training ) . When extending the number of epochs in Scratch-B , we also extend the learning rate decay schedules proportionally . One may argue that we should instead train the small target model for fewer epochs since it may converge faster . However , in practice we found that increasing the training epochs within a reasonable range is rarely harmful . In our experiments we found in most times Scratch-E is enough while in other cases Scratch-B is needed for a comparable accuracy as fine-tuning . Note that our evaluations use the same computation as large model training without considering the computation in fine-tuning , since in our evaluated methods fine-tuning does not take too long ; if anything this still favors the pruning and fine-tuning pipeline . Implementation . In order to keep our setup as close to the original papers as possible , we use the following protocols : 1 ) ff a previous pruning method ’ s training setup is publicly available , e.g . Liu et al . ( 2017 ) , Huang & Wang ( 2018 ) and He et al . ( 2018a ) , we adopt the original implementation ; 2 ) otherwise , for simpler pruning methods , e.g. , Li et al . ( 2017 ) and Han et al . ( 2015 ) , we re-implement the three-stage pruning procedure and generally achieve similar results as in the original papers ; 3 ) for the remaining two methods ( Luo et al. , 2017 ; He et al. , 2017b ) , the pruned models are publicly available but without the training setup , thus we choose to re-train both large and small target models from scratch . Interestingly , the accuracy of our re-trained large model is higher than what is reported in the original papers . This could be due to the difference in the deep learning frameworks : we used Pytorch ( Paszke et al. , 2017 ) while the original papers used Caffe ( Jia et al. , 2014 ) . In this case , to accommodate the effects of different frameworks and training setups , we report the relative accuracy drop from the unpruned large model . We use standard training hyper-parameters and data-augmentation schemes , which are used both in standard image classification models ( He et al. , 2016 ; Huang et al. , 2017 ) and network pruning methods ( Li et al. , 2017 ; Liu et al. , 2017 ; Huang & Wang , 2018 ; He et al. , 2018a ) . The optimization method is SGD with Nesterov momentum , using an stepwise decay learning rate schedule . For random weight initialization , we adopt the scheme proposed in ( He et al. , 2015 ) . For results of models fine-tuned from inherited weights , we either use the released models from original papers ( case 3 above ) or follow the common practice of fine-tuning the model using the lowest learning rate when training the large model ( Li et al. , 2017 ; He et al. , 2017b ) . For CIFAR , training/fine-tuning takes 160/40 epochs . For ImageNet , training/fine-tuning takes 90/20 epochs . For reproducing the results and a more detailed knowledge about the settings , see our code at : https : //github . com/Eric-mingjie/rethinking-network-pruning .

Summary: This paper shows through a set of experiments that the common belief that a large neural network trained, then pruned and fine-tuned performs better than another network that has the same size of the pruned one, but trained from scratch, is actually false. That is, a pruned network does not perform better than a network with the same dimensions but trained from scratch. Also, the authors consider that what is important for good performance is to know how many weights/filters are needed at each layer, while the actual values of the weights do not matter. Then, what happens in a standard large neural network training can be seen as an architecture search, in which the algorithm learns what is the right amount of weights for each layer.