Rethinking the Value of Network Pruning

1 INTRODUCTION . Over-parameterization is a widely-recognized property of deep neural networks ( Denton et al. , 2014 ; Ba & Caruana , 2014 ) , which leads to high computational cost and high memory footprint for inference . As a remedy , network pruning ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; Molchanov et al. , 2016 ; Li et al. , 2017 ) has been identified as an effective technique to improve the efficiency of deep networks for applications with limited computational budget . A typical procedure of network pruning consists of three stages : 1 ) train a large , over-parameterized model ( sometimes there are pretrained models available ) , 2 ) prune the trained large model according to a certain criterion , and 3 ) fine-tune the pruned model to regain the lost performance . Generally , there are two common beliefs behind this pruning procedure . First , it is believed that starting with training a large , over-parameterized network is important ( Luo et al. , 2017 ; CarreiraPerpinán & Idelbayev , 2018 ) , as it provides a highperformance model ( due to stronger representation & optimization power ) from which one can safely remove a set of redundant parameters without sig- nificantly hurting the accuracy . Therefore , this is usually believed , and reported to be superior to directly training a smaller network from scratch ( Li et al. , 2017 ; Luo et al. , 2017 ; He et al. , 2017b ; Yu et al. , 2018 ) – a commonly used baseline approach . Second , both the pruned architecture and its associated weights are believed to be essential for obtaining the final efficient model ( Han et al. , ∗Equal contribution . †Work done while visiting UC Berkeley . 2015 ) . Thus most existing pruning techniques choose to fine-tune a pruned model instead of training it from scratch . The preserved weights after pruning are usually considered to be critical , as how to accurately select the set of important weights is a very active research topic in the literature ( Molchanov et al. , 2016 ; Li et al. , 2017 ; Luo et al. , 2017 ; He et al. , 2017b ; Liu et al. , 2017 ; Suau et al. , 2018 ) . In this work , we show that both of the beliefs mentioned above are not necessarily true for structured pruning methods , which prune at the levels of convolution channels or larger . Based on an extensive empirical evaluation of state-of-the-art pruning algorithms on multiple datasets with multiple network architectures , we make two surprising observations . First , for structured pruning methods with predefined target network architectures ( Figure 2 ) , directly training the small target model from random initialization can achieve the same , if not better , performance , as the model obtained from the three-stage pipeline . In this case , starting with a large model is not necessary and one could instead directly train the target model from scratch . Second , for structured pruning methods with autodiscovered target networks , training the pruned model from scratch can also achieve comparable or even better performance than fine-tuning . This observation shows that for these pruning methods , what matters more may be the obtained architecture , instead of the preserved weights , despite training the large model is needed to find that target architecture . Interestingly , for a unstructured pruning method ( Han et al. , 2015 ) that prunes individual parameters , we found that training from scratch can mostly achieve comparable accuracy with pruning and fine-tuning on smaller-scale datasets , but fails to do so on the large-scale ImageNet benchmark . Note that in some cases , if a pretrained large model is already available , pruning and fine-tuning from it can save the training time required to obtain the efficient model . The contradiction between some of our results and those reported in the literature might be explained by less carefully chosen hyper-parameters , data augmentation schemes and unfair computation budget for evaluating baseline approaches . Our results advocate a rethinking of existing structured network pruning algorithms . It seems that the over-parameterization during the first-stage training is not as beneficial as previously thought . Also , inheriting weights from a large model is not necessarily optimal , and might trap the pruned model into a bad local minimum , even if the weights are considered “ important ” by the pruning criterion . Instead , our results suggest that the value of automatic structured pruning algorithms sometimes lie in identifying efficient structures and performing implicit architecture search , rather than selecting “ important ” weights . For most structured pruning methods which prune channels/filters , this corresponds to searching the number of channels in each layer . In section 5 , we discuss this viewpoint through carefully designed experiments , and show the patterns in the pruned model could provide design guidelines for efficient architectures . The rest of the paper is organized as follows : in Section 2 , we introduce background and some related works on network pruning ; in Section 3 , we describe our methodology for training the pruned model from scratch ; in Section 4 we experiment on various pruning methods and show our main results for both pruning methods with predefined or automatically discovered target architectures ; in Section 5 , we discuss the value of automatic pruning methods in searching efficient network architectures ; in Section 6 we discuss some implications and conclude the paper . 2 BACKGROUND . Recent success of deep convolutional networks ( LeCun et al. , 1998 ; Deng et al. , 2009 ; Girshick et al. , 2014 ; Long et al. , 2015 ; He et al. , 2016 ; 2017a ) has been coupled with increased requirement of computation resources . In particular , the model size , memory footprint , the number of computation operations ( FLOPs ) and power usage are major aspects inhibiting the use of deep neural networks in some resource-constrained settings . Those large models can be infeasible to store , and run in real time on embedded systems . To address this issue , many methods have been proposed such as low-rank approximation of weights ( Denton et al. , 2014 ; Lebedev et al. , 2014 ) , weight quantization ( Courbariaux et al. , 2016 ; Rastegari et al. , 2016 ) , knowledge distillation ( Hinton et al. , 2014 ; Romero et al. , 2015 ) and network pruning ( Han et al. , 2015 ; Li et al. , 2017 ) , among which network pruning has gained notable attention due to their competitive performance and compatibility . One major branch of network pruning methods is individual weight pruning , and it dates back to Optimal Brain Damage ( LeCun et al. , 1990 ) and Optimal Brain Surgeon ( Hassibi & Stork , 1993 ) , which prune weights based on Hessian of the loss function . More recently , Han et al . ( 2015 ) proposes to prune network weights with small magnitude , and this technique is further incorporated into the “ Deep Compression ” pipeline ( Han et al. , 2016b ) to obtain highly compressed models . Srinivas & Babu ( 2015 ) proposes a data-free algorithm to remove redundant neurons iteratively . Molchanov et al . ( 2017 ) uses Variatonal Dropout ( P. Kingma et al. , 2015 ) to prune redundant weights . Louizos et al . ( 2018 ) learns sparse networks through L0-norm regularization based on stochastic gate . However , one drawback of these unstructured pruning methods is that the resulting weight matrices are sparse , which can not lead to compression and speedup without dedicated hardware/libraries ( Han et al. , 2016a ) . In contrast , structured pruning methods prune at the level of channels or even layers . Since the original convolution structure is still preserved , no dedicated hardware/libraries are required to realize the benefits . Among structured pruning methods , channel pruning is the most popular , since it operates at the most fine-grained level while still fitting in conventional deep learning frameworks . Some heuristic methods include pruning channels based on their corresponding filter weight norm ( Li et al. , 2017 ) and average percentage of zeros in the output ( Hu et al. , 2016 ) . Group sparsity is also widely used to smooth the pruning process after training ( Wen et al. , 2016 ; Alvarez & Salzmann , 2016 ; Lebedev & Lempitsky , 2016 ; Zhou et al. , 2016 ) . Liu et al . ( 2017 ) and Ye et al . ( 2018 ) impose sparsity constraints on channel-wise scaling factors during training , whose magnitudes are then used for channel pruning . Huang & Wang ( 2018 ) uses a similar technique to prune coarser structures such as residual blocks . He et al . ( 2017b ) and Luo et al . ( 2017 ) minimizes next layer ’ s feature reconstruction error to determine which channels to keep . Similarly , Yu et al . ( 2018 ) optimizes the reconstruction error of the final response layer and propagates a “ importance score ” for each channel . Molchanov et al . ( 2016 ) uses Taylor expansion to approximate each channel ’ s influence over the final loss and prune accordingly . Suau et al . ( 2018 ) analyzes the intrinsic correlation within each layer and prune redundant channels . Chin et al . ( 2018 ) proposes a layer-wise compensate filter pruning algorithm to improve commonly-adopted heuristic pruning metrics . He et al . ( 2018a ) proposes to allow pruned filters to recover during the training process . Lin et al . ( 2017 ) ; Wang et al . ( 2017 ) prune certain structures in the network based on the current input . Our work is also related to some recent studies on the characteristics of pruning algorithms . Mittal et al . ( 2018 ) shows that random channel pruning ( Anwar & Sung , 2016 ) can perform on par with a variety of more sophisticated pruning criteria , demonstrating the plasticity of network models . In the context of unstructured pruning , The Lottery Ticket Hypothesis ( Frankle & Carbin , 2019 ) conjectures that certain connections together with their randomly initialized weights , can enable a comparable accuracy with the original network when trained in isolation . We provide comparisons between Frankle & Carbin ( 2019 ) and this work in Appendix A. Zhu & Gupta ( 2018 ) shows that training a small-dense model can not achieve the same accuracy as a pruned large-sparse model with identical memory footprint . In this work , we reveal a different and rather surprising characteristic of structured network pruning methods : fine-tuning the pruned model with inherited weights is not better than training it from scratch ; the resulting pruned architectures are more likely to be what brings the benefit . 3 METHODOLOGY . In this section , we describe in detail our methodology for training a small target model from scratch . Target Pruned Architectures . We first divide network pruning methods into two categories . In a pruning pipeline , the target pruned model ’ s architecture can be determined by either a human ( i.e. , predefined ) or the pruning algorithm ( i.e. , automatic ) ( see Figure 2 ) . When a human predefines the target architecture , a common criterion is the ratio of channels to prune in each layer . For example , we may want to prune 50 % channels in each layer of VGG . In this case , no matter which specific channels are pruned , the pruned target architecture remains the same , because the pruning algorithm only locally prunes the least important 50 % channels in each layer . In practice , the ratio in each layer is usually selected through empirical studies or heuristics . Examples of predefined structured pruning include Li et al . ( 2017 ) , Luo et al . ( 2017 ) , He et al . ( 2017b ) and He et al . ( 2018a ) When the target architecture is automatically determined by a pruning algorithm , it is usually based on a pruning criterion that globally compares the importance of structures ( e.g. , channels ) across layers . Examples of automatic structured pruning include Liu et al . ( 2017 ) , Huang & Wang ( 2018 ) , Molchanov et al . ( 2016 ) and Suau et al . ( 2018 ) . Unstructured pruning ( Han et al. , 2015 ; Molchanov et al. , 2017 ; Louizos et al. , 2018 ) also falls in the category of automatic methods , where the positions of pruned weights are determined by the training process and the pruning algorithm , and it is usually not possible to predefine the positions of zeros before training starts . Datasets , Network Architectures and Pruning Methods . In the network pruning literature , CIFAR-10 , CIFAR-100 ( Krizhevsky , 2009 ) , and ImageNet ( Deng et al. , 2009 ) datasets are the de-facto benchmarks , while VGG ( Simonyan & Zisserman , 2015 ) , ResNet ( He et al. , 2016 ) and DenseNet ( Huang et al. , 2017 ) are the common network architectures . We evaluate four predefined pruning methods , Li et al . ( 2017 ) , Luo et al . ( 2017 ) , He et al . ( 2017b ) , He et al . ( 2018a ) , two automatic structured pruning methods , Liu et al . ( 2017 ) , Huang & Wang ( 2018 ) , and one unstructured pruning method ( Han et al. , 2015 ) . For the first six methods , we evaluate using the same ( target model , dataset ) pairs as presented in the original paper to keep our results comparable . For the last one ( Han et al. , 2015 ) , we use the aforementioned architectures instead , since the ones in the original paper are no longer state-of-the-art . On CIFAR datasets , we run each experiment with 5 random seeds , and report the mean and standard deviation of the accuracy . Training Budget . One crucial question is how long we should train the small pruned model from scratch . Naively training for the same number of epochs as we train the large model might be unfair , since the small pruned model requires significantly less computation for one epoch . Alternatively , we could compute the floating point operations ( FLOPs ) for both the pruned and large models , and choose the number of training epoch for the pruned model that would lead to the same amount of computation as training the large model . Note that it is not clear how to train the models to “ full convergence ” given the stepwise decaying learning rate schedule commonly used in the CIFAR/ImageNet classification tasks . In our experiments , we use Scratch-E to denote training the small pruned models for the same epochs , and Scratch-B to denote training for the same amount of computation budget ( on ImageNet , if the pruned model saves more than 2× FLOPs , we just double the number of epochs for training Scratch-B , which amounts to less computation budget than large model training ) . When extending the number of epochs in Scratch-B , we also extend the learning rate decay schedules proportionally . One may argue that we should instead train the small target model for fewer epochs since it may converge faster . However , in practice we found that increasing the training epochs within a reasonable range is rarely harmful . In our experiments we found in most times Scratch-E is enough while in other cases Scratch-B is needed for a comparable accuracy as fine-tuning . Note that our evaluations use the same computation as large model training without considering the computation in fine-tuning , since in our evaluated methods fine-tuning does not take too long ; if anything this still favors the pruning and fine-tuning pipeline . Implementation . In order to keep our setup as close to the original papers as possible , we use the following protocols : 1 ) ff a previous pruning method ’ s training setup is publicly available , e.g . Liu et al . ( 2017 ) , Huang & Wang ( 2018 ) and He et al . ( 2018a ) , we adopt the original implementation ; 2 ) otherwise , for simpler pruning methods , e.g. , Li et al . ( 2017 ) and Han et al . ( 2015 ) , we re-implement the three-stage pruning procedure and generally achieve similar results as in the original papers ; 3 ) for the remaining two methods ( Luo et al. , 2017 ; He et al. , 2017b ) , the pruned models are publicly available but without the training setup , thus we choose to re-train both large and small target models from scratch . Interestingly , the accuracy of our re-trained large model is higher than what is reported in the original papers . This could be due to the difference in the deep learning frameworks : we used Pytorch ( Paszke et al. , 2017 ) while the original papers used Caffe ( Jia et al. , 2014 ) . In this case , to accommodate the effects of different frameworks and training setups , we report the relative accuracy drop from the unpruned large model . We use standard training hyper-parameters and data-augmentation schemes , which are used both in standard image classification models ( He et al. , 2016 ; Huang et al. , 2017 ) and network pruning methods ( Li et al. , 2017 ; Liu et al. , 2017 ; Huang & Wang , 2018 ; He et al. , 2018a ) . The optimization method is SGD with Nesterov momentum , using an stepwise decay learning rate schedule . For random weight initialization , we adopt the scheme proposed in ( He et al. , 2015 ) . For results of models fine-tuned from inherited weights , we either use the released models from original papers ( case 3 above ) or follow the common practice of fine-tuning the model using the lowest learning rate when training the large model ( Li et al. , 2017 ; He et al. , 2017b ) . For CIFAR , training/fine-tuning takes 160/40 epochs . For ImageNet , training/fine-tuning takes 90/20 epochs . For reproducing the results and a more detailed knowledge about the settings , see our code at : https : //github . com/Eric-mingjie/rethinking-network-pruning .

Summary: This paper reinvestigate several recent works on network pruning and find that the common belief about the necessity to train a large network before pruning may not hold. The authors find that training the pruned model from scratch can achieve similar, if not better, performance given enough time of training. Based on these observations, the author conclude that training a larger model followed by pruning is not necessary for obtaining an efficient model with similar performance. In other words, the pruned architecture is more important than the weights inherited from the large model. It reminds researchers to perform stronger baselines before showing complex pruning methods.