Rethinking Again the Value of Network Pruning -- A Dynamical Isometry Perspective

1 INTRODUCTION . Pruning is a time-honored methodology to reduce parameters in a neural network without seriously compromising its performance ( Reed , 1993 ; Sze et al. , 2017 ) . The prevailing pipeline of pruning comprises three steps : 1 ) pretraining : train a dense model ; 2 ) pruning : prune the dense model based on certain rules ; 3 ) finetuning : retrain the pruned model to regain performance . Most existing research focuses on the second step , seeking the best criterion to select unimportant weights so as to incur as less performance degradation as possible . This 3-step pipeline has been practiced for more than 30 years ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ) and is still extensively adopted in today ’ s pruning methods ( Sze et al. , 2017 ) . These said , several recent works ( Crowley et al. , 2018 ; Liu et al. , 2019 ) questioned the necessity of inheriting weights from a pretrained model because they empirically found the small model trained from scratch can match ( or sometimes outperform ) the counterpart pruned from the pre-trained large model . This acutely challenges the past wisdom as well as our common belief about pruning . As far as we know , there is no formal response to this critical conflict . A theoretical-level understanding of this problem is even more elusive . Meanwhile , the pruning community has been observing even more open questions . Specifically , ( Renda et al. , 2020 ; Le & Hua , 2021 ) found that the learning rate ( LR ) in finetuning holds a critical role in the final performance . A proper learning rate schedule ( e.g. , a larger initial LR 10−2 vs. 10−3 with step-decay schedule ) can improve the top-1 accuracy of a pruned ResNet-34 model ( He et al. , 2016 ) by more than 1 % on ImageNet ( Deng et al. , 2009 ) . This discovery calls for more attention being paid to the finetuning step when comparing different pruning methods . Unfortunately , they did not present more theoretical insights to explain its occurrence . This also remains an open question in the community up to date . In this paper , we will show these two open questions actually point to the same one . Specifically , we rerun the experiments of ( Crowley et al. , 2018 ; Liu et al. , 2019 ) and find simply using a larger finetuning LR ( 10−2 vs. 10−3 and decay it ) can significantly improve the final performance . Compared to the improved pruning performance , training from scratch does not compete or surpass pruning anymore ( see Tab . 1 and Tab . 2 on ImageNet ) . This observation invites many questions immediately : ( 1 ) Theoretical understanding : Why does this happen ? What is the theoretical reason behind it ? ( 2 ) Practical solution : If this is a problem , how to fix it ? Can the understanding of this problem lead us to better pruning algorithms ? This paper will present answers to all these questions . The key tool we employ to unveil the mysteries is dynamical isometry ( Saxe et al. , 2014 ) , which describes a kind of nice property in neural networks that are easy to optimize . We carefully design an explanatory experiment using a linear MLP ( multi-layer perceptron ) network to demonstrate how finetuning LR affects the final performance by affecting dynamical isometry . In brief , we observe the finetuning process can recover dynamical isometry ; a larger LR can help recover it faster ( or better ) , hence the better final performance . The proposed explanation is validated by our empirical results and resonates with many empirical observations . Furthermore , by the explanation , we learn dynamical isometry recovery is rather imperative . To achieve so , we present a very simple regularization-based method for pruning and show its effectiveness in recovering dynamical isometry on modern residual convolutional neural networks ( CNNs ) . Contributions . ( 1 ) We empirically demonstrate the questioning about the value of inheriting weights in structured pruning in previous works is inaccurate and point out that the direct cause is improperly using a small finetuning LR . Our finding justifies the value of inheriting weights in structured pruning . ( 2 ) On top of the empirical finding , more importantly , we present a theoretical explanation through examining the dynamical isometry of networks in pruning . This explanation is empirically validated by our carefully designed control experiments . ( 3 ) In addition to the theoretical understanding , we also propose a regularization-based method for dynamical isometry recovery . Despite its brutal simplicity , it is shown effective to recover the broken dynamical isometry on modern residual convolutional neural networks . 2 RELATED WORK . Conventional pruning . Pruning aims to remove as many parameters as possible in a neural network meanwhile maintaining its performance . There are many ways to categorize pruning methods . The most popular two are grouping by pruning structure and methodology . ( 1 ) In terms of pruning structure , pruning can be specified into unstructured pruning ( Han et al. , 2015 ; 2016 ) and structured pruning ( Wen et al. , 2016 ; Li et al. , 2017 ; He et al. , 2017 ) . For the former , a single weight is the basic pruning element . Unstructured pruning can deliver a high compression ratio ; whereas , without regularization , the pruned locations usually spread randomly in the network , which is hard to exploit for acceleration . On the opposite , structured pruning introduces certain patterns in the pruned locations , which benefit subsequent acceleration while can not achieve as much compression . Choices between unstructured and structured pruning depend on specific application needs . For structured pruning , there are still many sub-groups ( Mao et al. , 2017 ) . In the literature , without specific mention , structured pruning means filter pruning or channel pruning . This paper focuses on structured ( filter ) pruning because the “ no value of inheriting weights ” argument is mainly discussed in this context ( Liu et al. , 2019 ) . ( 2 ) In terms of pruning methodology ( i.e. , how to select unimportant weights to prune ) , pruning falls into two paradigms in general : importance-based and penalty-based . The former prunes weights based on some established importance criteria , such as magnitude ( for unstructured pruning ) ( Han et al. , 2015 ; 2016 ) or L1-norm ( for filter pruning ) ( Li et al. , 2017 ) , saliency based on 2nd-order gradients ( e.g. , Hessian or Fisher ) ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Theis et al. , 2018 ; Wang et al. , 2019a ; Singh & Alistarh , 2020 ) . The latter adds a penalty term to the objective function , drives unimportant weights towards zero , then removes those with the smallest magnitude . Note , the two groups are not starkly separated . Many methods take wisdom from both sides . For example , ( Ding et al. , 2018 ; Wang et al. , 2019b ; 2021b ) select unimportant weights by magnitude ( akin to the first group ) while also employing the regularization to penalize weights ( akin to the second group ) . There is no conclusion about which paradigm is better , yet empirically , the state-of-the-art pruning methods are closer to the second paradigm , i.e. , deciding weights via training instead of some derived formulas . Although no theories have formally discussed the reason , we can take a rough guess with the knowledge from this paper : Training can recover dynamical isometry , which is beneficial to subsequent finetuning . For more comprehensive literature , we refer interested readers to several surveys : an outdated one ( Reed , 1993 ) , some recent surveys of pruning alone ( Gale et al. , 2019 ; Blalock et al. , 2020 ) or pruning as a sub-topic under the general umbrella of model compression and acceleration ( Sze et al. , 2017 ; Cheng et al. , 2018a ; b ; Deng et al. , 2020 ) . Pruning at initialization ( PaI ) . Recent years have seen several new pruning paradigms . The most prominent one is pruning at initialization . Different from the conventional pruning , which prunes a pretrained model , PaI methods prune a randomly initialized model . Existing PaI approaches mainly include ( Lee et al. , 2019 ; 2020 ; Wang et al. , 2020 ; Frankle et al. , 2021 ; Ramanujan et al. , 2020 ) and the series of lottery ticket hypothesis ( Frankle & Carbin , 2019 ; Frankle et al. , 2020 ) . Interested readers may refer to ( Wang et al. , 2021a ) for a comprehensive summary about PaI . This topic is relevant to this work mainly because one PaI paper ( Lee et al. , 2020 ) also examines pruning using the tool of dynamical isometry . The similarity between our paper and theirs is that we both employ dynamical isometry as a tool to examine the property of network pruning . However , our paper is significantly different from theirs in many axes : ( 1 ) Basic setting . The most obvious difference is that we focus on pruning a pretrained model while ( Lee et al. , 2020 ) focuses on pruning at initialization ( PaI ) . They are two different tracks in pruning ( as such , PaI methods typically do not compare with the methods of pruning pretrained models ) and the latter was shown to consistently underperform the former ( Frankle et al. , 2021 ; Wang et al. , 2021a ) . ( 2 ) Motivation . Despite the same tool ( mean JSV ) , ( Lee et al. , 2020 ) uses it to select unimportant weights to prune ( i.e. , for a new pruning criterion ) , while we use it to analyze why finetuning LR has a significant impact on final performance . The role of finetuning LR in pruning is not mentioned at all in their paper . ( 3 ) Proposed technical method . ( Lee et al. , 2020 ) focuses on unstructured pruning , while we focuses on structured pruning . This further leads to fundamental difference when designing the dynamical isometry recovery ( DIR ) methods – In ( Lee et al. , 2020 ) , their proposed method is to use iterative optimization for approximated isometry ( due to the irregular sparsity ) ; while in our case , since the pruned filers can be completely removed from the network , one of our DIR method ( OrthP ) has closed-form solution and can achieve exact isometry . ( 4 ) Finally , in terms of empirical results , ( Lee et al. , 2020 ) only conducts experiments on MNIST ( LeCun et al. , 1998 ) and CIFAR ( Krizhevsky , 2009 ) , while we have extensive results on the large-scale ImageNet dataset ( Deng et al. , 2009 ) . 2.1 EMPIRICAL STUDY : LARGER FINETUNING LR IS CRITICAL . As far as we know , mainly two papers question the value of inheriting weights from a pretrained model : ( Crowley et al. , 2018 ; Liu et al. , 2019 ) . Both papers draw two similar conclusions . ( 1 ) Inheriting weights from a pretrained model in pruning has no value , i.e. , training from scratch the small model can match ( or outperform sometimes ) the counterpart pruned from a big pretrained model . ( 2 ) Given the fact of ( 1 ) , what really matters in pruning may lie in the pruned architecture instead of the inherited weight values . As such , both papers propose to view pruning as a form of neural architecture search ( Zoph & Le , 2017 ; Elsken et al. , 2019 ) . In this section , we first reexamine the empirical studies in ( Crowley et al. , 2018 ; Liu et al. , 2019 ) to show that the “ no value of inheriting weights ” argument is actually inaccurate owing to improper finetuning LR schedules . Reexamination of ( Liu et al. , 2019 ) . Before presenting results , here are some important comparison setting changes worth particular attention : ( 1 ) In ( Liu et al. , 2019 ) , they compare training from scratch with six pruning methods ( five structured pruning methods ( Li et al. , 2017 ; Luo et al. , 2017 ; Liu et al. , 2017 ; He et al. , 2017 ; Huang & Wang , 2018 ) and one unstructured pruning method ( Han et al. , 2015 ) ) . Here , we only focus on the L1-norm pruning ( Li et al. , 2017 ) on ImageNet . The main reason is that , L1-norm pruning is well-known a very basic filter pruning method . If we can show it outperforms training from scratch already , it will be no surprise to see other more advanced pruning methods also outperform training from scratch . In this sense , L1-norm pruning is the most representative method here for our investigation . ( 2 ) In ( Liu et al. , 2019 ) , they have two variants for the number of epochs in scratch training , “ Scratch-E ” and “ Scratch-B ” . For the former , different small models are trained for a fixed number of epochs ; for the latter , smaller models are trained for more epochs to maintain the same computation budget ( Scratch-B was shown to be better than Scratch-E in ( Liu et al. , 2019 ) ) . Also , they decay LR only to 10−3 following the official PyTorch ImageNet example1 . Here , we simply train all the networks for the same number of epochs but 1https : //github.com/pytorch/examples/tree/master/imagenet ensure the epochs are abundant ( 120 epochs ) and decay LR to a very small amount ( 10−5 ) . These two changes are to make sure the networks are trained to full convergence . As we will show , one primary cause possibly leading ( Liu et al. , 2019 ) to an inaccurate conclusion is exactly that the pruned networks are not fully converged ( see Tab . 1 ) . With the LR schedule changes , we rerun the experiments using the released code of ( Liu et al. , 2019 ) . Results are presented in Tab . 1 . In the implementations of ( Liu et al. , 2019 ) , the finetuned model is outperformed by the scratch training one , hence their “ no value of inheriting weights ” argument . We also reproduce their settings ( the two rows of “ 20 epochs , initial 10−3 , fixed ” in “ Our rerun ” ) for confirming their argument . However , the finetuning LR schedule “ 20 epochs , initial 10−3 , fixed ” is actually sub-optimal ; the network is not fully converged . Using the proper ones ( “ 90 epochs , initial 10−3 , decay ” or “ 90 epochs , initial 10−2 , decay ” ) , pruning outperforms training from scratch for both ResNet-34-A and ResNet-34-B . ( We note the pruned models even outperform the original models . This is probably because pruning reduces the network redundancy , thus curbing overfitting . This phenomenon is also widely observed in past pruning works ( Han et al. , 2016 ; Wen et al. , 2016 ; He et al. , 2017 ) especially under small pruning ratios as in Tab . 1 . ) Tab . 1 only presents two ResNet models and their speedups are actually quite small . To see if the finetuning LR effect still holds across the full spectrum of pruning ratios and on other types of networks , we vary the pruning ratios from 0.1 to 0.95 and include experiments on VGG11 BN ( Simonyan & Zisserman , 2015 ) . Results are presented in Tab . 2 . With a more proper finetune LR scheme ( column “ Pruned-Fintuned2 ” vs. “ Pruned-Fintuned-1 ” ) , the performance can be improved significantly . A clear pattern is , the larger the pruning ratio , the more of the improvement . Now , comparing the results of “ PrunedFintuned-2 ” to those of “ Scratch ” , we can see pruning outperforms scratch-training in most cases . Exceptions appear on ResNet-34/18 under extreme pruning ratios ( 90 % and 95 % ) . Despite them , we believe it is fair to say inheriting weights has value given the fact that 17/20 experiments in Tabs . 1 and 2 show pruning is better than training from scratch , especially under the pruning ratios of practical interest ( i.e. , non-extreme pruning ratios ) . Retrospectively , ( Liu et al. , 2019 ) concluded oppositely because they faithfully re-implemented the L1-norm pruning method just according to the description in the original paper ( Li et al. , 2017 ) : fixed LR 10−3 , 20 epochs , which turns out far from optimal as we know now . Reexamination of ( Crowley et al. , 2018 ) . Coincidentally , ( Crowley et al. , 2018 ) adopted a very similar finetuning LR scheme to ( Liu et al. , 2019 ) : They finetuned the pruned network with the lowest LR ( 8 ∗ 10−4 , close to 10−3 in ( Liu et al. , 2019 ) ) during scratch training and also fixed . Like the empirical study above , we reproduce the experiments of ( Crowley et al. , 2018 ) and rerun them with a larger initial LR ( 10−2 ) and decay it during finetuning . Detailed results are deferred to the Appendix ( Tab . 10 ) due to the limited length here . We summarize the observation here – Exactly the same as the case in ( Liu et al. , 2019 ) , when the proper finetuning LR is used , pruning actually outperforms the best scratch training scheme consistently . Up to now , the results above have shown that the “ no value of inheriting weights ” argument in previous works is largely attributed to sup-optimal finetuning settings . A larger LR ( e.g. , 10−2 ) can significantly improve the finetuning performance than a small one ( e.g. , 10−3 ) . In fact , we are not the only one to discover this . Previous works ( Renda et al. , 2020 ; Le & Hua , 2021 ) also reported similar observation . Nevertheless , they do not link the phenomenon with the “ value of inheriting weights ” argument and do not conduct systematical empirical studies as we do here . More importantly , neither of them presented theoretical explanations about its occurrence – next , we are about to bridge this gap . We present a faithful theoretical explanation through the lens of dynamical isometry .

Summary: There are recent works questioning the value of inheriting weights in structured neural network pruning as it is empirically observed training from scratch can match or outperform finetuning a pruned model. This paper mainly includes three components: 1) the authors reinvestigated the problem and demonstrates that the conclusion is inaccurate because of improperly small finetuning learning rates. They show finetuning with pruned weights actually outperforms training from scratch, when larger learning rates and longer training epochs are adopted. 2)  the authors explored dynamical isometry (DI) to understand how finetuning LR affects the final performance. They show that weight pruning breaks dynamical isometry and finetuning can recover it and a larger LR can recover faster. 3) They proposed to fully recover dynamical isometry in fitler pruning before finetuning.