Rethinking Again the Value of Network Pruning -- A Dynamical Isometry Perspective

1 INTRODUCTION . Pruning is a time-honored methodology to reduce parameters in a neural network without seriously compromising its performance ( Reed , 1993 ; Sze et al. , 2017 ) . The prevailing pipeline of pruning comprises three steps : 1 ) pretraining : train a dense model ; 2 ) pruning : prune the dense model based on certain rules ; 3 ) finetuning : retrain the pruned model to regain performance . Most existing research focuses on the second step , seeking the best criterion to select unimportant weights so as to incur as less performance degradation as possible . This 3-step pipeline has been practiced for more than 30 years ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ) and is still extensively adopted in today ’ s pruning methods ( Sze et al. , 2017 ) . These said , several recent works ( Crowley et al. , 2018 ; Liu et al. , 2019 ) questioned the necessity of inheriting weights from a pretrained model because they empirically found the small model trained from scratch can match ( or sometimes outperform ) the counterpart pruned from the pre-trained large model . This acutely challenges the past wisdom as well as our common belief about pruning . As far as we know , there is no formal response to this critical conflict . A theoretical-level understanding of this problem is even more elusive . Meanwhile , the pruning community has been observing even more open questions . Specifically , ( Renda et al. , 2020 ; Le & Hua , 2021 ) found that the learning rate ( LR ) in finetuning holds a critical role in the final performance . A proper learning rate schedule ( e.g. , a larger initial LR 10−2 vs. 10−3 with step-decay schedule ) can improve the top-1 accuracy of a pruned ResNet-34 model ( He et al. , 2016 ) by more than 1 % on ImageNet ( Deng et al. , 2009 ) . This discovery calls for more attention being paid to the finetuning step when comparing different pruning methods . Unfortunately , they did not present more theoretical insights to explain its occurrence . This also remains an open question in the community up to date . In this paper , we will show these two open questions actually point to the same one . Specifically , we rerun the experiments of ( Crowley et al. , 2018 ; Liu et al. , 2019 ) and find simply using a larger finetuning LR ( 10−2 vs. 10−3 and decay it ) can significantly improve the final performance . Compared to the improved pruning performance , training from scratch does not compete or surpass pruning anymore ( see Tab . 1 and Tab . 2 on ImageNet ) . This observation invites many questions immediately : ( 1 ) Theoretical understanding : Why does this happen ? What is the theoretical reason behind it ? ( 2 ) Practical solution : If this is a problem , how to fix it ? Can the understanding of this problem lead us to better pruning algorithms ? This paper will present answers to all these questions . The key tool we employ to unveil the mysteries is dynamical isometry ( Saxe et al. , 2014 ) , which describes a kind of nice property in neural networks that are easy to optimize . We carefully design an explanatory experiment using a linear MLP ( multi-layer perceptron ) network to demonstrate how finetuning LR affects the final performance by affecting dynamical isometry . In brief , we observe the finetuning process can recover dynamical isometry ; a larger LR can help recover it faster ( or better ) , hence the better final performance . The proposed explanation is validated by our empirical results and resonates with many empirical observations . Furthermore , by the explanation , we learn dynamical isometry recovery is rather imperative . To achieve so , we present a very simple regularization-based method for pruning and show its effectiveness in recovering dynamical isometry on modern residual convolutional neural networks ( CNNs ) . Contributions . ( 1 ) We empirically demonstrate the questioning about the value of inheriting weights in structured pruning in previous works is inaccurate and point out that the direct cause is improperly using a small finetuning LR . Our finding justifies the value of inheriting weights in structured pruning . ( 2 ) On top of the empirical finding , more importantly , we present a theoretical explanation through examining the dynamical isometry of networks in pruning . This explanation is empirically validated by our carefully designed control experiments . ( 3 ) In addition to the theoretical understanding , we also propose a regularization-based method for dynamical isometry recovery . Despite its brutal simplicity , it is shown effective to recover the broken dynamical isometry on modern residual convolutional neural networks . 2 RELATED WORK . Conventional pruning . Pruning aims to remove as many parameters as possible in a neural network meanwhile maintaining its performance . There are many ways to categorize pruning methods . The most popular two are grouping by pruning structure and methodology . ( 1 ) In terms of pruning structure , pruning can be specified into unstructured pruning ( Han et al. , 2015 ; 2016 ) and structured pruning ( Wen et al. , 2016 ; Li et al. , 2017 ; He et al. , 2017 ) . For the former , a single weight is the basic pruning element . Unstructured pruning can deliver a high compression ratio ; whereas , without regularization , the pruned locations usually spread randomly in the network , which is hard to exploit for acceleration . On the opposite , structured pruning introduces certain patterns in the pruned locations , which benefit subsequent acceleration while can not achieve as much compression . Choices between unstructured and structured pruning depend on specific application needs . For structured pruning , there are still many sub-groups ( Mao et al. , 2017 ) . In the literature , without specific mention , structured pruning means filter pruning or channel pruning . This paper focuses on structured ( filter ) pruning because the “ no value of inheriting weights ” argument is mainly discussed in this context ( Liu et al. , 2019 ) . ( 2 ) In terms of pruning methodology ( i.e. , how to select unimportant weights to prune ) , pruning falls into two paradigms in general : importance-based and penalty-based . The former prunes weights based on some established importance criteria , such as magnitude ( for unstructured pruning ) ( Han et al. , 2015 ; 2016 ) or L1-norm ( for filter pruning ) ( Li et al. , 2017 ) , saliency based on 2nd-order gradients ( e.g. , Hessian or Fisher ) ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Theis et al. , 2018 ; Wang et al. , 2019a ; Singh & Alistarh , 2020 ) . The latter adds a penalty term to the objective function , drives unimportant weights towards zero , then removes those with the smallest magnitude . Note , the two groups are not starkly separated . Many methods take wisdom from both sides . For example , ( Ding et al. , 2018 ; Wang et al. , 2019b ; 2021b ) select unimportant weights by magnitude ( akin to the first group ) while also employing the regularization to penalize weights ( akin to the second group ) . There is no conclusion about which paradigm is better , yet empirically , the state-of-the-art pruning methods are closer to the second paradigm , i.e. , deciding weights via training instead of some derived formulas . Although no theories have formally discussed the reason , we can take a rough guess with the knowledge from this paper : Training can recover dynamical isometry , which is beneficial to subsequent finetuning . For more comprehensive literature , we refer interested readers to several surveys : an outdated one ( Reed , 1993 ) , some recent surveys of pruning alone ( Gale et al. , 2019 ; Blalock et al. , 2020 ) or pruning as a sub-topic under the general umbrella of model compression and acceleration ( Sze et al. , 2017 ; Cheng et al. , 2018a ; b ; Deng et al. , 2020 ) . Pruning at initialization ( PaI ) . Recent years have seen several new pruning paradigms . The most prominent one is pruning at initialization . Different from the conventional pruning , which prunes a pretrained model , PaI methods prune a randomly initialized model . Existing PaI approaches mainly include ( Lee et al. , 2019 ; 2020 ; Wang et al. , 2020 ; Frankle et al. , 2021 ; Ramanujan et al. , 2020 ) and the series of lottery ticket hypothesis ( Frankle & Carbin , 2019 ; Frankle et al. , 2020 ) . Interested readers may refer to ( Wang et al. , 2021a ) for a comprehensive summary about PaI . This topic is relevant to this work mainly because one PaI paper ( Lee et al. , 2020 ) also examines pruning using the tool of dynamical isometry . The similarity between our paper and theirs is that we both employ dynamical isometry as a tool to examine the property of network pruning . However , our paper is significantly different from theirs in many axes : ( 1 ) Basic setting . The most obvious difference is that we focus on pruning a pretrained model while ( Lee et al. , 2020 ) focuses on pruning at initialization ( PaI ) . They are two different tracks in pruning ( as such , PaI methods typically do not compare with the methods of pruning pretrained models ) and the latter was shown to consistently underperform the former ( Frankle et al. , 2021 ; Wang et al. , 2021a ) . ( 2 ) Motivation . Despite the same tool ( mean JSV ) , ( Lee et al. , 2020 ) uses it to select unimportant weights to prune ( i.e. , for a new pruning criterion ) , while we use it to analyze why finetuning LR has a significant impact on final performance . The role of finetuning LR in pruning is not mentioned at all in their paper . ( 3 ) Proposed technical method . ( Lee et al. , 2020 ) focuses on unstructured pruning , while we focuses on structured pruning . This further leads to fundamental difference when designing the dynamical isometry recovery ( DIR ) methods – In ( Lee et al. , 2020 ) , their proposed method is to use iterative optimization for approximated isometry ( due to the irregular sparsity ) ; while in our case , since the pruned filers can be completely removed from the network , one of our DIR method ( OrthP ) has closed-form solution and can achieve exact isometry . ( 4 ) Finally , in terms of empirical results , ( Lee et al. , 2020 ) only conducts experiments on MNIST ( LeCun et al. , 1998 ) and CIFAR ( Krizhevsky , 2009 ) , while we have extensive results on the large-scale ImageNet dataset ( Deng et al. , 2009 ) . 2.1 EMPIRICAL STUDY : LARGER FINETUNING LR IS CRITICAL . As far as we know , mainly two papers question the value of inheriting weights from a pretrained model : ( Crowley et al. , 2018 ; Liu et al. , 2019 ) . Both papers draw two similar conclusions . ( 1 ) Inheriting weights from a pretrained model in pruning has no value , i.e. , training from scratch the small model can match ( or outperform sometimes ) the counterpart pruned from a big pretrained model . ( 2 ) Given the fact of ( 1 ) , what really matters in pruning may lie in the pruned architecture instead of the inherited weight values . As such , both papers propose to view pruning as a form of neural architecture search ( Zoph & Le , 2017 ; Elsken et al. , 2019 ) . In this section , we first reexamine the empirical studies in ( Crowley et al. , 2018 ; Liu et al. , 2019 ) to show that the “ no value of inheriting weights ” argument is actually inaccurate owing to improper finetuning LR schedules . Reexamination of ( Liu et al. , 2019 ) . Before presenting results , here are some important comparison setting changes worth particular attention : ( 1 ) In ( Liu et al. , 2019 ) , they compare training from scratch with six pruning methods ( five structured pruning methods ( Li et al. , 2017 ; Luo et al. , 2017 ; Liu et al. , 2017 ; He et al. , 2017 ; Huang & Wang , 2018 ) and one unstructured pruning method ( Han et al. , 2015 ) ) . Here , we only focus on the L1-norm pruning ( Li et al. , 2017 ) on ImageNet . The main reason is that , L1-norm pruning is well-known a very basic filter pruning method . If we can show it outperforms training from scratch already , it will be no surprise to see other more advanced pruning methods also outperform training from scratch . In this sense , L1-norm pruning is the most representative method here for our investigation . ( 2 ) In ( Liu et al. , 2019 ) , they have two variants for the number of epochs in scratch training , “ Scratch-E ” and “ Scratch-B ” . For the former , different small models are trained for a fixed number of epochs ; for the latter , smaller models are trained for more epochs to maintain the same computation budget ( Scratch-B was shown to be better than Scratch-E in ( Liu et al. , 2019 ) ) . Also , they decay LR only to 10−3 following the official PyTorch ImageNet example1 . Here , we simply train all the networks for the same number of epochs but 1https : //github.com/pytorch/examples/tree/master/imagenet ensure the epochs are abundant ( 120 epochs ) and decay LR to a very small amount ( 10−5 ) . These two changes are to make sure the networks are trained to full convergence . As we will show , one primary cause possibly leading ( Liu et al. , 2019 ) to an inaccurate conclusion is exactly that the pruned networks are not fully converged ( see Tab . 1 ) . With the LR schedule changes , we rerun the experiments using the released code of ( Liu et al. , 2019 ) . Results are presented in Tab . 1 . In the implementations of ( Liu et al. , 2019 ) , the finetuned model is outperformed by the scratch training one , hence their “ no value of inheriting weights ” argument . We also reproduce their settings ( the two rows of “ 20 epochs , initial 10−3 , fixed ” in “ Our rerun ” ) for confirming their argument . However , the finetuning LR schedule “ 20 epochs , initial 10−3 , fixed ” is actually sub-optimal ; the network is not fully converged . Using the proper ones ( “ 90 epochs , initial 10−3 , decay ” or “ 90 epochs , initial 10−2 , decay ” ) , pruning outperforms training from scratch for both ResNet-34-A and ResNet-34-B . ( We note the pruned models even outperform the original models . This is probably because pruning reduces the network redundancy , thus curbing overfitting . This phenomenon is also widely observed in past pruning works ( Han et al. , 2016 ; Wen et al. , 2016 ; He et al. , 2017 ) especially under small pruning ratios as in Tab . 1 . ) Tab . 1 only presents two ResNet models and their speedups are actually quite small . To see if the finetuning LR effect still holds across the full spectrum of pruning ratios and on other types of networks , we vary the pruning ratios from 0.1 to 0.95 and include experiments on VGG11 BN ( Simonyan & Zisserman , 2015 ) . Results are presented in Tab . 2 . With a more proper finetune LR scheme ( column “ Pruned-Fintuned2 ” vs. “ Pruned-Fintuned-1 ” ) , the performance can be improved significantly . A clear pattern is , the larger the pruning ratio , the more of the improvement . Now , comparing the results of “ PrunedFintuned-2 ” to those of “ Scratch ” , we can see pruning outperforms scratch-training in most cases . Exceptions appear on ResNet-34/18 under extreme pruning ratios ( 90 % and 95 % ) . Despite them , we believe it is fair to say inheriting weights has value given the fact that 17/20 experiments in Tabs . 1 and 2 show pruning is better than training from scratch , especially under the pruning ratios of practical interest ( i.e. , non-extreme pruning ratios ) . Retrospectively , ( Liu et al. , 2019 ) concluded oppositely because they faithfully re-implemented the L1-norm pruning method just according to the description in the original paper ( Li et al. , 2017 ) : fixed LR 10−3 , 20 epochs , which turns out far from optimal as we know now . Reexamination of ( Crowley et al. , 2018 ) . Coincidentally , ( Crowley et al. , 2018 ) adopted a very similar finetuning LR scheme to ( Liu et al. , 2019 ) : They finetuned the pruned network with the lowest LR ( 8 ∗ 10−4 , close to 10−3 in ( Liu et al. , 2019 ) ) during scratch training and also fixed . Like the empirical study above , we reproduce the experiments of ( Crowley et al. , 2018 ) and rerun them with a larger initial LR ( 10−2 ) and decay it during finetuning . Detailed results are deferred to the Appendix ( Tab . 10 ) due to the limited length here . We summarize the observation here – Exactly the same as the case in ( Liu et al. , 2019 ) , when the proper finetuning LR is used , pruning actually outperforms the best scratch training scheme consistently . Up to now , the results above have shown that the “ no value of inheriting weights ” argument in previous works is largely attributed to sup-optimal finetuning settings . A larger LR ( e.g. , 10−2 ) can significantly improve the finetuning performance than a small one ( e.g. , 10−3 ) . In fact , we are not the only one to discover this . Previous works ( Renda et al. , 2020 ; Le & Hua , 2021 ) also reported similar observation . Nevertheless , they do not link the phenomenon with the “ value of inheriting weights ” argument and do not conduct systematical empirical studies as we do here . More importantly , neither of them presented theoretical explanations about its occurrence – next , we are about to bridge this gap . We present a faithful theoretical explanation through the lens of dynamical isometry .

Summary: This paper challenges an existing argument that “inheriting the weights of the pruned network is not necessary for fine-tuning”. This paper provides empirical evidence that the previous experiments are not carried out with proper learning rates and training epochs to ensure the network convergence.   The authors conjecture that fine-tuning pruned network requires larger learning rates and longer training epochs because its dynamical isometry is broken. As a result, large learning rates (if training for shorter epochs) or long training epochs are required for the recovery of dynamic isometry.   Then the authors propose OrthP to re-initialize the pruned weights to completely recover the dynamic isometry in simple MLP. With OrthP, the fine-tuning process is less sensitive to different learning rate and training epochs.