Network Pruning That Matters:  A Case Study on Retraining Variants

1 INTRODUCTION . Training neural networks is an everyday task in the era of deep learning and artificial intelligence . Generally speaking , given data availability , large and cumbersome networks are often preferred as they have more capacity to exhibit good data generalization . In the literature , large networks are considered easier to train than small ones ( Neyshabur et al. , 2018 ; Arora et al. , 2018 ; Novak et al. , 2018 ; Brutzkus & Globerson , 2019 ) . Thus , many breakthroughs in deep learning are strongly correlated to increasingly complex and over-parameterized networks . However , the use of large networks exacerbate the gap between research and practice since real-world applications usually require running neural networks in low-resource environments for numerous purposes : reducing memory , latency , energy consumption , etc . To adopt those networks to resourceconstrained devices , network pruning ( LeCun et al. , 1990 ; Han et al. , 2015 ; Li et al. , 2016 ) is often exploited to remove dispensable weights , filters and other structures from neural networks . The goal of pruning is to reduce overall computational cost and memory footprint without inducing significant drop in performance of the network . A common approach to mitigating performance drop after pruning is retraining : we continue to train the pruned models for some more epochs . In this paper , we are interested in approaches based on learning rate schedules to control the retraining . A well-known practice is fine-tuning , which aims to train the pruned model with a small fixed learning rate . More advanced learning rate schedules exist , which we generally refer to as retraining . The retraining step is a critical part in implementing network pruning , but it has been largely overlooked and tend to vary in each implementation including differences in learning rate schedules , retraining budget , hyperparameter choices , etc . Recently , Renda et al . ( 2020 ) proposed a state-of-the-art technique for retraining pruned networks namely learning rate rewinding ( LRW ) . Specifically , instead of fine-tuning the pruned networks with a fixed learning rate , usually the last learning rate from the original training schedule ( Han et al. , 2015 ; Liu et al. , 2019 ) , the authors suggested using the learning rate schedule from the previous t epochs ( i.e . rewinding ) . This seemingly subtle change in learning rate schedule led to an important result : LRW was shown to achieve comparable performance to more complex and computationally expensive pruning algorithms while only utilizing simple norm-based pruning . Unfortunately , the authors did not provide the analysis to justify the improvement . In general , it is intriguing to understand the importance of a learning rate schedule and how it affects the final performance of a pruned model . In this work , we study the behavior of pruned networks under different retraining settings . We found that the efficacy from retraining with learning rate rewinding is rooted in the use of a large learning rate , which helps pruned networks to converge faster after pruning . We demonstrate that the success of learning rate rewinding over fine-tuning is not exclusive to the learning rate schedule coupling with the original training process . Retraining with a large learning rate could outperform fine-tuning even with some modest retraining , e.g. , for a few epochs , and regardless of network compression ratio . We argue that retraining is of paramount importance to regain the performance in network pruning and should not be overlooked when comparing two pruning algorithms . This is evidenced by our extensive experiments : ( 1 ) randomly pruned network can outperform methodically pruned network with only ( hyper-parameters free ) modifications of the learning rate schedule in retraining , and ( 2 ) a simple baseline such as norm-based pruning can perform as well as as other complex pruning methods by using a large learning rate restarting retraining schedule . The contributions of our work are as follows . • We document a thorough experiment on learning rate schedule for the retraining step in network pruning with different pruning configurations ; • We show that learning rate matters : pruned models retrained with a large learning rate consistently outperform those trained by conventional fine-tuning regardless of specific learning rate schedules ; • We present a novel and counter-intuitive result achieved by solely applying large learning rate retraining : a randomly pruned network and a simple norm-based pruned network can perform as well as networks obtained from more sophisticated pruning algorithms . Given the significant impact of learning rate schedule in network pruning , we advocate the following practices : learning rate schedule should be considered as a critical part of retraining when designing pruning algorithms . Rigorous ablation studies with different retraining settings should be made for a fair comparison of pruning algorithms . To facilitate reproducibility , we would release our implementation upon publication . 2 PRELIMINARY AND METHODOLOGY . Pruning is a common method to produce compact and high performance neural networks from their original large and cumbersome counterparts.We can categorize pruning approaches into three classes : Pruning after training - which consists of three steps : training the original network to convergence , prune redundant weights based on some criteria , and retrain the pruned model to regain the performance loss due to pruning ( Li et al. , 2016 ; Han et al. , 2015 ; Luo et al. , 2017 ; Ye et al. , 2018 ; Wen et al. , 2016 ; He et al. , 2017 ) ; Pruning during training - we update the “ pruning mask ” while training the network from scratch , thus , allowing pruned neurons to be recovered ( Zhu & Gupta , 2017 ; Kusupati et al. , 2020 ; Wortsman et al. , 2019 ; Lin et al. , 2020b ; He et al. , 2019 ; 2018 ) ; Pruning before training - Inspired by the Lottery Ticket Hypothesis ( Frankle & Carbin , 2019 ) , some recent works try to find the sparsity mask at initialization and train the pruned network from scratch without changing the mask ( Lee et al. , 2019 ; Tanaka et al. , 2020 ; Wang et al. , 2020 ) . In this work , we are mainly concerned with the first category i.e . pruning after training which has the largest body of work to our knowledge . Traditionally , the last step is referred to as fine-tuning , i.e. , continue to train the pruned model with a small learning rate obtained from the last epoch of the original model . This seemly subtle step is often overlooked when designing pruning algorithms . Particularly , we found that the implementation of previous pruning algorithms have many notable differences in their retraining step : some employed a small value of learning rate ( e.g . 0.001 on ImageNet ) to fine-tune the network ( Molchanov et al. , 2016 ; Li et al. , 2016 ; Han et al. , 2015 ) for a small number of epochs , e.g. , 20 epochs in the work by Li et al . ( 2016 ) ; some used a larger value of learning rate ( 0.01 ) with much longer retraining budgets , e.g. , 60 , 100 and 120 epochs respectively on ImageNet ( Zhuang et al. , 2018 ; Gao et al. , 2020 ; Li et al. , 2020 ) ; You et al . ( 2019 ) ; Li et al . ( 2020 ) respectively utilized 1-cycle ( Smith & Topin , 2019 ) and cosine annealing learning rate schedule instead of conventional step-wise schedule . Despite such difference , the success of each pruning algorithm is only attributed to the pruning algorithm itself . This motivates us to ask the question : do details like learning rate schedule used for retraining matter ? In this section , we strive to understand the behavior of pruned models under different retraining configurations and how they impact the final performance . Specifically , we conduct experiments with different retraining schedules on simple baselines such as ` 1-norm filters pruning ( Li et al. , 2016 ) and magnitude-based weights pruning ( Han et al. , 2015 ) . We show that the efficacy of several pruning algorithms can be boosted by simply modifying the learning rate schedule . More importantly , the performance gain by retraining can be remarkable : the accuracy loss can drop to zero and in some cases better accuracy than baseline models can be achieved . To analyze the effect of retraining a pruned network , we based on learning rate rewinding ( Renda et al. , 2020 ) and experiment with different retraining settings . Although in the previous work , Renda et al . ( 2020 ) demonstrated the efficacy of learning rate rewinding across datasets and pruning criteria , there is a lack of understanding of the actual reason behind the success of this technique . Here we hypothesize that the initial pruned network is a suboptimal solution , staying in a local minima . Learning rate rewinding succeeds because it uses a large learning rate to encourage the pruned networks to converge to another , supposingly better , local minima . Our experiment setups are as follows . Retraining techniques . To verify this conjecture empirically , we conduct experiments with different learning rate schedules including learning rate rewinding ( Renda et al. , 2020 ) while varying pruning algorithms , network architectures and datasets . In this work , we consider the following retraining techniques : 1 . FINE-TUNING ( FT ) Fine-tuning is the most common retraining techniques ( Han et al. , 2015 ; Li et al. , 2016 ; Liu et al. , 2019 ) . In this approach , we continue train the pruned networks for t epochs with the last ( smallest ) learning rate of original training . 2 . LEARNING RATE REWINDING ( LRW ) Renda et al . ( 2020 ) propose to reuse the learning rate schedule of the original training when retraining pruned networks . Specifically , when retraining for t epochs , we reuse the learning rate schedule from the previous t epochs , i.e. , rewinding . 3 . SCALED LEARNING RATE RESTARTING ( SLR ) : In this approach , we employ the learning rate schedule that is proportionally identical to the standard training . For example , the learning rate is dropped by a factor of 10× at 50 % and 75 % of retraining epochs on CIFAR , which is akin to original training learning rate adjustment . The original learning rate schedule can be found in Appendix A . 4 . CYCLIC LEARNING RATE RESTARTING ( CLR ) : Instead of using stepwise learning rate schedule as scaled learning rate restarting , we leverage the 1-cycle ( Smith & Topin , 2019 ) , which is shown to give faster convergence speed than conventional approaches . Note that for the last two strategies ( SLR and CLR ) , we warmup the learning rate for 10 % of total retraining budget . For simplicity , we always use the largest learning rate of the original training for learning rate restarting . Specifically , the learning rate is increased from the smallest learning rate of original training to the largest one according to cosine function . The detailed learning rate schedule of each technique is depicted in Figure 1 . See also Appendix F for the choice of warmup epochs . Pruning algorithms We consider the following dimensions in our experiments . For pruning methods , we use ` 1-norm filters pruning ( PFEC ) ( Li et al. , 2016 ) and ( global ) magnitude-based weights pruning ( MWP ) ( Han et al. , 2015 ) and evaluate them on the CIFAR-10 , CIFAR-100 and ImageNet dataset . We examine both variations of pruning namely one-shot pruning and iterative pruning when comparing the proposed retraining techniques . Furthermore , we also experiment the CLR schedule on HRank ( Lin et al. , 2020a ) , Taylor Pruning ( TP ) ( Molchanov et al. , 2019 ) and Soft Filter Pruning ( SFP ) ( He et al. , 2018 ) . Our implementation and hyperparameters of ` 1-norm filters pruning and magnitude weight pruning are based on the public implementation of Liu et al . ( 2019 ) , which is shown to obtain comparable results with the original works . For remaining algorithms , we use official implementations with hyperparameters specified according to their papers . The detailed configurations of training and fine-tuning is provided in the Appendix B for interested readers . Evaluation . For CIFAR-10 and CIFAR-100 , we run each experiment three times and report “ mean± std ” . For ImageNet , we run each experiment once . These settings are kept consistently across architectures , pruning algorithms , retraining techniques , and ablation studies unless otherwise stated .

Summary: This paper analyzes the role of learning rate in re-training after pruning, building on previous findings that changing the learning rate schedule of re-training can result in higher accuracy than low-learning-rate fine-tuning. The paper proposes several learning rate schedules to compare, specifically a cyclic learning rate (gradually ramping up to and back down from the maximum learning rate schedule used during the original training phase) and a compressed version of the original learning rate schedule, and shows that these learning rate schedules outperform standard fine-tuning and also learning rate rewinding, showing that the findings of prior work come from using a higher learning rate in general and not any specific schedule. The paper than shows that choice of re-training learning rate schedule can have more impact on final accuracy than choice of saliency metric.