Back to Basics: Efficient Network Compression via IMP

1 INTRODUCTION . Modern Neural Network architectures are commonly highly over-parameterized ( Zhang et al. , 2016 ) , containing millions or even billions of parameters , resulting in both high memory requirements as well as computationally intensive and long training and inference times . It has been shown however ( LeCun et al. , 1989 ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; Gale et al. , 2019 ; Lin et al. , 2020 ; Blalock et al. , 2020 ) that modern architectures can be compressed dramatically by pruning , i.e. , removing redundant structures such as individual weights , entire neurons or convolutional filters . The resulting sparse models require only a fraction of storage and floating-point operations ( FLOPs ) for inference , while experiencing little to no degradation in predictive power compared to the dense model . There is of course an inherent tradeoff between sparsity and model performance ; a very heavily pruned model will normally be less performant than its dense ( or moderately pruned ) counterpart , though it has been observed that pruning might have a regularizing effect and be beneficial to the generalization capacities ( Blalock et al. , 2020 ; Hoefler et al. , 2021 ) . One approach to pruning consists of removing part of a network ’ s weights after a standard training process , seemingly losing most of its predictive performance , and then retraining to compensate for that pruning-induced loss . This can be done either once ( One Shot ) , or the process of pruning and retraining can be repeated iteratively until the desired level of sparsity is reached . Although dating back to the early work of Janowsky ( 1989 ) , this approach was most notably proposed by Han et al . ( 2015 ) in the form of ITERATIVE MAGNITUDE PRUNING ( IMP ) . Because it is arguably one of the simplest pruning algorithms , IMP has been widely applied as a baseline comparison for other approaches ( Carreira-Perpiñán & Idelbayev , 2018 ; Ding et al. , 2019 ; Savarese et al. , 2020 ; Siegel et al. , 2020 ; Hoefler et al. , 2021 ) . As such , it is often subject to criticism , with the most commonly made claims arguing against the efficacy of IMP being the following : 1 . ) Inherent to the justification of many proposed alternatives is the claim , that sparsification should be part of the training . Methods of this type reach a sparse model at the end of training , ideally eliminating the need for further training . One desired benefit of doing so , is to improve the sparsity vs. performance tradeoff by reducing the impact of the actual ‘ hard ’ pruning , which results in a “ failure to properly recover the pruned weights ” ( Liu et al. , 2020 ) . It is argued that IMP achieves sub-optimal states since learning the pruning set throughout training “ helps find a better subset and hence prune more weights with no or little loss degradation ” ( Carreira-Perpiñán & Idelbayev , 2018 ) . Another frequently claimed advantage is that incorporating the sparsification into the training cuts down on computational cost by not requiring additional retraining epochs . Ding et al . ( 2019 ) for example advertise that there is “ no need for a time consuming re-training ” and Hoefler et al . ( 2021 ) argue that `` the sparsify-during-training schedule ( ... ) is usually cheaper than the train-then-sparsify schedule ” . 2 . ) IMP determines a single numerical threshold for pruning and applies it globally to every parameter , potentially resulting in very different levels of sparsity among the layers of the network . This behavior is often considered to be sub-optimal and it is argued that pruning should be layerdependent ( Liu et al. , 2020 ) , so more complex saliency criteria have been proposed ( Gale et al. , 2019 ; Lee et al. , 2020 ) . Kusupati et al . ( 2020 ) for example claim that “ uniform or heuristic nonuniform sparsity budgets ( ... ) have sub-optimal layer-wise parameter allocation resulting in a ) lower prediction accuracy or b ) higher inference cost ( FLOPs ) ” . 3 . ) While only the iterative approach , that is repeatedly removing only a small fraction of the parameters followed by extensive retraining , is said to achieve results on the Pareto frontier ( Renda et al. , 2020 ) , its iterative nature is also considered to be computationally tedious , if not impractical : “ iterative pruning is computationally intensive , requiring training a network 15 or more times consecutively for multiple trials ” ( Frankle & Carbin , 2018 ) , leading Liu et al . ( 2020 ) to trying to “ avoid the expensive pruning and fine-tuning iterations ” . Our interest lies in exploring these claimed disadvantages of IMP through rigorous and consistent computational experimentation with a focus on recent advancements concerning the retraining phase , see the results of Renda et al . ( 2020 ) and Le & Hua ( 2021 ) . This comparative study is in fact intended to complement both of these works , which focused on improving the sparsity-vs.performance tradeoff of IMP through improved learning rate schemes during training , by putting an additional spotlight on the total computational cost of IMP in a direct comparison with methods that are commonly assumed to outperform IMP in that aspect by avoiding retraining . Contributions . We empirically find that , using an appropriate learning rate scheme , only few retraining epochs are needed in practice to achieve most of the sparsity vs. performance tradeoff of IMP . We also find that the global selection criterion not only finds sparsity distributions on par with but , somewhat surprisingly , often better than those of more sophisticated layer-dependent pruning criteria . Finally , we conclude that , using an appropriate learning rate scheme , IMP performs well even when compared to state-of-the-art approaches that incorporate sparsification into the training without or with only little computational overhead . That is , not only can IMP find some of the best performing architectures at any given sparsity level , but due to the compressed retraining time it does so without needing to leverage a longer running time even when compared to methods typically considered to be superior to IMP in that particular aspect . Outline . Section 2 contains a complete overview over related works , including a brief summary of all pruning methods and approaches considered here . In Section 3 we provide the computational results and their interpretation by first addressing how IMP can develop its full potential within a restricted computational envelope in Subsection 3.1 and Subsection 3.2 . We then use the resulting lessons in order to draw up a fair comparison to methods that incorporate pruning into their training in Subsection 3.3 . We conclude with some discussion in Section 4 . 2 OVERVIEW OF PRUNING METHODS AND METHODOLOGY . While the sparsification of Neural Networks includes a wide variety of approaches , we will focus on Model Pruning , i.e. , the removal of redundant structures in a Neural Network . More specifically , our results will be limited to unstructured pruning , that is the removal of individual weights , as opposed to its structured counterpart , where entire groups of elements , such as neurons or convolutional filters , are removed . We will also focus on approaches that start with a dense network and then either prune the network during training or after training as already discussed in the introduction . Following Bartoldson et al . ( 2020 ) , we will also refer to methods of the former category as pruning stable , since the final pruning should result in a negligible decrease in performance , where methods of the latter category are referred to as unstable . For a full and detailed survey of Pruning algorithms we refer the reader to Hoefler et al . ( 2021 ) . Pruning unstable methods are exemplified by ITERATIVE MAGNITUDE PRUNING ( IMP ) ( Han et al. , 2015 ) . In its original form , it first employs standard network training , adding a common ` 2- regularization term on the objective , and then removes all weights from the network whose absolute values are below a certain threshold . The network at this point commonly loses some or even all of its learned predictive power , so it is then retrained for a fixed number of epochs . This prune-retrain cycle is usually repeated a number of times ; the threshold at every pruning step is determined as the appropriate percentile such that , at the end of given number of iterations , a desired target sparsity is met.1 In the following we will first discuss two particular details of IMP that have been the focus of recent research : the questions of ( a ) how to select the parameters to be pruned and ( b ) how to retrain . We will then conclude this section by briefly outlining the pruning stable methods we have selected for this comparison and establish how to fairly compare them to IMP . 2.1 RETRAINING APPROACHES . Let us first consider the learning rate scheme used during retraining . The original approach by Han et al . ( 2015 ) is commonly referred to as FINE TUNING ( FT ) : suppose we train for T epochs using the learning rate schedule ( ηt ) t≤T and retrain for Trt epochs per prune-retrain-cycle , then FT retrains the pruned network for Trt epochs using a fixed constant learning rate , most commonly ηT . It was first noticed by Renda et al . ( 2020 ) that the precise learning rate schedule during retraining can have a dramatic impact on the predictive performance of the pruned network . Motivated by WEIGHT REWINDING ( WR ) ( Frankle et al. , 2019 ) , they proposed LEARNING RATE REWINDING ( LRW ) , where one retrains the pruned network for Trt epochs using the last T − Trt learning rates ηT−Trt+1 , . . . , ηT . Le & Hua ( 2021 ) argued that the reason behind the success of LRW is the usage of large learning rates and proposed SCALED LEARNING RATE RESTARTING ( SLR ) , where the pruned network is retrained using a proportionally identical learning schedule , i.e. , by compressing ( ηt ) t≤T into the retraining time frame of Trt epochs with a short warm-up phase . They also introduced CYCLIC LEARNING RATE RESTARTING ( CLR ) based on the the 1-cycle learning rate schedule of Smith & Topin ( 2017 ) . We think that the nature of the proposed retraining methods indicates that the retraining phase is , at its core , similar to the usual training phase . Following this rationale , the success of LRW , SLR and CLR over FT should be attributed to the existence of both a large- and small-step retraining regime . In fact , a large initial and exponentially decaying learning rate has become the standard practice for regular training ( Leclerc & Madry , 2020 ) . Note that such a scheme is employed not just by SLR and CLR , but also by LRW if Trt is sufficiently large to model the decaying learning rate schedule of the original training phase . The conventional approach to explaining the success of decaying learning rate schedules comes from an optimization perspective , i.e. , an initially large learning rate accelerates training and avoids local minima , while the gradual decay helps to converge to an optimum without oscillation around it . However , an active line of research has theoretically supported the usage of large learning rates and separating training into a large- and small-step regime from a generalization perspective ( Jastrzębski et al. , 2017 ; Li et al. , 2019 ; You et al. , 2019 ; Leclerc & Madry , 2020 ) . Put more succinctly : retraining is training and therefore requires that some effort is put into tuning the learning rate scheme . LRW , SLR and CLR provide some good heuristic guidance for how to effectively do so without an insurmountable amount of hyperparameter tuning . While One Shot IMP , that is IMP with a single prune-retrain cycle , is a viable approach to model pruning , only the iterative approach ( with multiple prune-retrain cycles ) has been shown to achieve 1There is another way to view this relation : one can fix a given percentile to be pruned in every iteration and then simply repeat the prune-retrain cycle until either a desired level of sparsity is reached or the performance degradation exceeds a given threshold . This is in fact how it appears to be used by Han et al . ( 2015 ) and how it is for example presented by Renda et al . ( 2020 ) . While this reframing may appear trivial , it in fact highlights a strength of IMP that we will further emphasize when contrasting it with pruning stable approaches . state-of-the-art accuracy-vs.-sparsity tradeoffs ( Han et al. , 2015 ; Renda et al. , 2020 ) . This iterative approach however commonly consists of a significant amount of prune-retrain cycles , each needing the full original training time , that is T = Trt , resulting in several thousand epochs worth of total training time . Renda et al . ( 2020 ) for example suggested the following approach : train a network for T epochs and then iteratively prune 20 % percent of the weights and retrain for Trt = T epochs using LRW , i.e. , use the same learning rate scheme as during training , until the desired sparsity is reached . We note that for T = Trt the learning rate scheme of SLR becomes essentially identical to that of LRW . For a goal sparsity of 98 % and T = 200 original training epochs , the algorithm would therefore require 18 prune-retrain-cycles for a massive 3800 total retrain epochs . In Subsection 3.1 we will study the effect of the number of prune-retrain cycles , the number of retraining epochs and the learning rate scheme on the performance of the pruned network to establish whether IMP truly requires this massive amount of computational investment to develop its full potential .

Summary: The paper studies a fundamental and important research approach in network pruning: iterative magnitude pruning (IMP). Previously IMP is criticized to be time-consuming, layer-independent and sub-optimal in performance. In this paper, extensive empirical studies are conducted to show that under proper learning rates, IMP can have close performance with more advanced pruning approaches, with little training time increased.