Back to Basics: Efficient Network Compression via IMP
Authors: Max Zimmer
Sebastian Pokutta
Christoph Spiegel
Venue: ICLR 2022 Submitted (2022)
Keywords: Magnitude pruning
Sparsity
IMP
Model Compression

Abstract: Network pruning is a widely used technique for effectively compressing Deep Neural Networks with little to no degradation in performance during inference. Iterative Magnitude Pruning (IMP) (Han et al., 2015) is one of the most established approaches for network pruning, consisting of several iterative training and pruning steps, where a significant amount of the network’s performance is lost after pruning and then recovered in the subsequent retraining phase. While commonly used as a benchmark reference, it is often argued that a) it reaches suboptimal states by not incorporating sparsification into the training phase, b) its global selection criterion fails to properly determine optimal layer-wise pruning rates and c) its iterative nature makes it slow and non-competitive. In light of recently proposed retraining techniques, we investigate these claims through rigorous and consistent experiments where we compare IMP to pruning-during-training algorithms, evaluate proposed modifications of its selection criterion and study the number of iterations and total training time actually required. We find that IMP with SLR (Le & Hua, 2021) for retraining can outperform state-of-the-art pruning-during-training approaches without or with only little computational overhead, that the global magnitude selection criterion is largely competitive with more complex approaches and that only few retraining epochs are needed in practice to achieve most of the sparsity-vs.-performance trade-off of IMP. Our goals are both to demonstrate that basic IMP can already provide state-of-the-art pruning results on par or outperforming more complex or heavily parameterized approaches and also to establish a more realistic yet easily realisable baseline for future research.

Under review as a conference paper at ICLR 2022
BACK TO BASICS: EFFICIENT NETWORK COMPRESSION VIA IMP
Anonymous authors Paper under double-blind review
ABSTRACT
Network pruning is widely used technique for effectively compressing Deep Neural Networks with little to no degradation in performance during inference. Iterative Magnitude Pruning (IMP) (Han et al., 2015) is one of the most estab- lished approaches for network pruning, consisting of several iterative training and pruning steps, where significant amount of the network's performance is lost after pruning and then recovered in the subsequent retraining phase. While com- monly used as a benchmark reference, it is often argued that a) its iterative nature makes it slow and non-competitive, b) it reaches suboptimal states, in particular because it does not incorporate sparsification into the training phase, and c) its global selection criterion fails to properly determine optimal layer-wise pruning rates. In light of recently proposed retraining techniques, we investigate these claims through rigorous and consistent experiments. We find IMP to be surpris- ingly efficient, achieving its full potential with significantly less than the extensive amount of retraining usually considered necessary (Renda et al. 2020). When paired with the compressed learning rate scheme suggested by Le & Hua (021)). we find that it can not only perform on par with more complex or heavily pa- rameterized state-of-the computational overhead even when using its original global magnitude selection criterion. This casts doubt on the commonly claimed advantages of imposing an implicit bias during training to avoid retraining.
approaches, but it does so without or with only little
INTRODUCTION
Modern Neural Network architectures are commonly highly over-parameterized (Zhang et al.) 2016), containing millions or even billions of parameters, resulting both high memory require- ments as well as computationally intensive and long training and inference times. It has been shown however (LeCun et al. 1989 Hassibi & Stork 1993] Han et al. 2015 Gale et al.] 2019) Lin et al.) 2020 Blalock et al. 2020) that modern architectures can be compressed dramatically by pruning, 1.c,, removing redundant structures such as individual weights, entire neurons or convolutional fil- ters. The resulting sparse models require only fraction of storage and floating-poin operations (FLOPs) for inference, while experiencing little to no degradation in predictive power compared the dense model. There of course an inherent tradeoff between sparsity and model perfor- mance; a very heavily pruned model will normally be less performant than its dense (or moderately pruned) counterpart, though has been observed that pruning might have regularizing effect and be beneficial to the generalization capacities (Blalock et al. 2020 Hoefler et al. 2021).
One approach pruning consists of removing part of network's 's weights after a standard training process, seemingly losing most of its predictive performance, and then retraining to compensate for that pruning-induced loss. This can be done either once (One Shot), or the process of pruning and retraining can be repeated iteratively until the desired level of sparsity is reached. Although dating back to the early work of Janowsky (1989), this approach was most notably proposed by Han et al. (2015) in the form of ITERATIVE MAGNITUDE PRUNING (IMP). Because it is arguably one of the simplest pruning algorithms, IMP has been widely applied as a baseline comparison for other approaches Sarreira-Perpiñaa & Idelbayev| 2018 Ding et al.] 2019; Savarese et al. 2020 (iieell eet al. 2020; Hoefler et al. (2021). As such, it is often subject to criticism, with the most commonly made claims arguing against the efficacy of IMP being the following:

---

Under review as a conference paper at ICLR 2022
1.) Inherent to the justification of many proposed alternatives is the claim, that spansification should be part of the training. Methods of this type reach a sparse model at the end of training, ideally eliminating the need for further training. One desired benefit of doing so, is to improve the spar- sity vs. performance tradeoff by reducing the impact of the actual 'hard' pruning. which results in "failure to properly recover the pruned weights" (Liu al., (020). It is argued that IMP achieves sub-optimal states since learning the pruning set throughout training "helps find a better subset and hence prune more weights with no or little loss degradation" (Carreira-Prppiiae & Idelbayev 2018). Another frequently claimed advantage is that incorporating the sparsification into the training cuts down on computational cost by not requiring additional retraining epochs. Ding et al. (2019) for example advertise that there is "no need for a time consuming re-training" and Hoefler et al. (2021) argue that "the sparsify-during-trame schedule".
schedule (...) usually cheaper than the train-then-sparsify
2.) IMP determines single numerical threshold for pruning and applies it globally to every pa- rameter, potentially resulting in very different levels of sparsity among the layers of the network. This behavior is often considered to be sub-optimal and is argued that pruning should be layer- dependent (Liu et al., 2020), so more complex saliency criteria have been proposed (Gale et al.) 2019 Lee et al. 2020). Kusupati et al. (2020) for example claim that "uniform or heuristic non- uniform sparsity budgets (...) have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FI.OP)"".
3.) While only the iterative approach, that is repeatedly removing only a small fraction of the pa- rameters followed by extensive retraining, is said to achieve results on the Pareto frontier (Renda) al. 2020), its iterative nature is also considered be computationally tedious, not impractical: "iterative pruning is computationally intensive, requiring training a network 15 or more times con- secutively for multiple trials" (Frankle & Carbin 2018), leading Liu et al. (2020) to trying to "avoid the expensive pruning and fine-tuning iterations"
Our interest lies in exploring these claimed disadvantages of IMP through rigorous and consis- tent computational experimentation with focus on recent advancements concerning the retraining phase, see the results of Renda et al.] (2020) and [e & Hual (202T). This comparative study is in fact intended to complement both of these works, which focused on improving the sparsity-vs.- performance tradeoff of IMP through improved learning rate schemes during training, by putting an additional spotlight on the total computational cost of IMP in a direct comparison with methods that are commonly assumed to outperform IMP in that aspect by avoiding retraining.
Contributions. We empirically find that, using an appropriate learning rate scheme, only few retraining epochs are needed in practice to achieve most of the sparsity vs. performance tradeoff of IMP. We also find that the global selection criterion not only finds sparsity distributions on par with but, somewhat surprisingly. often better than those of more sophisticated layer-dependent pruning criteria. Finally, we conclude that, using an appropriate learning rate scheme, IMP performs well even when compared to state-of-the-aar approaches that incorporate sparsification into the training without or with only little computational overhead. That is, not only can IMP find some of the best performing architectures at any given sparsity level, but due to the compressed retraining time does so without needing to leverage longer running time even when compared to methods typically considered to be superior to IMP in that particular aspect.
Outline. Section 2] contains a complete overview over related works, including a brief summary of all pruning methods and approaches considered here. In SSection we provide the computational results and their interpretation by first addressing how IMP can develop its full potential within a restricted computational envelope in [Subsection 3.1 and Subsection 3.2] We then use the resulting lessons in order to draw up fair comparison to methods that incorporate pruning into their training We conclude with some discussion /Sectio 4
in Subsection 3.3
2 OVERVIEW OF PRUNING METHODS AND METHODOLLOGY
While the sparsification of Neural Networks includes wide variety of approaches. we will focus on Model Pruning, i..., the removal of redundant structures in a Neural Network. More specifically, our results will be limited to unstructured pruning, that is the removal of individual weights, opposed
2

---

Under review as a conference paper at ICLR 2022
to its structured counterpart, where entire groups of elements, such as neurons convolutional filters, are removed. We will also focus on approaches that start with a dense network and then either prune the network during training after training already discussed in the introduction. Following [Brrtllsoon et al. (2020), we will also refer to methods of the former category as pruning stable, since the final pruning should result in negligible decrease in performance, where methods of the latter category are referred to as unstable. For a full and detailed survey of Pruning algorithms we refer the reader to Hoetler al. (202T).
Pruning unstable methods are exemplified by ITERATIVE MAGNITUDE PRUNING (IMP) (Han et al., 2013). In its original form, it first employs standard network training, adding a common Ez- regularization term on the objective, and then removes all weights from the network whose absolute values are below certain threshold. The network at this point commonly loses some or even all of its learned predictive power, so it is then retrained for a fixed number of epochs. This prune-retrain cycle is usually repeated number of times; the threshold at every pruning step is determined as the appropriate percentile such that, at the end of given number of iterations, desired target sparsity is met I In the following we will first discuss two particular details of IMP that have been the focus of recent research: the questions of (a) how select the parameters be pruned and (b) how to retrain. We will then conclude this section by briefly outlining the pruning stable methods we have selected for this comparison and establish how fairly compare them to IMP.
2.1 RETRAINING APPROACHES
Let us first consider the learning rate scheme used during retraining. The original approach by Han et al. (2013) is commonly referred to as FINE TUNING (FT): suppose we train for T epochs using the learning rate schedule (7) )ssT and retrain for Ta epochs per prume-retrain-eycle then FT retrains the pruned network for Tn epochs using a fixed constant learning rate, most commonly yr- It was first noticed by Renda al., (2020) that the precise learning rate schedule during retraining can have a dramatic impact on the predictive performance of the pruned network. Motivated by WEIGHT REWINDING (WR) (Frankle et al.) 2019). they proposed LEARNING RATE REWINDING (LRW), where one retrains the pruned network for TF epochs using the last T - Tr. learning rates Wr-Te+1..... T/r Le & Hua (22T)) argued that the reason behind the success of LRW is the usage of large learning rates and proposed SCALED LEARNING RATE RESTARTING (SLR), where the pruned network is retrained using a proportionally identical learning schedule, i.e., by compressing (NO)ST into the retraining time frame of Tn epochs with short warm-up phase. They also introduced CYCLIC LEARNING RATE RESTARTING (CLR) based on the the 1-cycle learning rate schedule of Smith& & Topin (017).
We think that the nature of the proposed retraining methods indicates that the retraining phase is, at its core, similar to the usual training phase. Following this rationale, the success of LRW, SLR and CLR over FT should be attributed to the existence of both a large- and small-step retraining regime. In fact, large initial and exponentially decaying learning rate has become the standard practice for regular training (Leclere & Madry 2020). Note that such a scheme is employed not just by SLR and CLR, but also by LRW if Tx is sufficiently large to model the decaying learning rate schedule of the original training phase. The conventional approach to explaining the success of decaying learning rate schedules comes from an optimization perspective, i.e., an initially large learning rate accelerates training and avoids local minima, while the gradual decay helps to converge to an optimum without oscillation around it. However, an active line of research has theoretically supported the usage of large learning rates and separating training into a large- and small- step regime from a generalization perspective (Jastrzębski et al. 2017 Li et al., 2019 You et al., 2019 Leclere & Madry| 2020). Put more succinetly: retraining is training and therefore requires that some effort is put into tuning the learning rate scheme. LRW, SLR and CLR provide some good heuristic guidance for how effectively do so without an insurmountable amount of hyperparameter tuning.
While One Shot IMP, that is IMP with a single prune-retrain cycle, is a viable approach to model pruning, only the iterative approach (with multiple prune-retrain cycles) has been shown to achieve
1 'There is another way to view this relation: one can fix a given percentile to be pruned in every iteration and then simply repeat the prune-retrain cycle until either a desired level of sparsity is reached or the performance degradation exceeds given threshold. This is in fact how it appears to be used by Han et al. (2015) and how it is for example presented by Renda et al. (2020). While this reframing may appear trivial, it in fact highlights strength of TMP that will further emphasize when contrasting it with pruning stable approaches.
3

---

Under review as a conference paper at ICLR 2022
tradeoffs (Han et al. 2015 Renda et al. 2020). This iterative approach however commonly consists of a sigmificant amount of prune-retrain cycles, each needing the full original training time, that is T Ta. resulting in several thousand epochs worth of total training time. Renda et al. (2020) for example suggested the following approach: train a network for 7 epochs and then iterativlly prune 20% percent of the weights and retrain for T₁ = T epochs using LRW, i.e. use the same learning rate scheme as during training, until the desired sparsity is reached. We note that for T Tn the learning rate scheme of SLR becomes essentially identical to that of LRW. For a goal sparsity of 98% and T 200 original training epochs, the algorithm would therefore require 18 prune-retrain seanee-rtrain-ycllee cyclles for a massive 3800 total retrain epochs. In SuSubsectio 3.1 we will study the effect of the number of prune-retrain cycles, the number of retraining epochs and the learning rate scheme on the performance of the pruned network to establish whether IMP truly requires this massive amount of computational investment to develop its full potential.
dfate-of-the-art
2.2 PRUNING SELECTION CRITERIA
IMP in its original form treats all trainable parameters single vector and computes global threshold below which parameters are removed, independent of the layer they belong to. This simple approach, which will refer to as GL.OBAL, has been subject criticism for not determining optimal layer-dependent pruning rates and for being inconsistent (Liu et al. 2020). Fully-connected layers for example have many more parameters than convolutional layers and are therefore much
& Idelbayev 2018). Further, it has been observed that the position of a layer can play a role in whether that layer is amenable to pruning: often first and last layers are claimed to be especially relevant for the classification performance (Gale et al. 2019). On the other hand, in which layers pruning takes place significantly impacts the sparsity-induee
less sensitive to weight removal (Han et al. 2015 Carreira-
theoretical speedup (Blalock et al., 2020). Lastly, the non-negative homogeneity of modern ReLU-based neural network architectures (Neyshabur et al., 2015) would also seem indicate certain amount of arbitrariness to this heuristic selection rule, at least strong dependence on the network initialization rule and optimizer used, as weights can be rescaled to force it fully remove all parameters of layer, destroying the pruned network without having affected the output of the unpruned network.
Determining which weights remove is hence crucial for successful pruning and several methods have been designed to address this fact. Zhu & Gupta (2017) introduced the UNIFORM allocation. in which a global sparsity level is enforced by pruning each layer to exactly this sparsity. Gale al. (211)) extend this approach in the form of UNIFORM+ by (a) keeping the first convolutional layer dense and (b) pruning at most 80% of the connections in the last fully-connected layer. [Evei et al. (2020) propose reformulation of the ERDÔS RENYT KER NEL. (ERK) (Mocanu et al.) 2018 to take layer and kernel dimensions into account when determining the layerwise sparsity distribution. In particular, ERK allocates higher sparsity to layers with more parameters. Finally, Lee et al. (2020) propose LAYER-ADAPTIVE MAGNITUDE- -ASSEE PRUNING (LAMP), an approach which takes an fz-distortion perspective by relaxing the problem of minimizing the output distortion time of pruming with respect to the worst-case input. We note that we follow the advice of Evci et al. (2020) and Dettmers & Zettlemoyer (2019) and do not prune biases and batch- noaeaaaormalizttion parameters, since they only amount to a negligible fraction of the total weights, however keeping them has a very positive impact on the performance of the learned model.
We will compare these approaches in [Subsection 3.2 with a focus on the impact of the retraining phase. Since Le & Hual (2021) found that SLR can be used to obtain strong results even when pruning convolutional filters randomly, i.e., by assigning random importance scores to the filters instead of using the magnitude criterion or others, we are interested in understanding the importance of the retraining technique when considering different sparsity distributions.
2.3
A FAIR COMPARISON TO PRUNING STABLE METHODS
Pruning stable algorithms are defined by their ability to find a well-perforning pruned model during the training procedure. They do so by inducing a strong implicit bias during training, either by gradual pruning, i.e., extending the pruning mask dynamically, constraint-optine optimizaioo techniques to learn an almost sparse structure throughout training. LC by Curreira-Perpiiae & Idelbayev (2018) and GSM by Ding et al. (2019) both employ modification
by employing regularization- and
4

---

Under review as a conference paper at ICLR 2022
of weight decay and force the k weights with the smallest score more rapidly towards zero, where k is the number of parameters that will eventually be pruned and the score is the parameter magnitude or its product with the loss gradient. Similarly, DNW by Wortsman al. (2019) zeroes out the smallest k weights in the forward pass while still using a dense gradient. CS by Savarese et al. (2020), STR by Kusupati et al. (2020) and DST (Liu et al. 2020) all rely on the creation of additional trainable threshold parameters, which are applied to sparsify the model while being regularly trained alongside the usual weights. Here, the training objectives are modified via penalty terms to control the sparsification. GMP (Zhu & Gupta, (017) follows tunable pruning schedule which sparsifies the network throughout training by dynamically extending and updating a pruning mask. Based on this idea, DPF by Lin et al. (2020) maintains a pruning mask which is extended using the pruning schedule of ZZu & Gupta 2017). but allows for error compensation by modifying the update rule to use the (stochastic) gradient of the pruned model while updating the dense parameters.
The first claimed advantage of pruning stable methods is the lack of need for retraining, as they produce sparse and well-performing model at the end of regular training. This however comes at price: the implicit biases in pruning stable algorithms result in computational overhead when compared to the usual network training, i.e., find a sparse solution throughout training through ex- tensive regularization, masking or other methods, the per-iteration computational cost is increased. An overview of the computational overhead of the previously introduced methods can be found in Table 3 the appendix. Neither training epochs nor iterations are therefore an appropriate measure when comparing algorithms, since what matters the total time required obtain a sparse network. Gale et al. (2019), Evci et al. (2020) and Lin et al. (2020) consider this when presenting their re- sults. The second elaimed adaatte of pruning stable methods is that they find better pruning set than unstable methods that cannot recover mistakenly pruned weights (Liu et al., 2020) and whose heuristic selection mechanism is disadvantaged compared to methods that learn the pruning thresh- olds (Kusupati et al. 22220). While it seems reasonable to believe that the less performance lost at pruning, the better, Bartoldson et al. (2020) showed that this rationale might not be entirely sound: pruning seems to behave similar to noise injection and there exists a generalization-stabllleee siaiiinnstaabltty-aaaeee indicating that less stability can actually be beneficial to generalization.
Especially given the previously discussed recent advancements in better recovering pruning-induced losses during retraining, both the narrative that the need for retraining is a clear computational dis- advantage and that pruning stable methods find better pruning sets stand to be reevaluated. We will do so in Subsection 3.3
EXPERIMENTAL RESULTS
Let us outline the general methodological approach to computational experiments in this section, in- cluding datasets, architectures, metrics and optimization strategies used. Experiment-speciffe details are found in the respective subsections. We note that, given the surge of interest in pruning, [Blalock] let al. (2020) proposed experimental guidelines in the hope of standardizing the experimental setup. We aim to follow these guidelines whenever possible and encourage others to do so as well. All experiments performed throughout this computational study are based on the PyTorch framework (Paszke et al., 2019), using the original code of the methods whenever possible. All results and metrics were logged and analyzed using Weights Biases (Biewald) 2020). We plan on publicly releasing our implementations and general setup for the sake of reproducibility.
We restrict ourselves to the task of image recognition on the publicly available CIFAR-10 and CIFAR-100 (Krizhevsky et al. 2009) datasets as well as ImageNer (Russakovsky et al. 2015). All experiments are performed using deep convolutional neural network architectures, particular on Residual Networks (He et al.] (2015). Wide Residual Networks (WRN) (Zagoruyko & Komodakis, 2016) and VGG-16 (Simonyan & Zisserrann 2014). We use Stochastic Gradient Descent (SGD) with momentum as an optimizer throughout the experiments and exclusively use stepped learning rate schedules. We therefore also decided to forego CLR in favor of SLR despite the recommen- dations of Le & Hual (2021), , though we have included results using CLR in the appendix. Exact parameters can be found in Table 2 in the appendix.
The focus of our analysis will be the tradeoff between the model sparsity and the final test accuracy. As a secondary measure, we will also consider the theoretical speedup (Blalock et 2020) induced by the sparsity, which is defined as the ratio of the FL.OPs needed to evaluate the sparse over the
5

---

Under review as a conference paper at ICLR 2022
dense model assuming theoretical matrix multiplication algorithm capable of taking full advantage of unstructured sparsity, see (Sussection A. for full details. We decided to include since it can help to give a more differentiated picture of the inherent tradcoffs.
ResNet-56 on CIFAR- 10
90.0% of weights prumed
98.0% wrights pruned
0.333
0.525
0.030
best accurracy
0.025
0.875
0.850
Donse
0.915
FT
Renda al. (FT)
Renda al.
0.910
50
100
150
200
250
300
100
150
200
250
300
350
lotal petraining spochs
total reiraining epocls
Figure Test accuracy achieved by IMP relation to the total number of epochs required for retraining using either FT or SLR. For each point on the x-axis, we show the highest mean accuracy achieved by any configuration of grid search using up to this many total retraining epochs.
3.1 THE COMPUTATIONAL COST OF IMP
In this part will treat the number of retrain epochs per prune-retrain cycle Tn well as the total amount of such cycles J as tunable hyperparameters for IMP and try to determine the tradeoff between the predictive performance of the final pruned network and the total number of retrained epochs J T'. As baseline performance for a pruned network, we will use the approach suggested by Renda et al. (2020) as serves good benchmark for the current potential of IMP. We also include variant of this approach using FT during retraining. We will use the original global pruning criterion of Han et al. (2015) throughout this part.
In Figure I we present the results of our computations for ResNet-56 trained on CIFAR-10 with a moderate and high target sparsity of respectively 90% and 98%. The parameters for the retrain phase were optimized using grid search over Tn E (10, 15, .0. 60} and J E (1,2.....0 .6. and we consider both FT and SLR learning rate schemes during retraining. The weight decay values, including those used for the pruned and unpruned baselines, were individually tuned for each datapoint using a grid search over le-4, 2e-4 and 5e-4. All of our results are averaged over seeds with max-min- min-bands indicated.
Summarizing the results, we find that SLR significantly outperforms FT at both levels sparsity. While the general efficacy of tuning the learning rate scheme of IMP during retraining has previ- ously been sufficiently demonstrated by |Renda et al. (2020) and Hua (202)). the fact that IMP achieves its respective potential with significantly less than the total number of retraining epochs usually budgeted for its full iterative form has to our knowledge not been previously formally estab- lished: SLR comes within half percentage point of the pruned baseline within a mere 100 retraining epochs. This stands in stark contrast to the full 2000 and 3600 retraining epochs required to establish the respective baseline. Considering the pruned baseline using FT during retraining, this fact largely also does not seem to be a particular consequence of the learning rate scheme used during retrain- ing. Unlike commonly assumed, IMP with the right learning rate scheme during retraining therefore seems at least competitive with other methods not just with respect to the predictive power of the pruned network obtains but also with respect to its total computational cost. We also note that we found IMP with SLR and shortened retrain phase to benefit from significantly larger weight decay values than the baseline method, see extended results in Subsection B. in the appendix.
3.2 THE IMPORTANCE THE SPARSITY DISTRIBUTION
We compare the original global pruning criterion of IMP to the previously introduced proposed alternatives. Figuuee 2 reports the test accuracy in relation to the level of sparsity in the One Shot setting for CIFAR-TO0, where the network is retrained for 30 epochs, and ImageNet, where it is

---

Under review as a conference paper at ICLR 2022
WideResNet on CIFAR-100
Fine Tuning (FT)
Sealed Learning Rate Restarting (SLR)
70%
605
0est acouraty
1ost arauracy
60%
40%
50%
Uniform (Zhu Cupta,
Uniform (Gale
20%
40%
ERK (Evri et al., 2019)
30%
LAMP (Lee 2020)
0%
90%
92%
94%
96%
95%
90%
92%
94%
96%
989
amonnt of wrights prunol
Amousst weigltts pruned
ResNet-50 on ImageNet
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
741%
700%
155t accuniyy
Irst womnuyy
72%
70%
Global
Unifforn (Zlm & Gupta, 2017)
500
Unifocrnt (Gale al., 2019)
66%
BRK (Kvvi at al., 2019)
LAMP (Lee et al,, 2000)
64%
70%
78%
80%
85%
9095
95%
70%
75%
80%
85%
90%
96%
weights pramed
weights pruned
Figure 2: Performance-vss www.nnnncee----ppsaeee spaarisiy tradeoffs for the One Shot setting on CIFAR- (top) and Im- ageNet (bottom). We compare the sparsity allocation methods w.r.t. the different retraining tech- niques FT (left) and SLR (right).
retrained for 10 epochs. We tested both FT (Han et al. 2015) and SLR (Le & Hua 2021) to see if the learning rate scheme during retraining has any impact on the performance of the pruning selection scheme. The CIFAR-100 100 results are averaged over three seeds and max-min-bandd are indicated. Extended plots including the CIFAR-10 dataset well as the gruning-induced theoretical speedups can be found in [Subsection B.2 in the appendix. For ImageNet, the results are based on a single seed.
Surprisingly the simple global selection criterion performs at least on par with the best out of all tested methods at any sparsity level for every combination of dataset and architecture tested here when considering the sparsity of the pruned network as the relevant measure. Using SLR during training compresses the results by equalizing performance, but otherwise does not change the overall picture. We note that the results on CIFAR- 100 using FT largely track with those reported by Lee et al. (2020), with the exception of the strong performance of the global selection criterion. Apart retraining
from slightly different network architectures, epochs, e.g., 100 instead of 30, and that they use AdamW (Loshchilov & Hutter, 2019) instead of SGD. Comparing the impact different optimizers can have on the pruning selection schemes seems like a potentially interesting direction for future research.
note that they used significantly
While the wewpiittyvss-eeformma tradeoff has certainly been important part of the justification of modifications to global selection criterion, let us also directly address two further points that are commonly made in this context. First, the global selection criterion has previously been reported to suffer from a pruning-induced induced collapse at very high levels of sparsity in certain network architectures that avoided by other approaches. This phenomenon has been studied in the pruning before training literature and was coined layer-collapse by Tanaka et al. (200)), who hypothesize that it can be avoided by using smaller pruning steps since gradient descent restabilizes the network after pruning by following a layer-wise magnitude conservation law. To verify whether these observations also hold in the pruning after training setting, we trained a VGG-16 network on CIFAR-10 10, as also done by [eee al.] (2020), both in the One Shot and in the iterative setting. The results are reported in Figure iiin the appendix and show that layer collapse is clearly occurring for both FT and SLR for the global selection criterion sparsity levels above 99% in the One Shot setting, but disappears
7

---

Under review as a conference paper at ICLR 2022
entirely when pruning iteratively. This indicates that layer collapse, while genuine potential issue, can be avoided even using the global selection criterion. We also remark that SLR needs less prune- retrain-cycles to avoid layer-collapse than FT, possibly indicating that the retraining strategy impacts the speed of restabilization of the network in the hypothesis posed by Tanaka et al. (2020).
The second important aspect to consider is that layer-dependent selection criteria are also intended to address the inherent tradeoff not just between the achieved sparsity of the pruned network and its performance, but also the theoretical computational speedup. We have included plots highlighting the achieved performance in relation to the theoretical speedup in [Subsection B.2 in the appendix. The key takeaway here is that for both the ResNet-56 and the WideResNet network architecture, there is overall surprisingly little distinction between all five tested methods, with Uniform+ and ERK taking the lead and the global selection criterion performing well to average. For the ResNet- 50 architecture however a much more drastic separation occurs, with Uniform performing the best, followed by Uniform+ and then the global selection criterion. Overall, the picture is significantly less clear. Our results however indicate that the global selection criterion at the very least seems to offer a good balance in this inherent tradeoff.
Table 1: ResNet-56 on CIFAR-10- 10: Comparison between IMP and pruning stable methods for goal sparsity levels of 90%, 95% and 98%, denoted in the main columns. Each subcolumn denotes the Top-1 accuracy, the theoretical speedup and the actual sparsity achieved by the method. Each row corresponds one method, where we denote the time needed when compared to regular training next to the method's 's name. All results include standard deviations where zero or near-zero values are omitted. The two highest accuracy values are highlighted for each sparsity level.
90 %
95%
98 %
Method
Time
Accuracy
Speechup
Sparsity
Accuracy
Speedup
Spursity
Accuracy
Speeclup
Spursity
IMP
1.15x
92.79
10.21
60.04
90.00
91.62 10.29
#0.6
95.00
87.93 10.03
28 =1.7
98.00
IMP+
1.50k
93.25
20.14
1
90.00
92.57 20.18
20.6
95.00
89.86 10. 14
29 11.6
98.00
GMP
1.05x
92.84
10:42
10
90.00
92.12 10.17
95.00
89.65 10.31
50
98.00
GSM
1.17x
90.83
+0.24
90.24
88.91 10.15
11 #0.1
95.24
85.35 +0.24
23 #0.9
98.24
DPF
1.03x
93.32
90.00
92.68 #0.14
=0.1
#0.23
29
DNW
1.05x
#1.83
6 8
90.00
91.95 #0.06
7 20.3
95.09
34,87 +43.08
26 22,8
98.10
1.31x
90.51
10.16
1:0.1
90.00
#0.5
85,11 10.51
16
STR
1.35x
89.25
:1.23
8 8.008
15 10.76
89.77 a1.75
31 #10.3
95.11 a0.28
85. 10.26
66 #4.9
98 00 a0.04
CS
1.67x
10.30
=0.3
90.52
91.36
#2.9
95.38 10.19
90.04 20.36
50 =7.2
98.12 10.06
DST
2.41X
92.41
=0.28
10 =0.7
89. ,5 40.41
87
18
94.42
88.22 a0.36
53 #3.6
98.00 10.21
3.3
COMPARING IMP TO PRUNING STABLE APPROACHES
Using the lessons from the previous two sections, we finally compare IMP to recent pruning stable approaches. Given the high computational demand of tuning ten different methods, we limit our- selves to ResNet-56 networks trained CIFAR-10 10 as well WideResNet for CIFAR-100. For IMP we employ the global selection criterion and retrain using SLR, where the number of prune- retrain-cycles J well as the number of retraining epochs per cycle Tn were tuned using grid search over Tn E {10, 15, 20, 25} and J E (1,2,3, 3, 4} for CIFAR-10 and over Tn E {5, 10, 15} and E [1,2,3) 3} for CIFAR- 100. We will also consider a restricted version of IMP, where we impose the additional constraint J Ti < 30 for CIFAR-10 and J. Tr < 20 for CIFAR- to obtain a method that will be largely comparable to pruning stable methods w.r.t. its total runtime. In the following, we denote the restricted version by IMP and the unrestricted version by IMP+.
The hyperparameters of each of the pruning stable methods were likewise tuned using manually de- fined grid searches, resorting to the recommendations of the original publications whenever possible, see SSbsecttinn A.2 for exact details. Whenever possible, that is for GMP, GSM, DNW, DPF, and LC, we give the methods predetermined sparsity levels of 90%, 93%, 95%, 98%, 99% and 99.5%, the same as given to IMP. Tuning the remaining methods in a way that allows for a fair comparison however is significantly more difficult, since none of them allow to clearly specify desired level of sparsity but instead require one to tune additional hyperparameters as part of the grid search. Despite our best efforts, we were only able to cover part of the desired sparsity range using STR and DST. In addition, we noticed that each of these method can have some variance in the level of sparsity achieved even for fixed hyperparameters, so we list the standard deviation of the final sparsity with
8

---

Under review as a conference paper at ICLR 2022
respect to the random seed initialization. All results are averaged over three seeds for CIFAR-10 and over two seeds for CIFAR-100.
One of the main intentions behind pruning stability is to eliminate the need for computationally expensive retraining. We start by verifying this assumption using the pruning stable approaches and allowing them to be retrained for further 30 epochs using FT. F...ue 9 (anuree 1T]in the appendix show the increase in accuracy of the pruned networks after retraining. We observe that LC and CS are often unable to find an actually sparse model throughout training and hence greatly benefit from retraining. The remaining methods however mostly stay true to their claimed pruning stability, but nevertheless can profit from retraining. For the sake of fair comparison, we will therefore use the accuracy reached after retraining when it exceeds the original one for all pruning stable methods in this section while only referring to the original train time when comparing runtimes for all methods except LCC, CS and of course IMP.
Table now reports the computational cost, final test performance, theoretical speedup and actually achieved sparsity of all ten methods for CIFAR-10. Full results for both CIFAR-10 and CIFAR-100 are reported in SSbsection B.3] in the appendix. Regarding the computational costs, we see that only few methods consistently outperform IMP. The results show that for CIFAR-10 only GMP, DPF and DNW deliver a pruned network faster than the restricted form of IMP. While IMP+ is about 30% slower than the restricted form, it still beats CS and DST with respect to its runtime. The network architecture however seems to have a significant impact on the performance of the individual methods, since for CIFAR-100 with WideResNet architecture, GMP, DPF and STR deliver a pruned network faster than the restricted form of IMP, while IMP+, here about 12% slower than the restricted form, beats GSM, DNW and LC with respect its runtime.
Regarding the achieved accuracy-vs- sparsity tradeoff, the results show that IMP with SLR and a tuned retrain phase is able perform on par with the best pruning stable methods considered here. For CIFAR-10, 10, only DPF is able to outperform IMP+ throughout all levels of sparsity. usually by less than 0.3%. All other methods fail to compete at the highest level of sparsity and otherwise best about match the performance of IMP+. The restricted version of IMP does perform noticeably worse than IMP+ but still compares favorable to most other methods in the comparison throughout all levels of sparsity. For CIFAR-100 however, DPF does not exhibit the same strong behavior despite receiving some additional attention during hyperparameter tuning, see appendix for details. No method is able to consistently outperform even the restricted version of IMP. Somewhat surprisingly, we note that IMP is also able to get favorable theoretical speedups compared to other methods for both CIFAR-10 and CIFAR-100. We emphasize again that IMP manages obtain these results without a strong computational overhead as commonly assumed.
4 DISCUSSION
The goal of this work is to demonstrate that IMP is surprisingly efficient, achieving its full potential with significantly less than the extensive amount of retraining usually considered necessary. We also find that it can provide state-of-the-arr pruning results when paired with an appropriate learning rate scheme during retraining, further supporting the results of Renda al. (2020) and Le & Hua (202 In our opinion, these results contradict the common narrative in the literature and imply that there is currently insufficient empirical evidence to support the commonly claimed benefits of imposing an implicit bias during training- We hope that our findings establish a more realistic yet easily realisable baseline against which such claims can be compared in the future.
We would also like to acknowledge clear limitation of the study: our experiments were limited to image recognition tasks on particular network architectures. Likewise, we have limited our ex- ploration to the case of unstructured pruning. Extending the results to include, e.g., NLP well as a wider variety of network architectures and structured pruning tasks would help paint a more complete picture.
9

---

Under review as a conference paper at ICLR 2022
5 REPRODUCIBILITY
Reproducibility is of utmost importance for any comparative computational study such as this. All experiments were based on the PyTorch framework and use publicly available datasets. The implementation of the ResNet-56 network architecture is based on https::/githn wttp:////saararreseeeeoeooe py, the implemen-
com/ me meliketoy wi www.:rreeeett.pooo. reree the implementation of the VGG-16 network archi- com/ jawee- -ecooo--ee// wwweeeeeeeeeeeeaeeeeeeeareeeeeee wwtp:eerraaaattteesaaeaeaeeee and the implementation of the Resnet-50 network ar- wttg/vis.. wttg//vs...n//taalleee//eeo /ineex html Regard- ing the pruning methods, the code was taken from the respective publications whenever pos- sible. Regarding the different variants of magnitude pruning such as ERK Uniform+, we closely followed the implementation Lee et al. (2020) available at ptttps::/ggiiii sparsity For metrics such the theoretical speedup, of [Blalock et al. (2020), see All parameters considered are specified in Section 3 or in the Appendix. We also plan on publicly releasing the code used for our imple- mentations and general setup for the sake of reproducibility and will make it available during any
tation of the WideResNet network architecture is based on httpss://githe
tecture is based on httpp:tttttpptttpeeeeeeee
chitecture is taken from https:://ttposeeeeeeeeeee
com/
Maeho-lee) Leleellaerree aeyyer-adaptio
we relied on the implementation in the Sernnkkeenn----aewwweee com/ http/shiinkkbnee wttm///////rrrrreeeeee
/etts::/gittie
submission of this work.
REFERENCES
Brian Bartoldson, Ari Morcos, Adrian Barbu, and Gordon Erlebacher. The generalization-stabiiyy tradeoff in neural network pruning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. Curran Associates, Inc., 2020. URL 9ea 9551 de88bc (
neurips. pdff
C0/ cc/paper papppapppa//
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: www. wandb. com/ Software available from wandb.com.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Dhillon, D. Papailiopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 129 URL https: /ttp:s://roccceeee ngs www.mlls..e........m..........mm mlsys.........mmmm.mmmmmlm...m.. www.ssssssperr//////22 org//aaper/222//F1ee pdff
146, 2020.
Miguel Á. Carreira-Perpinán and Yerlan Idelbayev. Learning-oompression algorithms for neural net pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.048400 July 2019.
Xiaohan Ding, Guiguang Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, and Ji Liu. Global sparse momentum sgd for pruning very deep neural networks. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, URL htttps://proeeedingg neuriis.....p..... wt/pppppr//211////ieeee. http://wwe.............c...oaaaa 23316ckassderTaadd411771aaa--ea2
Inc., 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2943 2952. PMLR, 13-18 Jul 2020. URL https: httpss:/procceedingg. mlr. press/ http://111vvvvell h
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2018.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.0161,, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 1922009900 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL
neurips pttpscccppp//211//11122 pdf
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In S. Hanson, Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume 5.
1993. URL htttts::/ppooeeeee
neurips. Pape- pdf
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification, In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102.00554
January 2021.
Steven A. Janowsky. Pruning versus clipping in neural networks. Phys. Rev. A. 39:6600-6603, Jun 1989. doi: 10. 00.1100/hhysReev.39..66
Stanistaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, November 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterizatioo for learnable sparsity. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Confer- ence Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 5544-5555. PMLR, 13-18 Jul 2020. URL hntt:::/ http:////occeeennssceee mlr. wtt.pppsss/.ls///// kusupati20a. html
Duong Hoang Le and Binh-Son Hua. Network pruning that matters: case study on retraining variants. In International Conference on Learning Representations, 2021. URL https::/
openreview net (dbouummid-Cb5AMMMge
Guillaume Leclere and Aleksander Madry. The two regimes of deep network training. arXiv preprint arXiv:2002. 10376, February 2020.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky (ed.), Advances Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], p98 598-605.... Morgan Kaufmann, 1989. URL htlhtttpp/ppper/
n ps pascc/aape/// 2500 opt ima ma I bra i damage
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-brat pruning. In International Conference on Learning Representations, October 2020.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL htt.stttssss http:///n/aaape//oooreeeeeeeeeee mew.eeddiinnnnnnnissssss..u

---

Under review as a conference paper at ICLR 2022
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K.H. So. Dynamic sparse training:
Find efficient sparse network from scratch with trainable masked layers. In International Confer- ence on Learning Representations, 2020. URL https:://penrevieww wwt.::/wwweeeoummmmm...e
BJJIbGJrtDB
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Communications, 9(1), June 2018. doi: 10.1038/ $41467-018-04316-3
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL httpppppp/ppccccciggg..s.. ieeesnnenrree wwtpiips..ppppp.pppeeeeee cc//paper/2111//Eee. bdbca288fee Cabca288fee7192EE. Ttw.:E//EEEEEEE11011212220000-00
pdf
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1 15(3):211-252, 2015. doi: 10. 122777/11163-------81116
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsi- fication. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 11380- 11390. Curran As- sociates, Inc., 2020. URL 23004190b11 wtwp://7777aaa11111aaaaaaaa - Paper pdf
neurips wwt.eeeps....pppeeeeeeeeeeeeeeee cc/ppaperr2222//11eeee
Jonathan W. Siegel, Jianhong Chen, and Jinchao Xu. Training sparse neural networks using com- pressed sensing. arXiv preprint arXiv:2008.0966 August 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 4rXiv:1409.1556. September 2014.
Leslie N Smith and Nicholay Topin. Super-convergencee Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for -ooii- Operations Applications, volume 1 11006, pp. 1100612. International Society for Optics and Photonics, 2017.
Hidenori Tanaka, Daniel Kunin, Daniel I.. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems 2020, June 2020.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Asso- ciates, Inc., 2019. URL https::/pproceedggg.uu. wwwrsis...........p...... http:///ww....iiiss../////////// pctps/////ppee///////lle////// htw.::bcaaaabeeedecccccccc
Paper. pdf
Kaichao You, Mingsheng Long, Jianmin Wang, and Michael Jordan. How does learning rate decay help modern neural networks? arXiv preprint arXiv: 018.00 August 2019.
12

---

Under review as a conference paper at ICLR 2022
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07116, May 2016.
Wide residual networks.
arXiv preprint
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1rX1v:1611.03550, November 2016.
Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, October 2017.
13

---

Under review as a conference paper at ICLR 2022
A TECHNICAL DETAILS AND TRAINING SETTINGS
A. TECHNICAL DETAILS
We define pruning stability follows.
Definition A.1 [Bartoldson et al. (2020). Let Epe and post be the test accuracy before pruning and after pruning the trained model, respectively. We define the pruning stability of a method as
ppe
Epost
A stability =1
E [0, 1].
1pre
Pruning stable methods are sparsification algorithms that learn a sparse solution throughout training such that Astability 1. For example, methods that perform the forward-pase using an already sparsified copy of the parameters (e.g. DNW by /Wortsman et al.] 2219), will have Astabilty 1, since the 'hard' pruning step only consists of an application of the present pruning mask, which has no further effect. Methods that actively drive certain parameter groups towards zero more rapidly (such as Perrpinn & Idelbayev 2018] Ding et al., 2019) will have a pruning stability close to 1, since the projection of (magnitude) pruning at the end of training will perturb the parameters only slightly.
Crucial to analysis are the tradeoff's between the model sparsity, the final test accuracy and the theoretical speedup induced by the sparsity (Blalock et al. 2020). Theoretical speedup is metric measuring the ratio in FLOPs needed for inference comparing the dense and sparse model. More precisely, let Fa be the number of FLOPs the dense model needs for inference, and let similarly be F, the same number for the pruned model, given some sparsity sA] The theoretical speedup is defined as Fa/ F; and depends solely on the position of the zero weights within the network and layers, not on the numerical values of non-zero parameters. While we focused on the model sparsity in the main body of the text, we include plots regarding the theoretical speedup here in the appendix, as suggested by Blalock al. (2020).
A.2 TRAINING SETTINGS AND HYPERPARAMETERS
Table 2: Exact training configurations used throughout the experiments for IMP. For pruning sta- ble methods we use same settings, with the following exceptions: (1) since momentum plays crucial part in the justification of GSM, tune it over 0.9, 0.99 and 0.995 and (2) any addi- tional hyperparameters of the respective methods as well as weight decay were tuned as indicated in Subsection A.2 We note that others have reported accuracy of around 80% for WRN28x 10 trained on CIFAR 100 that we were unable to replicate. The discrepancy most likely due to an inconsistency in Py PyTor'''s 's dropout implementation.
Network (number of weights)
Learning rate ()= training epoch)
Unpruned Test accuracy
Dataset
Epochs
Batch size
Momentum
(0.1
E [1,90].
CIFAR-10
ResNet-56i (850 K) VGG-16 6 (138 Mio)
200
128
0.9
0.01
E [91, 180).
93.51% +0.3% 93.8% #0.2%
00.001
I E [181, 200)
0.1
t 10).
0.02
[61, 120),
CIFAR-100
WRN28x10 (37 Mio)
200
128
0.9
76.7% #0.2%
0.004
[121, 160),
0.0008
E [161, 200)
(0.1
I [1,5].
0.1
& [5,30],
ImageNet
ResNet-50 (26 Mio)
90
256
0.9
0.01
t [31, 60),
76.17% +0.03%
E [61, 80).
0.0001
E [81,90]
We list the hyperparameter grids used for each pruning stable method taking part in the comparative study.
TTo compute the number of FI.OPs, sample single batch from the test set. The code to compute the thcoretical speedup has been adapted from the repository of the ShrinkBench framework (Blalock et al. 2020)
14

---

GMP Under review as a conference paper at ICLR 2022
Table 3: Overview of sparsification methods. CS, STR and DST control the sparsity implicitly via additional hyperparameters. IMP is the only method that is pruning instable by design, i.e., loses its performance right after the ultimate pruning. Further, IMP the only method that is sparsity agnostic throughout the regular training; the sparsity does not play role while training to convergence. All other methods require training an entire model when changing the goal sparsity. The computational overhead refers the per-iteration overhead during regular training. We denote by n the number of trainable parameters, while S 7t the number of parameters that remain after pruning the goal sparsity.
Sparsity specifiable
Pruning stable
Sparsity agnostic training
Overhead during Training
IMP
N/A
((n) + O(alog n)) every time the mask updated
GSM
O(n an-ianmmn(
LC
O in-minnk,,n--- A3)
DPF DNW
((n) + O(n every iterations
[fn-min(k,
((n) + add. backprop
STR
X
((n) add. backprop
DST
X
(ns) + add. backprop
A.2.1 RESNET-56 CIFAR-10
For each method (including IMP) we tune weight decay over le-4, 5e-4, 1e-3 and 5e-3 and keep momentum fixed 0.9, Since momentum plays crucial part in the justification of GSM, we tune over 0.9, 0.99 and 0.995. Since the learning rate schedule might need additional tuning, we vary the initial learning rate between 0.05, 0.15, 0.1 and 0.2 for all methods except CS. The decay of the schedule follows the same pattern as listed in Table 2 Since CS required the broadest grid, we fixed the learning rate schedule the one in Table Otherwise, we used the following grids.
GMP
Equally distributed pruning steps: (20, 100).
GSM
Momentum: f0.9, 0.95, 0.99}.
LC
We only tune the weight decay and, similar the recommendation by Carreira-PPrpiinnn & Idel- bayev (2018), increase over time as Ao 1.1,, where is increased over time and Åo is the initial weight decay. For the retraining phase deactivate weight decay.
DPF
As for GMP, we tune the number of pruning steps, i.e., {20, 100), and the weight decay.
DNW
We only tune the weight decay, since there are no additional hyperparameters.
CS
As recommended by Savarese et (222)) fix the temperature B at 300. Oth- erwise, we perform the following grid search. We set the mask initialization so E
(-0.3, -0.25, -..2, -0.1, -0.05, 0, 0.05, 0.1, 0.2, 0.25, 0.3) and the e penalty (le-8, 1e-7).
STR
We tune the initial threshold value 8init E (-100 50, -5. -, -1,0, -,0,-,,-,,,1,,,,5500 50, 100)
an extended
grid search, we also used weight decays in (5c-05, le-4) and varied Siait (-40, -30, -20, 1300-10) 10).
DST
We tune the sparsity regularization parameter a E {5e-6, 1e-5, 5e-5, le-4, 5e-4). In an extended grid search, used weight decays in (0, le-4) and tuned over (1e-7, 5e-7, le-6).
15

---

Under review as a conference paper at ICLR 2022
A.2.2 WIDERESNET ON CIFAR-100
For each method (including IMP) tune weight decay over lc-4, 2e-4 and 5e-4 and keep momen- tum fixed at 0.9. Since momentum plays a crucial part in the justification of GSM, we tune it over 0.9 and 0.95. For CIFAR-10, we noticed that especially DPF. DST and GMP benefited from an additional tuning of the learning rate schedule. We hence vary the initial learning rate between 0.05, 0.15 and 0.1 for these three methods and also for CS to broaden the grid which resulted inferior performance compared to CS on CIFAR-10. The decay of the schedule follows the same pattern as listed Table Otherwise, we used the following grids.
GMP
Equally distributed pruning steps: (20, 100}.
GSM
Momentum: {0.9, 0.95, 0.99}.
LC
We only tune the weight decay and, similar to the recommendation by Carreira-Perpiiaee & Idel- bayev (2018), increase over time as Å 1.1', where j is increased over time and Ag is the initial weight decay. For the retraining phase deactivate weight decay.
DPF
As for GMP, we tune the number of pruning steps, i.e., {20, 1000. and the weight decay.
DNW
We only tune the weight decay, since there are no additional hyperparameters.
CS
As recommended by SSavarsse et al. (2020). we fixed the temperature 3 at 300, but in- creased it 500 upon noticing the pruning instability of CS this dataset. Oth- erwise, we perform the following grid search. We set the mask initialization so E {-0.3, 0.25, -0.2, -0.1, -0055, -0.03, -0001, 0.005 -0.003, 00.001, 0} and the l penalty to fle-9, le-8, lc-7}.
STR
We tune the initial threshold value Sinit E {-5000, grid search, we also varied Sinit E {-200, - 1500, 1000, 800, -50, -00 -25 -10,03
-500). In extended
DST
We
tune the sparsity-controlling regularization parameter a E (5e-6, le-5, 5e-5, le-4, 5e-4). In extended grid search, we tuned a over f1e-6, 3--6, 8e-6, 3e-5, 8e-5, 2e-, 3e-4, 4e-4, 5e-4, 6e-4, 7 8e-4, 9--4 le-3).
16

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL PLOTS
B.I
THE COMPUTATIONAL COST OF IMP
ResNet-56 on CIFAR-10 (weight decay 0.0001)
90.0% of weights pruned
98.0% of wrights prunod
0.34
tost accurre
0.03
Led accuras
0.900
0.875
0.800
0.91
0.90
0.800
50
100
150
200
250
300
350
50
100
150
200
250
300
350
zetraining epocias
retraining epochs
ResNet-56 on CIFAR-10 (weight decay 0.0002)
00.0% weights praned
wrights prunod
0.925
0.90
0.900
0.92
0.875
0.850
0.825
0.900
50
150
250
350
50
100
150
200
250
300
350
total eporias
total retraining apoclis
ResNet-56 on CIFAR-10 (weight decay 0.0005)
90.0% of weights pruned
98.0% of weights praned
0.99
0.925
Les: aeturee
0.900
0.92
0.875
0.850
SLR
..... Renda et al. (PT)
0.91
Renda et
PT
0.825
150
250
350
250
350
total retroining eporhs
total retraining epochs
Figure 3: Test accuracy achieved by IMP in relation to the total number of epochs required for retraining using either FT or SLR. Each row of the plots shows a different weight decay as indicated in the title. The pruned baselines were established by using the approach of Renda al. (2020) using the respective weight decay and either its original form, using LRW, or with a modified version using FT.
The plots of Figure show the results of Figure I for each of the three weight decay factors le-4, 2e-4 and 5e-4 independently. Clearly, the choice of the weight decay affects both the accuracy of the dense model as well the accuracy of the pruned model when using the approach by Renda et al. (2020). both with LRW and FT. For the dense models with weight decays lc-4, 2c-4 and 5e- we obtain respective average performances (40.26%). For the respective results of the algorithm by |Rnda et al. (2020) we observe test accu- racies of 92.91% % (+0.39%), 93.08% (#0.54%) and 91.99999 1.94% (+0.16% 16%) in the case of goal sparsity of 90%, while the results at goal sparsity of 98% are 89.95% (+0.23%), 90.18% (+0.89%) and 89.63% (6%.16%). Similarly, for the algorithm by Renda et al. 2020) with FT instead of LRW, we observe test accuracies of 90. 74% (+0.43%), 91.86% (#0.54%) and 92.28% (#0.02%) in the of a goal sparsity of 90%, while the results at a goal sparsity 98% are 87.75% (#0.4%), 87.85% (+0.58%) and 99.03% (+0.09%).
92.01 % (+0.41% 1)., 92.86% (+0.49%) and 93.53%
17

---

Under review as a conference paper at ICLR 2022
Clearly, that SLR benefits from larger weight decay, while the approach by Renda et al. (2020) is suffering from an increased penalty on the weights. Although SLR not able to reach the pruned baseline in the case of le-4 weight decay within the given retraining time frame, note that SLR easily outperforms the LRW-based proposal by |Rnda et al. (2020) when considering the weight decays that also lead to the best performing dense model, which is strong indicator that it is preferable to use SLR and shortened retraining timeframe.
B.2 THE IMPORTANCE OF THE SPARSITY DISTRIBUTION
In the case of ResNet-56 on CIFAR-10 and VGG-16 on CIFAR-10, we report the weight decay con- fig with highest accuracy, where we optimized over the values le-4, 5c-4 and le-3. For WideResNet on CIFAR-100 and ResNet-50 on ImageNet we relied on weight decay value of lc-4 for both architectures.
Figure 4 Figure 5 and Figure 6 compare the different sparsity allocation methods with respect to the retraining strategy for ResNet-56 on CIFAR-10, WideResNet on CIFAR-100 and ResNet-50 on ImageNet. Similarly, Figure F
7sswws the accuracy vs. theoretical speedup tradeoffs on a logarithmic scale. These plots also show that some methods sparsify convolutional layers more aggressively than the global approach, resulting in higher theoretical speedups. However, despite its simplicity, the global approach performs on par with respect to managing the accuracy vs. speedup tradcoff, where observe that for ResNet-50 ImageNet it even outperforms methods such as LAMP and ERK regarding both objectives, performance and speedup.
ResNet-56 on CIFAR-10
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
80%
80%
test acearary
1ost arcurac
Global
60%
Unifoom (Zhm & Cupta, 2017)
Unitorm | (Gale al., 2010)
ERK (Evri et al., 2019)
40%
LAMP (Lee et al., 2020)
90%
92%
144%
90%
98%
90%
92%
94%
91%
98%
weights prumod
weiglata pruned
Figure 4: Sparsity-vs. ww..eee..eraaeeeeeee tradcoffs for ResNet-36 on CIFAR-10 10 for IMP in the One Shot setting for FT (left) and SLR (right) retraining methods. The plot includes max-min confidence intervals.
WideResNet on CIFAR- 100
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
70%
60%
2at aoturacy
19t waurann
60%
Global
40%
Uniform (Zhu Gupta, 2017)
Unilorune (Gale
.0%
al., 2019)
GAMP 2020)
90%
94%%
98%
90%
92%
94%
96%
96%
antomn of wrigltss praned
aunonza of weightts proned
Figure 5: performmneec tradeoffs for WideResNet CIFAR- 100 for IMP in the One Shot setting for FT (left) and SLR (right) as retraining methods. The plot includes max-min confi- dence intervals.
18

---

Under review as a conference paper at ICLR 2022
ResNet-50 on ImageNet
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
740%
0est acouraty
19st aecuracy
72%
600
Unitoom (Zhn Cupta,
68%
60%
Uniform (Gale
66%
ERK (Evri et al., 2019)
LAMP (Lee 2020)
40%
70%
70%
S075
8536
90%
90%
70%
75%
80%
80%
90%
96%
Amsnnt wrights prunol
Amounst weigltts pruned
tradeoffs for ResNet-50 on ImageNet for IMP in the One Shot setting for FT (left) and SLR (right) as retraining methods.
Figure 6:
ResNet-56 on CIFAR-10
Fine Tuning (FT)
Scaled laarning Rate Restarting (SIR)
80%
t05 noruraey
t00: naturay
Global
Uniform (Zba & Gopta, 2017;
60%
Uniforms (Gale al., 2019)
40%
ERK (Evdi es al., 2010)
(Lee 2020)
10a
theoectioa speedup
theoresical speedup
WideResNet on CIFAR-100
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
70%
600%
tsst acctroocy
1ndt aecuracy
60%
40%
50%
20%
40%
30%
0%
10'
10'
thoometieal speedup
thvoretical speedug
ResNet-50 on ImageNet
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
74%
700%
72%
test accaiiugg
70%
60%
66%
64%
10a
10
theorctical speedup
thcoretical speedup
Figure 7: -e-thorrrc-tttheeticcclllllll speedup tradcoffs for ResNet-56, WideResNet and ResNet-50 on the respective datasets. All plots depict the shot setting.
19

---

Under review as a conference paper at ICLR 2022
VGG-16 on CIFAR-10 (One Shot)
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
50%
r2st acouraty
1ost securacy
Global
800
80%
Uniform (Zhu Cupta, 2017)
Uniform (Gale 2019)
70%
ERK (Evr et al., 2019)
LAMP (Lee 2020)
60%%
60%
90%
92%
94%
96%
95%
90%
92%
94%
96%
989
amonnt of weights pranol
Amounst weightts pruned
VGG-16 on CIFAR-10 (One Shot)
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
800%
90%
test accumiy
Irst wemmury
20%
70%
600
00%
102
10P
10
10a
theoretical speedup
theuresical speedup
VGG-16 on CIFAR-10 (Iterative)
Fine Tuning (TT)
Scaled Learning Rate Restarting (SLR)
90%
tast sccaracy
90%
Global
80%
85%
Uniform (Zlm Gupla, 2017)
Uniterm: (Gale al., 2010)
70%
ERK (Evri et al., 2019)
205
80%
LAMP (Lee 2020)
90%
94%
90%
94% Amount wiights pruned
amommt of weights primod
16 on CIFAR-10 (Iterative)
Fine Tuning (FT)
Scaled Learning Rate Restarting (SLR)
05%
90%
tett accuarry
Indt acnnary
50%
85%
Global
Uniforma (Zha & Gupto,
85%
Uniformet (Gale al. 2019)
73%
ERK (Evel al., 2019)
70%
LAMP (Lee al., 2020)
10a
10a
10a
10a
theoretical speedup
theuresical speedup
Figure 8: Performance-vs. -paarsty and on CIFAR-10 for IMP in the One Shot (above) and Iterative (below) setting for FT (left) and SLR (right) as retraining methods. In the One Shot setting the model is retrained for 30 epochs after pruning and the iterative setting consists of 3 rrune-retrain cycles with 10 epochs each. For One Shot we observe layer-collapse while the iterative splitting into less severe pruning steps avoids the problem. Note that the total amount of retraining epochs between the two settings is identical here.
tradeoffs for VGG-16 16
20

---

Under review as a conference paper at ICLR 2022
B.3 COMPARING IMP PRUNING STABLE APPROACHES
Table extends Table by showing the results of the full sparsity range between 90% and 99.5%.
The same results can be seen visualized in Figure 10 As described in the main section, Figure 9 shows the actual pruning stability and increase accuracy after retraining with FT. Apart from LC, all methods can be considered pruning stable with pruning stability close to 100%. However, note that some methods can benefit from retraining. To allow fair comparison, we hence always considered the maximum of the performances before and after retraining, while measuring the com- putational time needed only for the regular training time, ignoring the time needed for retraining.
ResNet-56 CIFAR-10
Accuracy increase by retraining
5%
GSM
DST
GMP
DNW
4%
DPF
LC
STR
CS
iacreuse in acturaay
3%
29
19
0%
985
90%
92%
94%
96%
989
100%
weights pruned
Pruning stability
80%
pranin stabiiity
60%
40%
GSM
GMP
DNW
100%
DPF
LC
STR
CS
900
925
100%
weights primed
Figure 9: Increase in accuracy after retraining (above) as well as pruning stability (below) for ResNet-56 trained on CIFAR- using different pruning stable methods. Each method was retrained for 30 epochs using FT. Each datapoint corresponds to the hyperparameter config with highest ac- curacy directly after pruning when considering 5% sparsity intervals between 88% and 100%. The confidence bands indicate the min-max-deviation around the mean with respect to different initial- ization seeds.
21

---

Under review as a conference paper at ICLR 2022
Table 4: ResNet-56 on CIFAR-10: Results of the comparison between IMP and pruning stable methods for the sparsity range between 90% and 99.5%. The columns are structured follows: First the method is stated, where IMP+ denotes the unrestricted version of IMP. Secondly, we denote the time needed when compared to regular training of dense model, e.g. LC needs 1.14 times as much runtime as regular training. The following columns are substructured as follows: Each column corresponds to one goal sparsity and each subcolumn denotes the Top-1 accuracy, the theoretical speedup and the actual sparsity reached. All results include standard deviations, where we omit zero or close zero results. Missing values (indicated by -) correspond to cases where were unable to obtain results the desired sparsity range, i.e., there did not exist a training configuration with average final sparsity within .2% interval around the goal sparsity and the closest is too far away or belongs to another column.
%
93%
Method
Time
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
IMP
1.15x
92.79
#0.21
6 #0.4
90.00
92.18 #0.15
10.5
93.00
IMP+
1.50x
93.25
#0.14
#0.4
90.00
92.89 20.13
+0.5
93.00
GMP
1.05x
92.84
#0.42
10
90.00
92.50 40.10
14
93.00
GSM
1.17x
90.83
+0.24
5
90.24
89.92 10.14
#0.2
93.24
DPF
1.03x
93.32 #0.11
90.00
93.23
90.5
93.00
DNW
1.05x
91.81 +1.83
6 #0.8
90.00
91.97 #0.20
10.2
93.09
LC
1.31x
90.51 #0.16
5 #0.1
90.00
89.67 #0.55
40.6
93.00
STR
1.35x
89.25
+1.23
#0.8
90.15 #0.76
90.27 40.86
20
93.01 40.58
CS
1.67x
91.87
+0.30
#0.3
90.52 #0.76
89.64 +1.43
16
92.78 40.15
DST
2.41x
92.41
#0.28
#0.7
89.55 #0.41
95%
98%
Method
Time
Accuracy
Specdup
Sparsity
Accuracy
Speedup
Sparsity
IMP
1.15x
91.62
+0.29
#0.6
95.00
87.93 #0.03
28 +1.7
98.00
IMP+
1.50x
92.57
20.18
13 #0.6
95.00
89.86 +0.
29 +1.6
98.00
GMP
1.05x
92.12 10.17
20
95.00
89.65 40.31
50
98.00
GSM
1.17x
88.91
+0.15
#0.1
95.24
85.35 40.24
23 +0.9
98.24
DPF
1.03x
92.68
#0.14
12 #0.1
95.00
90.49 +0,23
29 +1.2
98.00
DNW
1.05x
91.95 #0.06
7 #0.3
95.09
34.87 +43.08
26 +2.8
98.10
LC
1.31x
89. #0.60
8 40.5
95.00
85.11 #0.51
16
98.00
STR
1.35x
89.77
110.3
95.11 #0.28
89.15 #0.26
+4.9
98.00 #0.04
CS
1.67x
91.36 +0.23
21 +2.9
95.38 10.19
90.04 +0.36
50 47.2
98. 12 +0.06
DST
2.41x
89.17
18
94.42
88.22 40.36
53 +3.6
98.04 #0.21
99%
99.5%
Method
Time
Accuracy
Specdup
Sparsity
Accuracy
Speedup
Sparsity
IMP
1.15x
83.47 #0.22
51 +2.4
99.00
76.09 +0.20
94 +5.3
99.50
IMP+
1.50x
86.03
#0.31
52 #2.7
99.00
79.81 +0.68
87 +5.8
99.50
GMP
1.05x
62.80
+1.11
100
99.00
36.95 40.32
197
99.50
GSM
1.17x
81.09
#0.04
36 42.2
99.24
74.54 +1.01
65 44.3
99.74
DPF
1.03x
86.76
+0.33
63 +3.7
99.00
80.03 +0.64
146 +34.3
99.50
DNW
1.05x
83.67
#0.24
15 #0.1
99.17
34.71 #24.17
14.4
99.67
LC
1.31x
81.63
20.74
30
99.00
74.44 42.03
64
99.50
STR
1.35x
83.68
+0.94
159 #31.9
99.13 #0.02
77.34 42.68
420 #167.4
99.66 +0.09
CS
1.67x
86.55 +0.92
69 #7.1
98.90 +0.02
DST
2.41x
86.99
98.36
B.4 RESULTS USING CLR
Although our focus lies on analyzing SLR, as explained Section 3] we include results using CLR in this section: Figure complemensss Figure 3 and Figure 14/as well as Figure 14 show the effect of CLR onto the pruning selection criteria of interest throughout this work. We do not observe significant differences between SLR and CLR.
22

---

denote the Under review as a conference paper at ICLR 2022
Table 5: WideResNet CIFAR-100: Results of the comparison between IMP and pruning stable methods for the sparsity range between 90% and 99.5%. The columns are structured as follows: First the method is stated, where IMP+ denotes the unrestricted version of IMP. Secondly, time needed when compared to regular training of a dense model, e.g. LC needs 1.14 times as much runtime as regular training. The following columns are substructured as follows: Each column corresponds to one goal sparsity and each subcolumn denotes the Top-1 accuracy, the theoretical speedup and the actual sparsity reached. All results include standard deviations, where we omit zero or close to zero results. Missing values (indicated by -) correspond to cases where we were unable to obtain results the desired sparsity range, i.e., there did not exist a training configuration with average final sparsity within .25% interval around the goal sparsity and the closest one is too far away or belongs to another column.
90% %
93%
Method
Time
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
IMP
1.10x
77.75
+0.37
90.00
77.61 #0.07
11 40.2
93.00
IMP+
1.23x
77.72
+0.47
8
90.00
77.85 #0.37
11 #0.2
93.00
GMP
1.00x
75.84
+0.25
90.00
75.09 #0.47
14
93.00
GSM
2.08x
74.46
+0.35
6
90.02
74.34 #0.39
9
93.02
DPF
1.08x
76.74
+0.22
10
90.00
76.45 #0.22
10 +0.2
93.00
DNW
1.94x
77.73
+0.29
7
90.00
76.74 #0.05
7
93.00
LC
2.17x
74.45
+0.30
90.00
73.23 +0.67
60.01
93.00
STR
1.05x
73.06
+1.22
15
90.97
74.11 #0.21
13 #0.2
92.36 10.
CS
1.21x
73.50
40.47
89.82 +0.09
73.52 #0.21
10
92.96 40.03
DST
1.14x
18.81
+2.05
20 #0.4
90.31 +0.08
66.62 #1.68
23 +1.4
92.89 #0.11
95 %
98% %
Method
Time
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
IMP
1.10x
77.56
20.22
15 #0.2
95.00
75.93 #0.33
34 40.3
98.00
IMP+
1.23x
77.59
+0.34
#0.3
95.00
76.27 #0.23
35 40.5
98.00
GMP
1.00x
74.52
+0.19
19
95.00
66.25 #1.31
49
98.00
GSM
2.08x
73.47
+0.25
95.02
70.28 #0.95
23 40.1
98.02
DPF
1.08x
75.73
+0.28
16 #0.4
95.00
73.05
29
98.00
DNW
1.94x
75.69
#0.21
8
95.00
72.65 #0.28
17
98.01
LC
2.17x
72.41
+0.54
8 +0.2
95.00
70.06 #0.06
17 +0.5
98.00
STR
1.05x
70.66
+0.52
94.40 #0.02
62.24 #2.50
61 +0.5
97.85
CS
1.21x
72.81
+0.13
12
95.22
72.29 #0.34
28 40.5
97.99
DST
1.14x
69.33
+0.28
25 49.8
95.22 11.12
68.46
40
97.96
99 %
99.5%
Method
Time
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
IMP
1.10x
72.03
+0.52
+1.3
99.00
65.29 +0.87
130 +2.3
99.50
IMP+
1.23x
73.72
+0.15
71 +0.9
99.00
67.61 +1.38
140 +1.4
99.50
GMP
1.00x
36.33
49.76
100
99.00
16.19 27.59
200
99.50
GSM
2.08x
66.14
+0.32
32
99.02
59.64 #1.35
62 41.
99.52
DPF
1.08x
70.82
20.64
59 #0.4
99.00
60.23 #1.61
143 +3.8
99.50
DNW
1.94x
67.42 40.84
21 #0.8
99.02
62.10 #1.16
42
99.52
LC
2.17x
65.58 +0.69
33 40.3
99.00
59,60 #0.32
74 40.8.
99.50
STR
1.05x
17.32 15.67
142 #2.9
98.79
10.04 +0.88
271 +8.5
99.33 20.01
CS
1.21x
70.61 10.14
41 +0.3
98.75
68.27 #0.62
82 42.6
99.40
DST
1.14x
48.75
98.42
23

---

Under review as a conference paper at ICLR 2022
ResNet-56 CIFAR-10
Test accuracy
95%
Teat accuracy
83%
80%
GSM
DNW
GMP
L.C
7356
DPF
CS
STR
IMP
DST
TMP
70%6
92%
90%
9895
100%
Amounn of weights prined
Theoretical speedup
80x
GSM
DNW
IMP
60x
DST
Theureical speedup
40x
30x
10k
88%
90%
92%
94%
96%
98%
100%
Amount of weights praned
Figure 10: Test accuracy (above) and theoretical speedup (below) of IMP in comparison different pruning stable methods when training a ResNet-56 network on CIFAR-10. All methods were trained achieve sparsity levels of 90%, 93%, 95%, 98%, 99% and 99.5% with the exception of CS, STR and DST, where additional hyperparameter searches were necessary to obtain the curves shown here. Each datapoint corresponds the hyperparameter config with highest accuracy when considering .5% sparsity intervals between 88% and 100%. This holds similarly for the theoretical speedup, where points are selected by highest accuracy as well. The confidence bands indicate the min-max- deviation around the mean with respect to different initialization seeds.
24

---

Under review as a conference paper at ICLR 2022
WideResNet on CIFAR-100
Accurary increase by retraining
-
GMP
DST
60%
GSM
DNW
DPF
CS
50%
STR
LC
increcse atcaraty
40%6
30%
200
10%6
09
92%
96%
989
100%
of weights pruned
Pruning stability
80%
pruniis stabillyy
40%
-
GMP
DST
20%
GSM
DNW
DPF
CS
STR
LC
0%
88%6
900
92%
94%
100%
axnount of weights prnmed
Figure 11: Increase in accuracy after retraining (above) well as pruning stability (below) for WideResNet trained on C10 using different pruning stable methods. Each method was re- trained for 30 epochs using FT. Each datapoint corresponds the hyperparameter config with high- est accuracy directly after pruning when considering .5% sparsity intervals between 88% and 100%. The confidence bands indicate the min-ma min-max-o max-deviation around the mean with respect to different initialization seeds.
25

---

Under review as a conference paper at ICLR 2022
WideResNet CIFAR-100
Test accuracy
73%
24-
70%6
Teat ancuracy
63%
6056
53%
GMP
DNW
50%
GSM
CS
DPF
LC
439
STR
IMP
DST
TMP
4056
68%
92%
90%
989
10055
amount of weights prined
Theoretical speedup
80x
GMP
DNW
GSM
IMP
60xc
DST
Theureical speedup
40x
30x
10k
88%
90%
92%
94%
96%
98%
100%
Amouun of weights praned
Figure 12: Test accuracy (above) and theoretical speedup (below) of IMP in comparison different pruning stable methods when training a WideResNet network on CIFAR-100. All methods were trained achieve sparsity levels of 90%, 93%, 95%, 98%, 99% and 99.5% with the exception of CS, STR and DST, where additional hyperparameter searches were necessary to obtain the curves shown here. Each datapoint corresponds the hyperparameter config with highest accuracy when considering .5% sparsity intervals between 88% and 100%. This holds similarly for the theoretical speedup, where points are selected by highest accuracy as well. The confidence bands indicate the min-max-dee -ev--aittiiiiion around the mean with respect to different initialization seeds.
26

---

Under review as a conference paper at ICLR 2022
ResNet-56 on CIFAR-10 (weight decay 0.0001)
90.0% of weights pruned
98.0% of wrightss pramed
0.94
0.925
155t anC1599
0.93
Lte acCIERY
0.900
0.800
0.91
0.925
01.00
0.800
50
100
150
200
250
300
350
50
100
150
200
250
300
350
todal retroizing eportas
total reiraining epochts
ResNet-56 on CIFAR-10 10 (weight decay 0.0002)
90.0% of weights pruned
08.0% weights pruned
0.925
0bst secitaic
0.913
accuraty
0.900
0.92
0.875
0.850
0.825
0.90
0.800
30
100
130
200
250
350
50
100
150
200
250
300
350
total epadia
total
spocha
ResNet-56 CIFAR-10 (weight decay 0.0005)
90.0% of weights pruned
0% of weights pruned
0.925
tott acturaiy
0.92
0.875
SLR
Renda et al. (PT) CLR
0.91
Ronda et
0.825
total retroining epadia
total retraining epocha
Test accuracy achieved by IMP in relation to the total number of epochs required for training using either CLR or SI.R. Each row of the plots shows a different weight decay as indicated in the title. The pruned baselines were established by using the approach of |Renda al. (2020) using the respective weight decay and either in its original form, using LRW, or with a modified version using FT.
Figure 13:
27

---

Under review as a conference paper at ICLR 2022
ResNet-56 on CIFAR-10
Cyclic Learning Rate Restarting (CLR)
90%
80%
70%
tsrt accuracy
60%
50%
Global
Uniformn (Zhm & Capta, 2017)
40%
Uniformn+ (Gale al., 2019)
ERK (Evei et al., 2019)
30%
LAMP (Lee et al., 2020)
amount of weights praned
Cyclic Learning Rate Restarting (CLR)
90%
80%
test benunicy
70%
60%
50%
40%
30%
10F
theoretical speedup
Figure 14: Performance--s--spasiie (above) and tradeoffs for ResNet-56 CIFAR-10 for IMP in the One Shot setting with CLR the retrain- ing method. Each line corresponds to one pruning selection approach of interest. The model is retrained for 30 epochs after pruning.
(below)
28

---

Under review as a conference paper at ICLR 2022
WideResNet on CIFAR-100
Cyclic Learning Rate Restarting (CLR)
70%
60%
tsst ascrray
50%
40%
Global
Uniformn (Zhm & Capta, 2017)
Uniformn+ (Gale al., 2019)
30%
ERK (Evei et al., 2019)
LAMP (Lee et al., 2020)
amount of weights pruned
Cyclic Learning Rate Restarting (CLR)
70%
test acmuniyy
50%
40%
30%
10
10a
theoretical speedup
Figure 15: senformmncce-s.-sppriii (above) and porfommaee-s.thhoootiiillllll thereeticcal--speuup (below) tradeoffs for WideResNet CIFAR-100 for IMP in the One Shot setting with CLR the retraining Eethod.E ach line corresponds to one pruning selection approach of interest. The model is retrained for 30 epochs after pruning.
29


Reviews:
[{"review_id": "fw4ZhEv-_L", "reviewer": "Reviewer_WMwJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The clarity in writing throughout most of the paper helps the reader quickly assimilate the contents and understand the intent of the paper.\n- The level of detail, across the experiments as well as the figures, provided is of high quality and much appreciated.\n- Explanation of the different approaches to pruning, their background and clearly advocating for the missing link is commendable.\n\nWeaknesses\n- The first paragraph on the arguments against IMP (Pg. 2, bullet point 1), reads slightly incoherently, like a mix of multiple ideas. A more clear and point by point description of sub-optimal states, re-training epochs and others would be extremely helpful.\n- Additionally, \"Many proposed improvements...\" is followed by a single citation in the same bullet point 1. Adding multiple citations would be more appropriate with that statement.\n- Pg. 4 Paragraph of Pruning Approaches: The nomenclature of \"pruning approaches\" is slightly counterintuitive when discussing ideas related to how pruning methods limit the overall compression of different layers. An alternative heading would summarize the contents more appropriately.\n- While the intention of Section 3.1 is to highlight the low computational overhead of IMP in matching baseline performance, the intermediate outcome of SLR being better than FT traces the takeaways from Renda et. al (2020) and slightly weakens the contribution. Further, it isn't exactly clear what is the overhead when comparing the final set of results for IMP vs. SOTA pruning approaches. Comparison to a potentially weak baseline isn't quite enough and I encourage the authors to provide those results in a bid to strengthen their argument.\n- Some of the key takeaways from Section 3.2 are very similar to that of Tanak et al. (2020) and Lee et al.(2020). Specifically, that of layer-collapse and the SLR vs.FT performance comparison. Leaning on these outcomes, in text and across image captions, weakens the novelty of the content. I encourage the authors to emphasize on IMP-specific behavior and insights. \n- In Section 3.3, the last paragraph, the statement of IMP can be applied to already trained models while other methods in the experiment cannot isn't fair since the experiment uses pruning-during training approaches and not the general set of pruning methods.\n\nAfter Rebuttal\n\nI would like to thank the authors for their timely and pertinent discussions and revisions to the manuscript. \nAs I described previously the consistent references and experimental structure borrowed from existing work hinder the novelty of the work. \nThe authors have consistently clarified the difference in setup and mentioned that the exact outcomes differ from already existing work. \nWhile the implementation and proposed work are focused on resurrecting IMP as a valid SOTA baseline, more in-depth work in alternative settings (unseen in existing literature) would be one possible way to help further highlight the novelty aspect of IMP. \n", "summary_of_the_paper": "This work focuses on highlighting the strengths of Iterative Magnitude Pruning (IMP). Specifically, that it is capable of achieving strong performance when compared to more complex pruning approaches. The work explores the common arguments against IMP like, a) it reaches sub-optimal states since training doesn't compensate for sparse structures, b) it fails to identify optimal layer-wise pruning ratios and c) it is expensive, slow and non-competitive. The critical outcome shown is that IMP, with a global selection criterion and extremely small overhead, remains highly competitive with common state-of-the-art pruning approaches, both in sparsity, performance and theoretical speedup. ", "summary_of_the_review": "The proposed work does an extremely good job of explaining the landscape of pruning and setting up the missing link of IMP not being explored sufficiently. However, the experimental takeaways follow similar patterns to Tanaka et al.(2020) and Renda et al.(2020), which weaken the novelty of the work. The work sounds more like an extension of IMP into existing experimental frameworks and results than a purely novel instance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "euJX83T1hNi", "reviewer": "Reviewer_dL1d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n\n1. The paper revisits IMP, a basic yet important pruning approach. The empirical findings are promising: IMP can be trained on par with stable pruning approaches using reasonable training time under proper learning rate schemes.\n\n2. Extensive studies with detailed experimental setups are provided. Code is provided for reproducibility.\n\n\nWeakness:\n\n1. It is not explained why SLR is combined. I wonder if IMP can be improved when combined with other training tricks (e.g., CLR) w.r.t. the discussed aspects. And will there be any improvement if other baselines (e.g., Uniform/ERK/LAMP in Figure 2) are combined with SLR?\n\n2. The writing is not very focused, making it kind of hard to follow by the general audience, especially for Section 2. More technical details can be introduced for IMP and SLR, both of which are studied throughout the experiments.\n\n3. While the paper provides empirical studies for IMP, the technical contribution can be minor, as most components are existing methodologies.\n\n\nDetailed comments:\n\n- The title can be misleading? The empirical results show that only under proper learning rate schemes (SLR) can IMP achieve better performance with less training time. However, SLR is not a basic technique.\n\n- While the authors draw the conclusions based on the improved accuracy with SLR, the reasons for improvement are still unclear nor well explained. It would be more inspiring to explicitly study why previous IMP are sub-optimal, and how SLR can improve IMP.\n\n- Are the rest baselines in Figure 1-3 trained with the same learning rate scheme (FT or SLR)? Otherwise, it can be unfair to compare with IMP+SLR.\n\n- Reference formats are not formal and inconsistent.\n\n- It can be restrictive to study unstructured pruning as well as theoretical speed-up with FLOPs reduction in practice.", "summary_of_the_paper": "The paper studies a fundamental and important research approach in network pruning: iterative magnitude pruning (IMP). Previously IMP is criticized to be time-consuming, layer-independent and sub-optimal in performance. In this paper, extensive empirical studies are conducted to show that under proper learning rates, IMP can have close performance with more advanced pruning approaches, with little training time increased. ", "summary_of_the_review": "The empirical findings are promising. However, it is still not clear in what way SLR can improve IMP, and the reasons behind the improved accuracy / reduced training time.\n\n\n=== Post rebuttal ===\nThanks for the authors' response and paper revision, which addresses some of my concerns. While the paper introduces inspiring findings on how SLR (or CLR) help IMP, most components are from existing techniques. I choose to keep my score, and encourage the authors to provide more in-depth analysis behind the improvement by SLR, via either new methodologies or perspectives.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FnIPJtYTjlv", "reviewer": "Reviewer_fSGB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is motivated by a simple question: whether the basic IMP method is enough for weights pruning? Through the whole paper, the authors conducted lots of experiments to show that IMP with a good retraining is as good as many recently proposed pruning methods. The authors also show that using IMP we can also directly determine the layerwise sparsity through global ranking, instead of using more complex method to determine the sparsity for each layer.\n\nAlthough the authors did lots of experiments validating the claim, the experiments are kind of limited. All the experiments are in image classification, with larger size models. I am wondering how is the result on compact models like mobilenet. Besides CNNs, weights pruning are actually used for other architectures too, e.g., LSTMs and transformers. How are the comparison on those cases? I admit that this paper already conducted a comprehensive comparison, while it's hard to say IMP is enough for pruning with only a few models on Image classifications showing that. In addition, many new pruning algorithms are actually built for NLP and speech tasks.", "summary_of_the_paper": "This paper investigated several recently proposed pruning stable approaches and compared them with the basic iterative magnitude based pruning (IMP), and observe that IMP actually performs on-par or even better in the experiments. The authors investigated the retraining approaches, the computation cost, the importance of sparsity allocation (including layer-collapse) and compared IMP (+retraining) to different pruning methods.", "summary_of_the_review": "I like the author's view and the motivation to make a realizable and unified baseline for weights pruning. However, I think it's better to have more evidence on different models and tasks to validate this conclusion. If the proposed claim can be further validated, I think it will be a significant contribution to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an analysis of the well known method of Iterative Magnitude Pruning (IMP) for DNN compression. The problem tackled is undoubtably an important one, and IMP is likely one of the most known solutions for DNN compression. As such, there is no doubt that the paper is well motivated. In addition to the motivated task, the reviews indicate that the paper is well written and provides a thorough review of the related literature, making the paper easy to read and follow.
The main weakness of the paper seems to its novelty, as it seems that similar analyses have been done in the past. This issue was raised by the reviews and remained after the correspondence with the authors:

 WMeJ: “As I described previously the consistent references and experimental structure borrowed from existing work hinder the novelty of the work”, dL1d: “While the paper introduces inspiring findings on how SLR (or CLR) help IMP, most components are from existing techniques”.

Given the discussion and concerns related to the novelty of the paper, I feel that the paper requires too major of a revision to be accepted, either improving its core analysis, or presenting it in a better way that clearly distinguishes it from previous art.
Decision: Reject