How I Learned to Stop Worrying and Love Retraining
Authors: Max Zimmer
Christoph Spiegel
Sebastian Pokutta
Venue: ICLR 2023 poster (2023)

Abstract: Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing a significant amount of their performance after pruning and then recovering it in the subsequent retraining phase. Recent works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance of the learning rate schedule during the retraining phase and propose specific heuristics for choosing such a schedule for IMP (Han et al., 2015). We place these findings in the context of the results of Li et al. (2020) regarding the training of models within a fixed training budget and demonstrate that, consequently, the retraining phase can be massively shortened using a simple linear learning rate schedule. Improving on existing retraining approaches, we additionally propose a method to adaptively select the initial value of the linear schedule. Going a step further, we propose similarly imposing a budget on the initial dense training phase and show that the resulting simple and efficient method is capable of outperforming significantly more complex or heavily parameterized state-of-the-art approaches that attempt to sparsify the network during training. These findings not only advance our understanding of the retraining phase, but more broadly question the belief that one should aim to avoid the need for retraining and reduce the negative effects of ‘hard’ pruning by incorporating the sparsification process into the standard training.

Published as a conference paper at ICLR 2023
HoW I LEARNED TO STOP WORRYING AND LOVE RETRAINING
Max Zimmer', Christoph Spiegel' & Sebastian Pokutta'
1 Department for Al in Society, Science, and Technology. Zuse Institute Berlin, Germany 2 Institute of Mathematics, Technische Universität Berlin, Germany 2 Lmmeer, spiegel, pokutta| @ziib deib.de
ABSTRACT
Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing significant amount of their performance after pruning and then recovering it in the subsequent retraining phase. Recent works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance of the learning rate schedule during the retraining phase and propose specific heuristics for choosing such a schedule for IMP (Han et al., 2015). We place these findings the context of the results of Li et al. (2020) regarding the training of models within fixed training budget and demonstrate that, consequently, the retraining phase can be massively shortened using simple linear learning rate schedule. Improving on existing retraining approaches, we additionally propose method adaptively select the initial value of the linear schedule. Going a step further, we propose similarly imposing a budget on the initial dense training phase and show that the resulting simple and efficient method is capable of outperforming significantly more complex or heavily parameterized statc-of-the-art approaches that attempt to sparsify the network during training. These findings not only advance our understanding of the retraining phase, but more broadly question the belief that one should aim to avoid the need for retraining and reduce the negative effects of 'hard' pruning by incorporating the sparsification process into the standard training.
INTRODUCTION
Modern Neural Network architectures are commonly highly werrparamettrrzzd (Zhang et al., 2016), containing millions or even billions of parameters, resulting in both high memory requirements as well as computationally intensive and long training and inference times. It has been shown however (LeCun et al., 1989; Hassibi & Stork, 1993; Han et al., 2015; Gale et al., 2019; Lin et al., 2020; Blalock et al., 2020) that modern architectures can be compressed dramatically by pruning, i.e., removing redundant structures such as individual weights, entire neurons convolutional filters. The resulting sparse models require only a fraction of storage and floating-por point operations (FLOPs) for inference, while experiencing little to degradation in predictive power compared the dense model. Although it has been observed that pruning might have a regularizing effect and be beneficial the generalization capacities (Blalock et al., 2020), very heavily pruned model will normally be less performant than its dense (or moderately pruned) counterpart (Hoefler et al., 2021).
One approach to pruning consists of removing part of network's weights from the model architecture after a standard training process, seemingly losing most of its predictive performance, and then retraining to compensate for that pruning-induced loss. This can be done either One Shot, that is pruning and retraining only once, or the process of pruning and retraining can be repeated iteratively, Although dating back the early work of Janowsky (1989), this approach was most notably proposed by Han et al. (2015) in the form of ITERATIVE MAGNITUDE PRUNING (IMP). In its full iterative form, for example formulated by Renda et al. (2020), IMP can require the original train time several times over to produce a pruned network, resulting in hundreds of retraining epochs on top of the original training procedure and leading to its reputation for being computationally impractical (Liu et al., 2020; Ding et al., 2019; Hoefler et al., 2021; Lin et al., 2020; Wortsman et al., 2019). This as well as the belief that IMP achieves sub-optimal states (Carreira- Periinan & Idelbayev. 2018; Liu al., 2020) is one of the motivating factors behind methods that similarly start with initially dense

---

Published as a conference paper at ICLR 2023
model but incorporate the sparsification into the training. We refer to such dense-to-sparse methods as pruning- sraning-sttall (Bartoldson et al., 2020).
Motivated by recent results of Li et al. (2020) regarding the training of Neural Networks under constraints on the number of training iterations, we challenge these commonly held beliefs by rethinking the retraining phase of IMP within the context of Budgeted Training and demonstrate that it can be massively shortened by using a simple linearly decaying learning rate schedule. We further demonstrate the importance of the learning rate scheme during the retraining phase and improve upon the results of Renda et al. (2020) and Le & Hua (2021) by proposing a simple and efficient approach to also choose the initial value of the learning rate, a problem which has not been previously addressed in the context of pruning. We also propose likewise imposing budget on the initial dense training phase of IMP, turning into a method capable of efficiently producing sparse, trained networks without the need for a pretrained model by effectively leveraging cyclic linear learning rate schedule. The resulting method able to outperform significantly more complex and heavily parameterized state-of-the-ar approaches, that aim to reach pruning-stabiliin at the end of training by incorporating the sparsification into the training process, while using less computational resources.
Contributions.
The major contributions
follows:
1. We empirically find that the results of Li et al. (2020) regarding the Budgeted Training of Neural Networks apply to the retraining phase of IMP, providing further context for the results of Renda et al. (2020) and Le & Hua (2021). Building on this, we find that the runtime of IMP can be drastically shortened by using simple linear learning rate schedule with little to no degradation in model performance.
2. We propose novel way to choose the initial value of this linear schedule without the need to tune additional hyperparameters in the form of ADAPTIVE LINEAR LEARNING RATE RESTARTING (ALLR). Our approach takes the impact of pruning well as the overall retraining time into account, improving upon previously proposed retraining schedules on a variety of learning tasks.
3. By considering the initial dense training phase part of the same budgeted training scheme, we derive a simple yet effective method in the form of BUDGETED IMP (BIMP) that can outperform many pruming-son stable approaches given the same number of iterations to train network from scratch.
We believe that our findings not only advance the general understanding of the retraining phase, but more broadly question the belief that methods aiming for pruning-stabile stability are generally preferable over methods that rely on 'hard' pruning and retraining both in terms of the quality of the resulting networks and in terms of the speed at which they are obtained. We also hope that BIMP can serve as a modular and easily implemented baseline against which future approaches can be realistically compared.
Outline. Section 2 contains a summary of existing literature and network pruning approaches. also contains reinterpretation of some of these results in the context of Budgeted Training as well as a technical description of the methods we are proposing. In Section 3 we will experimentally analyze and verify the claims made in the preceding section. We conclude this paper with some relevant discussion in Section 4.
2 PRELIMINARIES AND METHODOLOGY
While the sparsification of Neural Networks includes a wide variety of approaches, we will focus on the analysis of Model Pruning, i.e., the removal of redundant structures Neural Network. We focus on performing unstructured pruning, that is the removal of individual weights, further also providing experiments for its structured counterpart, where entire groups of elements, such as convolutional filters, are removed. We will also focus on approaches that follow the dense-to-sparse paradigm, i.e., that start with a dense network and then either sparsify the network during training or after training, as opposed to methods that prune before training (e.g. Lee et al., 2019) or dynamic sparse training methods (c.g. Evci et al., 2020) where the networks are sparse throughout the entire training process. For full and detailed survey of pruning algorithms we refer the reader to Hoefler et al. (2021).
2

---

Published as a conference paper at ICLR 2023
Pruning-unstabe methods exemplified by ITERATIVE MAGNITUDE PRUNING (IMP) (Han et al., 2015). In its original form, it first employs standard network training. adding a common b regularization term on the objective, and then removes all weights from the network with magnitude below a certain threshold. The network at this point commonly loses some or even all of its learned predictive power, so it is then retrained for a fixed number of epochs. This prune-retrain eycle is usually repeated a number of times; the threshold at every pruning step is determined as the appropriate percentile such that, at the end of a given number of iterations, a desired target sparsity is met. Renda et al. (2020) suggested the following complete approach: train network for T epochs and then iteratively prune 20% percent of the remaining weights and retrain for Trt- = T epochs until the desired sparsity is reached. For goal sparsity of 98% and T = 200 original training epochs, the algorithm would therefore require 18 prune-retrain-eycles for a massive 3800 total retrain epochs.
2.1 RETHINKING RETRAINING AS BUDGETED TRAINING
There has been some recent interest in the learning rate schedule used during retraining. The original approach by Han et al. (2015) is commonly referred to as FINE TUNING (FT): suppose we train for T epochs using the learning rate schedule (M))sT and retrain for Tve epochs per grune-retrain-cycke then FT retrains the pruned net- work using a constant learning rate of fr, i.e., the last learning rate used during the original training. Renda et al. (2020) note that the learning rate schedule during retraining can have dramatic impact on the predictive per- formance of the pruned network and propose LEARNING RATE REWINDING (LRW), where one retrains the pruned network for Tve epochs using the last Trr learning rates TT-TT UT during each cycle. L. & Hua (2021) further improved upon these results by proposing SCALED LEARNING RATE RESTARTING (SLR), where the pruned network is retrained using proportionally identical sched- ule, i.e., by compressing (7e Jegr into the retraining time frame of Tr epochs with short warm-up phase. They also introduced CYCLIC LEARNING RATE RESTARTING (CLR) based on the 1-cycle learning rate schedule of Smith & Topin (2017), where the original schedule (commonly stepped one) is replaced with cosine based one starting at the same initial learning rate 77, likewise including a short warm-up phase. Figure depicts the aforementioned schedules for a retraining budget of 60 epochs.
0.10
FT
0.08
LRRN
SL.
CLR
L.R
0.01
0.11
20
retralising spoxlas
The different learning rate schedules for IMP when retraining for 60 epochs, assuming a stepped learning rate schedule during an initial training lasting for 200 epochs.
Figure 1:
LRW was proposed as a variant of WEIGHT REWINDING (WR) (Frankle et al., 2019), suggesting that its success due to some connection to the Lottery Ticket Hypothesis. Le & Hua (2021) already gave a more grounded motivation when introducing SLR by noting that its main feature is the "usage of large learning rates". ! By proposing CLR and motivating it through the 1-cycle learning rate schedule of Smith& & Topin (2017), they also already demonstrated that there is no particular significance to basing the learning rate schedule on the one used during the original training.
We think that the results of Li et al. (2020) regarding the training of Neural Networks within fixed iteration budget (Budgeted Training) provide some relevant further context for the varying success achieved by these methods as well as indications on how one can further improve upon them, in particular when retraining is assumed to be significantly shorter than the original training, that is when Tne < T. Li et al. (2020) study training when resource budget is given in the form of fixed number of epochs or iterations that the network will be trained for, instead of following the common assumption that training is executed until asymptotic convergence is achieved to some
'Note that large initial and then (often exponentially) decaying learning rate has become the standard practice for regular training (Leclerc & Madry, 2020). The conventional approach explaining the success of such schedules from an optimization perspective is that an initially large learning rate accelerates training and avoids local minima, while the gradual decay helps to converge to an optimum without oscillating around it. However, there are also indications that the usage of large learning rates and the separation of training into a large- and small-step regime help from generalization perspective (Jastrzębski et al., 2017; Li et al., 2019; You al., 2019; Leclere & Madry, 2020).
3

---

Published as a conference paper at ICLR 2023
satisfactory degree. Specifically, they empirically determine what learning rate schedules are best suited to achieve the highest possible performance within a given budget. The two major takeaways from their results are follow:
possible
1. Compressing any given learning rate schedule to fit within specific budget significantly outperforms simply truncating it once the budget is reached. Li al. (2020) refer to this compression as of & Hua (2021) 1) that SLR outperforms LRW, since SLR is simply the BAC of the original training schedule while LRW ís a truncated version of it (albeit truncated 'from the back' instead of the front).
CONVERSION (BAC). This clearly aligns with the findings
2. Certain learning rate schedules are more suited for a wider variety of budgets than others. In particular, their results indicate that linear schedule performs best when tight budget is given, closely followed by a cosine based approach. This provides an explanation for why CLR outperforms SLR when the original learning rate schedule follows more traditional recommendations.
Put succinetly, the empirical results of Li et al. (2020) regarding the learning rate schedule in a budgeted training context seem to closely resemble the development and improvement of retraining schedules in the context of pruning. Hence, we claim that retraining should first and foremost be considered under the aspect of Budgeted Training and that lessons derived in the latter setting are generally applicable in this context. Motivated by the findings of Li et al. (2020), we therefore propose LINEAR LEARNING RATE RESTARTING (LLR) leveraging a linear learning rate schedule during retraining: LLR linearly decays the learning rate during each retrain cycle from an initial value of 1/ to zero after a short warm-up phase. This effectively results in cyclic learning rate schedule when pruning and retraining in the iterative setting, which has previously been found to help generalization (Smith, 2017). Going one step further, we also propose dynamically adapting the initial value of the retraining schedule by relating it not just to the initial learning rate during the original training but also to the impact of the previous pruning step, resulting ADAPTIVE LINEAR LEARNING RATE RESTARTING (ALLR). While previous works have focused on the actual schedule of the learning rate during retraining. the initial value has only implicitly been dealt with. FT chooses the last learning rate value iTr, which is typically the smallest. On the other hand, SLR and CLR rely on the initial value corresponding to the maximum value v/i of the original schedule, which the authors attribute the success of their methods to. The initial value of LRW is implicitly chosen in proportion to the retraining time by truncating the original schedule from the back.
Existing works have shown that to find minima that generalize well, the learning rate should exhibit a large-step and a small-step retraining regime (Jastrzębski et al., 2017; Li et al., 2019; You et al., 2019; Leclerc & Madry, 2020). When choosing the initial value of the retraining schedule, the two characteristics of prune-retrain cycle have to be taken into account: its length and the impact of pruning. Given tight retraining budget it might occur that large initial steps cannot be compensated adequately, while a too small learning rate (possibly over long retraining) period might be insufficient to recover large pruning-inducee stepsize must therefore address the following question: how much of an increase in loss do we have to compensate for and do we have sufficient time to properly perform both large-step and small-step learning rate regime? To that end, ALLR discounts the initial value 7/2 by factor d E [0. 1] to account for both the available retraining time (similar to LRW, where the magnitude of the initial learning rate naturally depends on Trt) and the performance drop induced by pruning. Since measuring the decrease in train accuracy would require an additional evaluation epoch and is thus undesirable (cf. Appendix C.I for an ablation study). ALLR achieves this goal by first measuring the relative Lg-norm change in the weights due to pruning, that is after pruning a s E (o. 1] fraction of the remaining weights, we compute the normalized distance between the weight vector W and its pruned version WP in the form of
performance degradation. An adaptive way of choosing the initial
IW - WPll2 ||||2 vs
di
E 1].
where normalization by vs ensures that di can actually attain the full range of values in [0, 1J. We then determine d2 = Tre/T to account for the length of the retrain phase and choose d/ as the initial learning rate for ALLR where d max d1, dz). This approach effectively interpolates between the recommendations of Renda et al. (2020) and Le & Hua (2021) based on a computationally cheap proxy. Appendix C.1 contains several ablation studies to justify our design choices.
4

---

Published as a conference paper at ICLR 2023
In Section 3.1, we will verify our claims by empirically comparing the retraining schedules. namely FT. LRW, SLR, CLR, LLR and ALLR, against one another as well as against tuned versions of their underlying (constant, stepped, cosine or linear) schedules. We then study in Section 3.2 to what degree the retraining phase of IMP can be shortened when leveraging the proposed schedules.
2.2 PRUNING-STABILITY: TRYING TO AVOID RETRAINING
muning-stabble algorithms are defined by their attempt to find well-performing pruned model from an initially dense one during the training procedure so that the ultimate 'hard' pruning step results in almost no drop in accuracy and the retraining phase becomes superfluous. They do so by inducing strong implicit bias during some otherwise standard training setup, either by gradual pruning, i.c., extending the pruning mask dynamically, or by employing regularization- and constraint- toptimiiiioooiiiitiiion techniques to learn an almost sparse structure throughout training. Many methods also rely on some kind of 'soft' pruning for this, e.g., by zeroing out weights or strongly pushing them towards zero, but not fully removing them from the network architecture during training.
Let us briefly summarize a variety of methods that have been proposed in this category over the last couple of years: LC CCarreira-Perpinne employ modification of weight decay and force the weights with the smallest score more rapidly towards zero, where k is the number of parameters that will eventually be pruned and the score is the parameter magnitude or its product with the loss gradient. Similarly, DNW (Wortsman et al., 2019) zeroes out the smallest k weights in the forward pass while still using dense gradient. CS (Savarese et al., 2020), STR (Kusupati et al., 2020) and DST (Liu et al., 2020) all rely on the creation of additional trainable threshold parameters, which are applied sparsify the model while being regularly trained alongside the usual weights. Here, the training objectives are modified via penalty terms to control the sparsification. GMP (Zhu & Gupta, 2017; Gale et al., 2019) follows a tunable pruning schedule which sparsifies the network throughout training by dynamically extending and updating a pruning mask. Finally, based on this idea, DPF (Lin al., 2020) maintains pruning mask which is extended using the pruning schedule of Zhu & Gupta (2017), but allows for error compensation by modifying the update rule to use the (stochastic) gradient of the pruned model while updating the dense parameters.
& Idelbayev, 2018) and GSM (Ding et al., 2019) both
The two most commonly claimed advantages of following:
pruning-stable methods compared to IMP are the
1. They result in a pruned model faster when training from scratch since they avoid the expensive iterative prune-retrain cycles. Ding et al. (2019) for example advertise that is there is "no need for time consuming retraining", Liu et al. (2020) try to "avoid the expensive pruning and fine-tuning iterations", Hoefler al. (2021) state that sparsifying during training "is usually cheaper than the train-then thenn-parssffy schedule", Lin et al. (2020) argue that IMP is "computationally expensive" and "outperformed by algorithms that explore different sparsity masks instead of single one", Wortsman et al. (2019) try to "train sparse Neural Network without retraining or fine-tuning" and Frankle Carbin (2018) (in the context of the Lottery Ticket Hypothesis) state that "iterative pruning is computationally intensive, requiring training a network 15 or more times consecutively".
2. They produce preferable results either because they avoid 'hard" pruning due to the particular implicit bias they employ. Liu et al. (2020) for example state that 'hard' pruning methods suffer from a "failure to properly recover the pruned weights" and Carreira-Perpinaae & Idelbayev (2018) argue that learning the pruning set throughout training "helps find a better subset and hence prune more weights with no or little loss degradation".
While many of the previously listed methods perform well and achieve sfate-of-the arr results, so far little empirical evidence has been given to support that the claimed advantages of pruning-stability have actually been achieved. To verify this, we propose BUDGETED IMP (BIMP), where the same lessons we previously derived from Budgeted Training for the retraining phase of IMP are applied to the initial training of the network. More specifically, given budget of T epochs, we simply train network for some Ta < T epochs using a linearly decaying learning rate schedule and then apply IMP with the proposed schedules the output for the remaining 7 To epochs.
5

---

Published as a conference paper at ICLR 2023
Table 1: ResNet-50 on ImageNet: Performance of the different learning rate translation schemes for One Shot IMP for target sparsities of 70%, 80% and 90% and retrain times of 2.22% (2 epochs), 5.55% (5 epochs) and 11.11 1.11% % (10 epochs) of the initial training budget. The first, second, and third best values are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 70%
Model spursity 80%
Model sparsity 90%
Budget:
2,225
5.55%
11.11%
2.22%
5.55%
11.11%
2.22%
5.55%
11.11%
FT
73.51 +0.04
73.98 +0.04
74.44 +0.11
70.45
+0.20
71.81 +0.11
72.68 +0.07
56.75 =0.01
61.60 +0.30
64.61 +0.21
LRW
73.50 10.04
73.99
74.45 10.11
70.45
71.82 50.12
72.67
56.75 .0.01
10.23
SLR
70.93 20.01
72.58 20.03
73.69 #0.11
70.48 20.04
72.37 =0.02
73.44 20.18
67.19
20.23
69.45 20.01
70.80 #0,00
CLR
72.22 10.09
73.58 10.08
74.49 10.04
71.96 10.09
73.30 10,08
74.24 20.05
68.72 20.06
70.60 20.15
71.51 40.13
L.R (ours)
72.39 20.13
73.65 +0.05
74.34 #0.02
72.07 #0.09
73.41 =0.05
#0.10
68.90 =0,05
71.53
ALLR (ours)
73.69 20.03
74.37 20.05
74.89 +0.04
72.96 =0.15
74.02
=0.08
74.71 20.04
69,56 =0.07
71.19 20.01
71.99 20.07
The resulting method is capable of obtaining a pruned model from a random initialization within any given budget T while still maintaining all of the key characteristics of IMP, most notably the fact that (1) we 'hard' prune and do not allow weights to recover in subsequent steps, (2) we do not impose any particular additional implicit bias besides common weight decay term during either training or retraining, and (3) we follow a prescribed static training schedule with the exception of adapting the initial learning rate to the impact of the last pruning in the case of ALLR. This clearly delineates BIMP from the previously listed methods and allows us to compare them on equal terms by giving all methods the same budget of total of T epochs, independent of whether they are spent on 'normal' training or retraining. In Section 3.3 we thoroughly compare our proposed approach to previously listed pruning-stable methods in fair setting. We remark that the implicit biases of many pruning-stable approaches can result in a substantial computational overhead that we are deliberately ignoring here by comparing methods on a per-epoch basis and therefore giving these throughput of
methods the advantage in the comparison. However, we include the images-per-seonn the individual algorithms which highlights that BIMP is among the most efficient approaches.
Finally, let us remark that there has been significant amount of attention on how to select the specific weights to be pruned. Ranking weights for pruning based on the magnitude of their current values has established itself as the approach of choice (Lee et al., 2019), where specific criteria have been proposed that take the particular network architecture into consideration (Zhu & Gupta, 2017; Gale et al., 2019; Evei et al., 2020; Lee et al., 2020). We have verified some of these results in Appendix C.2 and will stick to the simple global selection criterion used by Han et al. (2015) for BIMP.
EXPERIMENTAL RESULTS
Let us outline the general methodological approach to computational experiments in this section, including datasets, architectures and metrics. Experiment-speciin details are found in the respective subsections. We note that, given the surge of interest in pruming, Blalock et al. (2020) proposed experimental guidelines in the hope of standardizing the experimental setup. We aim follow these guidelines whenever possible. All experiments performed throughout this computational study are based on the PyTorch framework (Paszke et al., 2019), using the original code of the methods whenever available. All results and metrics were logged and analyzed using Weights & Biases (Biewald, 2020). We have made our code and general setup available at for the sake of reproducibility.
We perform extensive experiments on image recognition datasets such as ImageNet (Russakovsky et al., 2015), CIFAR-10/100 (Krizhevsky et al., 2009), the semantic segmentation tasks COCO (Lin et al., 2014) and CityScapes (Cordts et al., 2016) as well as neural machine translation (NMT) on WMT16 (Bojar et al., 2016). In particular, we employed ResNets (He et al., 2015), Wide ResNets (WRN) (Zagoruyko& & Komodakis, 2016), VGG (Simonyan & Zisserman, 2014), the transformer- based Max (Tu et al., 2022) architecture, as well as PSPNer (Zhao et al., 2017) and DeepLabV3 (Chen et al., 2017) in the case of SityScapes and COCO, respectively. For NMT, we used a T5 transformer (Raffel et al., 2020) available through HuggingFace (Wolf et al., 2020). Exact parameters can be found in Appendix A, where we also define what can be considered a 'standard' training setup for each setting that we rely on whenever not otherwise specified. The focus of our analysis will
6

---

Published as a conference paper at ICLR 2023
ResNer 50, 710% Sparsit
RoxNect 30, 90% Sparsity
ResNee 50, 90%% Spansity
-1.5%
bet acereay drecties
2.8
+4.7%
-2.0%%
.6.7%
LaW
LINK
LAW Stutt
4.83
Stut
CLR
-4.0%
SL.I
CCR
+677%
CLR
LLR (oxm)
LLB (cXD)
7.075
CLR (oaa0))
5.88
-0.0%
-8075
20
30
toral corritatug
19tel rnttaiiding spoda
total
Figure 2: ResNet-50 on ImageNet: Performance of different retraining schedules compared to the dense model shown over the total number of epochs used for retraining including both One Shot and iterative magnitude pruning. Results are averaged over two seeds with max- max-min-bande indicated and the plots depict sparsity 70%, 80% and 90% from left right.
be the tradeoff between the model sparsity and the final test performance, being the accuracy in the case of image classification, the mloU (mean intersection over union) for segmentation, or the BLEU score (Post, 2018) for NMT. a secondary measure, we will also consider the theoretical speedup (Blalock et al., 2020) induced by the sparsity, see Appendix A for full details. We use a validation set of 10% of the training data for hyperparameter selection.
3.1 LEARNING RATE SCHEDULES DURING RETRAINING
Table contains part of the results regarding the comparison between FT, LRW, SLR, CLR and our proposed approaches LLR and ALLR for ImageNet in the One Shot setting. First of all, find that retraining after pruning is in fact a Budgeted Training scenario, as the insights for normal dense training (Li et al., 2020) transfer comparing the translated schedules to versions of constant, stepped exponential, cosine and linear learning rate schedules, where the initial learning rate after pruning was tuned using grid search (ef. Appendix B.1, also containing additional results, exact parameter grids and the results for other datasets). In general, linear and cosine based schedules clearly outperform the constant and stepped ones, with slight advantage of LLR over CLR.
the retraining case. This is further observable when
However, for short retraining times and for the medium sparsity range, the fixed restarting schedules CLR and LLR fail to yield results competitive to FT and LRW, since too large initial learning rate is detrimental given a restricted retraining budget. ALLR is able to consistently improve upon previous approaches, which becomes especially noticeable in the small retraining budget regime, where for larger budgets the approaches begin to converge. We think that ALLR is a suitable drop-in replacement when performing retraining. We similarly observe the strength of ALLR in Figure/ depicting the highest achievable test accuracy for each number of total retraining epochs, including both the One Shot as well iterative pruning case, Appendix B. includes the full results on different tasks and datsets, longer retraining budgets as well as the structured pruning setting, where we remove convolutional filters based on their respective norm (Li et al., 2016).
3.2 BUDGETING THE RETRAINING PHASE
In this part we will treat the number of retrain epochs per prume-retrain cycle Tre well as the total amount of such cycles J as tunable hyperparameters for IMP and try to determine the tradeoff between the predictive performance of the final pruned network and the total number of retrained epochs J Tr.. As baseline performance for pruned network, will use the approach suggested by Renda et al. (2020) as it serves as a good benchmark for the current potential IMP.
In Figure 3 we present the envelope of the results for ResNet-56 trained on CIFAR-10 with target sparsities of 90%, 95%, and 98%, respectively. The parameters for the retraining phase were optimized using grid search over T,, E { (10, 15, 20. 60) and J E 21, ... 6) using ALLR. We find that IMP is capable of achieving what has previously been considered its full potential with significantly less than the total number of retraining epochs usually budgeted for its iterative form.

---

Published as a conference paper at ICLR 2023
Table 2: ResNet-56 on CIFAR-IO (above) and ResNet-50 on ImageNet (below): Comparison between BIMP and pruning-stable methods when training for goal sparsity levels of 90%, 95%, 99% (CIFAR- 10) and 70%, 80%, 90% (ImageNet), denoted in the main columns. Each subcolumn denotes the Top-1 accuracy, the theoretical speedup and the actual sparsity achieved by the method. Further, we denote the aaaggs-per--ecoon
throughput during training, i.e., a higher number indicates faster method. All results are averaged over multiple seeds and include standard deviations. The first, second, and third best values are highlighted.
CIFAR-10
Model sparsity 90%
Model spuritty 95%
Model spurityy 99%
Method
img/s
Accuracy
Speedup
Sparsity
Accuracy
Speodup
Spursity
Accuracy
Specdup
Sparsity
BIMP (ours)
3638
93.35 =0.13
7-+112
90:00
=0.32
+1.88
00 #0:00
87.17
99:00
GMP
3536
92.84 =0.42
10 20,0
90.00 20,00
92.12 20.17
20 +1100
95.00 20.00
86. #0:04
77 +0.0
99.00 20,00
GSM
3251
91.27 =0.60
11 +2.5
90.00
90.07 +1.67
+5.0
95.00 +0.00)
83.00
77.1889
99.00
DPF
3560
93.32 s0.11
7.000
90.00 10.00
92.68 a0.14
12
95.00) +0.00
86.76 10.33
63 43.7
99:00 s0.00
DNW
3335
91.81 +1.83
6.20.8
90.00 40,00
91.95 20.06
7 7,0.3
95.09 +0.00
83.67 20:24
15.1
99. 20,00
LC
3467
90.51 =0.16
5:11.11
90.00
89.16
95.00
81.63
=1.5
99.00
STR
2864
89:25 +1:23
80.88
90.15 +0.76
89.77 +1.75
+10.3
95.11 +0.28
+0.94
159 +31.9
99. +0.02
CS
2725
91.87 =0.30
13 20.3
90.52 20.76
91.36 20.23
21 12.9
95.38 10.19
86. 20.92
69 =7.1
98. 20.02
DST
1972
92.41
=0.28
10 20.7
89.55
89.17
+0.0
94.42 #0.00)
86.99
63
98.36
ImageNet
Model sparsity 70%
Model sparsity 80%
Model sparsity 90%
Method
imgls
Accuracy
Speclup
Spanity
Accuracy
Speodup
Sparsity
Accuracy
Specdupp
Spansity
BIMP (ours)
1454
75.62 =0.02
10.0
70.00 20.00
75.08 20.16
3 30.0
80:00 10.00
73.53 20:05
10,0
90.00 20.00
GMP
1425
74.62 =0.8
+0.00
#0.17
40.0
80.00 #0.00
72.80
7
GSM
1349
73.69 -0.70
2 +0.11
70:00 +0.00
72.75
+0.62
4+0.3
80.00 +0.00
70.08
+0.94
9-018
90:00 +0.00
DPF
1456
75.59 20.07
10.0
70.00 10,00
75.30
:0.02
310.0
80.00 10,00
74.05 10.05
6 6:000
90.00 20.00
DNW
75.60 =0.01
20.0
=0.01
30.0
$0.00 #0.00
5
L.C.
1436
75.03 =0.20
20.00
70.00 +0.00
73.87
+0.62
3+0.0
80.00 +0.00
67.57 +2.71
5+000
90.00 +0.00
STR
1396
70.66
3000
75.34 +0.01
70.70 20.13
40.0
80.93 #0.101
70.13
800,
90.00
DST
1219
74.63
20
4.401
70.00 +0.00
73.16
6:001
80.00 +0.00
71.35 +0.09
+0.4
90.00 +0.00
More concretely, for all three levels of sparsity, 90%, 95%, and 98%, IMP meets the baseline laid out by Renda et al. (2020) after only around 100 epochs of retraining, instead of requiring the 2000, 2800, and 3600 epochs used establish that baseline, respectively. In fact, given the superiority of linear learning rate schedule over a more commonly used stepped one in the case of budgeted training (Li et al., 2020), ALLR clearly continues and exceeds the stepped learning rate schedule baseline. Additional results and exact parameter grids are included in Appendix B.2.
Given that we have just established that the retraining phase of IMP takes well to enforcing budget when us- ing an appropriate learning rate schedule and that Li et al. (2020) already established that 'normal' training can be significantly shortened through a linear learning rate sched- ule, it is reasonable assume that the same holds for the original training phase without strongly impacting the pruning and retraining part and therefore the ultimate prod- uct of IMP.
Menuray devialin
0.07%
40.555
-1.005
-1.5%
Baseline (Reuda al.
Sparnity 90,0%
Spurxity 96,0%
Spursity 98 98.0%
To verify this, we trained ResNet-56 on CIFAR-10 using linearly decaying learning rate schedule from between 5% up to 100% of 200 epochs, which we consider the 'full' training. and then apply IMP with ALLR on the resulting network both One Shot and iteratively for target sparsities of 90%, 95%, and 98%. Figure 7 in the appendix shows the results for the iterative setting, where we retrain for three cycles of 15 epochs each.
-2.595
100
150
250
300
tocal retralning epocla
Figure 3: ResNet-56 on CIFAR-I0: En- velope of the performance of IMP us- ing ALLR compared to the baseline of Renda et al. (2020) shown over the total number of epochs used for retraining.
We can see that IMP takes well to budgeting the initial training period, both in the One Shot and in the iterative setting, with the target sparsity seemingly having little influence on how much the initial training can be compressed.
8

---

Published as a conference paper at ICLR 2023
3.3 THE EFFICACY OF PRUNING-STABILITY
We conclude by comparing the performance of BIMP to pruning-stable approaches. To that end, we train models on ImageNet, CIFAR-100 and CIFAR-10 and give all methods an equal budget of 90 epochs (200 for CIFAR) to derive a pruned model from a randomly initialized one. For BIMP, we employ ALLR and treat the initial training length To as well as the number of prune-retrain-cyclle as hyperparameters, ensuring that the overall epoch budget I is not exceeded. The hyperparameters of each of the pruning-stane -table methods were likewise tuned using manually defined grid searches, resorting to the recommendations of the original publications whenever possible, see Appendix B.1. For GMP, GSM, DPF, DNW, and LC we give the methods prespecified sparsity levels, the same as given to BIMP. Tuning the remaining methods in way that allows for a fair comparison however is significantly more difficult, since none of them allow to clearly specify a desired level of sparsity but instead require tuning additional hyperparameters as part of the grid search. Despite our best efforts, we were only able to cover part of the desired sparsity range using STR and DST. For CS in the case of TetgeNet, we were unable to tune the hyperparameters. In addition, methods can have some variance in the level of sparsity achieved even for fixed hyperparameters, so we list the standard deviation of the final sparsity with respect to the random seed initialization. We note that in the original works, LC and GSM were applied to pretrained models. To allow fair comparison, we applied LC and GSM to both randomly initialized as well as pretrained models and chose the best results for each sparsity, giving them a larger budget than the others. Further, we noticed that some pruning-stable methods can profit from retraining. For CIFAR-10 10 and CIFAR-100, we hence retrain all methods excluding BIMP for 30 epochs using FT and use the accuracy reached after retraining when it exceeds the original one. All results are averaged over multiple seeds with standard deviation indicated.
noticed that each of these
Table 2 reports the final test performance, theoretical speedup, and actually achieved sparsity of all methods for CIFAR-10 and ImageNet, where we defer full results to Appendix B.4. The results show that BIMP is able to outperform many of the pruning-staile stable methods considered here. For ImageNet, DNW consistently performs on par or better than BIMP, albeit at the price of needing roughly twice as long for training, cf. the images-per-second throughput. Surprisingly, despite broad hyperparameter grid search, most methods seem to be in disadvantage compared to BIMP, with DPF being a both efficient and strong competitor. BIMP obtains these results within the same number of overall training epochs and we are ignoring the computational overhead of some of the more involved methods. We note that the authors of STR report better results on ImageNet, which we unfortunately were unable to replicate in our experimental setting (cf. Appendix B. for the exact hyperparameter grid).
In Appendix C.3, we have included an ablation study where we compare BIMP to several modifica- tions of GMP not previously suggested in the literature, since GMP is the closest in design to BIMP out of all pruning-stable methods considered here. Most notably this includes variants of GMP with both global and cyclical linear learning rate schedule as well as 'hard' pruning variant.
4 DISCUSSION AND OUTLOOK
The learning rate is often considered to be the single most important hyperparameter in Deep Learning which nevertheless still remains poorly understood, certainly from a theoretical perspective but also still from an empirical one. Our work therefore provides important building block in which we established that, counter to the often explicitly stated belief that IMP is inefficient, many significantly more complex and sometimes strenuously motivated methods are outperformed by perhaps the most basic of approaches when proper care is taken of the learning rate.
Despite providing strong retraining alternative with ALLR, we emphasize that the main goal of this work is not to suggest yet another acronym and claim that it is the be-all and end-all of network pruning, but to instead (a) hopefully focus the efforts of the community on understanding more basic questions before suggesting convoluted novel methods and (b) to emphasize that IMP can serve as a strong, easily implemented, and modular baseline. We think the modularity here is of particular importance, as individual aspects can easily be exchanged or modified, e.g., when, what, how, and how much to prune or how to retrain, in order to formulate rigorous ablation studies.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY
Reproducibility is of utmost importance for any comparative computational study such as this. All ex- periments were based on the PyTorch framework and use publicly available datasets. The implementa- tion of the ResNet-56 and ResNet-18 network architecture is based ctthub.com/JJGO///ee and github. whtaaailooooomyaakpptoorroa
the WideResNet network architecture is based --ferarr 10, respectively, the implementation of wattuu..ooo/mmiiettooooooo wwww-eeeeee..ytooree the Nom/jaeho-leellayrr--aa
implementation of the VGG-16 network architecture is based on sparsity and the implementation of the Resnet-50 network architecture is taken from PyTorch. Regarding the pruning methods, the code was taken from the respective publications whenever possible. Regarding the different variants of magnitude pruning such ERK or UNIFORM+, closely followed the implementation of Lee et al. (2020) available at adaptive- sparrsstspastt...... For metrics such as the theoretical speedup, we relied on the implementation in the ShrinkBench- www.eerorrrrrokkkkkk of Blalock et al. (2020), see Satpaubbcoomm//G/////nnnoooeeeee our code and general setup available at github.com////B-1111BB for the sake of reproducibility.
We have made
ACKNOWLEDGEMENTS
This research was partially supported by the DFG Cluster of Excellence MATH+ (EXC-2046/1, project 390685689) funded by the Deutsche Forschungsgemeinschaft (DFG). We would like to thank Berkant Turan for his support in conducting the NMT experiment using the HuggingFace library.
REFERENCES
Brian Bartoldson, Ari Morcos, Adrian Barbu, and Gordon Erlebacher. The generalization-stabill tradeoff in neural network pruning- In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20852-20864. Curran Associates, Inc., 2020. URL httptttt//////ceeeeime nwwrips...pp.... cc/ http://///./.eeeeee0eeeee.eee...
paper.pdÉ.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: www. wandb. com/. Software cofttarrevvaree available from wandb.com.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Dhillon, Papailiopoolos, and V. Sze (eds.), Proceeding of Machine Learning and Systems, volume 2, 129-146, 2020. URL https: http:://p//ccei//eeeeoeoo mlsys.
org/papes 20/// ftto0/iile/.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz. Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Mau Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http:/ www. web. . org/ antho ogy /W/M/////// /W1 2301.
Miguel A Carreira-Perpiin Perpenna and Yerlan Idelbayev. Learning- compressio algorithms for neural pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. June 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
10

---

Published as a conference paper at ICLR 2023
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 1rXiv:190.044800, July 2019.
Xiaohan Ding, Guiguang Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, and Ji Liu. Global sparse momentum sgd for pruning very deep neural networks. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' d'Alché-Bue Systems, volume 32. Curran Associates, Inc., 2019. URL ht /ttetts:::/ppcceeeeeeees pttp///rrceeeeeeeeeoee/oe/o neurips .dd..
Fox, and Garnett (eds.), Advances in Neural Information Processing
htc/paper/201 / file/ F341 1133485444400 B5544a55555771111c@556 (10778114114411
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 19943-2952 PMLR, 13-18 Jul 2020. URL https: proceedies.e.s...eoooee.......se mlr.
press/v119/ecctl
html.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.0160 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 09574, 2019.
Song Han. Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL https: htttps://proceett http://poooeeeengeeee/ee neeteeedddnnnnnnnreepeeeeee neurips (eee3eeed3d22bceee2222222222
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume 5. Morgan-Kauffann 1993, URL https: /tttttts///prrceeiinng www.unnggn.neuirpeee neurips............. cc/ paper/1992 file//ill//eeee hetiee//33eeeee 30tee3333334338333 6ab3 (ewaaab3e224443ba5e5e5
Paper.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden. and Alexandra Peste. Sparsity in deep learning: Pruming and growth for efficient inference and training in neural networks. arXiv preprint 102.05544 January 2021.
Steven A. Janowsky. Pruning versus clipping in neural networks. Phys. Rev. A, 39:6600-6603,, Jun 1989. doi: 10. 00.1103/PhyyRRevA39..6660
Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey Three factors influencing minima in sgd. arXiv preprint 1rX1v:1711.04623, November 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 4rXiv:1412.6988. 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateck Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 1 119 Proceedings of Machine Learning Research, pp. 5544 55555. PMLR, 13-18 Jul 2020. URL https: prwceedooeeeeeeooooooooocc ngs. . ml r, www.ss/sssvve...... 19/1199//kao 9/wssp/tt 120a html.
11

---

Published as a conference paper at ICLR 2023
Duong Hoang Le and Binh-Son Hua. Network pruning that matters: A case study on retraining
variants. In International Conference on Learning Representations, 2021. URL https://
openreview net/
Guillaume Leclere and Aleksander Madry. The two regimes of deep network training, arXiv preprint arXiv::002.10337,, February 2020.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky (ed.), Advances in Neural Information Processing Systems 2, (NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pp- 598-605. Morgan Kaufmann, 1989. URL http:/ petttt://pppere ppapers.
nips. httpcc/pppppee. eetaaprr2500-- 200/2200-000 www.-opttiaaaaaaaaaaaaaaaa damage.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer- sparsity for the magnitude-based pruning. In International Conference on Learning Representations, October 2020.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Singleshot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019. URL
http::///ppeeeeviiec net wtt.eeeeeeeee/euueeee/ (-trrrummio--122ejjAaa
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. August 2016.
Mengtian Li, Ersin Yumer, and Deva Ramanan. Budgeted training: Rethinking deep neural network training under resource constraints. In International Conference on Learning Representations, 2020. URL https: wpwpreweeneeeeeeeeee/eeeee
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: httpp/ppooeeenggeeeee/ee/e neur ips. co/ paper/2019 file /tppr///2..//LLe///eeababzz (tw.:/2222277777e22222
bee5d7dde Paeeee-apeereeeeeeee pdÉ.
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
Tsung-Yu Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-1 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K.H. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In International Confer- ence on Learning Representations, 2020. URL https: / http///pprrrr/eeee/e ew. net/ forum? d= DR.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Decebal Constantin Mocanu, Flena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1), June 2018. doi: 10.10388441447-0118.04411..
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bund E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL h http://:////pppppptoeeeceoooeect newrips bdbca288 fee7 7 F22 F
www.arisscccpppppee wwtpccppppppppceecccoeeeeeeeeepp 9/ /file/
12

---

Published as a conference paper at ICLR 2023
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- org/ anthology/
sociation for Computational Linguisties. URL https: W1 839...
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http:// /ttt:://mmr.../ otg/papers
html..
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211 10:211-255,, 2015. doi: 10.1007/s1 -266-011-081.....
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsi- fication. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1 11380-11390 Curran As- sociates, Inc., 2020. URL https: h/tpoeeeii///ooeeinnggs. neurips wwweurips.ccpppper/eeeee httpp/cppe//.//////..... file/ pdf.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint I4X9v 1556, September 2014.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference applications of computer vision (WACV), pp. 464-472. IEEE, 2017.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- DDtiiiiDmain Operations Applications, volume 11006, pp. 1100612. International Society for Optics and Photonies, 2017.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural net- works without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems 2020, June 2020.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Purt XXIV, pp. 459 459-479. Springer, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick vOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Sfate-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 118553/1/2020.00 http://2000mmmmmmmmm........ URL hetps://aclanthot
org/ 2020. emnlp demos.6 6.
Mitchell Wortsman, Ali Farhadi. and Mohammad Rastegari. Discovering neural wirings. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 2010396ca$abee aet.aaaabaafaaecccc2222222222222
wtw.:eeddddinnnnnnnnieeeeeeeeeee wwt.nnip.....pppree////...scpepe hctps/caaaa/e//11///1.... Papei-appppeeee capper.pde. pdf.
Kaichao You, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. How does learning rate decay help modern neural networks? arXiv preprint arXiv:1908.00
August 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1rXiv:1605.07144 May 2016.
13

---

Published as a conference paper at ICLR 2023
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.08 035300, November 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
Michael Zhu and Suyog Gupta. To prune, or not prune: Exploring the efficacy of pruning for model compression. arXiv preprint 1rXiv:1710.01878, October 2017.
14

---

Published as a conference paper at ICLR 2023
A TECHNICAL DETAILS AND TRAINING SETTINGS
A.1 TECHNICAL DETAILS AND GENERAL TRAINING SETTINGS
We define truming-stabiling theoretical speedup well as several pruning selection criteria for IMP, i.c. different criteria that are used to select weights for pruning. We will analyze the impact of such criteria under different retraining schedules in Appendix C.2. Further, Table shows the default training settings used throughout this work.
Definition A.1 (Bartoldson et al. (2020)). Let fpre and (poss be the test aceuracy before pruning and after pruning the trained model, respectively. Assuming ppost I (pre. we define the gruning-stabilie of a method as
stabiiity == Epn Epost
E [0, 1).
lpr
Pruning-stable methods are sparsification algorithms that learn a sparse solution throughout training such that Asability 1. For example, methods that perform the forward pass using an already sparsified copy of the parameters (e.g. DNW by Wortsman et al., 2019), will have Asaaility 1, since the 'hard' pruning step only consists of application of the present pruning mask, which has no further effect. Methods that actively drive certain parameter groups towards zero more rapidly (such as Carreira- Periiian & Idelbayev, 2018; Ding et al., 2019) will have pruning-stability close to 1, since the projection of (magnitude) pruning at the end of training will perturb the parameters only slightly.
Crucial to our analysis are the tradeoffs between the model sparsity, the final performance (measured by final test accuracy, BLEU scores or IoU) and the theoretical speedup induced by the sparsity (Blalock et al., 2020). The theoretical speedup is a metric measuring the ratio in FLOPs needed for inference comparing the dense and sparse model. More precisely, let Fa be the number of FLOPs the dense model needs for inference, and let similarly be F, the same number for the pruned model, given some sparsity s. The theoretical speedup is defined as Fa/ F, and depends solely on the position of the zero weights within the network and layers, not the numerical values of non-zero parameters.
IMP in its original form treats all trainable parameters as a single vector and computes a global threshold below which parameters are removed, independent of the layer they belong to. This simple approach, which we will refer to as GLOBAL, has been subject to criticism for not determining optimal layer-dependent pruning rates and for being inconsistent (Liu al., 2020). Fully-connected layers for example have many more parameters than convolutional layers and are therefore much less sensitive weight removal (Han et al., 2015: it has been observed that the position of a layer can play a role in whether that layer is amenable to pruning: often first and last layers are claimed to be especially relevant for the classification performance (Gale et al., 2019). On the other hand, in which layers pruning takes place significantly impacts the sparsity-induces theoretical speedup (Blalock et al., 2020). Lastly, the non-negative homogeneity of modern ReLU-based Neural Network architectures (Neyshabur et al., 2015) would also seem to indicate a certain amount of arbitrariness to this heuristic selection rule, or at least a strong dependence on the network initialization rule and optimizer used, as weights can be rescaled to force it to fully remove all parameters of a layer, destroying the pruned network without having affected the output of the unpruned network.
Perpinán & Idelbayev, 2018). Further,
Determining which weights to remove is hence crucial for successful pruning and several methods have been designed to address this fact. Zhu & Gupta (2017) introduced the UNIFORM allocation, in which a global sparsity level is enforced by pruning each layer to exactly this sparsity. Gale et al. (2019) extend this approach in the form of UNIFORM+ by (a) keeping the first convolutional layer dense and (b) pruning at most 80% of the connections in the last fully-connected layer. Evci et al. (2020) propose reformulation of the ERDÓS RÉNYI KERNEL (ERK) (Mocanu et al., 2018) to take the layer and kernel dimensions into account when determining the layerwise sparsity distribution. In particular, ERK allocates higher sparsity to layers with more parameters. Finally, Lee et al. (2020) propose LAYER- LAYER-ADAPTIVE ADAPTIVE MAGNITUDE BASED PRUNING (LAMP), an approach which takes an fz-distortion perspective by relaxing the problem of minimizing the output distortion at time of pruning with respect to the worst-case input. We note that we follow the advice of Evci et al. (2020)
TTo compute the number of Fl.OPs, sample single hatch from the test set. The code to compute the theoretical speedup has been adapted from the repository of the ShrinkBench framework (Blalock et al., 2020).
15

---

Published as a conference paper at ICLR 2023
Table Exact training configurations used throughout the experiments for IMP. We note that others have reported an accuracy of around 80% for WRN28x10 trained on CIFAR- 100 that we were unable to replicate. The discrepancy is most likely due an inconsistency in PyTorch's dropout implementation. For experiments involving gradient clipping. For COCO and CityScapes architectures, we rely on pretrained backbones and report the common mean Intersection-over- Union (loU) metric measured on the validation set. For the NMT task we report the BLEU score on the test set, where we limit the sequence length to 128 throughout.
we used label smoothing well as
Tanaset
Netwerk (number of weights)
Fpochs
Rutch size
Momentuum
Leaming nite (1 trrining quech)
Unprinnal test aocmmey/lot//81
RosNes-5G
0.1
[1,
CIBAR-IU
ResNeet- Mie) VGGG 16 (138 Mio)
200
128
0.9
0.01
1.. [91, 180.
95.0% me
[181, 200]
95.88% 216
(0.1
L. [1,00.
120).
CIBAR-1UU
WRN22K10 (37 Mooj
09
7V
0.004
Ze [121, 161)..
70.7%
[161,200 200)
0..1
E [),
0.1
ImageNNt
ResNeet 50 (26 Mio)
90
256
0.
0.01
e [31, 60)
76.17% SIALSS
0.0001
e [81, 90)
0.2g
(1,
0.2
E [20, 60)
ImageNet
MasVTT (31 Mis)
256
0.
74
120].
75.0%
0.002
E [/21, (60).
200]
0.05
E 20,
COCO
DespLabN33 (40
0.9
71
0.005
26/
0.0006
1 [27. 3xy
0..15)
[1. 20
0.1
1 [20, 100).
(101,200).
CityScapes
PSPNet (68 Mioj
300
12
09
0.001
1. 2701
58.3 IoU 20.5
0.0001
E (971,290)
0.00001
e (201.
lo, 1.0).
0.1
1 [].4].
WMT16 (EN-DE)
T5-imall Min)
0.
1 [2.3).
24.56
0.001
E [3.4).
0.0001
1 (4.5]
and Dettmers & Zettlemoyer (2019) and do not prune biases and batch-normalizationn parameters, since they only amount to negligible fraction of the total weights, however keeping them has very positive impact on the performance of the learned model. Further, for the computations involving GMP. we similarly employ the global selection criterion since we found yield better results than UNIFORM+.
We will compare these approaches in Appendix C.2 with a focus on the impact of the retraining phase. Since Le & Hua (2021) found that SLR can be used to obtain strong results even when pruning convolutional filters randomly, i.e., by assigning random importance scores the filters instead of using the magnitude criterion or others, we are interested in understanding the importance of the retraining technique when considering different sparsity distributions.
For experiments involving the pruning of convolutional filters instead of weights, we follow Li et al. (2016) and remove filters using an Lg-norm criterion, enforcing a uniform distribution of sparsity among the layers.
16

---

Published as a conference paper at ICLR 2023
B EXTENDED RESULTS AND COMPLETE TABLES
B.1 LEARNING RATE SCHEDULES DURING RETRAINING
This section contains the complete results regarding the performance of different learning rate schedules for retraining. For fixed schedules, i.e., FT, LRW, SLR, CLR, LLR and ALLR, only the weight decay parameter is tuned and the best configuration reported. For the tuned schedules, i.e. constant, stepped (BAC), cosine and linear, we tune the weight decay, the initial value of the learning rate as well as the length warm-up as follows. For ResNet-18 and ResNet-56 trained on CIFAR-10. 10, weight decay is tuned using grid search over Ie-4, 2c-4, and 5e-4 and for the tuned schedules in the lower half of the table the initial value is chosen using a grid search over {0.001, 0.005, 0.01, 0.05, 0.1, 0.5), where in the iterative case and the warm-up is tuned over either zero or ten percent of the retraining budget. For WideResNet on CIFAR-100 weight decay is tuned over 2e-4, and 5e-4, the initial value is chosen using a grid search over {0.0008, 0.004, 0.02, 0.05, 0.1, 0.5). where we vary the warmup length between zero and ten percent of the retraining budget. For ResNet-50 on ImageNet, Max ViT ImageNet, DeepLab V3 on COCO, PSPNet on CityScapes and T5-smal on WMT16, we only present the results for fixed schedules. For ResNet-50 on ImageNet, we keep the weight decay fixed at le-4, while for the other four aforementioned architecture-dattee
the same value for each cycle,
pairs, we set the weight decay to le-5.
As indicated in the caption, each table displays the results for one architecture-datee in the One Shot or the iterative setting. If no pruning method is indicated, we report the results of magnitude pruning with a global selection criterion. Whenever we perform filter pruning, we indicate it the caption of the table and rely on uniform selection of filters by their L2-norm.
pair either
Table 4: ResNet-56 on CIFAR-I0 (One Shot): Performance of the different learning rate translation schemes (above) compared to tuned schedules (below) for IMP in the One Shot setting for target sparsity of 50%, 90%, 95%, 99% and a retrain time of 2.5%, 5%, 10%. and 25% of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over two seeds with the standard deviation indicated.
Model spursity 50%
Model sparsity 90%
Budget:
2.5%
5%
10%
25%
2.5%
5%
10%
25%
FT
93.42 20.33
93.36 =0.42
93.40 40.36
93.62 =0.21
90.72 #0.22
91.34 40.07
91.51 =0.09
92.02 #0.11
LRW
93.40 20.30
93.34 10.43
93.42
93.38 20.01
90.76
91.32 20.08
91.47 20.02
92.44
SLR
92.00 20.23
92.62 10.18
92.88
20.28
93.29 20.02
91.05 20.09
91.61 40.04
92.09 20.08
92.27 20.26
CLR
92.72 10.14
93.
#0.12
93.42 +0.23
93.63 =0.03
91.66 =0.09
92.18 #0.02
92.27 20.13
92.56 =0.28
LLR
92.62 10.04
93.06 20.35
93.37
10.16
93.55
20.26
91.67 20.06
92.19 10.20
92.30 20.20
92.64
10.11
ALLR
93.18 =0.45
93.46 #0.21
93.55
#0.18
93.82 #0.01
91.72 =0.00
92.00
#0.04
92.44 #0.12
92.45 #0.31
constant
93.42
93.36 20.35
93.40
93.55 20.25
90,92
91.33 20.01
91.55 20.01
91,91
20.04
stepped
93.54
20.37
93.58 20.33
93.47
20.33
93.64 20.28
91.93
10.10
10.18 92.30
92.43
20.01
92.61
10.15
cosine
93.50
93.48 #0.41
93.62 #0.25
93.71 =0.22
92.06 =0.02
92.36 +0.06
92.67 20.15
92.66 #0.02
linear
93.50 20.35
93.54 20.35
93.70 40.31
93.75 20.39
92.01 20.01
92.25 40.08
92.43 20.10
92.69 20.08
Model sparsity 95%
Model sparsity
99%
Budget:
2.5%
5%
10%
25%
2.5%
5%
10%
25%
FT
87.61 =0.09
88.50 10.17
89.59 +0.03
90.27 #0.54
64.68
+2.53
68.97 +2.14
73.48 #1.13
77.54 +0.31
LRW
87.61 #0.04
88.56 +0.
89.60
91.03 #0.08
64.66
73.44 #1.17
81.38
SLR
88.38 =0.45
89.54 +0.23
90.43 20.08
90.75 =0.48
#0.16 77.05
78.98 40.04
80.23
=0.28
81.42 +0.03
CLR
89.45 =0.38
90.22 #0.05
90.76 #0.10
91.04 14.14
77.96 =0.08
79.40 +0.08
80.71 =0.38
81.95 #0.06
LLR
89,52 20.08
90.31
+0.34
90,92 a0.28
91.16 20.40
78.11 20.07
79.63 10.55
81.04 20.27
81,58 =0.12
ALLR
89.62 #0.25
90.42 =0.14
90.97
91.25 #0.46
77.96
79.49 #0.48
80.96 =0.13
81,94
constant
88.19 20.66
88.81 20.06
89.56
90.34 20.59
73.33 =1.04
74.91
40.40
77.59
20.49
78.90 20.45
stepped
89,81 =0.25
90.31 +0.08
90.89 10.13
91.09 =0.45
77.33 #0.35
79.01
40.13
80.75 =0.06
81.86 #0.26
cosine
89.84 =0.03
90.50 +0.40
91.17 +0,21
91.32 =0.37
+0.14 77.81
79.75 +0.25
#0.11 80.87
82,11 +0.21
linear
89.84 10.13
90.63 +0.30
90.98 40.25
91.29 20.39
78.11 +0.32
80.00 40.15
81.14 20.01
82.12 40.08
17

---

Published as a conference paper at ICLR 2023
Table 5: ResNet-56 CIFAR-10 (Iterative): Performance of the different learning rate translation schemes (above) compared to tuned schedules (below) for IMP in the iterative setting for target sparsity of 90%, 95%, 99% and retrain times indicated in the Budget row. Here 2 x 2.5% indicates two prune-retrain cycles, each of which having length equal to 2.5% of the overall training budget. The first, second, and third best values for the translation schemes highlighted. Results averaged over two seeds with the standard deviation indicated.
Model sparsity 90 %
Budget:
2 x 2.5%
2 X 5%
2 x 10%
x 2.5%
3 x 5%
3 10%
FT
90.96 +0.09
91.49 20.01
91.83 20.11
91.22 20.10
91.83 20.04
91.81 =0.30
LRW
90.88 #0.14
91.47 #0.07
91.99 #0.08
91.28 #0.08
91.60 +0.04
91.74 #0.20
SLR
91.33 #0.01
92.08 #0.07
92.41 #0.10
91.62
92.05 #0.01
92.49 #0.23
CLR
92.05 #0.08
92.47 20.08
92.69 #0.42
91.92 #0.11
92.73 +0.09
92.69 #0.08
LLR
91.79 #0.13
92.27
#0.38
92.76 +0.22
92.10 #0.35
+0.31 92.60
92.81 #0.35
ALLR
91.95 #0.32
92.26
#0.13
92.78 +0.33
92.20 #0.47
92.66 +0.35
92.79 #0.03
constant
91.10 #0.21
91.45
#0.11
91.86 #0.10
#0.13
91.52
#0.04
91.88 #0.19
stepped
91.80
#0.26
92.10
+0.33
92.62
+0.08
91.81
#0.22
92.52
+0.43
92.77 #0.13
cosine
92.01 20.01
92.30 20.15
92.77
20.54
92.22
20.21
92.74
20.11
92.89 20.08
linear
92.08 20.04
92.42 40.20
92.94 20.04
92.07 20.06
92.67 40.26
92.80 #0.12
Model sparsity 95%
Budget:
2 2.5%
2 x 5%
2 x 10%
x 2.5%
3 x 5%
3 10%
FT
87.80 =0.28
89.07 #0.21
89.72 #0.08
88.76 #0.11
89.47 #0.01
90.30 #0.11
LRW
87.81 #0.18
89.15 #0.10
89.86 +0.09
88.72 #0.04
89.66 #0.19
90.24 #0.31
SLR
89.80 =0.28
90.47 20.08
91.09 +0.21
90.20
91.07 40.34
91.60 #0.03
CLR
90.45 #0.20
91.25 40.35
91.36 20.13
90.94 20.04
91.61 40.16
91.98 #0.30
LLR
90.82 #0.30
91.03
20.04
91.27 #0.13
90.90 #0.56
91.42 #0.00
91.96 #0.22
ALLR
90.69 #0.08
91.39 #0.02
91.44 #0.51
91.05 #0.09
91.56 #0.17
91.90 #0.04
constant
87.83 #0.35
89.11
#0.24
89.80 +0.24
88.57 #0.25
89.92 #0.45
90.22
#0.09
stepped
90.17 #0.25
90.79
#0.21
91.39
#0.12
#0.13 90.25
91.10 +0.11
91.75 #0.21
cosine
90.51 #0.07
91.08
20.23
91.45
#0.06
91.03 20.15
91.57
20.21
92.09 #0.50
linear
90.56 #0.19
91.14 20.08
91.48 +0.23
90.95 +0.28
91.34 10.18
91.81 #0.36
Model sparsity 99%
Budget:
2 2.5%
2 x 5%
2 10%
2.5%
3 5%
3 10%
FT
65.95 +1.50
71.08 +1.70
76.46 +0.23
75.42 #0.16
78. +0.10
79.92 #0.30
LRW
65.78 +1.43
71.45 +1.18
76.35 #0.18
#0.11
77.94 +1.12
79.94 #0.87
SLR
81.11 #0.69
82.11
#0.39
82.60 +0.17
82.60
#0.44
83.50 +0.15
84.34 #0.35
CLR
81.41 10.18
82.08
+0.43
82.86 +0.08
83.21 20.16
84.24 40.21
84.88 20.04
LLR
81.25 #0.01
81,92 20.08
82.60 #0.03
#0.15 83.36
#0.11 84.18
84.48 #0.20
ALLR
81.24 +0.27
82.25 #0.31
83.16 +0.00
83.12
84.50 40.15
85.11 +0.91
constant
73.91 #0.13
75,37
#0.44
78.38
+0.86
78.03 #1.27
80.05 #0.33
80.73 #0.52
stepped
81.12 #0.06
82.18
+0.54
82.76 +0.04
83.54 +0.08
84.48 #0.18
cosine
81.42 40.23
82.09
40.17
82.56 40.30
82.89 10.62
84.16 +0.24
84.97 20.43
linear
81.17 #0.35
82.14 #0.47
82.75 +0.28
83.31 +0.08
84. +0.72
85.09 #0.48
18

---

Published as a conference paper at ICLR 2023
Table ResNet-18 on CIFAR-10 (One Shot): Performance of the different learning rate translation schemes (above) compared to tuned schedules (below) for IMP in the One Shot setting for target sparsity of 70%, 90%, 95%, 98% and retrain time of 5%, 10%, and 25% of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 70% %
Model sparsity 90%
Budget:
5%
10%
25%
5%
10%
25%
FT
94.97 #0.12
95.09
#0.24
95.23 #0.01
94.63 #0.08
94.75 #0.30
94.77 #0.07
LRW
95.05 #0.16
95.08
#0.23
95.05 +0.01
94.62 #0.08
94.74 +0.30
94.91 #0.20
SLR
94.21 #0.10
94.82 16.16
94.86 #0.06
94.24 #0.07
94.66 +0.20
94.90 #0.41
CLR
94.70 20.07
95,02 20.14
95.27 20.01
20.01 94.65
94.88 20.15
94.97 20.08
LLR
94.58 20.13
20.16 94.82
95,00 10.06
94.61 20.25
94.87 10.14
95.03 20.04
ALLR
94.99 #0.03
95.08 20.08
95.34 #0.15
94.66 #0.01
94.94 20.15
95.02 #0.07
constant
94.97 #0.12
95.09
#0.25
95.25 #0.01
94.62 10.12
94.73 #0.33
94.73 #0.04
stepped
95.08
#0.13
95.19 +0.12
95.19 +0.15
14.8840.22
94.86 +0.06
95.05 10.16
cosine
95.08
#0.18
95.16 10.16
95.35 #0.01
#0.13 10.13
95.15 #0.24
95.20 #0.35
linear
95.18 #0.14
95.20 #0.11
95.34 40.13
94.98 10.18
95.06 +0.26
95.18 #0.20
Model sparsity 95%
Model sparsity 98%
Budget:
5%
10%
25%
5%
10%
25%
FT
93.72 +0.14
94.11 20.17
94.14 10.06
91.01 20.54
92.38 40.05
93.03 20.13
LRW
93.74 #0.10
94.09
20.18
94.49 20.06
91.01 20.58
92.37 40.03
93.95 20.13
SLR
94.26 #0.17
94.38
#0.03
94.52 #0.08
93.18 20.08
93.62 #0.12
93.61 #0.16
CLR
94.38 #0.35
40.02 94.60
94.79 10.18
93.41 +0.25
94.06 20.05
93.95 #0.11
LLR
#0.19 94.45
94.71
#0.01
94.81 #0.13
93.50
#0.01
#0.25 93.72
94.22 #0.04
ALLR
94.43 #0.13
94.59 #0.25
94.68 #0.07
93.48 #0.06
93.86 +0.23
93.97 #0.08
constant
93.69 #0.11
94.15
#0.11
94.19 #0.06
92.09
#0.21
92.38
+0.13
93.14 #0.16
stepped
94.42
#0.13
94.55
10.18
94.72 +0.28
93.44 10.18
93.82 +0.13
93.86 #0.22
cosine
94.50
20.05
94.73
20.01
94.80 10.01
20.14
94.01
10.10
94.06 20.02
linear
94.68 #0.17
94.70 40.01
94.98 20.10
93.67 20.10
94.09 40.01
94.22 20.03
19

---

Published as a conference paper at ICLR 2023
Table 7: WideResNet CIFAR-100 (One Shot): Performance of the different learning rate translation schemes (above) compared to tuned schedules (below) for IMP in the One Shot setting for target sparsity of 90%, 95%, 99% and retrain time of 2.5%, 5%, 10% of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 90
Budget:
2.5%
5%
10%
FT
73.96 =0.28
74.70 +0.08
74.68
#0.18
LRW
73.95 #0.24
74.70 +0.07
74.67 #0.19
SLR
75.82 #0.48
76.09
#0.26
76.05
#0.16
CLR
75.69 #0.19
76.00 #0.17
75.96 #0.06
LLR
75.62 #0.02
76.25
#0.07
75.82 #0.36
ALLR
75.71 #0.05
75.84 #0.07
75.85 #0.41
constant
74.95 #0.21
74.87
+0.85
75.44
#0.70
stepped
77.23 20.36
77.40
#0.14
77.72 #0.45
cosine
77.34 10.15
77.85
+0.28
77.76
#0.01
linear
77.22 20.21
77.20 40.25
77.42 #0.02
Model sparsity 95 %
Budget:
2.5%
5%
10%
FT
68.32 10.16
69.54
+0.43
70.11 #0.43
LRW
68.28 10.14
69.59
+0.38
70.09
#0.54
SLR
75.10 20.35
75.73 +0.35
75.49
=0.21
CLR
75.40 20.18
75.62
+0.35
75.51
#0.13
LLR
75.32 #0.29
75.95
#0.26
75.69
#0.29
ALLR
75.02 #0.69
75.84 #0.25
75.61
#0.21
constant
70.41 #0.34
70.86 #1.11
71.27
#0.88
stepped
75.92 20.08
77.09
#0.28
77.67
#0.21
cosine
76.33 20.02
77.20
+0.35
77.35
20.66
linear
76.58 #0.28
76.95 +0.31
77.17 #0.52
Model sparsity 99 %
Budget:
2.5%
5%
10%
FT
16.24 #1.12
17.92
+2.20
19.63
=0,96
LRW
16.27 +1.17
17.87 +1.97
19.79
#1.19
SLR
60.51 +1.38
+0.09 64.50
68.40 #0,57
CLR
62.91 #0.56
66.27 +0.59
68.67
#0.84
LLR
63.09 #0.65
66.24 +0.57
68.77 #0.64
ALLR
62.93 20,53
65.74 #0.51
68.72 #0.04
constant
56.85
59.58 +1.61
61.05
#0.58
stepped
61.64 #0.20
65.93 +0.28
69.63
#0.59
cosine
63.52 #0.21
67.22 #0.19
70.23 #0.69
linear
63.93 #0.36
07.3440.77
69.50 #0.71
20

---

Published as a conference paper at ICLR 2023
Table 8: ResNet-50 on ImageNet (One Shot): Performance of the different learning rate translation schemes for IMP in the One Shot setting for target sparsity of 70%, 80%, 90%, 95% and a retrain time of 2.22% (2 epochs), 5.55% (5 epochs), 11.11% % (10 epochs), 22.22% (20 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. seeds with the standard deviation indicated.
Results averaged
Model sparsity 70%
Budget:
2.22%
5.55%
11.11%
22.22%
FT
73.51 =0.04
73.98 #0.04
74.44 #0.11
74.75 #0.03
LRW
73.50 #0.04
73.99 #0.04
74.45
#0.11
75.36 #0.03
SLR
70.93 #0.01
72.58 20.03
#0.11 73.69
74.59 +0.05
CLR
72.22 #0.09
73.58 +0.08
#0.04 74.49
74.98 +0.07
LLR
72.39 #0.13
73.65
+0.05
74.3440.00
74.80 +0.07
ALLR
73.69 20.03
74.37 #0.05
74.89 #0.04
75.27 20.16
Model sparsity 80%
Budget:
2.22%
5.55%
11.11%
22.22%
FT
70.45 20.20
71.81 40.11
72.68
#0.07
72.82 #0.13
LRW
70.45 +0.20
71.82 20.12
20.07 72.67
74.24 40.02
SLR
70.48 #0.04
72.37 +0.02
13.44.0.11
73.95 #0.13
CLR
71.96 #0.09
73.30
+0.08
74.24 #0.08
74.38 +0.05
LLR
72.07 #0.09
73,41
#0.05
74.23
#0.10
74.12 #0.02
ALLR
72.96 #0.15
74.02 #0.08
74.71 #0.04
74.43 +0.09
Model sparsity 90%
Budget:
2.22%
5.55%
11.11%
22.22%
FT
56.75 #0.01
61.60 +0.30
64.61
#0.21
65.90 +0.10
LRW
56.75 =0.01
61.61 +0.30
64.60
=0.23
70.41 +0.13
SLR
67.19 +0.23
69.45 20.01
70.80
+0.09
71.24
10.04
CLR
68.72 #0.06
70.60
#0.15
71.51 #0.13
71.68 #0.05
LLR
68.90
#0.05
70.48
#0.01
71.53 #0.09
71.45
#0.13
ALLR
69.56 #0.07
71.19 +0.01
71.99 +0.07
71.76 +0.09
Model sparsity 95%
Budget:
2.22%
5.55%
11.11%
22.22%
FT
27.24 40.30
40.91
40.05
48.56 10.17
51,69 40.08
LRW
27.24 20.30
40.91 40.04
48.56 20.17
62.55 40,09
SLR
61.42 #0.10
64.60 10.17
#0.00 66.31
66.93 +0.16
CLR
63.35 #0.11
66.0740.11
67.34 +0.09
67.44 40.03
LLR
63.58 +0.06
66.02 #0.16
67.20 #0.17
67.34 +0.04
ALLR
63.75 #0.03
66.22 +0.23
67.43 #0.10
67.75 #0.13
21

---

Published as a conference paper at ICLR 2023
Table 9: ResNet-50 on ImageNet (Iterative): Performance of the different learning rate translation schemes for IMP in the iterative setting for target sparsity of 70%, 80%, 90% and retrain times as indicated in the Budget row. Here x 2.22% indicates two prune-retrain cycles, each of which having length equal to 2.22% of the overall training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over two seeds with the standard deviation indicated.
Model sparsity 70 %
Budget:
2 x 2.22%
2 x 5.55%
x 11.11%
3 x 2.22%
3 x 5.55%
3 x 11.11%
FT
73.59 =0.04
74.16 #0.02
74.50 +0.10
73.77 #0.01
74.24 +0.03
74.67 #0.09
LRW
73.59 =0.04
74.16 #0.02
74.50 +0.10
73.77 #0.01
74.24 +0.03
74.67 +0.09
SLR
70.71 10.14
72.36 +0.06
73.67 #0.05
70.94 #0.14
72.35 +0.05
73.55 #0.06
CLR
72.30 =0.04
73.52 +0.00
74.43
#0.08
72.27 #0.05
73.50
#0.01
74.56 #0.11
LLR
72.46 #0.15
73.59
#0.05
74.42
#0.10
72.40
#0.02
73.80
#0.08
74.63 #0.06
ALLR
74.19 20.08
74.80 20.01
75.19 +0.03
73.47 #0.12
74.70 20.15
75.24 20.08
Model sparsity 80% %
Budget:
2 x 2.22%
2 5.55%
2 x 11.11%
3 x 2.22%
3 x 5.55%
3 x 11.11% %
FT
70.14 #0.02
71.55 #0.04
72.48 +0.07
70.63 #0.08
71.85 #0.04
72.82 #0.12
LRW
70.14 #0.02
71.55 #0.04
72.48 +0.07
70.63 #0.08
71.85 +0.04
72.82 #0.12
SLR
69.98 #0.13
71.96 #0.13
73.15 +0.06
70.33 #0.05
#0.15 72.05
73.13 #0.04
CLR
71.75 #0.04
73.09
#0.06
74.09
+0.13
71.89
#0.02
#0.00 73.29
74.27 #0.05
LLR
71.81 #0.04
73.32
#0.02
73.91
#0.04
72.04 #0.05
73.29
#0.04
74.34 #0.04
ALLR
73.10 #0.02
73.99 +0.25
74.69 +0.16
72.50 #0.16
74.48 +0.06
75.02 +0.04
Model sparsity 90 %
Budget:
2 x 2.22%
2 5.55%
2 11.11%
3 2.22%
3 x 5.55%
3 11.11 %
FT
56.96 =0.06
61.90 +0.12
64.80 +0.12
59.42 #0.17
63.72 +0.15
66.30 =0.14
LRW
56.96 20.06
61.90 10.12
10.12 64.80
59.42 20.17
63.72 10.15
66.30 10.14
SLR
68.16 #0.02
70.16 20.15
71.68 #0.07
#0.13 68.49
70.61
#0.13
72.09
#0.12
CLR
69.69 #0.01
71.38 #0.01
72.41 #0.04
70.18 #0.01
71.79
#0.12
72.93
#0.05
LLR
69.88 +0.24
71.35 #0.07
72.24
+0.09
70.29
#0.07
71.85
+0.04
72.91 40.08
ALLR
70.00 +0.08
71.77 20.05
72.75 +0.05
70.23 #0.16
71.45 40.13
73.14 +0.07
22

---

Published as a conference paper at ICLR 2023
Table 10: VaxViT on ImageNet (One Shot): Performance of the different learning rate translation schemes for IMP in the One Shot setting for target sparsity of 75%, 80%, 85%, 90% and a retrain time of 1% (2 epochs), 2.5% (5 epochs), 5% (10 epochs). 10% (20 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over seeds with the standard deviation indicated.
Model sparsity 75%
Budget:
1%
2.5%
5%
10%
FT
76.64 =0.58
76.90 #0.15
77.04
#0.23
77.02 #0.45
LRW
76.64 =0.58
76.90 #0.15
77.04 =0.23
77.02 #0.45
SLR
75.29
=0.29
76.31 #0.21
76.68 #0.16
76.99 40.66
CLR
76.10 =0.49
76.98
#0.05
77.33 #0.15
77.52 #0.54
LLR
76.21 =0.45
77.04
#0.00
77.39
#0.13
77.50
+0.52
ALLR
77.11 20.63
77.59
+0.09
77.62 #0.01
77.39 20.55
Model sparsity 80%
Budget:
1%
2.5%
5%
10%
FT
75.10 20.76
75.79
+0.29
75.94 20.08
76.23 20.67
LRW
75.10 20.76
75.79 20.29
75.94 20.08
76.23 10.67
SLR
74.89 #0.45
75.79 +0.07
#0.12 76.37
76.59 +0.63
CLR
75.73 =0.44
76.61
+0.09
#0.23 77.13
77.22 10.64
LLR
75.82 #0.52
76.57
#0.02
77.22 #0.07
77.21 #0.64
ALLR
76.70 #0.60
77.21
#0.04
77.37
#0.08
77.31 #0.66
Model sparsity 85%
Budget:
1%
2.5%
5%
10%
FT
71.49 #0.95
72.97
+0.40
73.61
=0.24
74. 14 +0.58
LRW
71.49 =0.95
72.97 +0.40
73.61
=0.24
74. +0.58
SLR
74.02 20.29
75.15
+0.21
75.75 +0.09
75.86 +0.69
CLR
74.85 =0.40
76.11 #0.18
76.55 #0.06
76.59 +0.58
LLR
75.05
#0.34
76.05
#0.03
76.56 #0.02
76.59
#0.51
ALLR
75.79 #0.55
76.55
+0.18
76.85 +0.02
76.74 #0.65
Model sparsity 90%
Budget:
1%
2.5%
5%
10%
FT
60.03 +1.76
64.56
+0.67
66.62
+0.75
68.42 40.77
LRW
60.03 #1.76
64.56
+0.67
66.62
20.75
68,42 40.77
SLR
71.44 #0.10
#0.11 73.03
73.87
#0.03
74.14 +0.64
CLR
72.62 #0.05
74.25
+0.09
74.97
+0.03
75.10 20.64
LLR
72.82 #0.10
74.27 #0.26
74.96 10.10
75.03 +0.72
ALLR
73.43 #0.26
74.66 +0.22
75.22 #0.04
75.28 +0.53
23

---

Published as a conference paper at ICLR 2023
Table 11: Max ViT on ImageNet (Iterative): Performance of the different learning rate translation schemes for IMP in the iterative setting for target sparsity of 75%, 80%, 85%, 90% and retrain times as indicated in the Budget row. Here 2 2.5% indicates two prume-retrain cycles, each of which having length equal to 2.5% of the overall training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 75%
Budget:
2 x 1%
x 2.5%
3 x 1%
3 x 2.5%
FT
76.14 #0.21
76.84 +0.19
76.85 #0.36
76.93 #0.24
LRW
76.14 #0.21
76.84 20.19
76.85 #0.36
76.93 +0.24
SLR
74.89 #0.07
76.08
#0.10
75.32
#0.71
75.98 +0.29
CLR
75.93 #0.08
76.97
#0.04
76.43 =0.40
76.97 #0.12
LLR
76.14 #0.11
77.23
+0.19
76.58
#0.31
77.13 +0.07
ALLR
76.81 #0.28
77.56 40.25
77.37 20.55
77.32 #0.27
Model sparsity 80%
Budget:
2 x 1%
x 2.5%
3 x 1%
3 2.5%
FT
74.75 20.24
75.69 40.13
75.48 20.30
75.85 10.19
LRW
74.75 20.24
75.69 20.13
75.48 +0.30
75,85 10.19
SLR
74.46 #0.12
75.76 +0.03
74.92
40.47
75.59
+0.13
CLR
75.49 20.01
76.68
#0.20
#0.29 76.11
76.52
20.04
LLR
75.66
#0.04
76.78
#0.07
76.10 #0.31
76.80
#0.15
ALLR
76.45 #0.14
77.11 #0.22
77.01 #0.42
77.15 #0.36
Model sparsity 85%
Budget:
2 1%
2.5%
x 1%
2.5%
FT
71.07 #0.45
72.85
+0.36
72.14 14 +0.12
73.12 +0.06
LRW
71.07 #0.45
72.85 +0.36
72.14+0.11
73.12 #0.06
SLR
73.62 20.11
+0.09 75.05
73.95 20.43
74.85 +0.24
CLR
74.68 #0.20
76.09
#0.10
75.34 #0.40
75.95 #0.31
LLR
74.89
+0.27
76.22
#0.17
75.47
+0.40
76,21 40.02
ALLR
75.43 #0.08
76.45 +0.08
76.14 #0.33
76.53 40.12
Model sparsity 90%
Budget:
2 x 1%
2 x 2.5%
3 x 1%
3 x 2.5%
FT
59.72 =1.15
64.38 20.64
61.75 +0.35
64.96 40.20
LRW
59.72 #1.15
64.38 20.64
61.75 +0.35
64.96 +0.20
SLR
71.04 10.19
13.0440..0
72.12
#0.43
73.50 +0.00
CLR
72.39 +0.36
40.16 74.33
73.31 +0.32
74.47 40.17
LLR
72.58
#0.16
74.36
#0.00
73.69
#0.05
74.78 +0.03
ALLR
72.89 #0.27
74.65 +0.03
73.94 #0.02
74.84 #0.11
24

---

Published as a conference paper at ICLR 2023
Table 12: DeepLabV3 on COCO (One Shot): Performance (measured mloU) of the different learning rate translation schemes for IMP in the One Shot setting for target sparsity of 50% 80% and retrain time of 3.33% (1 epochs), 6.66% (2 epochs), 16.66% (5 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over two seeds with the standard deviation indicated.
Model sparsity 50%
Budget:
3.33%
6.66%
16.66%
FT
63.10 #0.15
63.07
#0.08
63.11 #0.07
LRW
63.10 20.15
63.07
#0.08
63.13 #0.10 10
SLR
60.84 20.66
61.14 #0.16
62.12 20.16
CLR
61.72 #1.03
62.72 20.79
63.13 #0.26
LLR
61.73 #1.09
62.69 +1.11
63.41
40.37
ALLR
63.02 40.36
63.21 #0.27
63.58 20.08
Model sparsity 60%
Budget:
3.33%
6.66%
16.66%
FT
62.89 20.17
62.89
20.07
62.99 20.06
LRW
62.89 #0.17
62.89 +0.07
63,07 20.08
SLR
60.82 #0.69
61.56 #0.51
62.33 10.17
CLR
61.93 +0.90
62.63
#0.93
63.17 20.03
LLR
61.90 #0.92
62.83 #0.97
63.20 #0.49
ALLR
62.87 #0.43
63.09 +0.32
63.46 #0.12
Model sparsity 70%
Budget:
3.33%
6.66%
16.66%
FT
62.24 #0.22
62.31 #0.11
62.56
+0.05
LRW
62.24 +0.22
62.31
+0.11
62.75 #0.06
SLR
60.82 20.39
61.32 20.81
62.11 20.04
CLR
61.66 20.74
+0.85 62.56
62.99 #0.00
LLR
61.62 #0.78
62.64
#0.66
63.11 #0.41
ALLR
62.53 20.54
62.78 #0.45
63.08 #0.06
Model sparsity 80%
Budget:
3.33%
6.66%
16.66%
FT
59.84 20.09
60.30 40.18
61.08 40.00
LRW
59.84 20.09
60.31
10.18
61.80 #0.24
SLR
60.04 20.79
60.91
+0.25
61.69 +0.59
CLR
61.02 +0.95
61.74 +0.68
62.69 #0.22
LLR
61.24 +0.90
61.85 +0.47
62.62 #0.21
ALLR
61.68 +0.75
62.05 +0.93
62.55 #0.03
25

---

Published as a conference paper at ICLR 2023
Table 13: DeepLabV3 on COCO (Iterative): Performance of the different learning rate translation schemes for IMP in the iterative setting for target sparsity of 50 80% and retrain times as indicated in the Budget row. Here 2 x 6. 6.66% indicates two retteiretrain cycles, each of which having length equal to 6.66% the overall training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over seeds with the standard deviation indicated.
Model sparsity 50%
Budget:
2 x 6.66%
2 10%
3 x 6.66%
3 10%
FT
63.10 #0.01
63.26 #0.39
63.18 #0.10
63.30 #0.38
LRW
63.10 #0.01
63.26 #0.39
63.18 =0.10
63.30 #0.38
SLR
61.50 #0.69
61.86 +0.54
61.54 #0.45
62.39 +0.35
CLR
62.72 #0.42
62.65 #0.27
62.70
#0.07
63.16 +0.34
LLR
62.97 #0.63
62.85
#0.14
63.06 =0.28
63.32 40.44
ALLR
63.27 20.38
63.02 #0.69
63.35 #0.20
63.44 +0.29
Model sparsity 60%
Budget:
2 6.66%
10%
3 x 6.66%
10%
FT
62.93 20.04
63.12 10.47
63.03 20.17
63.19 20.46
LRW
62.93 #0.04
63.12 20.47
63.03 #0.17
63.19 20.46
SLR
61.44 #0.53
62.02 +0.01
61.81 #0.02
62.70 #0.00
CLR
62.65 #0.55
62.67
#0.04
62.69 #0.17
63.35 +0.23
LLR
62,82 #0.48
62.52 #0.38
62.86 #0.23
63.43 #0.17
ALLR
63.19 #0.34
63.00 #0.53
63.27 #0.16
63.38 #0.37
Model sparsity 70%
Budget:
2 x 6.66%
2 x 10%
3 x 6.66%
3 x 10%
FT
62.33 #0.04
62.58 +0.50
62.51 #0.11
62.71 +0.37
LRW
62.33 =0.04
62.58 #0.50
62.51 =0.11
62.71 +0.37
SLR
61.57 20.18
62.00
20.11
61.83 20.15
62.17 10.10
CLR
62.58 #0.77
62.51
#0.01
62.64 #0.02
62.72
#0.13
LLR
62.78
#0.64
62.38
#0.29
63.10
#0.23
63.05
#0.32
ALLR
62.81 #0.30
62.61 40.51
62.87 +0.23
63.17 #0.25
Model sparsity 80%
Budget:
2 6.66%
x 10%
3 x 6.66%
3 10%
FT
60.20 +0.07
60.86 +0.47
60.53 10.19
60.85 40.55
LRW
60.20 20.07
60.86 +0.47
60.53 +0.19
60.85 40.55
SLR
60.93 +0.87
61.56 +0.23
61.13 #0.04
61.37 +0.41
CLR
62.05 #0.86
61.98 #0.21
62.04 +0.38
62.27
40.38
LLR
62.29 +1.05
61.96 +0.23
62.50 #0.26
62.61 40.13
ALLR
62.21 #0.81
61.77 +0.72
62.19 #0.22
62.59 20.18
26

---

Published as a conference paper at ICLR 2023
Table 14: PSPNet on CityScapes (One Shot): Performance (measured mloU) of the different learning rate translation schemes for IMP in the One Shot setting for target sparsity of 60% 90% and retrain time of 0.66% (2 epochs), 1.66% (5 epochs), 3.33% (10 epochs). 5% (15 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 60%
Budget:
0.66%
1.66%
3.33%
5%
FT
58.03 =0.08
58.15 =0.22
58.18
#0.07
58.12 #0.10
LRW
58.03 #0.08
58.15 +0.22
58.18 #0.07
58.11 #0.13
SLR
57.27 #0.12
57.70 +0.79
58.10 #0.07
57.95 +0.19
CLR
57.58 =0.22
57.80
#0.25
58.23 #0.16
58.18
#0.72
LLR
57.52 #0.33
58.17
40.53
57.84 #0.05
58.30
+0.02
ALLR
57.91 #0.22
58.26 #0.20
58.28 #0.09
58.41 #0.21
Model sparsity 70%
Budget:
0.66%
1.66%
3.33%
5%
FT
56.46 20.07
56.71 40.16
56.84
#0.04
56.85 20.04
LRW
56.46 20.07
56.71 20.16
56.84 20.04
57.16 20.05
SLR
56.58 #0.24
57.52 +0.47
58.00 20.05
58.02 #0.07
CLR
57.25 =0.27
57.96
+0.04
$7.74+0.44
58.05 +0.32
LLR
57.37 #0.32
58.06 #0.28
58.18 #0.23
58.49 #0.09
ALLR
57.67 #0.10
58.09 #0.08
58.04 #0.29
58.30 +0.30
Model sparsity 80%
Budget:
0.66%
1.66%
3.33%
5%
FT
49.59 #0.18
51.08 +0.19
52. 17 #0.15
52.77
+0.38
LRW
49.59
10.19
51.08
+0.20
52.17 #0.15
54.34 +0.37
SLR
55.88 20.59
57.07
20.10
57.41 20.01
57.87
10.34
CLR
56.68 #0.12
57.68
#0.37
57.84 #0.11
58.16 #0.12
LLR
56.78
#0.18
57.53
#0.46
57.95
#0.07
58.31
#0.38
ALLR
56.89 #0.08
57.40
40.15
57.57 #0.21
58.05 #0.11
Model sparsity 90%
Budget:
0.66%
1.66%
3.33%
5%
FT
32.54 20.34
35.85 +0.22
39.14 #0.02
41,00 40.46
LRW
32.54 20.34
35.85 +0.23
39.14 +0.02
45.57 40.46
SLR
54.76 +1.59
+0.52 56.43
56.50 #0.02
56.53 +0.09
CLR
55.96 20.19
56.77
40.38
56.89 +0.30
57.34 +0.52
LLR
55.75 +0.23
56.99
+0.26
56.70 #0.14
57.59 +0.38
ALLR
56.12 #0.12
56.48 +0.07
56.85 +0.32
57.25 +0.06
27

---

Published as a conference paper at ICLR 2023
Table 15: PSPNet CityScapes (Iterative): Performance of the different learning rate translation schemes for IMP in the iterative setting for target sparsity 60 90% and retrain times as indicated in the Budget row. Here 2 x 1.66% indicates two prune retrairetrain cycles, each of which having length equal to 1.66% the overall training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over seeds with the standard deviation indicated.
Model sparsity 60%
Budget:
2 x 0.66%
2 x 1.66%
3 x 0.66%
3 1.66%
FT
58.14 10.19
58.17 =0.09
58.18 #0.07
58.16 #0.06
LRW
58.14 10.19
58.17 #0.09
58.18 =0.07
58.16 #0.06
SLR
57.24 #0.29
57.50 40.31
57.80 =0.45
55.79 +1.42
CLR
58.17 #0.21
58.15 #0.28
#0.19 57.98
57,49 +0.42
LLR
58.29 +0.33
57.60 +1.32
58.31 #0.29
58.23 40.81
ALLR
58.47 #0.32
58.46 #0.17
58.59 #0.20
58.51 +0.40
Model sparsity 70%
Budget:
2 0.66%
1.66%
3 x 0.66%
1.66%
FT
56.62 20.10
56.79
20.04
56.60 10.19
56.75
#0.12
LRW
56.62 20.10
56.79 20.04
56.60 10.19
56.75
10.12
SLR
57.24 #0.48
57.63 +0.28
57.06 #0.08
56.68
#0.18
CLR
58.18 #0.10
57.07 +0.59
58.05
#0.04
57.93 +0.55
LLR
58.09 #0.08
58.13
#0.72
58.07
#0.14
58,47 #0.97
ALLR
58.28 #0.02
58.28 #0.10
58.34 #0.06
58.41 #0.35
Model sparsity 80%
Budget:
2 x 0.66%
x 1.66%
3 x 0.66%
3 x 1.66%
FT
50.05 #0.24
51.25 +0.30
49.71 #0.73
51.22 +0.22
LRW
50.05 =0.24
51.25 +0.30
49.71 =0.73
51.22 +0.22
SLR
56.61 20.18
20.21 56.70
#0.11 56.45
55.35 +1.94
CLR
57.36 #0.11
57.01 #0.90
57.43
#0.70
57.49
+0.83
LLR
57.50
#0.27
57.14 #0.60
57.22
#0.71
57.65
#0.02
ALLR
57.64 #0.54
57.93 +0.24
57.61 +0.83
58.04 #0.32
Model sparsity 90%
Budget:
2 0.66%
2 x 1.66%
3 x 0.66%
3 1.66%
FT
33.09 +0.29
36.22 40.13
33.70 +0.87
37.69 40.44
LRW
33.08 #0.28
36.22 #0.13
33.70 20.87
37.69 40.44
SLR
55.37 +1.24
55.89 +0.77
55.47 +0.89
54.53 +0.88
CLR
56.44 +0.49
56.95
+0.45
55.56 +1.18
56.20
+0.28
LLR
56.48
+0.28
56.79 +0.97
55.84 +0.90
56.47 40.48
ALLR
56.23 #0.17
57.40 +0.27
56.35 +0.76
56.62 #0.16
28

---

Published as a conference paper at ICLR 2023
Table 16: T5-small WMT16 (de-en) (One Shot): Performance (measured test BLEU score) of the different learning rate translation schemes for IMP in the One Shot setting for target sparsity of 50% 70% and retrain time of 2% (0.1 epochs), 5% (0.25 epochs), 10% (0.5 epochs), 20% (1 epochs), 40% (2 epochs), 60% (3 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 50% %
Budget:
2%
5%
10%
20%
40%
60%
FT
23.76 #0.05
23.93
+0.20
24.24
#0.15
24.08 =0.07
24.06 +0.15
24.36 +0.09
LRW
23.76 20.05
23.93
+0.20
24.24
20.15
24.08 =0.07
24.68 10.19
24.92 10.12
SLR
21.66 +0.40
22.09
20.13
22.74
20.11
23.36 20.34
24.09
#0.36
24.61 20.03
CLR
22.69 20.06
23.29
40.17
23.79
#0.34
24.47 20.31
24.73
#0.06
25.28 10.19
LLR
22.97 #0.01
23.20
+0.15
24.00
#0.14
24.53
#0.22
25.11
#0.05
24.95 #0.43
ALLR
24.11 #0.17
24.29
20.02
24.51
#0.00
24.63 #0.21
25.27 40.30
25.17 +0.39
Model sparsity 60%
Budget:
2%
5%
10%
20%
40%
60%
FT
22.18
22.75 +0.03
23.07 #0.31
23.36 #0.03
23.47 #0.04
23.59 +0.39
LRW
22.18 10.18
22.75 +0.03
23.07 #0.31
23.36 +0.03
23.90 +0.06
24.62 +0.11
SLR
21.61 =0.08
21.95 +0.45
22.47 #0.10
23.19 =0.17
23.78 +0.05
24.18 +0.08
CLR
22.51 10.19
23.10
+0.15
23.51
+0.07
23.96 20.22
24.67 20.15
24.87 10.16
LLR
22.57 #0.05
23.19
+0.18
23.48
#0.24
23.96 #0.01
24.59
+0.19
24.60 +0.36
ALLR
23.52 +0.07
24.04 +0.01
24.32
+0.09
24.43 #0.20
24.91 #0.11
24.90 +0.25
Model sparsity 70%
Budget:
2%
5%
10%
20%
40%
60%
FT
18.20 #0.14
20.01
+0.49
20.75
#0.24
21.25 #0.21
21.84 +0.17
22.05 #0.17
LRW
18.20 10.14
20.01
+0.49
20.75
#0.24
21.25 +0.21
22.98 +0.24
24.22 10.13
SLR
20.71 +0.03
21.41
+0.53
22.27
+0.23
22.51 #0.09
23.36 +0.19
23.66 +0.01
CLR
21.68 20.09
22.68
40.22
23.19
20.08
23.52 20.00
23.98 10.16
23,99 10.09
LLR
20.21
22.44
+0.32
23.18
#0.28
33.16
23.64
#0.11
24.12 10.16
ALLR
22.33 +0.47
22.83 +0.08
23.72 +0.18
23.85 #0.21
24.36 #0.15
23.99 #0.14
29

---

Published as a conference paper at ICLR 2023
Table 17: ResNet-56 on CIFAR-10 (One Shot, Filter Pruning): Performance of the different learning rate translation schemes in the One Shot setting for target sparsity of 50 80%, and retrain time of 5% (10 epochs), 10% (20 epochs), 15% (30 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 50%
Budget:
5%
10%
20%
FT
90.78 #0.14
90.88 +0.31
91.36 #0.03
LRW
90.76 10.19
91.00 #0.15
91.80 #0.13
SLR
90.53 20.16
91.27 +0.08
91.78 #0.28
CLR
91.14 #0.06
91.90 #0.18
91.85 #0.25
LLR
91.30 40.13
91.74 #0.02
91.75
+0.24
ALLR
91.35 +0.18
91.84 10.19
92.15 +0.50
Model sparsity 60%
Budget:
5%
10%
20%
89.62 10.18
90.09 20.32
90.33 20.23
LRW
89.79 =0.23
90.13 20.37
91.21 10.13
SLR
89.47 #0.14
90.60 #0.06
91.02 #0.01
CLR
90.28 #0.33
91.19 10.19
91.54 #0.25
LLR
90.51 #0.10
91.35 #0.33
91.45 #0.01
ALLR
90.60 #0.28
91.33 +0.42
91.61 #0.34
Model sparsity 70%
Budget:
5%
10%
20%
FT
87.63 #0.18
88.03 #0.16
88.7 74 +0.46
LRW
87.61 =0.49
88.03
+0.46
90.23
SLR
88.34 20.09
89.41 #0.06
89.93 #0.26
CLR
88.97 20.02
89.98
#0.08
90.59 #0.33
LLR
89.06 #0.28
89.92 #0.13
90.72 #0.03
ALLR
89.38 20.04
90.03 #0.09
90.82 20.04
Model sparsity 80%
Budget:
5%
10%
20%
FT
78.32 43.99
81.80 40.69
84.08 +1.08
LRW
78.71 #3.55
82.09
10.16
85.73 +1.41
SLR
82.90 +2.82
86.38
+0.40
87.08 #0.76
CLR
84.47 +1.72
86.86
#0.76
87.67 40.73
LLR
84.45 +1.64
86.69
+0.75
87.51
#0,22
ALLR
84.66 +1.70
86.88 #0.77
87.68 #0.13
30

---

Published as a conference paper at ICLR 2023
Table 18: ResNet-50 on ImageNet (One Shot, Filter Pruning): Performance of the different learning rate translation schemes in the One Shot setting for target sparsity of 30 50%, and retrain time of 2.22% (2 epochs), 5.55% (5 epochs) and 11.11% (10 epochs) of the initial training budget. The first, second, and third best values for the translation schemes highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 30%
Budget:
2.22%
5.55%
11.11%
FT
67.28 +0.27
69.87 #0.21
71.23 #0.22
LRW
67.28 =0.27
69.87 +0.21
71.23 #0.22
SLR
66.21 #0.04
69.17 #0.00
70.81 #0.00
CLR
68.45 #0.01
70.88 +0.00
72.12 #0.02
LLR
68.73 #0.02
70.93
#0.09
72.08
#0.08
ALLR
69.11 20.05
71.23 20.10
72.31 20.04
Model sparsity 40%
Budget:
2.22%
5.55%
11.1 %
FT
60.87 20.04
65.58 +0.17
68.1 11 #0.04
LRW
60.87 20.04
65.58
20.17
68. 11 20.04
SLR
64.45 20.11
67.87
20.09
69.66 #0.26
CLR
66.91 #0.02
#0.10 69.68
71.25 10.14
LLR
67.02 #0.13
69.83
#0.04
71.04 #0.08
ALLR
67.52 +0.19
70.12 #0.04
71.32 #0.01
Model sparsity 50%
Budget:
2.22%
5.55%
11.11%
48.85 20.38
58.28 #0.16
62.93 40.01
LRW
48.85 20.38
+0.16 58.28
62.93 #0.01
SLR
61.81 #0.00
#0.16 66.08
68.32 40.12 12
CLR
64.50 #0.17
67.93
+0.21
69,82 #0.03
LLR
64.87
#0.20
68.12 #0.12
69.67 #0.06
ALLR
65.04 #0.09
68.32 #0.12
69.98 #0.08
31

---

Published as a conference paper at ICLR 2023
Table 19: ResNet-50 on ImageNet (Iterative, Filter Pruning): Performance of the different learning rate translation schemes the iterative setting for target sparsity of 30%, 40%, 50% and retrain times as indicated in the Budget row. Here 2 x 2.22% indicates two prune-retrain cycles, each of which having length equal to 2.22% of the overall training budget. The first, second, and third best values for the translation schemes are highlighted. Results averaged over two seeds with the standard deviation indicated.
Model sparsity 30%
Budget:
2 2.22%
5.55%
3 x 2.22%
5.55%
FT
68.18 #0.16
70.47 +0.12
68.77 #0.04
70.78 #0.12
LRW
68.18 10.16
70.47
#0.12
68.77 #0.04
70.78 #0.12
SLR
67.09 #0.07
69.54 +0.01
67.43 #0.10
69.51 #0.06
CLR
69.14 #0.11
71.07
#0.10
69.27 #0.01
71.16 #0.04
LLR
69.31 #0.12
71.16
#0.08
69.49 #0.07
71.30
#0.11
ALLR
69.32 #0.00
71.11 #0.09
69.60 #0.12
72.38 +0.07
Model sparsity 40%
Budget:
2 x 2.22%
2 x 5.55%
3 x 2.22%
3 x 5.55%
FT
62.83 #0.04
66.87 #0.10
63.94 =0.09
67.55 #0.01
LRW
62.83 20.04
66.87 10.10
63.94 20.09
67.55 +0.01
SLR
65.74 #0.02
68.32 20.05
66.23 20.04
68.54
+0.02
CLR
67.94 #0.05
70.14 #0.17
68.25 #0.01
70.18 #0.03
LLR
68.08 #0.03
70.17 +0.05
68.41 #0.14
70.38 10.13
ALLR
68.11 #0.04
70.17 #0.06
68.95 #0.02
71.90 #0.02
Model sparsity 50%
Budget:
2 2.22%
5.55%
3 x 2.22%
3 5.55%
FT
54.30 #0.11
61.16
+0.10
56.51 =0.27
62.44 40.21
LRW
54.30 #0.11
61.16 #0.10
56.51 #0.27
62.44 +0.21
SLR
63.85 =0.08
67.04 #0.13
64.59
#0.07
67.28
+0.17
CLR
66.07 #0.01
68.78 #0.20
66.71
#0.19
69.00
#0.16
LLR
66.40
#0.05
68.75 #0.06
66.87 #0.14
69.11 +0.10
ALLR
66.37 #0.03
68.63 40.16
67.86 +0.04
70.47 20.18
32

---

Published as a conference paper at ICLR 2023
Table 20: MaxViT on ImageNet (One Shot, Filter Pruning): Performance the different learning rate translation schemes in the One Shot setting for target sparsity of 30 50%, and retrain time of I % (2 epochs), 2.5% (5 epochs), 5% (10 epochs) and 10% (20 epochs) the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 30%
Budget:
1%
2.5%
5%
10%
FT
68.05 #0.08
70.96 +0.31
72.32
=0.20
73.99 +0.11
LRW
68.05 =0.08
70.96 20.31
72.32 =0.20
73.99 20.11
SLR
74.23 #0.16
74.80 #0.26
75.80
#0.12
76.76
#0.01
CLR
75.45 =0.10
76.11 +0.37
=0.20 76.85
77.50
+0.10
LLR
75.64 #0.21
76.19 #0.06
76.88 =0.10
77.55
+0.20
ALLR
76.31 #0.18
76.77 #0.19
77.19
#0.08
77.58 +0.03
Model sparsity 35%
Budget:
1%
2.5%
5%
10%
FT
64.18 20.19
68.38
10.19
70.50
=0.32
72.43 +0.07
LRW
64.18
=0.19
68.38
+0.19
70.50 #0.32
72.43 +0.07
SLR
73.70 #0.22
74.61
+0.33
75.57 #0.09
76.49 +0.05
CLR
74.98 #0.09
75.78
#0.31
76.75
=0.06
77.39
+0.04
LLR
75.24
#0.10
75.99
#0.42
76.78
#0.05
77.30
#0.01
ALLR
75.86 20.14
76.48
+0.22
77.01 +0.09
77.42 #0.14
Model sparsity 40%
Budget:
1%
2.5%
5%
10%
FT
58.65 #0.52
64.75 +0.50
67.78
#0.18
70.56 +0.36
LRW
58.65 20.52
64.75 +0.50
67.78
20.18
70.56 40.36
SLR
73.20 20.16
74.17 20.67
75.31
20.04
76.24 +0.02
CLR
74.52 #0.19
75.40
+0.39
76.27
#0.08
77.14 +0.04
LLR
74.70
=0.22
75.62
#0.27
76.38 #0.05
77.22 +0.08
ALLR
75.40 #0.18
76.18 +0.39
76.68 #0.01
77.11 40.01
Model sparsity
45%
Budget:
1%
2.5%
5%
10%
FT
51.20 #0.15
60.19
40.76
64.56
#0.17
68.34
+0.40
LRW
51.20 #0.15
60.19 +0.76
64.56
#0.17
68.34 +0.40
SLR
72.34 #0.09
73.71 +0.37
74.82 #0.04
75.80
+0.18
CLR
74.03 #0.16
75.05
#0.38
76.04
#0.06
77.01 #0.06
LLR
74.05 #0.08
75.19
#0.32
76.19 #0.08
77.00
#0.04
ALLR
74.75 #0.08
75.66 #0.31
76.31 #0.00
77.05 +0.06
Model sparsity 50%
Budget:
1%
2.5%
5%
10%
FT
39.85 #0.37
53.59 +1.31
60.00
#0.06
65.20
+0.45
LRW
39.85 20.37
53.59
+1.31
60.00
#0.06
65.20 +0.45
SLR
71.59 10.13
73.05
20.45
74.26
+0.07
75.62
10.11
CLR
73.16 #0.14
74.50
+0.44
75.59
=0.07
76.69 +0.07
LLR
73.41
+0.36
74.62
40.64
75.85
+0.09
76.69 +0.06
ALLR
74.04 #0.17
75.17
+0.42
75.99 +0.10
76.75 #0.08
33

---

Published as a conference paper at ICLR 2023
Table 21: ViX on ImageNet (Iterative, Filter Pruning): Performance the different learning rate translation schemes in the iterative setting for target sparsity of 30% 50% and retrain times as indicated in the Budget row. Here 2 x 2.5% indicates two prune-retrain cycles, each of which having length equal 2.5% of the overall training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 30%
Budget:
2 1%
2 x 2.5%
3 x 1%
3 2.5%
FT
68.56
71.84 40.31
70.01 20.98
72.15 40.78
LRW
68.56 #0.62
71.84 #0.31
70.01 20.98
72.15 +0.78
SLR
73.48 =0.37
75.03 +0.30
74.28 =0.48
75.19 +0.43
CLR
74.73 #0.19
76.30
#0.22
75.51 #0.52
76.62 20.15
LLR
75.01 20.11
76.56
#0.42
75.79
20.33
76.72 20.14
ALLR
75.93 #0.24
76.96 #0.18
76.40 #0.34
76.72 #0.27
Model sparsity 35%
Budget:
2 1%
2.5%
x 1%
2.5%
FT
65.35 #0.27
69.53 +0.19
67.29 =0.84
70.40 #0.71
LRW
65.35 #0.27
69.53 +0.19
67.29 #0.84
70.40 #0.71
SLR
73.04 #0.23
74.77
40.33
73.89
#0.56
74.90 #0.24
CLR
74.42 =0.27
76.06
#0.26
75.14
#0.43
76.28
#0.17
LLR
74.57
#0.01
76.25
#0.25
75.38
#0.31
76.47
#0.20
ALLR
75.62 #0.22
76.61 +0.28
76.17 #0.35
76.59 40.13
Model sparsity 40%
Budget:
2 x 1%
2 x 2.5%
3 x 1%
3 x 2.5%
FT
60.97 #0.11
66.62 +0.23
63.67 #0.76
67.77 +0.74
LRW
60.97 20.11
66.62 20.23
63.67 20.76
67.77 20.74
SLR
72.48 =0.09
74.51 +0.20
73.50 20.53
74.66 10.31
CLR
73.89 =0.28
75.70
+0.23
74.94 +0.55
76.17
40.31
LLR
74.29 #0.18
75.91 +0.28
75.19 +0.54
76.26
10.17
ALLR
74.99 #0.14
76.38 #0.33
75.68 #0.41
76.39 #0.41
Model sparsity 45%
Budget:
2 1%
2 2.5%
3 x 1%
3 2.5%
FT
55.23 +0.57
63.10 40.62
58.66 +1.45
64.53 +1.30
LRW
55.23 #0.57
63.10 +0.62
58.66 +1.45
64.53 +1.30
SLR
71.81 =0.19
74.01 +0.18
72.85 #0.44
74.41
+0.16
CLR
73.44 #0.27
75.37
#0.22
74.43
+0.49
75.71
#0.14
LLR
73.66
#0.15
75.52
#0.25
74.61 #0.53
75.84
#0.27
ALLR
74.40 #0.20
75.90 +0.29
75.07 #0.32
76.01 +0.40
Model sparsity 50%
Budget:
2 1%
2.5%
3 x 1%
2.5%
FT
48.40 +1.29
58.29 +0.95
53.10 +1.34
60.36 41.41
LRW
48.40 +1.29
58.29 40.95
53.10 +1.34
60.36
+1.41
SLR
71.08 #0.27
73.30
+0.30
72.30 20.58
73,88 40.19
CLR
72.74 +0.02
74.74
+0.24
73.83 +0.66
75.61 +0.42
LLR
73.05
#0.06
75.01 40.35
74.21 +0.46
75.55 +0.14
ALLR
73.77 #0.18
75.54 +0.29
74.54 #0.37
75.53 +0.37
34

---

Published as a conference paper at ICLR 2023
B.2 BUDGETING THE RETRAINING PHASE
Similarly to Figure 3 in the main part, Figure 4 and Figure display the envelope when retraining with LLR and ALLR, respectively. Here, the weight decay values, including those used for the baseline, were individually tuned for each datapoint using grid search over le-4, 2e-4 and 5e-4. All results are averaged over two seeds with max max--iin-bande indicated.
Although LLR and ALLR show slightly different behaviour, we note that both reach the performance of the baseline with significantly less retraining epochs than required to establish that baseline.
0.0%
tost accurroy devistion
-0.3%
-1.0%
-1.3%
Baseline (Renda al.)
Spansity 90.0%
-20%
Sparsity 95.0%
Sparsity 98.0%
-23%
0
100
200
300
total retraining epochs
Figure ResNet-56 on CIFAR-10: Envelope of the performance of IMP using LLR compared the baseline of Renda et al. (2020) shown over the total number of epochs used for retraining, Results are averaged over two seeds with max-min-bands indicated.
0.5%
tost acarzay deviation
0.0%
-0.5%
-1.0%
-1.5%
Baseline (Renda et al.)
Sparsity 90.0%
-2.0%
Spansity 95.0%
Spanity 98. 98.0%
-23%
retraining opochs
Figure 5: ResNet-56 on CIFAR-10: Envelope of the performance of IMP using ALLR compared to the baseline Renda al. (2020) shown over the total number epochs used retraining. Results are averaged over two seeds with max-min-bands indicated.
35

---

Published as a conference paper at ICLR 2023
B.3 BUDGETING THE INITIAL TRAINING
Figure and Figure show the behaviour of One Shot as well as iterative pruning when budgeting the initial training, where retraining performed with LLR and ALLR, respectively. Here, the initial training is budgeted between 5% up to 100% of 200 epochs, which we consider the 'full' training. The initial training follows a linearly decaying learning rate schedule which starts from 0.1. After initial training, IMP is applied One Shot (retraining for 30 epochs) or iteratively (3 cycles of 15 epochs each). The individual datapoints are given by the length of the initial training, namely 10, 25, 50, 75, 100, 125, 150, 175 and 200 training epochs. We keep weight decay fixed at 5e-4.
0.0%
test ROCHLION deviation
-1.0%
-2.0%
+3.0%
training baseline
Sparsity 90%
-1.0%
Sparsity 95%
-5.0%
Sparsity 98%
20%
40%
60%
80%
100%
fraction of originsl training
0.0%
test accurscy devintion
-1.0%
-1.3%
Fall training buseline
-2.0%
Spursity 90%
Sparsity
-2.5%
Sparsity 98%
40%
60%
80%
100%
fraction of original training
Figure 6: ResNet-56 on CIFAR-10: Test accuracy achieved by IMP when retraining One Shot for 30 epochs (above) iteratively for three cycles of 15 epochs each (below) with LLR after budgeting the initial training length. Each line depicts a different goal sparsity and values are indicated as deviation from the performance of IMP when applied to a network trained with the full budget. Results are averaged over two seeds with max-min-banm indicated.
36

---

Published as a conference paper at ICLR 2023
0.0%
test aecurayy deviation
-2.5%
-7.5%
-10.0%
Full training baseline
Sparsity 90%
-12.5%
Spansity 90%
Sparsity 98%
-15.0%
60%
100%
fraction original training
0.3%%
0.0%%
test accurnyy deviation
-0.5%
-1.0%
-1.3%
training baseline
-2.0%
Sparsity 90%
-2.3%
Sparsity 95%
Sparsity 98%
-3.0%
20%
40%
60%
80%
100%
fraction of original training
Test accuracy achieved by IMP when retraining One Shot for 30 epochs (above) or iteratively for three cycles of 15 epochs each (below) with ALLR after budgeting the initial training length. Each line depicts different goal sparsity and values are indicated as deviation from the performance of IMP when applied to network trained with the full budget. Results are averaged over two seeds with max-min-bands indicated.
Figure 7: ResNet-56 on CIFAR-10:
B.4 COMPARISON TO STATE-OF-THE-ART PRUNING-STABLE METHODS
Table 22 lists all methods that take part in the comparative study between BIMP and state-of-the-art pruning-stable methods (cf. Section 3.3), where we added IMP as a method of reference. In the following subsections, we list the exact hyperparameter grids used for each pruning-stable method.
B.4.1 RESNET-56 CIFAR-10
For each method (including BIMP) we tune weight decay over Ic-4, 5c-4, 1e-3 and 5e-3 and keep momentum fixed 0.9. Since the learning rate schedule might need additional tuning, we vary the initial learning rate between 0.05, 0515, 0.1 and 0.2 for all methods except CS. The decay of the schedule follows the same pattern as listed in Table 3. Since CS required the broadest grid, we fixed the learning rate schedule to the one in Table 3. Otherwise, we used the following grids.
BIMP
Initial training budget epochs: {20, 60, 100). Number of pruning phases of equal length: {1, 2)
GMP
Equally distributed pruning steps: (20, 1003.
GSM
Momentum: (0.9, 0.95, 0.99).
37

---

Published as a conference paper at ICLR 2023
Table 22: Overview of sparsification methods. CS, STR and DST control the sparsity implicitly via additional hyperparameters. IMP is the only method that is pruning instable by design, i.e., it loses its performance right after the ultimate pruning. Further, IMP is the only method that sparsity agnostic throughout the regular training; the sparsity does not play role while training to convergence. All other methods require training an entire model when changing goal sparsity.
Sparsity specifiable
Pruning stable
Sparsity agnostic training
IMP
BIMP
GMP
GSM
LC
X
DPF
X
DNW
CS
X
STR
X
DST
X
X
LC
We only tune the weight decay.
DPF
As for GMP, tune the number of pruning steps, i.e., {20, 100), and the weight decay.
DNW
We only tune the weight decay, since there are additional hyperparameters.
CS
As recommended by Savarese al. (2020), fix the temperature 8 at 300. We tune the mask ini- tialization 80 over {-0.3, -0.25, -..2 -0.1, -0.05, 0, 0.05, 0.1, 0.2, 0.25, 0.3) and the & penalty À over {1e-8, 1e-7).
STR
We tune the initial threshold value Sinir (-100, -50, -5, -2, -,, 50, 100). In extended grid search, we also used weight decays in (5e-05, le-4) and varied Sinit E { (-40, -30, 200 - 10).
DST
We tune the sparsity-controlling regularization parameter a E (5e-6, le-5, 5e-5, le-4, 5e-4). In an extended grid search, used weight decays in {0, le-4) and tuned over {le-7, 5e-7, le-6).
B.4.2 WIDERESNET ON CIFAR-100
For each method (including BIMP) we tune weight decay over le-4, 2e-4 and 5e-4 and keep momentum fixed at 0.9. We vary the initial learning rate between 0.05, 0.1 and 0.15 for all methods and the learning rate schedule of Table but additionally include linear schedule with initial learning rate value set to O.1. Otherwise, we used the following grids.
BIMP
Initial training budget epochs: {20, 60, 100}. Number of pruming phases of equal length: {1, 3.
GMP
Equally distributed pruning steps: (20, 100).
GSM
Momentum: (0.9, 0.95, 0.99).
38

---

Published as a conference paper at ICLR 2023
LC
We only tune the weight decay.
DPF
Equally distributed pruning steps: (20, 100).
DNW
We only tune the weight decay, since there are no additional hyperparameters.
CS
As recommended by Savarese et al. (2020), we fixed the temperature 3 at 300, but increased it to 500 upon noticing the pruning instability of CS on this dataset. We tune the mask initialization Sy {-0.3, -0.25, -0.2, -0.1, -0.05, -0.03, -0.01, -0.005, -0.003, - 0.000, 0} and the ε penalty over (1e-9, le-8, le-7].
STR
We tune the initial threshold value Sinit E { -30000000000, 1000,, -500). In extended grid search, we also varied Sinit E {-200, - -500, -100, -0, -10,0, 0}.
DST
We tuned a over {5e-6, le-5, 5e-5, le-4, Se-4). In an extended grid search, we tuned a over {le-6, 3e-6, 8c-6, 3e-5, 8c-5, 2c-4, 3e-4, 4e-4, 5e-4, 6e-4, 7c-4, 8c-4, 9c-4, le-3).
B.4.3 RESNET-50 ON IMAGENET
If not otherwise specified, we fix the weight decay at le-4 and momentum at 0.9. For all methods, we use the learning rate schedule of Table but additionally include a linear schedule with the same initial learning rate value to rule out too strong dependence on the training schedule. Note that for GSM and LC, we choose the best value training from scratch or using a pretrained model. In the latter case, apply the methods for 10, 20 or 40 epochs. Otherwise, used the following grids.
ВIMP
Initial training budget epochs: {60, 75). Number of pruning phases of equal length: {1, 3).
GMP
Equally distributed pruning steps: (5, 9, 18, 45).
GSM
Momentum: {0.9,
Weight decay: (1c-4, le-5].
LC
Weight decay: (le-4, le-5).
DPF
Equally distributed pruning steps: (9, 18}.
DNW
Weight decay: (le-4, le-5}.
STR
Weight decay: (1c-5, 2c-5, 3c-5, 4c-5, le-4).
8ma. 20, 15, -13, 100 -5, 4, -22. 2].
DST
Weight decay: (1c-4, le-57.
o: (le-7, 5e-7, le-6, 2e-6, 5e-6, 8e-6, le-5, le-4)
39

---

Published as a conference paper at ICLR 2023
B.4.4 RESULTS
Table 23 and Table 24 show the full results when comparing BIMP pruning-stable methods in the case of ResNet-56 on CIFAR-10 and WideResNet on CIFAR-100, respectively.
Table 23; ResNet-556 on CIFAR-10. Results of the comparison between BIMP and pruning stable
methods for the sparsity range between 90% and 99.5% The columns are structured as follows: First the method is stated. Secondly, we denote the images-per-second throughput through training, i.e., a higher number indicates a faster algorithm. The following columns are substructured as follows: Each column corresponds to goal sparsity and each subcolumn denotes the Top-1 accuracy, the theoretical speedup and the actual sparsity reached. All results include standard deviations. Missing values (indicated by -) correspond to cases where were unable to obtain results in the desired sparsity range, i.e., there did not exist training configuration with average final sparsity within .25% interval around the goal sparsity and the closest one is too far away or belongs another column.
Model sparsity 90%
Model sparsity 93%
Method
# img/s
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
BIMP
3638
93.35 #0.13
7 #0.2
90.00 +0.00
93.00 #0.26
10 =0.8
93.00 #0.00
GMP
3536
92.87 +0.13
90.0
90.00
92.47 20.33
=0.0
93.00
GSM
3251
91.27 #0.69
11 #2.5
90.00 20.00
91.12 +1.07
13,5
93.00 #0.00
DPF
3560
93.32 20.11
7 +0.0
90.00 #0.00
93.23 +0.05
9 20.5
93,00 =0.00
DNW
3335
91.81 +1.33
60.88
90.00 +0.00
91.97 20.20
50.2
93.09 20.00
LC
3467
90.51 #0.16
#0.1
90.00 +0.00
89.67 #0.55
#0.6
93.00 #0.00
STR
2864
89.25 #1.23
#0.8
90.15 #0.76
90.27 20.86
20 #4.0
93.01 +0.58
CS
2725
91.87
+0.30
13 +0.3
90.52 +0.76
89.64 +1.43
#2.9
92.78 #0.18
DST
1972
92.41
+0.28
#0.7
89.55 #0.41
Model sparsity 95%
Model sparsity 98 %
Method
# img/s
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
BIMP
3638
92.57 #0.32
12 40.8
95.00 +0.00
90.43 #0.11
26 +4.9
98.00 40.00
GMP
3536
92.20 +0.07
18 +0.0
95.00 +0.00
90.08 #0.32
43 40.0
98.00 +0.00
GSM
3251
90.07 +1.67
21 +5.0
95.00
87.21 +1.24
44 +10.5
98.00 40.00
DPF
3560
92.68 #0.14
12.0..1
95.00 #0.00
90.49 +0.23
29
98.00 +0.00
DNW
3335
91.95 #0.06
70.3
95.09 +0.00
34.87 #43.08
26
98.10 18.10-a =0.00
LC
3467
89.16 #0.60
80.5
95.00 #0.00
85.11 #0.51
16.00
98.00 =0.00
STR
2864
89.77 #1.75
110.
95.11 #0.28
89.15 #0.26
66.
98.00
CS
2725
91.36 #0.23
21 42.9
95.38 40.19
90.04 40.36
50 47.2
98. 12 #0.06
DST
1972
89.17 #0.00
18 40.0
94.42 +0.00
88.22 40.36
53 3.6
98.04 #0.21
Model sparsity 99%
Model sparsity 99.5%
Method
# img/s
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
ВIMP
3638
87.17 #0.59
+7.2
99.00 +0.00
82.02 =0.07
91 #12.8
99.50 #0.00
GMP
3536
86.72 #0.04
77 +0.0
99.00 +0.00
80.26 +0.80
127 #0.0
99.50 +0.00
GSM
3251
83.00 +0.90
77 +18.9
99.00 +0.00
76.52 +1.82
156 141.4
99.50 +0.00
DPF
3560
86.76 +0.33
+3.7
99.00 +0.00
80.03 20.64
146 =34.3
99.50
DNW
3335
83.67 #0.24
15 +0.1
99.17 20.00
34.71 +24.17
34.4.
99.67 #0.00
LC
3467
81.63 #0.74
30 +1.5
99.00 +0.00
74.44 +2.03
64 #4.5
99.50 +0.00
STR
2864
83.68 #0.94
159 431.9
99.13 #0.02
77.34 +2.68
420 #167.4
99.66 #0.09
CS
2725
86.55 #0.92
69 47.1
98.90 +0.02
DST
1972
86.99 +0.00
63 40.0
98.36 40.00
40

---

Published as a conference paper at ICLR 2023
Table 24: WideResNet on CIFAR-I00: Results the comparison between BIMP and pruning stable methods for the sparsity range between 90% and 98%. The columns are structured as follows: First the method is stated. Secondly, we denote the images-per-second throughput through training, i.e., higher number indicates faster algorithm. The following columns substructured as follows: Each column corresponds to one goal sparsity and each subcolumn denotes the Top-1 I accuracy, the theoretical speedup and the actual sparsity reached. All results include standard deviations.
Model sparsity 90%
Model sparsity 93 %
Method
img/s
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
BIMP
2318
76.10 #0.00
#0.1
90.00 +0.00
75.92 =0.42
12 #0.2
93.00 #0.00
GMP
2278
76.09 =0.29
10.1
90.00 +0.00
74.91 10.16
9 =0.2
93.00 =0.00
GSM
2190
74.25 =0.31
70.00
90.00 #0.00
73.88 #1.19
40.0
93.00 #0.00
DPF
2299
75.44 #0.44
7 700.8
90.00 +0.00
74.78 +0.52
11
93.00 #0.00
DNW
258
76.84 20.72
70.1
90.00 +0.00
75.75 20.10
8 =0.1
93.00 +0.00
LC
2207
72.82 20.57
50.01
90.00 #0.00
68.87 20.13
6
93.00 +0.00
STR
2148
73.06 #1.22
+0.0
90.97 +0.00
74.11 +0.21
13 #0.2
92.36 #0.17
CS
2077
73.50 20.47
20.0
90.82 #0.09
73.52
40.21
10 20.0
92.96 20.03
DST
1868
72.84
20.42
10.1
90.00 +0.00
72.16 40.32
20 #0.3
93.00 +0.00
Model sparsity 95%
Model sparsity 98 %
Method
img/s
Accuracy
Speedup
Sparsity
Accuracy
Speedup
Sparsity
BIMP
2318
75.64 20.29
16 =0.2
95.00 20.00
73.73 +0.20
39 #0.3
98.00 =0.00
GMP
2278
74.80 20.60
20 20.0
95.00 40.00
72.07 40.03
24 #0.1
98.00 20.00
GSM
2190
73.25 #0.52
10.0.1
95.00 #0.00
68.79 +0.58
18 20.0
98.00 #0.00
DPF
2299
73.87 #0.69
+0.6
95.00 #0.00
72.18 #0.29
20.9
98.00 =0.00
DNW
258
74.94 +0.25
9 #0.2
95.00 #0.00
72.13 #0.12
17 #0.5
98.00 #0.00
LC
2207
60.29 #0.75
80.2
95.00 #0.00
29.30 +0.20
17 10.5
98.00 #0.00
STR
2148
70.66 #0.52
24
94.40 #0.02
65.17 40.64
22 20.1
98.50 #0.03
CS
2077
72.81
20.13
12 10.0
95.22 s0.00
72.29
20.34
28 20.5
97.99 #0.00
DST
1868
70.66 +0.33
245
95.00 +0.00
68.46 +0.00
#0.0
97.96 40.00
41

---

Published as a conference paper at ICLR 2023
C ABLATION STUDIES
C.I ALLR: THE INITIAL VALUE OF THE LEARNING RATE SCHEDULE
We analyze the impact of our choices regarding the design of ALLR. First of all, we justify the usage of the proxy to determine the initial learning rate. Recall that ALLR discounts the initial value vn by a factor d E [o. 1] to account for both the available retraining time (similar to LRW) and the actual increase in loss induced by pruning. It does so by choosing d = max(d), d2), where
dj = IW - 1 2 [W V
E [0,
(2)
measures the relative I2-norm change in the weights due to pruning and d2 = T,t/TT accounts for the length of the retrain phase in comparison to the original training length.
The motivation behind choosing these factors lies in handling different retraining scenarios. When choosing the initial value, two aspect have to taken into account.
1. The number of retraining epochs available, T,,, might be very limited. A large initial learning rate might yield too large oscillations of the loss, from which we cannot recover in too short retraining timeframe. To find highly generalizing minima, a large-step and a small-step retraining regime (Jastrzębski et al., 2017; Li et al., 2019; You et al., 2019; Leclere& & Madry, 2020). If Tf, v is too small, we do not have enough time to do both. This is the advantage of LRW: The magnitude of the initial learning rates depends directly on T₃+-
need both phases:
2. The pruning-induced decrease in accuracy might be very small, depending on the fraction of weights we remove. Pruning only a small fraction of the weights will most likely have little impact on the loss: in a highly paraaatttatte network, removing a small fraction of the weights will not drive the parameters far away from the current (local) optimum. In that case, convergence is accelerated by performing small learning rate steps. We expect that especially in the higher sparsity regime, where the loss increases dramatically by pruning, a larger initial learning rate is desirable to being able to compensate the drop in accuracy and approach the optimum faster.
In other words, we seek to choose the learning rate to initially be as large as possible (but bounded by the largest learning rate used throughout training), where as large as possible means taking two factors into account: How much increase in loss do we have to compensate and do we have enough time to properly perform both a large-step regime and a small-step regime?
The metric dy is an immediate proxy to the duration of retraining phase, motivated by LRW. Note that we clip d2 to not become larger than 1. When we are allowed to retrain at least as long as the initial training, we prefer the initial value of the original training, which is the maximum value in the problem setting. In any other case, we take a fraction of it. However, regarding the metric d1, we choose to measure the drop in I--norm induced by pruning instead of measuring the actual drop in accuracy, since the latter is only available after performing an entire forward pass on the train dataset.
To investigate whether this replacement is justified, we compare ALLR to an accuracy drop based variant of it, namely AccALLR, AccALLR works exactly like ALLR, but we instead choose di as follows:
di Acc($a) Acc(dss) Acc(
E {0,0,1 1].
where Acc($) denotes the train accuracy of model + and E.. t. denote the dense and sparse model, respectively. We clip dj between O and 1. The metric d1 measures the relative drop in accuracy compared to the dense model. It hence requires a complete forward pass of the sparse model on the training data before commencing retraining. For CIFAR-10, Table 25 shows that despite AccALLR having a slight advantage over the inssoienf ALLR, these often lie within the margin of the standard deviation. The discounting factor of ALLR taking the norm-drop into account is indeed a fell-functioning proxy for the pruning-indin induced drop in accuracy. Similarly, Table 26 shows the comparison on ImageNet in the iterative setting. Especially for moderate sparsities such as 70%
42

---

Published as a conference paper at ICLR 2023
that the accuracy is more precise metric when determining the initial learning rate, however it becomes unprecise at higher sparsities. High distortions of the parameters will lead to the accuracy dropping entirely to that of random classifier, resulting in the largest possible learning rate being taken as the initial value. ALLR gives a more robust estimate.
Table 25: ResNet-56 CIFAR-10 (One Shot): Performance of ALLR versus its accuracy-drop based variant AccALLR for IMP in the One Shot setting for target sparsity of 80%, 90%, 98% and a retrain time of 2% (2 epochs), 5% (5 epochs), 20% (20 epochs) of the initial training budget. The first and second best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 80%
Budget:
2%
5%
20%
ALLR
90.85 20.66
91.40 +1.02
91.96 #0.70
AccALLR
90.92 #1.17
91.27 #1.24
91.71 #1.02
Model sparsity 90%
Budget:
2%
5%
20%
ALLR
89.56 #0.55
90.16
+..24
90.87 #1.32
AccALLR
89.75 #1.36
90.34 +1.23
90.89 #1.21
Model sparsity 98%
Budget:
2%
5%
20%
ALLR
80.47 #1.09
82.60 #1.48
84.75 #1.24
AccALLR
80.11 #1.23
82.48 +1.72
84.89 #1.08
Table 26: ResNet-50 on ImageNet (Iterative): Performance of ALLR versus its accuracy-drop based variant AccALLR for IMP in the iterative setting for target sparsity of 70% and 90%. The first and second best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 70%
Budget: 2 2.22%
2 3.33%
3 x 2.22%
3 3.33%
ALLR
74.19 +0.03
74.60 #0.11
73.80 #0.01
74.71 #0.06
AccALLR
74.64 #0.21
75.03 #0.21
74.94 +0.09
75.23 #0.24
Model sparsity 90%
Budget: 2 2.22%
3.33%
3 x 2.22%
3 3.33%
ALLR
70.08 #0.03
71.77 40.18
69.96 +0.21
72.36 #0.19
AccALLR
69.19 +0.18
70.51 20.11
70.54 #0.15
72.28 +0.09
We further show the impact of the two factors dj and d2 of ALLR by considering all four cases of selectively disabling a subset of the metrics, that is, we compare ALLR to ALLRdI (ALLR only using d,), ALLRd2 (ALLR only using d2) and LLR (which is the same as applying no discounting factor at all) when training 56 on CIFAR-10. Table displays the performance of the four different variations when testing against the different scenarios as outlined above, i.e., with low to high sparsity (80-98%) and short to long retraining time (2-20% budget), where we stick to the One Shot setting. First of all, we observe ALLR consistenly performs best or second best among all variants. Note that the discounting factor d = max d2) of ALLR is attained at either dj or dy and ALLR always matches the performance of the better d1- or dy-disabling variant. Further, behaves exactly as designed and addresses the different retraining scenarios as outlined above. In the low sparsity, short retraining regime, both ALLR variants disabling one of the two discounting
observe that ALLR
43

---

Published as a conference paper at ICLR 2023
factor perform equally well and are superior than selecting large initial learning rate as LLR does. When the retraining time is increased, larger initial learning rates are more suited despite low sparsity, visible in the marginalized difference to LLR. With further decreasing sparsity and increasing retraining time, the effect of ALLRd1 would vanish to the benefit of ALLRd2. On the other hand, in and it is
the high sparsity regime, notice that it is beneficial start with high initial learning not sufficient to only include the length of the retraining time, as ALLRd2 shows with a difference of ten percent in accuracy its competitors. LRW would behave similarly in this particular case. Overall, ALLR addresses these issues by accounting both for the increase in loss as well as the available retraining time.
Table 27: ResNet-56 on CIFAR-10 (One Shot): Performance of the ALLR derivations for IMP in the One Shot setting for target sparsity of 90%, 98% and retrain time of 2% (2 epochs), 5% (5 epochs), 20% (20 epochs) of the initial training budget. The first, second, and third best values for the translation schemes are highlighted. Results are averaged over two seeds with the standard deviation indicated.
Model sparsity 80
Budget:
2%
5%
20%
ALLR
90.92 20.81
91.40
#1.07
91.82 =0.87
ALLRd1
90.73 +0.71
91.23 #1.19
91.94
+0.85
ALLRd2
90.78 +1.16
91.28 #1.19
91.66 #0.70
LLR
90.07 #0.72
90.84 #1.29
91.76 #0.95
Model sparsity 90%
Budget:
2%
5%
20%
ALLR
89.75 +0.89
90.20 +1.08
91.03 +1.15
ALLRd1
89.66 +1.18
90.10
#1.03
90.72 #1 +1.56
ALLRd2
87.94 +1.07
89.70 #1.29
90.85 #1.11
LLR
88.68 #1.12
89.89 #1.29
90.72 #1.17
Model sparsity 98%
Budget:
2%
5%
20%
ALLR
80.11 +1.28
82.59
+1.60
84.89 +1.2
ALLRd1
80.16 +1.19
82,51
#1.75
84.84 #1.04
ALLRd2
70.05 #0.31
79.04 #0.62
85.12 #1.08
LLR
79.75 +1.18
82.43 #1.60
84.55 +1.48
44

---

Published as a conference paper at ICLR 2023
C.2 PRUNING SELECTION CRITERIA
We compare the original GLOBAL pruning criterion of IMP to the previously introduced proposed alternatives. In the case of ResNet-56 on CIFAR-10 (Figure 8) and VGG-16 on CIFAR-10 (Figure 11), we report the weight decay config with highest accuracy, where we optimized over the values le-4, 5e-4 and 1e-3. For WideResNet on CIFAR- 100 (Figure 9) and ResNet-50 on ImageNet (Figure 10) we relied on a weight decay value of lc-4 for both architectures. The CIFAR- 10 and CIFAR-100 results are averaged over three seeds and max- max-min-bande are indicated. For ImageNet, the results are based on single seed.
We tested both FT (Han et al., 2015) and SLR (Le Hua, 2021) to see whether the learning rate scheme during retraining has any impact on the performance of the pruning selection scheme. Surprisingly the simple global selection criterion performs at least on par with the best out of all tested methods at any sparsity level for every combination of dataset and architecture tested here when considering the sparsity of the pruned network as the relevant measure. Using SLR during retraining compresses the results by equalizing performance, but otherwise does not change the overall picture. We note that the results on CIFAR-100 using FT largely track with those reported by Lee et al. (2020), with the exception of the strong performance of the global selection criterion. Apart from slightly different network architectures, we note that they used significantly more retraining epochs, e.g.. 100 instead of 30, and that they use AdamW (Loshchilov & Hutter, 2019) instead of SGD. Comparing the impact different optimizers can have on the pruning selection schemes seems like a potentially interesting direction for future research.
While the spursity-ssperrfrmmacc performance tradeoff has certainly been an important part of the justification of modifications to global selection criterion, let us also directly address two further points that are commonly made in this context. First, the global selection criterion has previously been reported to suffer from a pruning-indunn induced collapse at very high levels of sparsity in certain network architectures that is avoided by other approaches. This phenomenon has been studied in the pruning before truining literature and was coined layer-collapse by Tanaka et al. (2020), who hypothesize that it can be avoided by using smaller pruning steps since gradient descent restabilizes the network after pruning by following a layer-wise magnitude conservation law. To verify whether these observations also hold in the pruning after training setting, we trained a VGG-16 network on CIFAR-10 10, as also done by Lee et al. (2020), both in the One Shot and in the iterative setting. The results are reported in Figure 11 and show that layer collapse is clearly occurring for both FT and SLR for the global selection criterion at sparsity levels above 99% in the One Shot setting, but disappears entirely when pruning iteratively. This indicates that layer collapse, while a genuine potential issue, can be avoided even using the global selection criterion. We also remark that SLR needs less prune-retrain ccclles to avoid layer-collapse than FT, possibly indicating that the retraining strategy impacts the speed of restabilization of the network in the hypothesis posed by Tanaka et al. (2020).
The second important aspect to consider is that layer-dependent selection criteria are also intended to address the inherent tradeoff not just between the achieved sparsity of the pruned network and its performance, but also the theoretical computational speedup. Figure 8, Figure and Figure 10 include plots highlighting the achieved performance in relation to the theoretical speedup. The key takeaway here is that for both the ResNet-56 and the WideResNNt network architecture, there is overall surprisingly little distinction between all five tested methods, with Uniform+ and ERK taking the lead and the global selection criterion performing well to average. For the ResNet-50 architecture however much more drastic separation occurs, with Uniform performing the best, followed by Uniform+ and then the global selection criterion. Overall, the picture is significantly less clear. However, despite its simplicity, the global approach performs on par with respect to managing the accuracy vs. speedup tradeoff, where we observe that for ResNet-50 on ImageNet it even outperforms methods such as LAMP and ERK regarding both objectives, performance and speedup.
45

---

Published as a conference paper at ICLR 2023
Fine Tuning
Scaled Learning Rote Restarting
80%
Teet arrraay
805
Clobal
Unikers (2lra Cupes, 2017)
Unifers (Gala
405
ERK (Evd d.,, 2019)
46%
LAMP 2000;
94%
D006
Imoune teights praned
smoune wigghte praned
Fine Tuning (PT)
Scaled Learning Rate Restarting (SLR)
807
80%
Text 00-10444
606
Global
Unifors (Zlia Gupca, 2017)
60%
Unikorz (Gale ci al. 2019)
105
(Exci 2019)
LAMI (Lee et al., 2020)
41%
10'
11P
10'
speedtip
thocential spowdap
Figure 8: ResNet-56 CIFAR-10 (One Shot): Sparsity-vs. Searsity-vs.-perfooee performacce (above) and theoretical speedup-vs. perrpopo-sseeeroommaeeeee (below) tradeoffs in the One Shot setting with FT (left) and SLR (right) as retraining methods. Retraining is done for 30 epochs. The plot includes max-min confidence intervals.
Fine Tuning [PT]
Scaled Learning, Rate Restarting
606
text uu1ear)
60%
105
80%
Unifora (2hrs Uupea, 2017)
2064
40%
Uniforma (Gale
BRK (Evd dl, 2019)
33%
LAMP 2000)
06C
Emour eoightts praned
wsights praned
Fine Tuning (PT)
Scaled Learning Rate Restarting (SLR)
6073
teet anrumay
106
2063
40%
339
10'
102
theceetical speedip
thocentical speedup
Figure 9: WideResNet CIFAR-100 (One Shot): Sparsity-vs. vaarsitt-vs.-erfommaae (above) and theoretical specdup-vs.performance performance (below) tradeoffs in the One Shot setting with FT (left) and SLR (right) retraining methods. Retraining is done for 30 epochs. The plot includes max-min confidence intervals.
46

---

Published as a conference paper at ICLR 2023
[FT)
Scaled Learning
Bestarting
71%
70%
te2 Areiras
72%
806
70%
Uniferm (Zlra Gupes, 2017)
30%
Unifee
ERK (Eni sh., 2019)
106
8.%
95%
Euoous of witalts proued
tuoous wxigltt praned
Fine Tuning (PT)
Scaled Learning Rate Restarting (SLR)
707
606
605
61%
speedii
thocectical speedip
Figure 10: ResNet-50 ImageNet (One Shot): ceeesstyyvs--erformaan (above) and theoretical speedup-vs. perfforereee (below) tradeoff's in the One Shot setting with FT (left) and SLR (right) as retraining methods. Retraining is done for 10 epochs.
Fine Tusing (PT)
Scaled Learinng Rate Restarting (SLR)
906
teek ar110347
806
Global
Gupta, 2017)
* Unikeras (Gaze ol. 2010)
706
sl., 2019)
LAMP (Lee al., 2000)
5025
0695
50.8
52%
94%
50%
teights
prined
Fine Tuming (M'))
Scaled Learning Rate Restarting (SUR)
9096
90%
way
80%
Global
307
Uniform (Zlrs Gupes,
Unikerss (Gale et ol. 2019)
288
BRK dl,,
2076
80R
LAMP fLee ei al., 2020)
14755
0125
94%
5076
14775
10X
M236
91%
50%
weighte
Figure 11: VGG-16 on CIFAR-10: Berformance-vs--sparit tradeoff's in the One Shot (above) and Iterative (below) setting with FT (left) and SLR (right) as retraining methods. the One Shot setting the model is retrained for 30 epochs after pruning and the iterative setting consists of 3 prune-retrain cycles with 10 epochs each. For One Shot we observe layer-collapse while the iterative splitting into less severe pruning steps avoids the problem. Note that the total amount of retraining epochs between the two settings is identical here.
47

---

Published as a conference paper at ICLR 2023
C.3 COMPARING BIMP TO GMP
We note that BIMP has fair number of similarities to GMP (Zhu & Gupta, 2017) and we will therefore seek the direct comparison between the two. In particular, both methods effectively prune and retrain (although that terminology is commonly reserved to IMP) at distinct predetermined points during the overall training process. Let us start by highlighting the particular design decisions by which BIMP and GMP differ:
1. BIMP 'hard' prunes while GMP 'soft' prunes. In GMP, pruned weights are zeroed out during the forward and backward pass, but the mask is recomputed at the next pruning step, hence allowing for previously pruned weights to recover. However, it is unclear for example how the momentum buffer of SGD or the memory of optimizers like Adam (Kingma & Ba, 2014) are supposed to take this soft pruning into consideration, but we believe that this aspect overall contributes very little to explain the different performance of GMP and BIMP.
2. BIMP requires sufficiently long retraining time between two pruning points to recover from the last pruning step. GMP on the other hand can have much shorter distances between the equally distributed pruning points with as little as pruning every 100 training iterations (Zhu & &Gupta, 2017).
3. Both BIMP and GMP rely on magnitude pruning, however we have decided to use the simple GLOBAL selection criterion of Han al. (2015) for both while GMP was originally proposed using the UNIFORM selection and further improved by Gale et al. (2019) with the UNIFORM+ criterion, see Appendix C.2 for a discussion. The version of GMP included in the main part of this text in fact relies on the same global criterion as BIMP, since we found it to yield better results than UNIFORM+ with respect to the final accuracy.
4. BIMP relies on a cyclic linear learning rate schedule where the cycles coincide with the points at which the network is pruned, while GMP can use any kind of learning rate scheme that would commonly be employed for that kind of architecture and data set, i.c., normally not a cyclic one. Zhu & Gupta (2017) note that GMP can be quite sensitive to the learning rate. In the main body of the text we have mostly relied on common stepped learning rate schedule. Given the importance of the learning rate schedule, we have also tested version of GMP using both linearly decaying schedule, as well as a cyclic learning rate schedule similar to what we have suggested for BIMP, i.e., the cycles are chosen to exactly end at the next pruning step.
5. We have relied on the simple exponential pruning schedule suggested by Renda et al. (2020) for BIMP while GMP relies on a particular schedule defined by a cubic polynomial that effectively leads to pruning larger amounts initially and progressively smaller amounts later in training when compared to BIMP. While we think that the pruning schedule probably has a significant impact the performance of the pruming method and can possibly interact with the learning schedule in particularly interesting ways, we have so far not attempted exchanging the schedule in BIMP for either the one of GMP or novel one.
In Table 28 we have included the previously mentioned modifications of GMP and compared them to BIMP for WideResNet trained on CIFAR-100, In particular, for each variant of GMP we indicate which learning rate schedule we use and whether we prune 'hard' or 'soft'. The models are trained according the same settings as indicated in Table 3, where stepped refers to the stepped learning rate case. On the other hand, linear indicates a linear learning rate schedule and cyclic refers to a linearly decaying learning rate schedule that restarted after every pruning point. We use a weight decay value of le-4 and set the initial value of the learning rate to 0. For GMP we prune every 5 or 10 epochs, while for BIMP we fix the initial training length to a 100 epochs and split the remaining 100 epochs equally between to 4 cycles. All results are averaged over two seeds with standard deviations indicated.
We note that there seems to be surprisingly little difference between hard and soft pruning. The impact of the learning rate schedule is more nuanced: using a linearly decaying schedule throughout training can give slight increase in performance, albeit the classical stepped learning rate schedule seems to work better in the high sparsity regime. Whether the cyclic restarting of the learning rate works crucially depends on the distance between two pruning points and the impact of pruning, which BIMP with ALLR seems to leverage. For the medium sparsity of 90% a cyclical learning rate seems to be detrimental, while in the high sparsity regime we see improvements.
48

---

Published as a conference paper at ICLR 2023
Table 29 further reports results for ResNet-50 trained ImageNet, where we sticked to the 'hard' pruning for each GMP variant. Similarly, we use the same settings as in Table 3, set the weight decay to lc-4 and the initial value of the learning rate to 0.1. GMP prunes every 5 or 10 epochs, whereas BIMP leverages an initial training length 60 or 75 epochs, splitting the remaining 30 or 15 epochs into 1 to 4 cycles of equal length. Here, the stepped learning rate schedule seems to be in advantage over linear one, with the slight exception of the highest sparsity. Interestingly, the cyclic learning rate schedule seems to be in conflict with and too aggressive for the sparsification schedule GMP, Overall we think that there is a fair amount of nuance here that deserves further exploration and will probably require more diverse testbed draw any definitive conclusions.
Table 28: WideResNet on CIFAR-100: Comparison between BIMP and GMP variants for goal sparsity levels of 90% and 95%, denoted in the main columns. Each subcolumn denotes the Top- accuracy, the theoretical speedup and the actual sparsity achieved by the method. All results are averaged over two seeds and include standard deviations.
Model sparsity 90%
Model sparsity 95%
Method
Accuracy
Speedup
Sparsity
Accuracy
Specdup
Sparsity
GMP (stepped, hard)
75.19 40.25
8 +0.0
90.00
74.62 40.02
15 +0.2
95.00 +0.00
GMP (linear, hard)
75.44 +0.23
8 +0.0
90.00 #0.00
74.40 40.01
16 #0.1
95.00 #0.00
GMP (cyclic, hard)
75.09 #0.25
8 +0.0
90.00 #0.00
74.81 +0.47
16 #0.1
95,00 =0.00
GMP (stepped, soft)
75.32 #0.04
8 +0.0
90.00 #0.00
74.29
#0.32
15 10.2
95.00 #0.00
GMP (linear, soft)
75.52 #0.05
8 +0.1
90.00 +0.00
74.86
+0.34
16 16.0
95.00
GMP (cyclic, soft)
74.79 #0.12
8 +0.0
90.00 #0.00
74.65 10.19
16 #0.0
95.00 #0.00
BIMP
75.77 #0.23
8 #0.1
90.00 +0.00
75.30
+0.34
15 #0.1
95,00 =0.00
Table 29: ResNet-50 on ImageNet: Comparison between BIMP and GMP variants for goal sparsity levels of 70%, 80% and 90%, denoted in the main columns. Each subcolumn denotes the Top- accuracy, the theoretical speedup and the actual sparsity achieved by the method. All results are averaged seeds and include standard deviations.
Model sparsity 70%
Model sparsity 80%
Model sparsity 90%
Method
Accuracy
Speedup
Spanstt
Accuracy
Speedup
Sparsity
Accuracy
Spocdup
Sparsity
GMP (stepped)
74.62 20.08
2000
70.00 10.00
74.19 10.17
+0.0
80.00 10.00
72.74 20.06
7+000
90.00 10.00
GMP (cyelic)
72.91 +0.19
2
70.00 +0.00
72.67 +0.10
4 +0.0
80.00 +0.00
71.76 =0.00
7 7=001 I
90.00 +0.00
GMP (linear)
74.58 20.01
2000
70.00 10.00
73.90 10.04
40.0
80.00 10.00
72.80 20.03
7+0.1
90.00 10.00
BIMP
75.62 =0.02
2
70.00 +0.00
75.08 20.16
+0.0
80.00 +0.00
73.53 =0.05
6++0.0
90.00 +0.00
49


Reviews:
[{"review_id": "d5TBl_GOz5A", "reviewer": "Reviewer_cREN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the retraining steps in iterative magnitude pruning from the \"budgeted training\" perspective. This is used to derive an efficient learning rate schedule for the retraining phase.", "summary_of_the_review": "I think this is a very well-written paper with a solid contribution to the model pruning literature. I was slightly disappointed by the motivation for **A**LLR and would encourage the authors to respond to this during the rebuttal.", "strength_and_weaknesses": "### Strengths\n\n- The paper does a very good job of explaining existing work and positioning itself in that context. This makes the work accesible to readers not familiar with minute details of pruning methods.\n\n- While one could see the proposed technique as only a slight variation of some existing methods, the simple linear step size schedule is well-motivated from the \"budgeted training\" perspective. It is simpler than the schedules used in prior work while performing at least as good, and often better, across a wide range of \"retraining budgets\" and target sparsities.\n\n- The experimental comparison is conducted with great care. The experimental set-up is motivated and explained in detail, including hyperparameter settings and/or tuning protocols. The results are presented in very clearly and their significance is discussed in detail.\n\n- I want to explicitly commend the authors for making clear the limitations of the proposed method and to focus on generating useful insights rather than trying to coin another acronym.\n\n### Weaknesses\n\n- I find the motivation for **A**LLR quite unsatisfying. The relative distance computed in Eq. (1) strikes me as quite a poor proxy for what we are really interested in, which is to quote the paper \"how much of an increase in loss do we have to compensate for\". At the same time, this increase in loss could actually be computed at the expense of an additional validation epoch. In contrast to what's stated in the paper, I don't think that would be a big computational burden. The evaluation could be done on the validation set (or possibly even a subset thereof) and requires only a forward pass. Its cost should therefore be minor compared to the actual retraining for multiple passes over the training set.\n\n- The experiments are restricted to image classification tasks. For breadth, it would have been desirable to include another data modality and/or task.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TznjnC099M7", "reviewer": "Reviewer_Ybn7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of network pruning under a fixed training budget.  Earlier work on network pruning relied on computationally expensive iterative training & pruning regimes.  The authors show that when operating under a fixed training regime, a lot of savings can be found by eschewing complex learning rate schedules for a simple but aggressive linear learning rate schedule.  The authors further propose improvements to the initial choice of learning rate in the schedule, as well as adding more structure to the budget of the dense network training, which together result in real value per unit of computation.  They demonstrate good performance against alternative methods which induce sparsity during training, and challenge the community to reconsider the commonly held tenet that retraining is wasteful. ", "summary_of_the_review": "While the authors convinced me that the addition of budgeting can save IMP, I am less than convinced of the generality of their results to other settings than well-studied vision data sets.  I think that the authors should either extend their set of empirical results to encompass different problems (and different networks), or to reach into theory of generalization papers to try and support their claims for adaptive learning rates and budgeted retraining.", "strength_and_weaknesses": "### Strengths\n- The authors demonstrate a solid understanding of the literature on pruning\n- The figures and tables are quite clear, establishing the benefit of adaptive LLR in certain settings.\n- The experiments, within their original parameters, are extensive enough to demonstrate the advantage of LLR and ALLR over the competing methods.\n\n### Weaknesses\n- There are a number of claims in the paper that do not agree with the published literature.  For instance, in the first paragraph of the introduction, they claim that a heavily pruned model will normally be less performant than its dense (or moderately pruned) counterpart.  But this isn't always true.  For example, Lottery-Ticket hypothesis [works](https://proceedings.mlr.press/v119/frankle20a.html) show that pruning even up to 80 to 90% of the weights of an original network.  I think the authors should either be more careful in their claims, or more precise in their writing.\n- At the end of the introduction, the authors state their results build upon work by Renda et al, Le & HUa by proposing  how to choose the initial value of the learning rate, a problem not previously addressed.  Key works missing here are: *in a pruning context*.  This is a fine point to make, but a necessary one to avoid claiming that the problem of choosing an initial value of the learning rate for retraining a network has not previously been addressed.  For instance, every simple transfer learning problem has to solve this issue, by fine-tuning new classification weights (or more output-proximal representation layer weights) to solve a new task, which i do not see as fundamentally different than retraining following pruning.  I would walk back this claim. I also note that the reviewers of Le and Hua found the same criticism. \n- In section 2.1, the authors declare themselves for Li et al in the battle of ideas about re-training a learned network.  That may be so, but what I have yet to see here, (and to be fair in the short readings of any of the competitions like Renda et al., or Le & Hua), is the question of how this problem differs form the fine-tuning problem of any base model (or foundation model).  \n\nI think the authors would help separate their work from these other lines by clearly delineating what aspects of the retraining problem they think are the most salient, and then arguing why the budgeted approach yields the best results.\n\n- The connection of the learning rate scheduler to generalization (in Li et al as well) is empirical, and not well established.  Appealing to this as justification is wishful thinking. Over all, section 2.1 feels poorly structured.  Where does the related work summary end, and the authors’ own work begin?  It could be made more clear by the use of sub-sectioning to delineate the concepts of pruning, learning rate schedules, and generalization.\n\n- A minor quibble about equation (1): you cannot have the interval of $s$ include $0$, as that leaves $d_1$ undefined by virtue of division by zero.\n\n- It’s also unclear how this is supposed to stand as a proxy for the degradation in on-task training accuracy, which the authors claim in the preamble to equation (1). There might be a more grounded argument to be made here (perhaps approximating the function of $W$ -> accuracy for a fixed evaluation set?\n\n- The final paragraph of section 2 cursorily sketches out ideas in selecting weights to be pruned, but only considers magnitude based pruning.  There are competing approaches that are sufficiently different, and deserve to be mentioned in here, such as  [Tanaka et al.](https://proceedings.neurips.cc/paper/2020/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf) who propose a flow-based method for pruning weights.\n\n- Finally, I am sure I’m not the first to raise this issue, but there is surely an in-built bias resulting from the authors’ choice to study only multi-class classification in image data sets, using one family of convolutional networks.  It would be more convincing if the authors could have included some diversity in either their model families chosen, or their tasks considered.  The authors acknowledge this failing in their work  as a footnote in the concluding discussion, but if they are to follow in the steps of  Li et al 2020 (who use tasks from four different domains), then they should present more evidence that their method has an empirical benefit beyond that of this narrow domain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CHl49I0pgW6", "reviewer": "Reviewer_HL8u", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose that recent work on budgeted training in dense models applies in the context of both the dense training and sparse re-training phases of Iterative Magnitude Pruning (IMP) - perhaps the most common form of pruning. In particular the authors propose two methods of pruning/re-training consisting of relatively short total training duration compared to existing work, and yet achieving strong results compared to those much longer-trained baselines.", "summary_of_the_review": "Overall the paper is well-written, with a good background, experimental setup and analysis. The paper sets strong new baselines for \"pruning unstable\" methods, such as Iterative Magnitude Pruning. Although these results are used by the authors to compare with \"pruning stable\" methods, such as Dynamic Sparse Training methods (in general), in not comparing to the widely known DST methods with the best generalization results (in particular RiGL on ImageNet/ResNet-50) weakens this argument significantly, and avoiding comparing to such a well-known and strong baseline in general when its cited by the authors raises questions. The authors do make very good points on the relative computational complexity of such methods however which even in the face of RiGL results still leaves this work motivating, and of interest to the sparse neural network research community.", "strength_and_weaknesses": "# Strengths\n* Well-written and well-organized paper, with a clear motivation, excellent background/references\n* Method is simple and explained well overall\n* Figures and results are presented well\n* Empirical evaluation is (mostly) on reasonable datasets/models (exception being VGG, see below)\n* Questions existing wisdom in the sparse neural network community that \"pruning stable\" approaches (i.e. methods without hard/fixed masks during training) to the finding sparse neural networks are preferable to \"pruning unstable\" approaches (i.e. methods with hard masks).\n* Strong results mean this could be a new baseline for \"pruning unstable\" methods\n\n# Weaknesses \n* The authors significantly weaken the impact of their work by essentially over-claiming that their work shows that \"pruning stable\" methods are pointless, without the results to back it up. In a notable and questionable omission in the results, the authors do not compare to the state-of-the-art dynamic sparse training methods despite citing them, for example RiGL. On ImageNet/ResNet-50 the BIMP results are massively below those of RiGL. For example, in Table 2, at 90% sparsity, BIMP has a Top-1 accuracy of 70.25%, while RiGL achieves 72-76.4% (saying based on ERK/non-ERK and length of training). \n* The authors point out rightly that RiGL and other such methods are massively more computationally complex and require longer training - and if they instead focused on this argument in their results, rather than generalization alone while omitting RiGL, this would be a much stronger paper.\n* Intuition/reasoning behind the ALLR method of using the normalized distance between weight vectors to decide an initial learning rate is not made clear, and this reads as quite arbitrary/heuristic as-is.\n* VGG is much too over-parameterized to be in a sparsity-related paper in 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster