Uncovering the impact of hyperparameters for global magnitude pruning
Authors: Janice Lan
Rudy Chin
Alexei Baevski
Ari S. Morcos
Venue:  (2021)
Keywords: deep learning
pruning
understanding

Abstract: A common paradigm in model pruning is to train a model, prune, and then either fine-tune or, in the lottery ticket framework, reinitialize and retrain. Prior work has implicitly assumed that the best training configuration for model evaluation is also the best configuration for mask discovery. However, what if a training configuration which yields worse performance actually yields a mask which trains to higher performance?  To test this, we decoupled the hyperparameters for mask discovery (H_find) and mask evaluation (H_eval). Using unstructured magnitude pruning on vision classification tasks, we discovered the "decoupled find-eval phenomenon," in which certain H_find values lead to models which have lower performance, but generate masks with substantially higher eventual performance compared to using the same hyperparameters for both stages. We show that this phenomenon holds across a number of models, datasets, configurations, and also for one-shot structured pruning. Finally, we demonstrate that different H_find values yield masks with materially different layerwise pruning ratios and that the decoupled find-eval phenomenon is causally mediated by these ratios. Our results demonstrate the practical utility of decoupling hyperparameters and provide clear insights into the mechanisms underlying this counterintuitive effect. 

Under review as a conference paper at ICLR 2021
UNCOVERING THE IMPACT OF HYPERPARAMETERS FOR GLOBAL MAGNITUDE PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
A common paradigm in model pruning is to train model, prune, and then either fine-tune or, in the lottery ticket framework, reinitialize and retrain. Prior work has implicitly assumed that the best training configuration for model evaluation is also the best configuration for mask discovery. However, what if a training configura- tion which yields worse performance actually yields a mask which trains higher performance? To test this, we decoupled the hyperparameters for mask discovery (Hana) and mask evaluation (Heval). Using unstructured magnitude pruning on vi- sion classification tasks, we discovered the "decoupled find-eval phenomenon," " in which certain Hand values lead to models which have lower performance, but generate masks with substantially higher eventual performance compared us- ing the same hyperparameters for both stages. We show that this phenomenon holds across number of models, datasets, configurations, and also for one-shot structured pruning. Finally, we demonstrate that different Hand values yield masks with materially different layerwise pruning ratios and that the decoupled find-eval phenomenon is causally mediated by these ratios. Our results demonstrate the practical utility of decoupling hyperparameters and provide clear insights into the mechanisms underlying this counterintuitive effect.
INTRODUCTION
There has been significant progress in deep learning in recent years, but many of the best performing networks are extremely large (Kolesnkkvv et al. 2019 Brown al. 2020). This can be problematic due to the amount compute and memory needed to train and deploy such models. One popular approach is model pruming: removing weights (unstructured pruning) or units (structured pruning) from a trained network in order to generate a smaller network with near-cquivalent (and in some cases, better) performance (Blalock al. 2020 Lin et al. 2020 Liu et al.) 2017] He al. 2019b] Molchanov et al. 2019). One of the most commonly used heuristics is magnitude pruning. in which the lowest magnitude weights/units removed, as is simple and competitive with more complex
methods (Han et al. 2015 Gale et al. 2019).
The lottery ticket hypothesis (Frankle & Carbin 2018). related concept, posits that large network contains a smaller subnetwork at initialization that can be trained to high performance in isolation, and provides simple pruning method to find such winning lottery tickets (LTs). In addition to allowing the training of sparse models from scratch, the existence of LTs also suggests that over- parameterization is not necessarily required to train a network to high performance; rather, over- parameterization may simply be necessary to find good starting point for training.
When pruning a network, one must decide how much prune each layer. Should all layers be pruned equally? Or rather, should some layers be pruned more than others? Previous studies have shown that global pruning results better compression and performance than layerwise (or uniform) pruning (Frankle & Carbin 2018 Morcos et al.) 2019). This is because global pruning ranks all weights/units together independent of layer, granting the network the flexibility to find the ideal layerwise pruning ratios (LPR), which we define as the percent that each layer is pruned.
One can frame most pruning methods as having two phases of training: one to find the mask, and one to evaluate the mask by training the pruned model (whether by rewinding and re- re-training fine- tuning). Common practice for methods that rewind weights after pruning, such as lottery ticket prun- ing, has been to use the same hyperparameters for both finding and evaluating masks (see Blalock

---

Under review as a conference paper at ICLR 2021
et al. 2020) for extensive review). Methods that fine-tune weights after pruning typically train at a smaller learning rate than the training phase to find the mask (Han et al. 2013 Renda et al. 2020), but other iyperparameters are held constant.
Pruning methods also rest on the assumption that models with the best performance will generate the best masks, such that the optimal hyperparameters for mask generation and mask evaluation should be identical. However, what if the mechanisms underlying mask generation are not perfectly correlated with those leading to good performance? Consequentially, what if models which converge to worse performance can yield better masks? To test this, we explored settings for global magnitude pruning which different hyperparameters were used for mask generation and mask evaluation. In particular, we focused on three hyperparameters that are commonly adjusted in practice: learning rate, batch size, and weight decay. Using this paradigm, we make the following contributions:
Surprisingly, we found that the best hyperparameters to find the mask (Hima) are often different from the best hyperparameters
train the regular model or to evaluate the mask (Hevat: Figures [ [nd which we term the "decoupled find-eval phenomenon". Counter- intuitively, this means that models with worse performance prior to pruning can generate masks which leads to better performance for the final pruned model than mask generated by a higher performance model preepprunng..
and
We show that this phenomenon is not artifact of the particular setting we studied and also occurs for structured pruning (Figure other datasets, and other variants of LTs, including late rewinding, learning rate warmup, and learning rate rewinding (Figure 4
We found that different hyperparameters for mask generation led to materially different LPRs (Figures ba and AT8). Notably, we observed that a larger learning rate, smaller batch size, and larger weight decay (which resulted in worse masks) consistently pruned early layers far more than in masks found with the opposite.
Finally, we show that this phenomenon is causally mediated by the differences in LPR. When the LPR is fixed to a "good" LPR (i.e., that of a high performance mask), the pre- viously bad hyperparameters now lead to better performance and better mask generation (Figure 7.
The same is true for the inverse experiment.
2 METHODS
2.1 MODIFIED PRUNING PROCEDURES
Our main experiments are mainly based on the lottery ticket procedure from Frankle & Carbin (2018) along with follow up work (Renda et al. (020) for unstructured pruning. We used global iterative magnitude pruning (IMP) for LTs because it has been shown to perform better than one shot pruning, where the pruning procedure only done once rather than iteratively, and local (or uniform layerwise) pruning, where each layer has the same pruning ratio (Morcos et al. 2019 Frankle & Carbin 2018). We also investigated structured one-shot pruning, following Liu et al. (2017), which also prunes globally. All experiments magnitude pruning.
We define four different sets hyperparameters: Humpruned- which used for regular training of an unprumed model; Hund, used to find masks (i.e. the pre-training part of the pruning procedure); Heval, used for obtaining the final pruned model to be used for inference and evaluating the final perfor- mance of a mask; and HEr to refer to the hyperparameters optimized for LTs where H H Note that HEr is often different from Humprumed in practice and in previous literature (Liu al. 2019 Frankle & Carbin 2018). Our modified procedures can incorporate changes in any hyperparame- ters, but in this paper we focus on experiments with learning rate (LR), batch size (BS), and weight decay (WD). For the main experiments, we focus on one hyperparameter at time. When only one hyperparameter changed from Hand and Hoval, we will denote such as LRand, WDoval, etc.
Unstructured pruning To account for these distinct sets of hyperparameters, we slightly modified the procedure for unstructured IMP described in Algorithm A1 We emphasize that modi- fication requires no additional compute compared to the original LT LT procedure used practically. However, to generate additional data points for analysis in the present work, we separately evaluated
2

---

Under review as a conference paper at ICLR 2021
a)
Regular lottery tickets, LRere Liqyal 0.5
b)
Regular lottery tickets, LR LR 0.2
Iciley
goobul ranconn
presenve LFR
Teat ACCurnyy
Test SCUPSYY
63
lotbery ca preserve UP
6.0
0.0
999
0,909
Fraction wagnt pruned
Fraztion waigtt pruned
Figure 1: Test accuracy at various sparsity levels, with the regular lottery ticket procedure, i.e. LRand LReat-- "Lottery" is the selected mask (winning ticket), "glohal random" is a mask selected at random globally. and "preserve LPR" is random mask that has the same LPR as the winning ticket. (a) LR = 0.5 performed very poorly, even worse than the random baseline. (b) Curves would expect from a properly functioning lottery ticket. The pruned performance is much better, even though training a regular model without pruning better with LR as 0.5 than 0.2.
many of the intermediate masks as well. We pruned weights with the smallest magnitudes through- out the whole network (i.e., global magnitude pruning) by p 20% at each pruning iteration until we reached 99.9% pruned, for a total of 30 pruning iterations. For some experiments, we also mod- ified line 6 for techniques such as late rewinding and learning rate rewinding (Renda et al. 2020 The original LT algorithm is recovered in Algorithm Al when Heval = Hind-
Structured pruning For structured pruning, we consider one-shot pruning using the scale batch normalization layers described in Algorithm[AA A2] This also has additional compute requirement compared to the original, and the original is again recovered when Heval = Hind-
2.2 MODELS AND DATASETS
For our main results, we used ResNet-50 (He et al., 2016) on Tiny ImageNe [Le & Yang 2015 and we also ran experiments with ResNet-18 18 on Tiny Imagenet and ResNet-50 on MiniPlaces (Zhou et al. 2017). We only pruned convolutional layers, such that the fully connected output layer of ResNets remained unpruned. Unless otherwise stated, we used SGD with 0.9 momentum and trained for 1 110 epochs, decaying the learning rate with a gamma of 0.2 at epochs 70, 85, and 100. See Section A. for details. The defaults for Hand and Heval batch size of 512, learning rate of 0.2, and weight decay of le-4; adjustments will be explicitly noted.
2.3 RANDOM BASELINES
We compare the selected masks to two random baselines. The first, "global random"", selects a mask at random, uniformly throughout the whole network. This means that all layers have the same expected pruning ratio. The second, "preserve LPR", also selects a mask at random, but with the same LPR as the winning ticket that we compare to; this is equivalent to shuffling the winning ticket mask within each layer. Both baselines are evaluated with new randomly initialized weights. Shaded error bars throughout represent standard deviation across 5 runs.
3 RESULTS
3.1 FAILING LOTTERY TICKETS
The failure of large LRs for lottery tickets has been pointed out previously (Liu et al. 2019;; win- ning tickets were only better than random baselines at a small LR. Directionally consistent with this work, we found particularly extreme case: not only can random baselines match the performance of the winning tickets with large I.R they can be significantly better. We found these catastroph- ically failing lottery tickets by following the LT procedure using a learning rate of 0.5 (for both LRind and LRvval), as in Hamprunct- This is illustrated in Figure La] where the winning tickets (bluc) and the random baseline with the same LPR as the winning ticket (green) are both worse than the
Use of these datasets is only for noncommercial, research purposes, and not for training networks deployed in production or for other commercial
3

---

Under review as a conference paper at ICLR 2021
a)
Mosks evaluated with LRqual 0.5
b)
Masks evalusted BS
Masks evaluated with WD 56-4
0.5
0.5
0.5
0.4
Teat accumeyy
0.3
0.3
0.3
Ula
0.05
0.2
55m 225
0.2
MC
Ufee +.2
ES 512
MDDgc -104
URe a 0.5
0.1
Mi = 1134
MiDso Sei4
diba random
glotal random
giebal randon
0.0
preseme L7n
preserv UR
0.0
Presarve LPA
0,222
0.40
0,000
0.822
8.00
0.30
0,500
6.22
0.000
0.30
0.000
8422
0.80
6.
6.99
d)
Masks found with LR 0.05
Masks found with BSrea 1024
Masks found with WDtne 2e-5
Test accurscy.
0.4
0.4
0.3
0.3
LRuud = 0.05
0.
WDew
LR 0.2
WDwu
0.1
LRuu 0.5
gidba random
global random
ERR
0.2
0,000
0.022
0.908
6.0
0.00
0.500
0.90
Fraction of weights prunos
Fraction weights pruned
Fraction weights pruncd
Figure 2: Test accuracy for different combinations of Hand and Heat, adjusting learning rate (a, d), batch size (b. e), weight decay (c, f). Top row (a-c): given constant Hevat, we compare different values of Hand- Bottom row (d-f): given a constant Hand, we compare different values of Hoat- Random baselines have the same Hand and Heva as the value in the title. The points at 0% sparsity represent the unpruned model trained with the same Hand as the rest of each line. Overall, it is bad for Hand to have large LR, small BS, or large WD, but the opposite is true for Heul. See Figures A2 A6 for additional combinations.
global random baseline (orange) by over 25 percentage points at 96-98% sparsity. Surprisingly, this winning ticket is much worse than the global random baseline, rather than just having no significant improvement over the random baseline as found in Liu et al. (2019) and Frankle & Carbin (2018). The learning rate of 0.5 not unreasonably large- in fact, the unpruned model performs the best at LR = 0.5. This failure also arises for reasonable values of batch size and weight decay (Figure Al).
Prior works have addressed this problem by training LTs with smaller learning rates (HıT instead of Humprumed: where for both, Heval) Similarly in our experiments, dropping the LR from 0.5 to 0.2 resulted in winning tickets that substantially outperform the global random baseline and are somewhat better than the random baseline with the same LPR as the winning ticket (Figure [b). This is why, in the lottery ticket setting, HIT is typically different from Humpruned (Frankle & Carbin 2018). However, training the standard unprumed model with LR of 0.2 instead of 0.5 decreases performance from 52.5% to 48.9%, as depicted by comparing the leftmost points in Figure [] The accuracy is decreased for low sparsity pruned networks as well. Current practice therefore faces a difficult off between low sparsity and high sparsity model performance.
These findings raise questions in two directions. First, for practical purposes, can we get the best of both worlds by decoupling the hyperparameters used to find the mask (Hina) and to evaluate the mask (Hevat)? This is inspired by the fact that large LR/small BS/large WD does well at low sparsity levels but causes the pruned model to get worse sparsity increases, suggesting that there may be potential for better winning tickets. Second, from a scientific standpoint, how do hyperparameters affect pruning performance, and why does Hanpruned not work well for lottery tickets?
3.2 THE DECOUPLED FIND-EVAL PHENOMENON
To test the whether we can adjust hyperparameters to improve the performance of winning tick- ets, we decoupled Hnnd and Hoval by setting learning rate (LR), batch size (BS) and weight decay (WD) independently for mask generation and evaluation. Specifically, we evaluated LR LR E (0.05, 0.2, 0.5): BS BS E {256, 512, 1024]; and WDand. WD E {2e-5, 1c-4, 5c-4] for total of nine find-eval combinations for each hyperparameter.
4

---

Under review as a conference paper at ICLR 2021
Best coupled vs, decoupled,
b)
Best coupled vs, decoupled, B5
Best coupled vs. decoupled, WD
Lt
2.55
3.50
2.50
0.45
0.90
0.40
0.35
0.35
0.30
0.00
6.22
3.99
Fraction weights
Fraction weights pruned
Froction weights
Figure 3: Comparison of the best coupled vs. decoupled run, for all three hyperparametees. For clarity, each point at given sparsity level is mean and standard deviation of the best configuration of any decoupled (blue) coupled (orange) combination of Hand, Heva that sparsity level. This was done because the best Heval changes with sparsity level, and this was cleaner than plotting several lines.
Late rewinding to 2 epochs
b)
Warmup, LRoaa and
Learning rate rewinding
0.55
0.55
3.50
0.45
0.42
Tost ScCrryy
0.33
3.40
0.3
0.30
Teat
002
Jas
3.30
02
-2
02
| 5.2 1 -05
2.2
02
0.
2.5
02
0.222
0.008
0.50P
3.0
0.90
Fracion of teights pruned
Fraction of waggtts pruned
meights
Figure 4: Test accuracy on variants of LT, with two values of Lf and two values of L cach. (a) Late rewinding to 2 epochs instead of initialization. (b) Learning weights continue training from their values during pruning rather than rewinding back to an earlier value. The better Landa for pruned models is the smaller one (blue is better than green, orange is better than red), despite performing worse on the unpruned model, and the best LR starts high and then decreases.
warmup. (c) Learning rate rewinding, where
The results are shown in Figure 2 grouped from two different perspectives. In the top row (a-c), we compare different values of Hand for a constant Hovat, with one plot for each hyperparameter. These comparisons show that it is better to find masks with a smaller LRind (0.2 is slightly better than 0.05, but both are significantly better than 0.5). larger BS and smaller WD For ease of discussion, we will denote these as "good Ham.." "t In the bottom row (d-f), we compare different values of Heval for a constant Humd- Here, we observe the opposite effect: given a constant Hand, the best performance is obtained by a larger LReval, smaller BS and larger WDevati we will denote these as "good Heva"". More specifically, the best Heval matches Humpramed for lower sparsity levels but gradually shift to more intermediate values at higher sparsity levels (around 96-98%).
It is interesting to note that decreasing the batch size has similar effect increasing the learning rate, which supports previous studies about the relationship between the two (Smith et al. 2018 Goyal et al. 2017: He et al. 2019a) Increasing weight decay also has a similar effect as increasing the learning rate, corroborating with Zhang et al. 2019). Altogether, these results demonstrate two key findings: first, the good Hand values do not achieve the highest model accuracy, which means that hyperparameter tuning on a regular unpruned models (Hunpnmea) may not be relevant when carried over pruning. Second, the best Hand values are not the same as the best Heval values, thus it is important to decouple the Hand and Haval in LT pruning. The importance of decoupling can be seen clearly when we compare the best decoupled configuration to the best coupled (original LT) runs (Figure 3): decoupled is better by about 3-4 percentage points around sparsity levels 70-90%.
3.3 THE DECOUPLED LR PHENOMENON IS MAINTAINED ACROSS DIFFERENT MODELS, DATASETS, CONFIGURATIONS, AND STRUCTURED PRUNING
Other models and datasets To expand beyond 50 on Tiny ImageNet, we also evaluated Re on MiniPlaces (Figure A8) and a ResNet-18 18 on Tiny ImageNet (Figure A9). In both cases, we observed the same effect: the best Hand requires smaller LR, larger BS, and smaller WD compared to Humpaueed and Heval-
5

---

Under review as a conference paper at ICLR 2021
a)
Structured pruning, comparing learning
b)
Structured pruning, companing batch size
Structured pruning.compartig weight decay
Uier E4, aee 24
Bass Di
WDos Aio Wine -ed
Ben 126. 26. 1834
weights pranes
Figure 5: Hand should also be decoupled from Heval in one-shot structured pruning, Test accuracy for two values of Hind and two Hoval for four combinations each for learning rate (a), batch size (b), and weight decay (c). The best performing configuration requires decoupled values; though this only applies low sparsity levels for batch size, the pattern clearer for learning rate and weight decay. See Figures All A12 for full results.
Other variants of lottery ticket For the same model and dataset as our initial results (ResNet- 50 on Tiny ImageNet), we tried the following: using late rewinding, learning rate rewinding, and learning rate warmup. We ran late rewinding experiments from Renda et al. (2020), where instead of rewinding the weights to 0g in line 6 of Algorithm A1 we set the weights to their values at epoch 2. Following Frankle et al. (2019b), that we could linearly between modes without substantial drops in performance, though notably this was only true for small learning rates (Figures A14). Epoch 2 was chosen to be roughly similar to experiments in Morcos et al.] (2019), which showed strong winning ticket performance across a number of datasets. Here, we again see the same pattern, where the best LR smaller than the best standard LR (Figure 4a). For learning rate warmup, we warmed up the learning rate linearly over 1000 iterations from to the specified LRand- The effect of LR is the same as well, although it is more subtle (Figure 4b
also found that training from this point on was stable, such
Our learning rate rewinding experiments also encapsulate the fine-tuning experiments (Renda al. 2020). That is because both methods prune and then continue training the remaining weights from that point; the only difference that learning rate rewinding trains at large learning rate, and fine- tuning trains at small learning rate (due to the decay in learning rate in the training phase before pruning). Thus, we extended learning rate rewinding to different LRs. We saw the same effect, with even better overall performance: taking mask from small LRand and using learning rate rewinding to a high LReval improves accuracy by approximately 3 percentage points compared to weight rewinding (Figure 4c). This also explains why fine-tuning in |Renda et al. (2020) did worse than learning rate rewinding: a larger LReval is better, and fine-tuning is effectively the same as learning rate rewinding but with smaller and worse LReval-
Structured pruning We have demonstrated that our results in Section 3.2 are not merely an arti- fact of the precise setting we studied, but rather represent more generic feature of lottery tickets across models, datasets, and configurations. However, this phenomenon may still be limited to only lottery tickets. To test its applicability to pruning more generally, we also investigated structured pruning on R8 Tiny ImageNet in which entire filters are removed. Once again, we found that the best performing pruned networks required different Hind and Heval (Figure 5). though the improvement is more subtle. Like before, we found that it is generally better to find the mask with a smaller Lind and WDind and larger BSnnd. and evaluate with a larger LReval and WD and smaller BSa The decoupled configuration can increase the top-1 accuracy by up to 2 percentage points at moderate sparsity levels. Coupled hyperparameters were often best at high sparsity levels, but at that point, performance across all configurations were subtantially lower than the unpruned model. These results demonstrate that the phenomenon of decoupling hyperparameters does not just apply to lottery tickets, but also applies to additional methods of global magnitude pruning.
3.4 LAYERWISE PRUNING RATIOS CAUSALLY MEDIATE THE DECOUPLED LR PHENOMENON
we observed that using high LR for HIT resulted in performance below the global random baseline not only for the lottery ticket, but also for the "preserve LPR" random baseline. This suggests that LPR may be related to the decoupled find-eval phenomenon. To test this, examined the LPR over different pruning iterations for various settings. Consistent with our hypothesis, we found that LPR varied dramatically across different Hand (Figure C). Most strikingly, we observed
In Figure Ia

---

Under review as a conference paper at ICLR 2021
a)
compating
compartng
255
M V
Figure 6: Layerwise pruning ratios depend on Hand. Shown here is LPR masks found by the three different values of LRaad (a), BS (b), and WDnnd (c). Small LR, large BS, and small WD values tend to prune more in the earlier layers. See Figure A17 in the appendix for equivalent plots at other sparsity levels.
vs. for learning rate
b)
vs. bad
size
Good vs. LPR for weight decay
6.4
0.4
0.4
Teat RCCuRNyy
0.3
0.3
E.2 Dlaa 95
1024. 15
0.3
WD Fe-4 match JRR 205
63
0.
0.
coN
8.49
8,900
0,400
Fr of waights pruneo
Fracton of waggha stuned
Frack on of weightt prunee
Figure 7: Test accuracy when LPR constrained. For each of learning rate (a), batch size (b), and weight docay (c), the blue line represents pruning using bad Hand but LPR fixed to a run from good Hand. The orange line represents pruning using good Hand but LPR fixed to run from bad Hind, which performs much more poorly. For comparison, the green lines are regular pruning runs (from Figure 2 that have the same LPR as the blue line, except with different Hind used for finding the masks within each layer. All three lines in each plot are evaluated with the same Heval.
that the bad Hand values (large LR small BSand, and large WDnnd) tend to prune the earlier layers much more. This imbalance was consistent across different sparsity levels (Figure (17)
We have shown there is clear difference in LPR across various Hand, but is the change in LPR directly responsible for the performance of the pruned models? this is the case, we should be able change the performance of models just by constraining the model's LPR. To test this, we ran two types of experiments. Based on the results of Figure 2, we consider the LPR of masks from LRand = 0.2, BSndd= = 1024, and WD = 2e-5 as "good LPR Likewise, the LPR of masks from LR 0.5, BSand = 256, and WDand = 5c-4 can be considered as "bad LPR". We ran Algorithm [A] with bad Hand but forced the pruning step to prune fixed amount each layer, specifically, according to the good LPR from the runs with respective good Himd soaatch-ooodLPRR").. While the I.PR are fixed, the masks within each layer were still determined by the particular training configuration (the bad Hand in this case). We also ran the complementary procedure with good Hand values but pruning according to the bad LPR ("match-bad-LPR"))
If LPR causally mediates the decoupled LR phenomenon, we would expect to see match-good- LPR do well and match-bad-LPP do poorly, despite being trained with Hand which typically result in the opposite effect. This is in fact what we observed: match-good-LPP (blue) substantially outperformed the match-bad-LPPR (orange; Figure 7. By constraining LPR, we were able to account for the entire change in performance present in the decoupled find-eval phenomenon, demonstrating that this phenomenon is causally mediated by LPR. Furthermore, the match-good-LPPR does even better than the regular pruning runs with good Hand from Section 3.1 (green), indicating that the previously "bad" Hand were only bad because of their LPR. Interestingly, we observe this same effect when we constrain LPR to be the same for all layers. equivalent to local or uniform layerwise pruning (Figure A15] Figure A16). confirming the importance of LPR to the decoupled find-eval phenomenon.
Adjusting weight decay and learning rate together Thus far, we have adjusted each hyperpa- rameter independently, but these parameters can also be adjusted jointly. One particular result of interest is that when the weight decay is set to O, the learning rate barely affects the LPR (Figure 8] contrast with Figure ba)- while there is still some visible difference within each block of layers, the larger sections of the network are all pruned equally. Additionally, higher IR now perform very
7

---

Under review as a conference paper at ICLR 2021
decay
b)
pruned
m033
min55
Dis 325
2.2
Figure 8: LPR for different L LRand values at (a) 67% % pruned (b) and 89% pruned. When weight decay is set to zero, LRind has significantly smaller effect on LPR. This is in contrast to Figure 6a where the early layers get pruned much more at higher learning rates. As result, overall performance is no longer harmed by large Linnd: when comparing large range of Rind, all evaluated with reasonable LR 0.5 and WD le-4, the larger R.dd does better (c).
a)
Early pruning,
b)
Early pruning, Ui
spach
spack 1161
03
apach
0.8
126 (end)
Fracion waight
Fredioon
Figure 9: Early pruning: Pruning at epoch 20 (blue) is bad, but pruning at epochs 50 (orange) or 70 (green) is actually better than pruning at convergence (110 epochs, red) for LRa = 0.2. Pruning at 70 for LRand 0.05 also performs well. All masks are evaluated with LReaal 0.5, except for the points at 0% sparsity, which represent an unpruned model trained for the corresponding number of epochs.
well, even slightly better than moderate or small LRindd similar to the match-good- experiments above. This is in stark contrast to earlier results with weight decay of le-4.
3.5
EARLY PRUNING
For high Linnd LPR changes dramatically over training, while a low LRand leads to stable LPR throughout training, with the exception of the first layer (Figure A18). Furthermore, the distance between the early mask's s LPR and the actual mask's ' LPR decreases quickly and plateaus, likely due to the learning rate decay (Figure eet al. (2020). We found that early pruming can be even better than pruning convergence: pruning at either epoch 50 or 70 for LR = 0.2 actually improves performance compared to pruning at the end of training (Figure 9). There are two useful implications here. First, this is evidence of yet another hyperparameter that follows the same phenomenon: the best settings to train a mask for evaluation or to train standard unpruned model (110 epochs) not necessarily the same as the best settings to prune the model (50 epochs). Second, for the particular case of epochs spent training, this allows us to save compute. As mentioned in Section [ pruning with decoupled Hana and Heval requires no extra compute compared to the original LT procedure. In this case, Heval decreases the compute used to find masks by 55% while also gaining a slight increase in test accuracy.
This inspired us to experiment with pruming early, as in Yin
4 ADDITIONAL RELATED WORK
Magnitude pruming. in which weights or filters are removed according to their magnitude, was first introduced in Han et al. (2015), and has had great success, prompting a large field of study, recently summarized by Blalock et al. (2020). Structured pruning approaches have had great success since such techniques can accelerate CNN during test-time without requiring specialized hardware and software support. Li et al. (2016) have demonstrated the potential of channel pruning for improving the efficiency of CNNs at test time. Later, many methods were proposed to further improve the performance of structural pruning (Liu et al. 2017 He et al. 2019b] Molchanov et al. 2019 Fan et al. 2020). Related to our work, [iu et al. (2019) have demonstrated the importance of LPR over pre-trained weights for wide range of structural pruning methods. Complementing their results,
8

---

Under review as a conference paper at ICLR 2021
our findings further suggest that while both weights and LPR are important for pruning techniques, the value of weights is conditioned on good LPR, which is implicitly determined by hyperparameters when it comes to global magnitude pruning. They also pointed out the failure of large LRs for lottery tickets, resulting in lottery tickets being trained with smaller L.s..
The lottery ticket hypothesis argues that over- parreeeeemttiiiiionnnn is necessary to find good initializa- tion rather than for optimization itself (Frankle & Carbin 2018 Frankle et al.] 2019a). Since their introduction, number of works have attempted understand various aspects of this phenomenon, including their transferability (Morcos et al. 2019), the factors which make particular winning tick- ets good (Zhou et al. 2019), and linear mode connectivity (Frankle et al., (019b). The lottery ticket hypothesis has also been challenged by (Liu et al. 2019)), in part because of the requirement for small learning rates, as we address in this work.
Tanaka et al. (2020) observed that layer collapse, which they defined as entire layers being com- pletely pruned, is a key issue that causes some pruned models fail. This is along the same lines of the bad LPR that we observed, with a few key differences: we saw poor performance even before layer collapse, and skip connections ResNets should make layer collapse less of an issue. Further, they claim that iterative pruning avoids layer collapse, but we see failure cases in iterative pruning.
In [i et al. (2020), the authors also decouple the learning rate between pre-training and fine-tuning. somewhat analogous to how we decouple the learning rate between finding and evaluating mask; they are similar concepts applied to different tasks. As we analyze the impact of LRand, they analyze the significance of the learning rate for fine-tuning
different dataset.
There are several proposed methods to prune at initialization, which does not require training the model before finding the mask (Lee et al. 2019 Wang et al. [2000] Tanaka et al., (020). In contrast with our experiments, there is no Hind (beyond the initialization distribution), only Heval- While pruning initialization is more efficient resource-wisse |Frankle al. (2200) recently demonstrated FFrreekki,,
in extensive experiments that performance of these approaches is not on par with pruning after training. Hence, it is still critical to study methods that do not prune at initialization.
DISCUSSION
In this work, we proposed simple idea: use different hyperparameters for mask discovery and evaluation for pruning. We uncovered the counterintuitive decoupled find-eval phenomenon (Section 3.2), in which using hyperparameters that lead to worse model performance yield better masks. This effect is consistent across both lottery tickets and structured pruning (Section 3.3). We also demonstrated that values of Hand which correspond to effectively small learning rates also lead to materially different LPRs in which early layers are pruned less than later layers (Figure 6a). and that these LPR causally mediate the decoupled find-eval phenomenon (Section (.4). While T.PR not quite all you need, it does determine a very significant part of the final pruned performance and provides insights into the masks found by pruning.
Practical implications Our work also has several key practical implications to improve com- pressed model performance. Most notably, we found that tuning hyperparameters separately for the two phases of training can result in substantial performance gains. While this does introduce ad- ditional hyperparameter dimensions to search over, our experiments suggests several general guide- lines for tuning these parameters: Hand requires smaller LR and WD or larger BS; Heval is similar to Hanpruned at low sparsity levels; and Heval at high sparsity levels should gradually decrease LR and WD or increase BS. Alternatively, we found that simply turning off weight decay during Hind pro- duced good LPRs and, consequentially, good performance. Alternatively, don't 't use weight decay for finding masks and then the LR no longer matters. Finally, we observed that masks generated earlier in training often yielded better overall performance, suggesting that by pre-training for fewer epochs, both compute efficiency performance can be improved.
Limitations and future work While we showed that the decoupled find-eval phenomenon is causally mediated by LPR, we did not find clear answer to the question of why certain Hand leads to better LPR. In addition, a method to derive a "good" LRP would be very useful. We hope to investigate these question further in future work. Furthermore, it would be interesting to study additional hyperparameters and how they interact, and experiment with more datasets, architectures, and tasks beyond vision classification.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning?, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv, pp. arXiv-2005, 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. In International Conference on Learning Representations, 2020. URL
net htt.:iiwwweeeeeoeiiiiiaee..... fo
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis at scale. March 2019a. /RL
01611
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis, 2019b.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint 00X752009908876, 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. February 2019. URL URL http::/ttxxi..rrrrt.et/
992.9990.00000 09574
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet in I hour. CoRR, abs/1706.02677,, 2017. URL http:://axxiv.org//bbb//////
02677
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. In Advances in Neural Information Processing Systems, volume 32, pp. 1143-1152. Curran Associates, Inc., 2019a.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yang He. Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4343 2019b.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 1rXiv:1912.11370 2019.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. 2015.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learn- ing Representations, 2019,
net/ forummrrmmmmm Biierumnn---1V
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608..6088.00 000/00 2016.
10

---

Under review as a conference paper at ICLR 2021
Hao Li, Pratik Chaudhari, Hao Yang. Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Confer- ence Learning Representations, 2020. URL https:: /orrps:s//ppeeevieww http:/iwwwwweeeeeeeeeeeeeeeeeeee m net/forum?ic B1 B1ę8VkHFPH
Tao Lin, Sebastian U. Stich,, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL hetps:
7/ //openreview net wwt.ee/ooooo..d-oeeeeeee
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference Computer Vision, pp. 2736-2744 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019. https:
//openzeview. wwt.:eeeeooommmddr.....a.aa.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1264-112772 2019.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: gener- alizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Infor- mation Processing Systems, 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Samuel I.. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018. URL https:
/ ///pnnevviww.....mmmoeeor wwt.eeet/oooe wow.mmidddBBiylleee
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruming neural networks without any data by iteratively conserving synaptic flow, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse.
Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. URL
https://cpenreviiee net wtt.te/ooooom.dd.aaaaaaetce
Shihui Yin, Kyu-Hyoun Kim, Jinwook Oh, Naigang Wang, Mauricio Serrano, Jae-Sun Seo, and
Jungwook Choi. The sooner the better: Investigating structure of early winning lottery tickets, 2020. /t//:tttss///peeeeees.eeeee/tttt wtt.:n/ttttttorrrooooooososssss.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay
regularization. In International Conference on Learning Representations, 2019. URL https:
//openreview. berpeerrvieeeeet/ffooooeeee www.:/eet/oooummm.c.mm didimidddiBlli 3Rct
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Conference on Neural Information Processing Systems, 2019.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.I ADDITIONAL TRAINING DETAILS
Algorithm A1: Unstructured iterative magnitude pruning
Input:
Model parameters 0, pruning percentage per iteration p. number of total pruning
iterations N, fandk-innding hyperparameters Hima, mask-evaluating h hyperparameters
Heval
Output: Performance of trained pruned models
Initialize mask Mo as all ones
2 Randomly initialize model weights 0o for i I...N do
Train model 0 O M,-1 to convergence using hyperparameters Hind to obtain 0 Prune p% of parameters 0g, the ones with the smallest magnitudes globally and have not been masked out by M;-1, to create a binary mask M;
Rewind weights back to initial 0o
end
Evaluate final mask or other masks of interest (M;) by training 0 @ M, with hyperparameters Hevll and record its performance
Algorithm A2: Structured one-shot magnitude pruning
Input: Model parameters pruning percentage per iteration p, mask-finding hyperparameters Hind mask-evaluating hyperparameters Heval
Output: Performance of trained pruned model
Randomly initialize model weights A
Train model 0 using hyperparameters Hund to obtain O
Prune p% of parameters Otnall create binary mask M by pruning channels with smallest
magnitude in the scale for the batch normalization layers, i.e. 7
Evaluate the mask by training 0 M with hyperparameters Heval and record its performance
All unstructured pruning experiments used data augmentation Residdm and Ran- domHorrzonnallFipp from torchvision), batch size of 512, and weight decay of 0.0001. We pruned convolutional layers 20% at each pruning iteration, for a total of 30 iterations. Weights were initial- ized with Kaiming normal distribution.
ResNet-50 on Tiny ImageNet: we trained 110 epochs, decaying LR by gamma = 0.2 at epochs 70, 85, and 100. Experiments with warmup, late rewinding, and learning rate rewinding (Section 3.3) use the same hyperparameters, except for modifications explicitly mentioned in the main text.
ResNet-50 on MiniPlaces: we trained 120 epochs, decaying LR by gamma- 0.2 at epochs 80, 110, and 115.
ResNet- 18 on Tiny ImageNet: we trained 110 epochs, decaying LR by gamma 0.1 at epochs 80 and 100.
Structured pruming: we use the magnitude of y of batch normalization layers as the ranking metric to decide which filters to prune. For channels that are connected by residual connections, we sum their measures as they have to pruned jointly. We train the pruned network using 200 epochs for both pre-training and fine-tuning. We use batch size of 256, weight decay of 5e-5, cosine learning rate decay, linear learning rate warmup from 0 to the set learning rate within the first 5 epochs, SGD with 0.9 Nesterov momentum, and 0.1 label smoothing. All layers were initialized with Kaiming normal distribution. The algorithm is specified in A2 Note that we do one-shot pruning and do not rewind weights back to their initial values, contrary to the method for unstructured pruning.
Shaded error bars all figures represent standard deviations over runs with different random seeds.
12

---

Under review as a conference paper at ICLR 2021
A.2
ADDITIONAL FIGURES REFERENCED FROM SECTION 3
a)
Regular lottery tickets, BS 85
256
b)
Regular lottery tickets WD WD 55
85 - 512, WD 11-4, LR 0.2
0.5
lstrry
gitkal turblm
0.5
0.3
0.3
0.3
6.
0.
0.3
2ibb random
preseve LPA
0.990
6DW
0.122
0.00
Test accuracy at various sparsity levels, with the regular lottery ticket procedure, i.c. H Havat Smaller batch size (a) and larger weight decay (b) result in poorly performing lottery tickets, despite being better for the regular unpruned model, represented by the points 0% sparsity. (c) is the same Figure Ib copied over for comparison, which uses the default hyperparameters (hut not the best ones for training a regular model).
Figure A1:
a)
Masks evaluated with URer 0.05
b)
Masks evalusted with LRer 0.2
Masks evalusted with LRew 0.5
Test acciioo
Test Tociao
0.3
0.3
0.3
Dla 0.75
Mas 2.00
LPo 0.65
0.
pencerze
preseve 1i
64
areseve 19
0.02
5000
0.02
0.99
2000
0.02
3.9
Fraction wrights pruned
lnn aelghtt pruned
Fraction aelgh/ pruned
d)
Masks LReed
Mades with LRrad 0.2
f)
found with LRsss
sccurors
0.4
2.4
10 Sccuracy
1.4
02
6.2
+D.5
Lou 2.2
-02
Low 0.2
033
6.1
Lue -03
2lbbl randam
plaa ranzam
0.000
0.008
2.9
0.42
2.008
8.50
d.80
0.509
Fraction of weights praned
Fractio of weight pruned
Fraction weiget pranes
Figure A2: All combinations LRand and LReal. Top (a-c): given a constant LRevai, we compare different values of LR 0.1 does the best, followed closely by 0.2 then 0.05, and everything is significantly better than 0.5. Bottom (d-f): given a constant LRadd. we compare different values of LReval 0.5 does the best at lower sparsity levels and 0.2 at higher sparsity levels. Random baselines have the same LRand and LReaal the value in the title.
13

---

Under review as a conference paper at ICLR 2021
All learning rate combinations
LRqes 0.05, LRoal
0.4
LRess 0.05, LRoul 0.2
Test accuracy
LRqe 0.05, LR
0.3
LRess 0.2, URoai 0.05
URoes URove
LRee 0.2, URqa = 0.5
0.2
URiea URave
LRee 0.5. Uique 0.2
LRaes UR
0.0
0,000
0.632
e.sss
0,950
0,982
0,903
0,938
Fraction of weights pruned
Figure A3: combinations of leaminng rate plotted together comparison
Masks evalusted with BS
b)
Masks evalua led with BS
512
Masks evaluated with BS 1024
0.4
0.4
0.4
Tol
Tol
D5. 224
35
Blox 512
M $12
85 = 1024
25 1014
2S 1024
gibbal modion
gobai modion
0.000
Trection
Fractinn meights pruned
Fraction meight gruned
d)
Masks found with 85md 256
e)
Masks found with B5 512
Masks found with B5rr 1024
Tertacurasy
Tea accumetyy
0.3
02
0.2
1625
2Md
Ditserve LT1
praeer U1
560
0.209
5.977
weights pruned
Fraction weight pruncd
weight praned
Figure A4: All combinations of BSnnd and BS Top (a-c): given constant BS we compare different values of BSand showing that larger is better. Bottom (d-f): given constant BS compare different values of BSeaa. 256 does the best at low to medium sparsity levels and 512 higher sparsity levels. Random baselines have the same BSand and BS as the value in the title.
All batch size combinations
0.5
85pu 256, B5 256
0.4
05 05...
Test accuracy
85pu 256, 85 1024
0.3
0S 05...
8Spu 512, B5 512
B8 1024. BS... 256
0.1
BS 1024. BS... 1024
0.000
0.82
0,865
0,850
0,902
39993
8
Fraction of weights pruned
Figure A5: All combinations of batch size plotted together for comparison
14

---

Under review as a conference paper at ICLR 2021
Masks eveluated with ovd 2e-5
b)
Masks evalusted with WDex 1e-4
Masks evalusted with WDeva 5e-4
03
0.4
04
Teat S0L3CCu
04
W
W
WD Sp-4
Wioe 5p4
WDr
goEni modion
1888 modion
Frection
Fraction seights prumed
Fraction meight pruned
d)
Nosks found with WDad 2o-5
e)
Masis found with WDD 10-4
Masks found with WDnn So-4
Sccurscy
02
1bbd
gobel Fandom
00
Sitserv LF1
20
1.
Ereave LPA
0.500
0.09
6200
5.00
tiggtt oraned
WEIghS pruncd
welge prp.rrrr
Figure A6: All combinations of WD and WD Top (a-c): given constant WD compare different values of WDmnd. showing that smaller better. Bottom (d-f): given constant WDand, compare different values WD 5c-4 does best at lower sparsity levels and Ic-4 at higher sparsity levels. Random baselines have the same WDand and WD eval as the value in the title.
All weight decay combinations
0.5
WD
Test acuracy
0.4
WD 2e5, WD 1e4
WD WD
WD 1e-4, WD 26-5
1o-4
WD 1e-4, WD 5e-4
WD Se-4, WD es 1e4
5e-4, Se-4
0.000
0.632
0.865
0,550
0,922
0,993
0,988
589
Fraction weights prunea
Figure A7: All combinations of weight decay plotted together for comparison
15

---

Under review as a conference paper at ICLR 2021
A.2.1
OTHER MODELS AND DATASETS
MiniPlaces, LR
b)
MiniPlaces,
0.4
0.4
0.3
0.3
0.2
LRawl
0.1
LRowl
LRow
05
0.1
UPtd 05
0.180
0.832
0.950
8:382
0.993
0.880
8,82
0.950
8.982
0,99
Fraction weights pruned
Fraction weights pruned
MiniPlaces, LRevi 0.5
MiniPlaces, LRan 0.1
LRwa 0.1
0.5
0.55
LR 0.5
0.50
0.45
0.40
0.2
LRow m 0.02
0.1
0.5
LRaa
0.20
0.032
865
0,050
8882
0,993
0,000
0,632
0.000
982
0,993
Fraction weights pruned
Fraction of weights pruned
Figure A8: ResNet-50 on MiniPlaces with LRadd and LReaal from f0.02, 0.1, 0.5}. In (a), (b), and (c), we constant LReaa of 0.02, 0.I, and 0.5 respectively, with each plot showing all three values of LRund They show LReval 0.1 is best. Thus, (d) compare
different values of LReaal all with LR 0.1. demonstrate the hest R.R is 0.5, until very high sparsity levels where 0.1 becomes better, In line with other results, the best LR for standard training (0.5) is not the best I R for pruning.
a)
ResNet18 ImageNet, LReval 0.5
b)
ResNet18 Tiny ImageNet, LR
0.5
0.5
0.4
Test accurayy
0.3
LRow
LR 0.05
URes 0.2
LRee 0.2
0.8
0.3
global random
B-805
0,000
0.82
3933
6399
000
B452
0.933
8396
0.999
Proction weights pruhes
Fraction of weights pruned
Figure A9: ResNet-1 18 on Tiny ImageNet 0.5 in (a) and 0.8 in (b). For both values of LRevat. LRas of 0.05 and 0.2 are very similar and both perform significantly better than LR = 0.8. despite 0.8 performing the best on the unpruned model. LRnm of 0.8 also does worse than the random baseline (red).
LR {0.05, 0.2, 0.5), compared using constant LR
16

---

Under review as a conference paper at ICLR 2021
Structured prar ing. Uiea 0.1
b)
Stractured prutinng Lite 0.3
Stractured prating, Lite 0.4
Fraddon
Smuctured wdggtt
weight
d)
Structured pruning, LPea
UR
Structured praning, UReat 0.4
wna
TOSA WASEAO
x
Fraciion naighta pruncd
Pruction gira
Practon of nagll stunes
Figure A10: Structured pruning: All combinations of LRund and Leeal. Top (a-c): given constant LReval,
compare different values of L Bottom (d-f): given a constant Laadd compare diffcrent values of LReal- Structured pruning, 05
Structured 120
256
Structured pruning, 85 1024
1985
1934
3930
Tedt accurry
Text SCUuBRy
Text SCUueyy
46
Fradior of
Faciion of adgght
Praction inight
Structrre praning, life 128
Sincctude prining, B5 - 256
Structured praning, Bh 1004
Maa 254
255
Traction of neightee grused
Fracten give promed
neightt sruned
Figure A11: Structured pruning: All combinations of BSand and BSeal Top (a-c): given constant BS
compare different values of BS Bottom (d-f): given constant BSnnd, compare different values of BSeat. a)
Structured pruning, WD 1e-5
b)
Strudfured pruning, WDD, 3e-5
Structured pruning, WD 2.5
505
THE SCUUSEY
TeE
TUE
Prodtor of Weigtt pused Structured pruning. WDrw
Facion adgios alaned
Pagion neigh a praned
d)
Structured pruring. WDne 5c5
Structured pruning. WDre 2.544
tot sacunay
giva
Figure A12: Structured pruning: All combinations of WDrud and WD Top (a-c): given a constant WD compare different values of WD Bottom (d-f): given constant WDmmd compare different values of WD
17

---

Under review as a conference paper at ICLR 2021
A.2.2
OTHER VARIANTS
a)
Linear made connectivity,, 48.8% pruned
b)
Unner connectivity. 73.8% pruned
1.0
1.0
0.05, epochs
Test oroor
opochs
LR 0.05, no late rewinding
0.05, no late rewinding
0.7
LR 0.2, 2 epochs
0.7
LR 0.2. 2 spocts
0.6
0.6
0.5
0.5
0.0
0.2
0.4 0.6
0.8 1.0
0.0 0.2
0.4 Interpolation
0.6
0.8 1,0
Interpoiation
Figure A13: Instability analysis from Frankle et al. (2019b) of various networks. We measure the mode connectivity between two different networks trained with the same initialization but different SGD noise. Plot- ted is the test error we linearly interpolate between the two networks, for (a) 48.8% pruned and (b) 73.8% pruned. Using LR 0.05 (both L.Rind and L. as per a regular LT run) starting from epochs, the network contrast, LR 0.05 without late rewinding (orange)
stable, thus late rewinding 2 epochs is reasonable. and a larger LR= = 0.2 from 2 epochs (green) both are not stable.
LTs with LR = 0.05, late rewinding epochs
0.5
Test accuracy
0.3
0.2
winning ticket
0.1
global random
LPR
preserve mask
0.0
0.000
0.322
068
0,950
0.902
0.993
0.398
0.999
Fraction of weights pruned
Figure A14: Regular lottery tickets, with LRand LReval 0.05 and late rewinding to 2 epochs. compared to random baselines. The winning ticket does significantly better than random baselines, including "preserve mask", which is a random baseline that uses the same mask as the winning ticket but with reinitialized weights. This indicates that the network is stable at 2 epochs (Frankle et al. 2019b).
a)
Local pruning, LR 0.2
b)
Local pruning, LRwal = 0.5
Uiore
Liper
0.5
1Rr 022
LR 0.2
Lieae - 0.5
LPpe 0.5
accurccy
0.4
Test accuracc
0.4
0.3
0.2
0,0
0.000
0,022
0.665
0.6g0
0,3
0.998
0,989
0,900
0.83
0.990
0.983
0:999
0
Fraction weights pruned
Fraction weights pruned
Figure A15: Local pruning: pruning all layers the same amount. We compare LR E (0.1, 0.5) using
LR of 0.2 (a) and 0.5 (b). For both values of LR that the larger the LRasd, the better the pruned models perform.
18

---

Under review as a conference paper at ICLR 2021
Local pruning, BS 256
b)
Local pruning, WDeval 5e.4
05.
Win 5e-4
sccurccy
0.4
Test accuracy
0.4
0.3
0.3
0.2
0.1
0.1
0,000
0,32
0663
8,0
8
0,983
0,990
0,999
9,000
9,32
0.88
0,950
0
9
0,900
0,9
Fraction weights pruned
Fraction weights pruned
Figure A16: Local pruning: pruning all layers the same amount. Similar experiment setup Figure AIS
except comparing different values of BSnnd and WDnnd. While large BS and small WDand performed well global pruning, we see the opposite for local pruning.
A.2.3 ADDITIONAL LPR FIGURES
PIANEES comgur bay Teannng
prasedi compaing butch
PIANOOS COMPUTTIO WORSH dEcGe
97
%
Co
Winm
26
10
d)
LETT ut 93% pranedd Compating loarning rule
LFHt ut 93% prunco,, Compoar ny butch stze
Lmft at 52% pruned. com paring we decap
Us
Ki $1.
JAh 22% cwmpuringgggg.c.ooccoooc........ learning
Lnf ut 25% praced comseing butch
LmT 92% pruned compan
0.22
Figure A17: Comparison of LPR of masks found by the three different values of LR (a). BS (b), and
WD ind (c) 20, 89 and 99% sparsity (1, 10, and 20 pruning iterations respectively). Similar Figure 67% sparsity (5 pruning iterations), small LR, large BS, and small WD values tend to prune more in earlier layers.
19

---

Under review as a conference paper at ICLR 2021
a)
LPR throughout training, LRend 0.5
b)
LPR throughout trainingg LRou 0.05
0.95
M MM MM
0.9
wwW
parcnt eunainiee
0.85
0.7
opoch
0.75
epoch 20
epazt: 50
110
apacch
10
20
40
50
10
40
50
layers
Figure A18: Layerwise pruning ratios you were to prune at different epochs, for LRadd = 0.5 (a) and 0.05 (b), at the first pruning iteration (starting from unpruned model). In (a), the LPR change significantly throughout training, whereas for (b), LPR changes very slightly except for the first layer.
training
URiu
LRas 0.2
LRrs
UPR mean 3bs 6it
apochs
Figure A19: Distance between LPR if pruned at epoch actual LPR when pruned at the end, for different LRsmn-- Distance is calculated the absolute difference in pruning ratio for each layer. This is done the first pruning iteration (starting from an unpruned model). Smaller L. values start with much smaller distances; all LRad values start to plateau around epoch 70, which is when the learning rate decays.
20


Reviews:
[{"review_id": "_pwuiSGOM5f", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper evaluates explicitly decoupling the hyperparameters (specifically, learning rate) used to find pruning masks from those used to train pruned networks, finding that for global magnitude pruning lower learning rates (relative to the learning rate that results in the most accurate full-size network) result in masks that can train to higher accuracies; on the other hand, higher learning rates tend to train the pruned networks better. These results are shown to generalize across pruning and re-training techniques (including the standard lottery ticket regime, lottery tickets with warmup, weight rewinding, learning rate rewinding, and structured pruning). The paper then shows that these differences in performance are primarily due to differences in layerwise pruning rates that come from training with different learning rates.\n\n# Strengths\n\n- The paper proposes an interesting experiment, decoupling mask-finding hyperparameters from mask-training hyperparameters, with the potential to change how people think about network pruning, challenging the assumption that a higher-accuracy full network results in a higher-accuracy pruned network\n- The findings are well evaluated, convincingly showing that lower learning rates than those that result in the highest-accuracy full network result in better masks\n- The analysis is also strong, showing that the primary factor in the performance delta is the layerwise pruning rates\n\n# Weaknesses\n\n- Given that many of the findings are centered around performance when resetting the weights to the beginning of training, which is known to not work with large-scale networks, a significant amount more discussion of or replications of experiments of [1] is warranted.\n- Presentation of figures could also be significantly improved: for instance, by moving figures closer to text that references them, by collating high-level takeaways into tables, and otherwise making it easy to process the large amount of data in this paper\n- The early pruning subsection seems to refer to results that are not presented in the paper; given that the data is presented in the appendix, it'd be better to include this text in the appendix.\n\n# Overall recommendation\n\n7: Accept\n\n# Other comments and suggestions\n\n- The original training hyperparameters could be much more clearly presented in the main body of the paper\n- It would be helpful to show the original accuracy of the fully trained networks (at various different learning rates) on the plots\n\n# References used in review:\n\n[1] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. \"Linear Mode Connectivity and the Lottery Ticket Hypothesis\"\n\n\n# Update after author response\n\nThanks to the authors for their response. I appreciate the inclusion of Figure A13,  though more details (specifically, whether the plot for LR=0.2 is both LR_find and LR_eval, as I suspect but can't confirm) and more runs (specifically, the decoupled LR lines in Fig 4a that outperform LR=0.05, probably the LR_find=0.05 and LR_eval=0.2) would be appreciated. \n\nI don't agree with the authors' claims about standard LT (rewinding to iteration 0) being equally as valid of an object of study as rewinding to later iterations on these large-scale networks, because by the original definition of lottery ticket (a sparse randomly initialized subnetwork that matches the accuracy of the full network in the same amount of training time), there do not exist standard lottery tickets on ResNet-50 at nontrivial sparsities using the standard learning rate schedules, and the resultant \"lottery ticket\" network trains little better than a random subnetwork [1, Figure 10], implying that LR_find may not actually be a relevant hyper-parameter in this large-scale rewind-to-0 context. I still think more discussion of this point is also warranted when discussing results on ResNet-50 when rewinding to iteration 0.\n\nRegardless, I believe that the paper presents and thoroughly validates an interesting hypothesis, and I maintain that the paper should be accepted.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_pKb5rbcb4v", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper studies the effect of different learning rates on finding performant pruning masks that can be applied to train a sparsified neural network in a lottery-ticket-style framework (i.e. train the original random initialization with the resulting pruning mask applied). The study hereby focuses on a particular setting: find a pruning mask using iterative global magnitude pruning (IMP) applied to various ResNets trained on Tiny ImageNet. The paper finds that a small learning rate is beneficial in finding a performant pruning mask, although the resulting network's performance may be worse, while a large learning rate is better suited to then optimize the sparsified network in the subsequent lottery-ticket-style training procedure. The authors further hypothesize that the per-layer prune ratios (LPRs) found by small learning-rate IMP masks are the main driving factor in increasing the performance of the sparse network in the subsequent training process. This is corroborated with a range of experiments where the LPRs from small learning rate IMP pruning masks is used to find masks (with pre-specified LPRs) using larger learning rates. \n\n\n## Score\n\nI enjoyed reading the paper and found the resulting conclusions quite interesting. As pointed out by related work before [1, 2, 3] the learning rate can have a huge impact on the performance of lottery tickets and related experimental settings (such as lottery-ticket-style masks with random re-initialization of the weights). This paper provides additional experimental evidence for this phenomenon by separating the effects of training hyperparameter on the mask-finding procedure and the sparse-training procedure. \n\nHowever, I am not entirely convinced about the framing of the paper itself. I believe the experiments are valid and have merit on their own but in my opinion the authors mix two related but distinct concepts, namely: \n1.  Lottery Tickets (LTs): Pruning masks that occur (in the general setting as pointed out by [2, 3]) _early in training_ and _not at initialization_. Specifically, the LT hypothesis in the more general setting states that if the experiment is repeated _exactly the same_ starting from that early iteration in training but with the applied pruning mask, it does not harm the performance of the resulting network.\n2. Pruning at initialization: Inspired by LT [4, 5, 6] (but also concurred work [7] with LT), various authors have proposed pruning methods that sparsify the network at the time of random initialization and then train the resulting, sparsified network. These methods usually perform better than when the network is pruned uniformly at random. \n\nConsequently, the conclusions that are drawn in this paper are somewhat confusing, it is really hard to discern the generality of the results, and to understand what is actually observed. Are these observations related to LTs or are the authors proposing another method, i.e. small learning rate IMP, to find performant pruning masks at initialization? \n\nI think that must be clarified before this paper can be accepted in order to for the paper to be beneficial to the community as a whole and to ensure that the observations are not just spurious effects of the particular experimental settings. \n\n## Ways to Improve My Score\n\nMainly, I think the paper must be properly contextualized in a clear setting that either studies LTs in general or studies pruning at initialization. Right now it sits in the middle and thus it hards to draw appropriate conclusions as the reader of the paper. Specifically, I can see two avenues for the paper to improve its framing of the results: \n1. Show that the conclusions hold for a more _general_ LT setting. That is, use a version of Algorithm 1 that rewinds to a stable training phase, c.f. [3, 8], where it has been previously shown that valid LTs can be found using standard IMP. As a result that would naturally require the authors to repeat the experiments. \n2. Frame the procedure (small learning rate IMP) as another way to perform pruning at initialization, which performs well. That would require the authors to compare to other pruning at initialization methods [4-7] to understand the performance. Also this would raise the question of why one would use the method in the first place since it is computationally much more expensive than the methods of [4-7]. \n\nI have additional feedback in the \"Weaknesses\" section as discussed below. The minor feedback won't necessarily change my score but I believe can help you strengthen the paper upon publication. \n\n## Strengths\n\n* I commend the authors for a very clean and well-written paper. It is easy to follow, well-structured, and provides sufficient context. \n\n* The experimental study seems to be carried out at a high level. Hyperparameters are clearly summarized and the provided level of detail is sufficient to reproduce the experiments. \n\n* Each plot is clearly labeled and contains shaded error region (i.e. experiments were repeated multiple times). Plots are also all well-structured and easy to interpret.\n\n* The study on the effect of separate hyperparameters for sparse training and discovering sparse masks could be helpful in guiding future research on pruning. \n\n* The part about LPR being the main driven factor for the improved performance is really interesting. \n\n## Weaknesses\n\nOn top of the concerns I have previously raised, here are some additional points of feedback: \n\n* What is the reason to mostly rely on Tiny ImageNet for drawing the conclusions? Most pruning work considers CIFAR and ImageNet. Since the conclusions of the paper are dependent upon previous papers it would be helpful to consider the inclusion of at least ImageNet. \n\n* Why do the late rewinding experiments use epoch 2 as rewind epoch? I believe this is misleading. As stated in [3, 6] the \"stable phase\" of training where LTs can be observed in a general setting usually don't occur until later in the training. So these experiments don't really add value since epoch 2 is most likely not in the stable phase and so it might be confusing to the reader rather than helpful.\n\n* This is mostly related to my main points raised in the \"Score\" section. Since all of these experiments consider rewinding to an unstable phase of training (i.e. back to the random initialization or a very early in training), I am really uncertain whether these conclusions hold for LTs or pruning at initialization. To me personally, the main conclusion is that if I want to use IMP for finding a pruning mask at initialization, I should use a small learning rate. But then again. Why would I use IMP to find a pruning mask at initialization if there is more efficient methods [4-7] for that? \n\n* The work of [9] is drawing somewhat analogous conclusions about the importance of the distribution of LPRs for pruning at initialization. In particular, the authors conclude that mimicking the LPRs found during pruning at initialization with various methods is essentially sufficient to reproduce the accuracy of the resulting sparsely trained network. I couldn't find a version of [9] in a peer-reviewed venue, so it might count as concurrent submission but nonetheless I think it is crucial to compare the results. \n\n## Other Minor Feedback\n\n* Please clarify in the introduction that all observations are entirely limited to magnitude pruning. I don't think that any conclusions can be drawn for pruning in general and some of the text may hint at a more general phenomenon. \n\n* You could add the appendix directly to the main document. It will be easier to read and jump back and forth between the main body and the appendix. \n\n## References\n\n1. [Rethinking the Value of Network Pruning](https://openreview.net/forum?id=rJlnB3C5Ym)\n2. [The State of Sparsity in Deep Neural Networks](https://arxiv.org/abs/1902.09574)\n3. [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://arxiv.org/abs/1912.05671)\n4. [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://openreview.net/forum?id=SkgsACVKPH)\n5. [Pruning neural networks without any data by iteratively conserving synaptic flow](https://arxiv.org/abs/2006.05467)\n6. [Progressive Skeletonization: Trimming more fat from a network at initialization](https://arxiv.org/abs/2006.09081)\n7. [Snip: Single-shot network pruning based on connection sensitivity](https://openreview.net/forum?id=B1VZqjAcYX)\n8. [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://openreview.net/forum?id=S1gSj0NKvB)\n9. [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://arxiv.org/abs/2009.08576)", "extra_scores": {}, "extra_text": {}}, {"review_id": "YDc8vPE5OVD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nIn the lottery ticket hypothesis's general framework, network pruning's two phases of training (before and after the network is actually pruned) use the same hyperparameters - in particular, the learning rate schedule is identical in both phases.  Using different learning rates for these two phases, and, unintuitively, preferring a learning rate in phase 1 (finding the mask) that results in a *worse* dense model, can result in a better final model (after training with the mask in place).  It is also shown that the layerwise pruning ratios may be the key to understanding this behavior: finding the proper LPR is best done with a small learning rate, but given an LPR, a large learning rate is better able to determine the specific mask.\n\n##########################################################################\n\nReasons for score: \n\nI rated this submission as marginally below the acceptance threshold mainly because I'm struggling to find the practical benefit of the findings.  On one hand, decoupling learning rates seems to work for the limited results contained in the submission, but this boils down to \"here's an extra hyperparameter we should be testing.\"  (This hyperparameter is also not new in all contexts, only in the LTH.)  On the other hand, these results pointed to the LPR as being crucial for a good pruned model.  Back to the first hand, though, what do we do with this information?  The submission doesn't offer any guidance about where to go next, except that perhaps other hyperparameters should be similarly decoupled, which isn't the most comforting thought for practitioners.  Finally, the claim that decoupling learning rates is brand new should be tempered somewhat (see below).\n\n##########################################################################\n\nPros:\n\n+ This paper is well-written and fairly easy to follow with a straightforward organization. \n+ Decoupling learning rates in the LTH framework seems novel and useful.\n+ Not argued by the authors, but I'll add/clarify: while decoupling learning rates might not be new for all pruning methods (traditionally pruning+fine-tuning, as opposed to rewinding), it *is* novel to use a lower learning rate during the mask finding phase, which may result in a worse dense model, from which a better sparse model is created.  (I don't think this has been validated empirically anywhere for prune+fine-tune approaches, so maybe it's best left as future work and the claim re-worded to not run afoul of past work that uses different learning rates.)\n+ The importance of a good LPR is clearly motivated, and the experiments seem sufficient to prove the causal relationship.\n\n##########################################################################\n\nCons:\n\n- It's hard to tell what the practical benefit of these findings are.  I'd have expected a new SOTA result for some task with the LTH framework and decoupled learning rates, or at least a clear comparison of \"the best baseline LTH vs. the best decoupled LRs LTH over a range of sparsities,\" but I can't find such a result anywhere.\n- The discussion suggests practitioners incorporate \"simple additional sweeps of decoupled learning rates.\"  These sweeps may be simple (they may not, depending on the training framework), but they will almost certainly be very expensive for most production data sets.  (The cost is further compounded when considering the early pruning results of section 3.4.)  Useful guidance for limiting costly hyperparameter sweeps, either from a theoretical grounding or wider empirical studies, would make this requirement less painful.\n- Decoupling learning rates before and after pruning in general is nothing new as claimed in the introduction; past work (e.g. Han et al., 2015, referenced in the submission) specifically use different learning rates before and after pruning, so novelty/scope takes a slight hit. That said, applying it to the LTH framework is new, and it uncovered the dependence of a good model on finding a good LPR set.  (Also, as I argued *for* the submission above, it's new that the goal of the mask finding stage isn't necessarily to find the best-performing dense model; a worse dense model can lead to a better pruned model.)\n\n##########################################################################\n\nQuestions:\n\n- In section 3.2's late rewinding experiments, is the LR warmed up for both the find and eval phases?\n- At the end of 3.2's structured pruning: \"... decoupling LRs does not just apply to lottery tickets, but rather applies to global magnitude pruning in general.\"  Isn't this experiment still essentially the LTH framework, but with structure?  Application to \"magnitude pruning in general\" is too a broad claim, given past work (again, Han et al., 2015).  Or, if there was some experimental setup other than train, prune (with structure), rewind, etc., that is decidedly *not*-LTH, please clarify.\n- The text description of Figure 5 describes different pruning iterations -- what are these iterations?\n\n##########################################################################\n\nMinor suggestions:\n\n- Please add the particular network/data set used for each figure or table in the captions.\n- There's a double \"linear\" in the LR warmup description (just one is enough).\n- Similarly, the first sentence of 3.3 has a double \"also.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q6yMxe5Ctej", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThe authors propose to use a different learning rate (LR) when fine-tuning a pruned model (LR_eval) than the learning rate used for the original training (LR_find). They empirically demonstrate in a Lottery Ticket (LT) framework, using a ResNet50 on Tiny ImageNet, that the LR that produced the best model before pruning does not correspond to the LR that produces the best results after fine-tuning. They then further show this phenomenon on other LT setups (with late rewinding, LR warm-up, LR rewinding), architecture (ResNet18), and dataset (MiniPlaces), and for structured pruning as well. Finally, they empirically show how LR_find impacts the pruning ratio obtained at each layer, and that high LR_find seems to prune too aggressively the first layers.\n\n### Strengths:\n+ The paper tries to provide better understanding of pruning methods, rather than introducing a new heuristic, which is an extremely important research direction in network pruning.\n+ The results are presented on different architecture, datasets, LT variants, and on both structured and unstructured pruning setups.\n\n### Concerns:\n+  I have issues the motivation behind the proposed method. In particular, these few sentences (listed below) sound quite incorrect to me, at least in the LT framework, which is the framework used for in this empirical evaluation:\n   - _\"However, prior work has implicitly assumed that the best training configuration for model performance was also the best configuration for mask discovery.\"_ \n   - _\"Common practice rests on the assumption that models with the best performance will generate the best masks, such that the optimal hyper-parameters for mask generation and mask evaluation should be identical.\"_\n  - _\"If we had to use the same LR for everything, we would have to make a trade-off between a better unpruned performance and a better pruned model performance.\"_\n  - _\"If we had followed common practice in the past, we would have swept over some LR values over standard training, found that 0.5 performed the best, trained LT with 0.5 as the LR\\_find, and gotten poor performance.\"_\n\n  In a pruning scenario where we have to both train the network and then prune it, like in the LT framework, the hyper-parameters are optimized to produce the best performing models after the iterations of pruning and fine-tuning, not the best performing network before the first pruning!\n  Ideally, one would have to tune all the hyper-parameters, which means the hyper-parameters at each of the LT iterations, not only the ones of the last iteration as proposed, which is prohibitively expensive. However Renda et al. (2020) showed that using the same ones at each iteration is a good heuristic, and this is precisely what the experiments of this paper seems to show (the best overall results are: in Figure (2) LR_eval = LR_find = 0.2, in Figure (3) LR_find = LR_eval = 0.2, in Figure (A1) LR_find = LR_eval = 0.1, etc...). I do agree however that one should not first tune the hyper-parameters to obtain the best network, and then reuse these same hyper-parameters for the pruning stages, and your experiments indeed show that, but I don't think this is what practitioners do in practice.\n\n+ While the analysis of how the pruning ratio per layer changes depending on LR_find is quite interesting, it lacks a bit of insights. What is causing this? Are other pruning criteria also affected? One can imagine the neural networks trained with high learning rate have gradient propagation issues (Magnitude Pruning removes the smallest weights, so the ones that received the smallest gradient). On that note, how are the networks initialized (this information is missing from the document)?\n\n### Reasons for score:\nAs it stands, I don't think this paper brings much new insights, and I don't think the assumptions the authors made about current practices, which are the motivation for the propose method, are correct.\n\n### Questions:\n- At each iteration of LT, what is the model selected to compute the mask on? Is it the model obtained at the last iteration, or the model that was best performing on the validation set? I imagine this could have a big impact on the results, as a high learning rate might produce models drastically different.\n\n### Other comments:\nThere are a few inaccuracies in the paper:\n - (2.1) what are one shot and local pruning (it should be defined in the paper)?\n - What are error bars representing in the figure? Standard deviation? Standard error? How many seeds were used? This should be precised in the document.\n - (2.2) What does \"decaying the learning rate with a gamma of 0.2\" means? Again, the type of decay used should be precised.\n\nFinally, Footnote 1 is quite unnecessary.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the role of hyperparameters in the separate phases of a classic pruning pipeline: mask identification and retraining. Key observations include a set of the hyperparameters to search relative to a standard regime as well as the identification that the layerwise pruning rates from mask finding are intertwined with these hyperparameters and are what chiefly affects the eventual performance of the pruned network.

The pros of this paper are that it works against the contemporary wisdom that the default hyperparameters for a model are the best for finding a mask for the model. Instead, there are improvements to be had by identifying a set of hyperparameters that lead to worse overall model accuracy, but better masks. Second, the work shows that the layerwise pruning rates are the key elements of these hyperparameters effect. The rates can in fact be transferred to more poorly performing network configurations and improve performance.

The cons of this paper, as noted by the reviewers, are the somewhat unclear implications of the technique. The added guidance on directions to improve hyperparameters is valuable but does not necessarily provide a cost-effect strategy to find these. At its strongest, this guidance offers practitioners a recommendation to also consider hyperparameters for the initial model. 

The stronger, forward-looking implication is, instead, the connection to layerwise pruning rates. Specifically, while layerwise pruning rates have been demonstrated to be important in the literature (e.g., [1]), there has been a limited study into the exact nature of a good set of pruning rates versus a bad set of pruning rates.  Where this paper stops short of a clear result, is if were to connect excessive pruning of the earlier layers, or simply the layerwise rates themselves to another property of the network (e.g., gradient flow, or capacity) that indicates the improved eventual performance.

My Recommendation is to Reject. The paper's core experiments are well-executed. However, this final detail, closing the gap between the portability of these layerwise rates and a conceptual understanding, is a key missing component.  Once done, that will make for a very strong paper.

[1] AMC: AutoML for Model Compression and Acceleration on Mobile Devices. Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, Song Han. EECV, 2018


Decision: Reject