Comparing Rewinding and Fine-tuning in Neural Network Pruning
Authors: Alex Renda
Jonathan Frankle
Michael Carbin
Venue:  (2020)
Keywords: pruning
sparsity
fine-tuning
lottery ticket

Abstract: Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, we compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al., (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.


Published as a conference paper at ICLR 2020
COMPARING REWINDING AND TUNING IN NEURAL NETWORK PRUNING
Alex Renda MIT CSAIL
Jonathan Frankle
Michael Carbin
MIT CSAIL j frankle@csail iitiaaalmmmtit.... edu
MIT CSAIL
renda@csail. mit mit
edu
mcarbin@csail mit edu
ABSTRACT
Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al. (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-ard techniques.
INTRODUCTION
Pruning a set techniques for removing weights, filters, neurons, or other structures from neural networks (e.g.. Le Cun al., 1990; Reed, 1993; Han et al., 2015; Li et al., 2017; Liu al., 2019). Pruning can compress standard networks across a variety of tasks, including computer vision and natural language processing, while maintaining the accuracy of the original network. Doing so can reduce the parameter count and resource demands of neural network inference by decreasing storage requirements, energy consumption, and latency (Han, 2017).
We identify two classes of pruning techniques in the literature. One class, exemplified by regulariza- tion (Louizos et al., 2018) and gradual pruning (Zhu & Gupta, 2018; Gale et al., 2019), prunes the network throughout the standard training process, producing pruned network by the end of training.
The other class, exemplified by retraining (Han al., 2015), prunes after the standard training process. Specifically, when parts of the network are removed during the pruning step, accuracy typically decreases (Han al., 2015). It is therefore standard retrain the pruned network to recover accuracy. Pruning and retraining met; doing so often results in higher accuracy than pruning in one shot (Han et al., 2015). single iteration of the retraining based pruning algorithm proceeds follows (Liu al., 2019):
be repeated iteratively until target sparsity or accuracy threshold is
1. TRAIN the network to completion.
2. PRUNE structures of the network, chosen according to some heuristic.
3. RETRAIN the network for some time (t epochs) to recover the accuracy lost from pruning.
The most common retraining
trains the pruned weights for further t epochs at a fixed learning rate (Han et al., 2015), often the final learning rate from training (Liu et al., 2019).
Work the lottery ticket hypothesis introduces new retraining technique, weight rewinding (Frankle et al., 2019), although Frankle et al. do not evaluate as such. The lottery ticket hypothesis proposes that early in training, sparse subnetworks emerge that train in isolation to the same accuracy as the original network (Frankle Carbin, 2019). To find such subnetworks, Frankle et al. (2019) propose training to completion and pruning (steps and above) and then rewinding the unpruned

---

Published as a conference paper at ICLR 2020
weights by setting their values back to what they were earlier in training. If this pruned and rewound subnetwork trains to the same accuracy as the original network (reusing the original learning rate schedule), for their purposes- this validates that such trainable subnetworks exist early in training. For our purposes, this rewinding and retraining technique simply another approach for retraining after pruning. The selection of where to rewind the weights is controlled by the retraining time t; retraining for epochs entails rewinding to epochs before the end of training.
We also propose new variation of weight rewinding, learning rate rewinding. While weight rewinding rewinds both the weights and the learning rate, learning rate rewinding rewinds only the learning rate, continuing train the weights from their values at the end of training (like fine-tuning). This is similar to the learning rate schedule used by cyclical learning rates (Smith, 2017).
In this paper, we compare fine-tuning, weight rewinding, and learning rate rewinding as retraining techniques after pruning. We evaluate these techniques according to three criteria:
ACCURACY EFFICIENCY
The accuracy of the resulting pruned network.
The resources required to represent or execute the pruned network.
SEARCH COST
The cost to find the pruned network (i.e., the amount of retraining required).
The goal of neural network pruning is to increase EFFICIENCY while maintaining ACCURACY In this paper we specifically study PARAMETER- EFFIREECE the parameter count of the pruned neural network.' We also evaluate the SEARCH CoST of finding the pruned network, measured as the number epochs for which the network is retrained.
We compare the pruning and retraining techniques evaluated in this paper against pruning algorithms from the literature that are shown to be state-of-the-art by Ortiz et al. (2020). These state-of-the-ant algorithms complex to use, requiring network-specift. hyperparameters (Curreir-Perrpinán Idelbayev, 2018; Zhu & Gupta, 2018) or reinforcement learning (He et al., 2018).
Perpiñán &
Contributions.
We show that retraining with weight rewinding outperforms retraining with fine-tuning across networks and datasets. When rewinding to anywhere within a wide range of points through- out training, weight rewinding is drop-in replacement for fine-tuning that achieves higher ACCURACY for equivalent SEARCH COST.
We propose simplification of weight rewinding, learning rate rewinding, which rewinds the learning rate schedule but not the weights. Learning rate rewinding matches or outperforms weight rewinding all scenarios.
We propose a pruning algorithm based on learning rate rewinding with network-agnostic hyperparameters that matches state-of-the-art tradeoffss between ACCURACY and PAR PARAMETER- EFFICINCYY across networks and datasets. The algorithm proceeds as follows: train to completion, 2) globally prune the 20% of weights with the lowest magnitudes, 3) retrain with learning rate rewinding for the full original training time, and 4) iteratively repeat steps and until the desired sparsity is reached.
We find that weight rewinding can nearly match the ACCURACY of this proposed pruning algorithm, meaning that lottery tickets found by pruning and rewinding are state-of-the-art pruned networks.
We show that learning rate rewinding outperforms the standard practice of fine-tuning without requiring any network-specific hyperparameters in all settings that study. This technique forms the basis of a simple, ofate-of-the-ar pruning algorithm that we propose as a valuable baseline for future research and compelling default choice for pruning in practice.
METHODOLOGY
In this paper, we evaluate weight rewinding and learning rate rewinding as retraining techniques. We therefore do not consider regularization or gradual pruning techniques, except when comparing against otate-of-the-art. Creating a retraining based pruning algorithm involves instantiating each
We discuss other instantiations of EFFICIENCY, such FLOPs, in Section 6 and Appendix F.
2

---

Published as a conference paper at ICLR 2020
of the steps in Section (TRAIN, PRUNE, RETRAIN) from range of choices. Below, discuss the set of design choices considered in our experiments and mention other standard choices. Our implementation and the data from the experiments in this paper available at: https: /// thub. com/ www/lottery..eeeeoooooo ooteey-tttte eeneeeryytcieek ciket/ rewi indiing ng- iel120 r20- 1apppuu1ic
2.1 How DO WE TRAIN?
We assume that TRAIN is provided as the standard training schedule for network. Here, discuss the networks, datasets, and training hyperparameters used in the experiments in this paper.
We study neural network pruning on a variety of standard architectures for image classification and machine translation. Specifically, we consider ResNet-56 (He et al., 2016) for CIFAR- 10 (Krizhevsky, 2009), ResNet-34 and ResNet-50 (He al., 2016) for ImageNet (Russakovsky et al., 2015), and GNMT (Wu et al., 2016) for WMT16 EN-DE. Our implementations and hyperparameters are from standard reference implementations, as described in Table I, with the exception of the GNMT model. For GNMT, we extend the training schedule used in the reference implementation to reach standard BLEU scores on the validation set, rather than the lower BLEU reached by the reference implementation." This extended schedule uses the same standard GNMT warmup and decay schedule as the original training schedule (Luong et al., 2017), but expanded to span epochs rather than
Network
Optimizer
Learning truining spach)
accuracy
Nesterov SCID
0.9
0.1
E [u. 92)
CIFAR- 10
ResNet.56
852K
Rutch size: 128
0.01
LE [91, 136)
93.46 1: 0.21%
Weight decay: 0.0002
0.001
E [136, 182]
Fpochs: 182
Nesterov SGD
0.44 g
EE (0,
R
21.8M
0.9
0.4
E (0)
73.60 0.27% top
ImageNer
Batch size: 1024
0.04
[30,
ResNet-50f
25.5M
Weight decay: 0,0001
0.004
E (60, 80)
top-1
Epochs: 90
0.0004 E [s0, 90]
Adam A1 0.9
0.002
0.01 1-x
E [0, 0.16)
0.002
E [0.125, 8.75)
WMT16 EN-DE
G
165M
0.999
[3.76, 4.163)
26.87 + 0.23 BLEU
new. 2015:
0.0005
E [4.165, 4.58)
Epochs:
0.00025
e (4.58, 5)
Table 1: Networks, datasets, and hyperparameters. We use standard implementations available online and standard hyperparameters. All accuracies are in line with baselines reported for these networks (Liu al., 2019; He et al., 2018; Gale et al., 2019; Wu et al., 2016; Zhu & Gupta, 2018).
2.2 How DO WE PRUNE?
What structure do we prune?
UNSTRUCTURED PRUNING. Unstructured pruning prunes individual weights without consideration for where they occur within each tensor (c.g., Le Cun et al., 1990; Han et al., 2015).
STRUCTURED PRUNING. Structured pruning involves pruning weights in groups, removing neurons, convolutional filters, or channels (e.g., Li et al., 2017).
Unstructured pruning reduces the number of parameters, but may not improve performance on commodity hardware until a large fraction of weights have been pruned (Park et al., 2017). Struc- tured pruning preserves dense computation, meaning that it can lead immediate performance improvements (Liu et al., 2017). In this paper, we study both unstructured and structured pruning.
What heuristic do we use to prune?
MAGNITUDE PRUNING. Pruning weights with the lowest magnitudes (Han et al., 2015) is standard
"The reference implementation is from MLPerf 0.5 (Mattson et al., 2020) and reaches newstest-201. BLEU of 21.8. With the extended training schedule, reach more standard BLEU of 24.2 (Wu et al., 2016).
Phctps://githen httt:::://gghhaa..c//eeeeeee
"kstps://githe http:////shaaaacccccssssst/t//// http:///cw..ooeoe///////////////
'rttpa://oo /ttpp://githhh.... ccw/al...cmmmm//////////..m pamm///arr////atnnnen
a/goog e/cloud_yc v3. 3.g.t/ttnttt

---

Published as a conference paper at ICLR 2020
choice that achieves ofate-of-thheart ACCURACY versus EFFICIENCY tradeoffs (Gale al., 2019). For unstructured pruning. we prune the lowest magnitude weights globally throughout the network (Lee et al., 2019; Frankle & Carbin, 2019). For structured pruning, we prune convolutional filters by their L1 norms using the per-layer pruning rates hand-chosen by Li et al. (2017). 6 Specifically, we study ResNet-56-B and ResNet-34-A from Li et al. (2017).
We only consider magnitude- based pruning heuristics in this paper, although there are a wide variety of other pruning heuristics in the literature, including those that learn which weights to prune as part of the optimization process ge.g., Louizos et al., 2018; Molchanov et al., 2017) and those that prune based on other information (e.g.. Le Cun et al., 1990; Theis et al.. 2018; Lee al., 2019).
2.3 How DO WE RETRAIN?
Let Wa E IR be the weights of the network at epoch g. Let E {0, 1]d be the pruning mask, such that the element-wise product @ m denotes the pruned network. Let T be the number of epochs that the network is trained for. Let S[g] be the learning rate for each epoch g. defined such that Sig > 7] = S[T] (i.e., the last learning rate is extended indefinitely). Let TRAIN' m. g) be a function that trains the network Wo O m for / epochs according to the original learning rate schedule S, starting from epoch g-
retrains the unpruned weights from their final values for a specified number of epochs using a fixed learning rate. Fine-tuning is the current standard practice in the literature (Han et al., 2015; Liu al., 2019). is typical fine-tune using the last learning rate of the original training schedule (Li et al., 2017; Liu et al., 2019), convention we follow in our experiments. Other choices are possible, including those found through hyperparameter search (Han al., 2015; Han, 2017; Guan et al., 2019). Formally, fine-tuning for 1 epochs runs TRAIN' (Wr, m, T).
FINE-TUNING.
WEIGHT REWINDING. Weight rewinding retrains by rewinding the unpruned weights to their values from i epochs earlier in training and subsequently retraining the unpruned weights from there. It also rewinds the learning rate schedule to its state from I epochs earlier in training. Retraining with weight rewinding therefore depends on the hyperparameter choices made during the initial training phase of the unpruned network. Weight rewinding was proposed to study the lottery ticket hypothesis by Frankle et al. (2019). Formally, weight rewinding for t epochs runs TRAIN' (Wr-4, m, t).
LEARNING RATE REWINDING. Learning rate rewinding is a hybrid between fine-tuning and weight rewinding. Like fine-tuning, it uses the final weight values from the end of training. However, when retraining for & epochs, learning rate rewinding uses the learning rate schedule from the last t epochs of training (what weight rewinding would use) rather than the final learning rate from training (what fine-tuning would use). Formally, learning rate rewinding for / epochs runs TRAIN' (Wr, m, T l). We propose learning rate rewinding in this paper as a novel retraining technique.
In this paper, we compare all three retraining techniques. For each network, we consider ten retraining times & evenly distributed between 0 epochs and the number of epochs for which the network was originally trained. For iterative pruning, this retraining time is ran per pruning iteration.
2.4 DO WE PRUNE ITERATIVELY?
ONE-SHOT PRUNING. The outline above prunes the network to a target sparsity level all at once, known one-shot pruning (Li et al., 2017; Liu et al., 2019).
ITERATIVE PRUNING. An alternative to iterate steps and 3, pruning weights (step 2), retraining (step 3). pruning more weights, retraining further, etc., until target sparsity level is reached. Doing so is known as iterative pruning. In practice, iterative pruning typically makes it possible to prune more weights while maintaining accuracy (Han et al., 2015; Frankle & Carbin, 2019).
In this paper, we consider both one-shot and iterative pruning. When running iterative pruning, we prune 20% of weights per iteration (Frankle & Carbin, 2019). When iteratively pruning with weight rewinding, weights are always rewound to the same values Wr-t from the original run of training.
"To study multiple sparsity levels using these Hand-chosen rates, we extrapolate these per-layer pruning rates to higher levels of sparsity, by exponentiating each per-layer pruning rate p. (which denotes the resulting density of layer i) by l E 21,2, 4, 5). creating new per-layer pruning rates p;-
4

---

tradeoffs, and Published as a conference paper at ICLR 2020
2.5 METRICS
We evaluate pruned network according to three criteria.
ACCURACY is the performance of the pruned network on unseen data from the same distribution as the training set (i.e., the validation or test set). Higher accuracy values indicate better performance, and a rypical goal is to match the accuracy of the unpruned network. All plots show the median, minimum, and maximum test accuracies reached across three different training runs.
For vision networks, we use 20% of the original test set, selected at random, as the validation set; the remainder of the original test set is used to report test accuracies. For WMT16 EN-DE, we use newstest 2014 4 as the validation set (following Wu et al., 2016), and newstest2015 as the test set (following Zhu & Gupta, 2018).
EFFICIENCY is the resources required to represent or perform inference with the pruned network. This can take multiple forms. We study We specifically measure ratio of the pruned network. For instance, if the pruned network has 5% of weights remaining, then its compression ratio is 20x. Higher compression ratios indicate better PARAMETEE-EEITCIENNS EFTICIENCY We discuss other instantiations of EFFICIENCY in Section 6 and Appendix F
the parameter count of the network. relative to the full network with the compression
SEARCH COST is the computational resources required to find the pruning mask and retrain the remaining weights. We approximate SEARCH CoST using retraining time, the total number of additional retraining epochs. Fewer retraining epochs indicates a lower SEARCH COST. Note that this metric does not consider speedup from retraining pruned networks. For instance, a network pruned to 20x compression may be faster to retrain than if only pruned to 2x compression.
3 ACCURACY VERSUS
TRADEOFF
In this section, we consider the Pareto frontier of the tradeoff' between ACCURACY and PARAMETER- EFfiCieNcY using each retraining technique, without regard for SEARCH COST. In other words, we study the highest accuracy each retraining technique can achieve at each compression ratio. We find that weight rewinding can achieve higher accuracy than fine-tuning across compression ratios on all studied networks and datasets. We further find that learning rate rewinding matches or outperforms weight rewinding in scenarios. With iterative unstructured pruning, learning rate rewinding achieves state-of-the th-t--rr- ACCURACY versus PARAMETER-EPFICHA weight rewinding remains close.
Methodology. For each retraining technique, network, and compression ratio, we select the setting of retraining time with the highest validation accuracy and plot the corresponding test accuracy.
One-shot pruning results. Figure presents the results for one-shot pruning. At low compression ratios (when all techniques match the accuracy of the unpruned network), there is little diffcrentiation between the techniques. However, learning rate rewinding typically results in higher accuracy than the unpruned network, whereas other techniques only match the original accuracy. At higher compression ratios (when no techniques match the unpruned network accuracy), there is more differentiation between the techniques, with fine-tuning losing more accuracy than either rewinding technique. Weight rewinding outperforms fine-tuning in all scenarios. Learning rate rewinding in turn outperforms weight rewinding by a small margin.
Iterative pruning results. Figure presents the results for iterative unstructured pruming. As basis for comparison, we also plot the drop in accuracy achieved by statc-of-the-an art techniques (as described in Appendix C and shown to iterative pruning, weight rewinding continues to outperform fine-tuning, and learning rate rewinding continues to outperform weight rewinding. Learning rate rewinding matches the ACCURACY versus PARAMETER- EFFEEREE-EEY tradeoffs of state-of-the-art techniques across all datasets. In particular, learning rate rewinding with iterative unstructured pruning produces a ResNet-50 that matches the accuracy of the original network at 5.96x compression, to the best of our knowledge a new state- of-the--rt ResNet-50 compression ratio with no drop in accuracy. Weight rewinding nearly matches these state-of-the-art results, with the exception of high compression ratios on GNMT.
by Ortiz et (2020) as individual black dots. In
5

---

Published as a conference paper at ICLR 2020
One-shot ACCURACY versus PARAMETER-EFFICHES Tradeoff
Unstructured
Structured
ResNet-50 Unstructured
ReaNet-55 56 Structured
+1%
+1%
CIFAR-10
Accurac
Accuracy
0%
0%
-1%
-1%
-2%
-2%
55%
1
2.44x
4.77x
9.31x
1.16x
1.29x
1.43x
1,57
1.70x
Compression ratio
Compression ratio
ResNet-50 Unstructured
ResNet-34 Structured-A
+1%
+11%
ImageNet
A Accuracy
Accuracy
0%
0%
-1%
-1%
-2%
-3%
1.25X
2,44%
4.77x
9.31x
1,08x
1.15x
1.20x
1.24x 1
Compression ratin
Compression ratio
GNMT Unstructured
WMT16
A BLEUU
Learning rate rewinding
Weight rewinding
Fine-tuning
-6
1.25x
3.65x
7.45x
18.18x
Compression ratio
Figure 1: The best achievable accuracy across retraining times by one-shot pruning.
Iterative ACCURACY versus PARAMETER- Tradeoff
CIFAR-10 ResNet-56 Unstrnctured (iterative)
ImageNet ResNet-50 Unstrictured (iterative)
+1%
Areuracy
Accuracy
0%
0%
-1%
-1%
-2%
Carreirs-erpotii & Edellayees (20.8)
-2%
He al. (208)
-3%
-3%
1.56x
1.7xx
14.55x
41.1x
1.58x
244x
3.81x
5.96x
9.31x
Compression ratio
Compression ratio
WMT16 GNMT Unstructured (iterative)
BLEU
Learning rate rewinding
Weight rewinding
Zhm de Gogala (2017)
Fine- Einettnnng
-6
1.25x
3.05x
7.45x
18x18
Compression ratio
Figure 2: The best achievable accuracy across retraining times by iteratively pruning.
Takeaway, Retraining with weight rewinding outperforms retraining with fine-tuning across networks and datasets. Learning rate rewinding in turn matches or outperforms weight rewinding in all scenarios. Combined with iterative unstructured pruning, learning rate rewinding matches the tradeoff's between ACCURACY and PAR METER-- METER-EFFFCIEENCE achieved by more complex techniques. Weight rewinding nearly matches these state-of-the the-art tradeoffs.

---

Published as a conference paper at ICLR 2020
4 ACCURACY VERSUS SEARCH COST TRADEOFF
In this section, we consider the tradeoff between ACCURACY and SEARCH COST for each retraining technique across selection of compression ratios. In other words, we consider each method's ACCURACY given a fixed SEARCH COST. We show that both rewinding techniques achieve higher accuracy than tne-ttning for a variety of different retraining times t (corresponding to different SEARCH COSTS). Therefore, in many contexts either rewinding technique can serve as a drop-in in replacement for fine-tuning and achieve higher aceuracy. Moreover, we find that using learning rate rewinding and retraining for the full training time of the original network leads to the highest accuracy among all tested retraining techniques, simplifying the hyperparameter search process.
Methodology. Figures 3 (unstructured pruning) and 4 (structured pruning) show the accuracy of each retraining technique as we vary the amount of retraining time; that is, the tradeoff between ACCURACY and SEARCH COST. Each plot shows this tradeoff at specific compression ratio. The left column shows comparisons for No Accuracy Drop, which we define as the highest compression ratio at which any retraining technique can match the accuracy of the original network for any amount of SEARCH COST. The right column shows comparisons for /%/l BLEU Accuracy Drop, which we define the highest compression ratio at which any retraining technique gets within 1% accuracy or 1 BLEU of the original network. We include similar plots for all tested compression ratios in Appendix E. All results presented in this section are for one-shot pruning; Appendix E also includes iterative pruning results, which exhibit the same trends
Unstructured pruning results. Both rewinding techniques almost always match or outperform fine- tuning for equivalent retraining epochs. The sole exception is using weight rewinding and retraining for the full original training time, thereby rewinding the weights the beginning of training: Frankle et al. (2019) show that accuracy drops if weights are rewound too close to initialization, and we find the same behavior here. We define the rewinding safe zone as the maximal region (as a percentage of original training time) across all networks in which both forms of rewinding outperform fine-tuning for an equivalent SEARCH COST. This zone (shaded gray in Figure 3) occurs when retraining for 25% to 90% of the original training time. Within this region, either rewinding technique can serve as a drop-in replacement for mun-tunnng.
With learning rate rewinding, retraining for longer almost always results in higher accuracy. The same is true for weight rewinding other than when weights are rewound to near the beginning of training. On most networks and compression ratios, accuracy from rewinding saturates after retraining for roughly half of the original training time: while accuracy can continue to increase with more retraining, this gain is limited.
Structured pruning results. Structured pruning exhibits the same trends as unstructured pruning, 7 except that retraining with weight rewinding does result in drop in accuracy when retraining for the full training time (thereby rewinding to the beginning of training). This is consistent with the findings of Liu al. (2019), who show that fine-tuning after structured pruning provides no accuracy advantage over reinitializing and training the pruned network from scratch. Liu et al. (2019) indicate that initialization less consequential for retraining after structured pruning than for it is for retraining after unstructured pruning. Since weight rewinding and learning rate rewinding only differ in initialization before retraining, we expect and observe that they achieve similar accuracies when used for retraining after structured pruning.
Takeaway. Both weight rewinding and learning rate rewinding outperform fine-tuning across a wide range of retraining times, thereby serving drop-in replacements that achieve higher accuracy anywhere within the rewinding safe zone. To achieve the most accurate network, retrain with learning rate rewinding for the full original training time (although accuracy saturates after retraining for about half of the original training time).
"On the CIFAR-IO ResNet-56 Structured-B at 1% Accuracy Drop, we find that learning rate rewinding reaches lower accuracy than fine-tuning when retraining for 30 epochs. At this retraining time, the techniques are identical: the learning rate in the last 30 epochs of training, which learning rate rewinding uses, is the same as the final learning rate, which fine-tuning uses. The observed accuracy difference at that point therefore appears be result of random noise, and is not characteristic of the retraining techniques.
7

---

Published as a conference paper at ICLR 2020
Unstructured ACCURACY versus SEARCH COST Tradeoff
No Accuracy Drop
1%/1 BLEU Accuracy Drop
ResNet-56 Unstractured, 4.77x
ResNet-56 Unstructured, 9.31 x
+1%
+1%
CIFAR-10
A Accamacy
Accuracy
0%
0%
-1%
-1%
25%
-2%
-3%6
-3%
67
153
96
253
182
Retraining epochs
Retraining epochs
ResNet-50 50 Unstructured, x.05x
ResNet-50 Unstructured, 5.96x
ImageNet
A Accurary
0%
Accuracy
0%
496
4%
.6%
-6%
Retraining epochs
Retraining epuchs
GNMT Unstructured, 3.82x
GNMT Unstructured, 5.96x
WMT16
BLEU
A BLEU
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.50
1.25
2.00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
Learning nite rewinding
Weight rewinding
Fino-tuning
Rewinding sufe
Figure 3: Accuracy curves across networks and compression ratios using unstructured pruning.
Structured ACCURACY versus SEARCH COST Tradcoff'
No Accuracy Drop
1% Accuracy Drop
Ros Structured-B, 1.15x
RosNet-56 Structured-B, 1.70x
CIFAR-10
A Accuracy
Arcuracy
-2%
-2%
-1%
-1%
.6%
10
39
67
96
125
153
182
10
39
96
125
153
182
Retraining epochs
Retraining epochs
ResNet-34 Structured-A, 1.08x
ResNet-34 Structure-AA 1.26x
ImageNet
Acmmayy
0%
Acouncy
0%
22%
-2%
-6%
63
Retraining epochs
Retraining epochs
Learning rate rewinding
Weight rewinding
Fine-tuning
Rewinding safe zone
Figure 4: Accuracy curves across networks and compression ratios using structured pruning.
8

---

Published as a conference paper at ICLR 2020
ACCURACY versus PARAMETER-EEFICIENN Tradeoff from Algorithm
CIEAR-10 RosNet-56 Unstructured (iterative)
ImageNet ResNet-50 Unstructured (iterative)
Accurscy
Accurocy
-1%
-1%
-2%
Carretri-Prruuii Idelbwyer (20.8)
-2%
(AUs)
-3%
-3%
1.56x
1.77x
14.55x
4411
1.56x
2
9.31x Compression ratio
3.81x
5.96x
Compression ratio
WMT16 GNMT Unstructured (iterative)
BLEUU
+ Our
Our pruning algorithm
Zii & Grapta (2017)
-6
1.25x
3.05x
7.45x
18.18x
Compression ratio
Figure 5: ACCURACY versus PARAMETER- CHAMETTEREEFICIENY tradeoff of our pruning algorithm.
OUR PRUNING ALGORITHM
Based on the results Sections 3 and 4, we propose a pruning algorithm that is on the state-of-the-art ACCURACY versus PARAMETER-EPaE EFFIRIECEF Pareto frontier. Algorithm I presents instantiation of the pruning algorithm from Section 2 using aggoorc- hyperparameters:
Algorithm 1 Our pruning algorithm
1. TRAIN to completion.
2. PRUNE the 20% lowest-magnitude weights globally.
3. RETRAIN using learning rate rewinding for the original training time.
4. Repeat steps 2 and iteratively until the desired compression ratio reached.
Figure presents an evaluation of our pruning algorithm. Specifically, compare the ACCURACY versus PARAMETER CHRAMETER-EFFFCIENCC tradeoff achieved by our pruning algorithm and by state-of-the-art baselines. This results in the same state-of-the ofet------h-ttt behavior seen in Section 3, without requiring any per-compression-rattee raaio hyperparameter search. We evaluate other retraining methods in Appendix D.
The hyperparameters for our pruning algorithm are shared across all networks and tasks we consider: there are neither layer-wise pruning rates nor pruning schedule to select, beyond the network- agnostic 20% per-iteration pruning rate from prior work (Frankle Carbin, 2019). Moreover, our pruning algorithm matches the accuracy of pruning algorithms that require more hyperparameters and/or additional methods, such as reinforcement learning (He et al., 2018; Carreira- Perpiñán Idelbayev, 2018; Zhu & Gupta, 2018).
DISCUSSION
Weight rewinding. When retraining with weight rewinding, the weights are rewound to their values from early training. This means that after retraining with weight rewinding, the weights themselves receive more gradient updates than in the original training phase. Nevertheless, weight rewinding outperforms fine-tuning and competitive with learning rate rewinding. losing little accuracy even though it reverts most of training. These results show that when pruning, it is not necessary train the weights for a large number of steps; the pruning mask itself is a valuable output of pruning.
Learning rate rewinding. We propose learning rate rewinding, alternative retraining technique that achieves state-of-the-art ACCURACY versus PARAMETER EFFICIENCY - tradeoffs. this paper do not investigate why the learning rate schedule used by learning rate rewinding achieves higher

---

Published as a conference paper at ICLR 2020
accuracy than that of the standard of sparse neural networks can shed light on why learning rate rewinding achieves higher accuracy than standard fine-tuning and can help derive other techniques for the training of sparse networks (Smith, 2017; Dettmers& & Zettlemoyer, 2019).
schedule. We hope that further work on the optimization
The retraining techniques we consider reuse the hyperparameters from the original training process. This choice inherently narrows the design space of retraining techniques by coupling the learning rate schedule of retraining to that of the original training process. There may be further opportunities to improve performance by decoupling the hyperparameters of training and retraining and considering other retraining learning rate schedules. However, these potential opportunities come with the cost of added hyperparameter search.
tradeofl's with our pruning algorithm requires substantial SEARCH COST. Our pruning algorithm requires T (1++ k) total training epochs reach compression ratio / 0.8*, where T is the original network training time, and k is the number of pruning iterations. In contrast, on CIFAR-IO (T' = 182 epochs) Carreira-Perpinán & Idelbayev (2018) employ a gradual pruning technique followed by fine-tuning, training for a total of 317 epochs to reach any compression ratio. On ImageNet (T' = 90 epochs), He et al. (2018) retrain the ResNet-50 for 376 epochs to match the accuracy of the original network at 5.13x compression. On W6 (T = 5 epochs), Zhu & Gupta (2018) use a gradual pruning technique that trains and prunes over the course of about 11 epochs to reach any compression ratio.
SEARCH COST. Achieving state-of-the-art ACCURACY versus CORAMETER-EITICIER
The SEARCH COSTS of these other methods do not take into account the per-network hyperparameter search that each method required to find the settings that produced the reported results, nor the cost of the pruning heuristics themselves (e.g.. training a reinforcement learning agent to predict pruning rates). In addition to optimizing ACCURACY and PARAMETER- PARAMETER-EEFICIENG EFFICIENCY, we believe that pruning research should also consider SEARCH COST (including hyperparameter search and training time).
EFFICIENCY. In this paper we study PARAMETER EFFECEERE::C the number of parameters in the network. This provides a notion of scale of the network (Rosenfeld al., 2020) and can serve as an input for theoretical analyses (Arora et al., 2018). There are other useful forms of EFFICIENCY that we do not study in this paper. One commonly studied form is INFERENCE- EFFICIEEC, the cost of performing inference with the pruned network. This is often measured in floating point operations (FLOPs) or wall clock time (Han, 2017; Han et al., 201 that both rewinding techniques outperform fine-tuning after structured pruning (which explicitly targets INFERENCE- INFERENCE-EFFRIIENCYY In Appendix F, we show that iterative unstructured pruning and retraining with either rewinding technique results in networks that require fewer FLOPs to execute than those found by iterative unstructured pruning and retraining with fine-tuning.
Sections 3 and 4, we demonstrate
Other forms of EFFICIENCY include CTORAGE-EFFICIENTE (Han al., 201 COMMUNICATION- EFFICIENCY (Alistarh et al., 2017), and
(Yang et al., 2017).
The Lottery Ticket Hypothesis. Weight rewinding first proposed by work on the lottery ticket hypothesis (Frankle & Carbin, 2019; Frankle et al., 2019), which studies the existence of sparse subnetworks that can train in isolation to full accuracy from near initialization. We present the first detailed comparison between the performance of these lottery ticket networks and pruned networks generated by standard fine-tuning. From this perspective, our results show that the sparse, lottery ticket networks that Frankle et al. (2019) uncover from early in training using weight rewinding can train full accuracy at compression ratios that are competitive for pruned networks in general.
CONCLUSION
We find that both weight rewinding and learning rate rewinding outperform fine-tuning as techniques for retraining after pruming. When we perform iterative unstructured pruning and retrain with learning rate rewinding for the full original training time, we match the ACCURACT ACY versus PARAMETER- EFFICIENCY tradeoffs of more complex techniques requiring network-speciffe hyperparameters. We believe that this algorithm is valuable baseline for future research and compelling default choice for pruning in practice.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication efficient sgd via gradient quantization and encoding. In Conference on Neural Information Processing Systems, 2017.
Sanjeev Arora, Rong Gc, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, 201 8.
Riyadh Baghdadi, Abdelkader Nadir Debbagh, Kamel Abdous, Benhamida Fatima Zohra, Alex Renda, Jonathan Frankle, Michael Carbin, and Saman Amarasinghe. Tiramisu: A polyhedral compiler for dense and sparse deep learning. In SysML. Workshop, Conference on Neural Information Processing Systems, 2019.
Miguel Á net pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
and Yerlan Idelbayev. Clearning-Comprest algorithms for neural
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance, arXiv preprint urXiv:1901 04840, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis, arXiv preprint arXiv: 19126667,, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, arXiv preprint arXiv: 190v:1902,,99774 2019.
Hui Guan, Xipeng Shen, and Seung-Hwan Lim. Wootz: A compiler-based framework for fast cnn pruning via composability. In ACM SIGPLAN Conference Programming Language Design and Implementation, 2019.
Song Han. Efficient Methods and Hardware for Deep Learning. PhD thesis, Stanford University, 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Conference on Neural Information Processing Systems. 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William Dally. Fie: Efficient inference engine compressed deep neural network. In International Symposium on Computer Architecture, 2016a.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference Learning Representations, 2016b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: Automl for model compression and acceleration on mobile devices. In European Conference on Computer Vision, 2018.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Yann Le Cun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Conference on Neural Information Processing Systems, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr SNIP: Single-shot network pruning bassed on connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
11

---

Published as a conference paper at ICLR 2020
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In International Conference on Computer Vision, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018.
Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. Neural machine translation (seq2seq) tutorial. https://githubb.cm http:///giilub.coo//eeesrr//oo/i 2017.
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu- Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim Hazelwood, Andrew Hock, Xinyuan Huang. Bill Jía, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma. Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Caro Wu, Lingjie Xu, Cliff Young, and Matei Zaharia. MIperf training benchmark. In Conference on Machine Learning and Systems, 2020.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference Machine Learning, 2017.
Jose Javier Gonzalez Ortiz, Davis Blalock, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Conference on Machine Learning and Systems, 2020.
Jongsoo Park, Sheng Li. Wei Wen. Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. In International Conference on Learning Representations, 2017.
Russell Reed. Pruming algorithms survey. IEEE Transactions on Neural Networks, 1993.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
Leslie N. Smith. Cyclical learning rates for training neural networks. IEEE Winter Conference on Applications of Computer Vision, 2017.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning, arXiv preprint arXiv: 881880011,.72220 2018.
Yonghui WU, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi. Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, tukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation, arXiv preprint 1rXiv: 1et.:0609904444 08144, 2016.
Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In International Conference on Learning Representations Workshop Track. 2018.
12

---

Published as a conference paper at ICLR 2020
A ACKNOWLEDGEMENTS
We gratefully acknowledge the support of Google, which provided us with access to the TPU resources necessary to conduct experiments on ImageNet and WMT through the TensorFlow Research Cloud. In particular, we express our gratitude to Zak Stone.
We gratefully acknowledge the support of IBM, which provided us with access to the GPU resources necessary to conduct experiments on CIFAR-10 through the MIT-IBM Watson Al Lab. In particular, we express our gratitude to David Cox and John Cohn.
This work was supported in part by cloud credits from the MIT Quest for Intelligence. This work was supported in part by the Office of Naval Research (ONR N00014-17-1-2699, This work was supported in part by DARPA Award #HR001 18000199
B APPENDIX TABLE OF CONTENTS
Appendix C. We describe the state-of-the-art baselines that we compare against Section 3.
Appendix D. We evaluate other retraining methods using the algorithm described in Section 5.
Appendix F. We extend the results in the main body of the paper to include more networks, pruning techniques, baselines, and ablations.
Appendix F. We discuss theoretical speedup via FLOP reduction, rather than just compression.
C STATE-OF-THE-AN BASELINES
In Sections 3 and 5, we compare rewinding against state-of-the-art techniques from the literature. In this section, we describe the techniques we compare against. We specifically consider the techniques from the literature on the Pareto frontier of the ACCURACY versus PARAMETER EFFICIENCY where ACCURACY is measured as relative loss in accuracy to the original network, and PARAMETER- EFFICIENCY is measured by compression ratio. As there is consensus the definition of state-of-the-art in the pruning literature, we base our search on techniques listed in Ortiz et al. (2020).
-ARAMETER-EPHCIENCYY curve,
CIFAR-10 ResNet-56: Carreira-Perpiiánn & Idelbayev (2018). For the CIFAR-10 ResNet-56, compare against "Learning Compression" (Carreira-Perpiñan & Idelbayev, 2018). This technique is selected as the most accurate technique at high sparsities, from Ortiz al. (2020).
Curreira-Perpiñaa & Idelbayev (2018) use unstructured gradual global magnitude pruning, derived from an alternating optimization formulation of the gradual pruning process which allows for weights to be reintroduced after being pruned. The pruning schedule is hyperparameter, and the paper does not explain how the chosen value was found. After gradual pruning, Carreira-Perpinae & Idelbayev fine-tune the remaining weights.
ImageNet ResNet-50:
He al. (2018). For the ImageNet ResNet-50,, we compare against AMC (He et al., 2018). This technique is not listed in Ortiz et al. (2020), but achieves less reduction in accuracy at higher sparsity than other techniques, losing 0.02% top-1 accuracy at 5.13x compression.
He et al. iteratively prune ResNet-50, using manually selected per-iteration pruning rates (pruning by 50%, then 35,, then 25%, then 20%, resulting in a network that is 80.5% sparse, a compression ratio of 5.13x) ). and retrain with fine-tuning for 30 epochs per iteration. On each pruning iteration, He et al. use a seinforeement-learnin approach to determine layerwise pruning rates.
WMT16 EN-DE GNMT: Zhu & Gupta (2018). For the WMT16 EN-DE GNMT model, compare against Zhu & Gupta (2018). This technique is not listed in Ortiz et al. (2020), as Ortiz et al. do not consider the GNMT model. To confirm that this technique is state-of-ther of-the--rt model, we extensively searched among papers citing Zhu & Gupta (2018) and Wu et al. (2016) and found results claiming better ACCURACY versus PARAMETER EFFECIENCY tradeoff curve.
the GNMT
Zhu & Gupta search across multiple pruning techniques, and ultimately use a pruning technique that prunes each layer at equal rate, excluding the attention layers. Zhu & Gupta use training
13

---

Published as a conference paper at ICLR 2020
ACCURACY versus PARAMETER-EPFICIENS Tradeoff from Algorithm
CIFAR-10 RosNet-20 Unstructured (iterative)
CIEAR-10 RosNet-56 Unstructured (iterative)
Accuracy
A Accuracy
0%
-1%
-1%
-2%
.2%
Correco-espinda ldelaaee (alla)
-3%
-3%
1,20x
2
4.77x
9.31x
1.50x
4.77
14,35x
4141x
Compression ratio
Compression ratio
CIFAR-10 ResNet-110 Unstructured (itcrative)
ImageNet RosNet-50 Unstructured (iterative)
Accuracy
A Acouracy
0%
0%
-1%
-1%
-2%
22%
(208)
-3%
-3%
1.25x
2.44x
4.7Tx
9.31
1.56x
2.4xx
3.81x
5.96x
9.31
Compression ratio
Compression ratio
WMT16 GNMT Unstructured (itcrative)
BLEU
Retraining with Weight rewinding
Retraining with Fine tuning
Zhm & Gugaa (2017)
Retraining with Learning rate rewinding
-6
1.25x
Compression ratio
Figure 6: ACCURACY versus PARAMETER EPREEIEECC tradeoff of our pruning algorithm.
algorithm that gradually prunes the network as it trains, using specific polynomial to decide pruning rates time, rather then fully training then pruning. Zhu & Gupta (2018) use larger GNMT model than defined the MI.Perf benchmark, with 21 M parameters to only 165M parameters in ours. Therefore, model at given compression ratio from Zhu & Gupta (2018) has remaining parameters than model at given compression ratio using the GNMT model in this paper.
D OTHER INSTANTIATIONS OF OUR PRUNING ALGORITHMS
In this appendix, we present comparison of the algorithm presented Section 5 to instantiations of that algorithm with other retraining techniques. Specifically, we compare against retraining with weight rewinding for 90% of the original training time, learning rate rewinding for the original training time (as presented in the algorithm in the main body of the paper), or fine-tuning for the original training time.
Algorithm 2 Our pruning algorithm
1. TRAIN to completion.
2. PRUNE the 20% lowest-magnitude weights globally.
3. RETRAIN using either weight rewinding for 90% of the original training time, learning rate rewinding for the original training time, fine-tuning for the original training time.
4. Repeat steps 2 and 3 iteratively until the desired compression ratio is reached.
Figure 6 presents evaluation of our pruning algorithm. We find that retraining with weight rewinding performs similarly well to retraining with learning rate rewinding, except for at high sparsities the GNMT.
14

---

Published as a conference paper at ICLR 2020
E ADDITIONAL NETWORKS AND BASELINES
In this appendix, we include results for more networks, pruning techniques, baselines, and ablations. We consider larger set of networks than the paper, more structured pruning techniques from Li et al. (2017), a reinitialization baseline from Liu et al. (2019), and a natural ablation of rewinding, where rewind the weights but use the learning rate of fine-tuning.
METHODOLOGY
Retraining techniques. We include two baselines of the techniques presented in the main body of the paper, using the notation from Section 2. For convenience, we duplicate that notation here.
Neural network pruning is an algorithm that begins with a randomly initialized neural network with weights Wo E Rd and returns two objects: weights W Rd and pruning mask m {0, 1] such that W @ m is the state of the pruned network (where @ is the element-win product operator). Let Wg be the weights of the network at epoch g. Let be the standard number of epochs for which the network is trained. Let S[g] be the learning rate schedule of the network for each epoch g. defined such that Sig > T = S (i.e., the last learning rate is extended indefinitely). Let TRAIN (W, m, g) be a function that trains the weights of W that are not pruned by mask for t epochs according to the original learning rate schedule S, starting from step g.
Low-LR WEIGHT REWINDING. The other natural ablation of weight rewinding (other than learning rate rewinding) is to rewind just the weights and use the learning rate that would have been used in fine-tuning. Formally, Low-LR weight rewinding for t epochs runs TRAIN' TRAIN'(Wr-1, (Wr-i,m,T)
REINITIALIZATION, We also consider reinitializing the discovered pruned network and retraining it by extending the original training schedule to the same total number of training epochs fine- tuning trains for. Liu et al. (2019) found that for many pruning techniques, pruning and fine-tuning results in the same worse performance simply training the pruned network from scratch for an equivalent number of epochs. To address these concerns, we include comparisons against random reinitializations of networks with the discovered pruned structure, trained for the original T training epochs plus the extra epochs that networks were retrained for, Formally reinitializing and retraining for epochs is sampling a new W/ E TRd then running T'RAINT+ (W6, m, 0).
In this baseline, consider the discovered structure from training and pruning the original network according to the given pruning technique. For unstructured pruning, this discovered structure is the specific structure left behind after magnitude pruning: for structured pruning, this discovered structure is the structure determined by the layerwise rates derived in Li et al. (2017). We note that in both of these cases, the resulting structure is determined by having trained the network, whether that occurs explicitly (as with unstructured pruning) or implicitly (as Li et al. determine layerwise pruning rates by pruning individual layers of trained network). We therefore expect this to perform least as well as randomly pruning the network before any amount of training, since the pruned structure incorporates knowledge from having already trained the network at least once.
Networks, Datasets, and Hyperparameters. We include two more CIFAR-10 vision networks in this appendix: ResNet-20 and ResNet- 10. We also include several more structured pruning results from these networks, again given by Li et al. (2017): ResNet-56- (A.B} on CIFAR- 10, ResNet- 1 10- {A.B} on CIFAR-10, and 34-Nctt-34(AABB)) on ImageNet. The networks and hyperparameters are described in Table 2.
Data. All plots are collected using the same methodology described in the main body of the paper. We also include the data from the networks presented in the main body of the paper for comparison. On each structured pruning plot, we plot the accuracy delta observed by Liu et al. (2019).
RESULTS
ACCURACY versus CHAAMETER-EFTIIEEEE tradeoff results (Figures 7 and 8). Low-LR weight rewinding results in large drop in accuracy relative to the best achievable accuracy, and a
nctps: http:///ww..oee//tt/////////////
Phetps: http://ggttbecceee
(rttpa: // httpah..co//aaarr///aaaattttioia
w/g.ogg////o/ooooooooooooooo/... e/cloud_ve hadddv3.8//mmme 8/gnntttpuv--9
15

---

Published as a conference paper at ICLR 2020
Dataset
Network
#Params
Optimizer
Learning training spoch)
Accuracy
271K
Nesterov SGD
91.71 0.23%
0.9
0.1
L [0, 91)
CIFAR-10
56
852K
Batch size:
0.01
LE [91, 136)
93.44 0.21%
Weight decay: 0,0022
0.001
[136, 182]
ResNet-
1.72M
Epocha: 182
0.23%
Nesterov SGD
0.4 +
E [0,
R4
0.9
0.4
(5,30)
73.60 0.27% 1og-1
ImugeNet
Rutch size: 1024
0.04
16 [30, 60)
ResNer-50P
25.5M
Weight decag: 0,0001
0.004
IE [00, a0)
76.17 0.14%
Epachs: 90
0.0004
(80,
Adaan P1 0.9
0.002
0.01
-5
E [0, 0.125)
0.002
[0.125, 8.73)
WMT16 EN-DF
GNNM 10
165M
0.999
0.001
[3.75, 4.165)
26.87 + 0.23 BLEU
newakeac
Batch
0.0005
E [4.165,
Fpachs:
0.00025
E [4.58, 6)
Table 2:
Networks, datasets, and hyperparameters. We use standard implementations available online and standard hyperparameters. All accuracies are in line with baselines reported for these networks (Liu al., 2019; He et al., 2018; Gale et al., 2019; Wu al., 2016; Zhu & Gupta, 2018).
small drop in accuracy compared to standard fine-tuning. With unstructured pruning, reinitialization performs poorly relative to all other retraining techniques. With structured pruning. reinitialization performs much better, roughly matching the performance of rewinding the weights and learning rate. This is expected from the results of Liu et al. (2019), which find that reinitialization comparatively performs well with structured pruning techniques.
ACCURACY versus ACRH COST tradeoff results (Figures and 10). Low-LR weight rewind- ing has markedly different behavior than other techniques when picking where to rewind to. Specifi- cally, when performing low-LR weight rewinding, longer training does not always result in higher accuracy. Instead, accuracy peaks at different points different networks and compression ratios, often when rewinding to near the middle of training.
Reinitialization typically saturates in accuracy with the original training schedule, and does not gain a significant boost in accuracy from adding extra retraining epochs. When performing structured pruning, this means that reinitialization achieves the highest accuracy with few retraining epochs, although rewinding the learning rate can still achieve higher accuracy than reinitialization with sufficient training.
16

---

Published as a conference paper at ICLR 2020
One-shot ACCURACY versus PARIMETER-EFFICIENYY Tradeoff
Unstructured
Structured-A
Structured-B
RosNet-20 Unetruttadd
2
LTTx
0.50
Re Unstucturee
BesNet-S6 Struetrne-A
ResNet-66 Stractured-D
CIFAR-10
Accurary
Compressoo mill
Compresiin natlo
BeeNet 110 Usa cucaree
ReeNet- 110 Struckured-A
ReeNet 110 Stractured-
A Nootraa
1241)
(20.5)
Compresou
Comperssion
Comprosion
ImageNet
ResNet-53 Unetructured
ResNes-34 StruetnddAA
ResNet-34 Straetured-II
Austurayy
Camproeia eulko
Chespassin nilla
Chmpueoiin ralln
GNMT Unstrnctured
WMT16
Loariing zute rewinding
Low: LR weight rewinding Reinitadiring
Wright resrindin
Fin-ttming
Cimpuooius
Figure 7: The best achievable accuracy across retraining times by one-shot pruning.
17

---

Published as a conference paper at ICLR 2020
Iterative ACCURACY versus PARAMETER-EPPICIEN Tradeoff
CIFAR-10 ResNett-2 Unstructured (iterative)
CIEAR-10 ResNet-56 Unstructured (iterative)
+1%
+1%
Acmirey
Accuracy
0%
0%
-1%
-1%
2%%
.2%
Carrern Carmero-Perpilll Idelboeee (ar1s)
CIFAR-10
-3%
-3%
1.25x
2.4xx
4.77x
9.31x
1.56x
4.77x
14.55x
4441x
Compression ratio
Compression ratio
CIFAR-10 ResNet-110 Unstructured (itcrative)
+1%
4 Accuracy
1,26x
2
4.77x
9.31
Compression ratio
ImageNet ResNet-50 50 Unstructured (iterative)
ImageNet
+1%
Aceuracy
0%
-1%
-2%
Ho al (2018)
-8%
1.56x
2.4XX
3.81
5.96X
9.31x
Compression ratio
WMT16 GNMT Unstructurod (itcrative)
WMT16
A BLEU
Zhn Cupta (2017)
1.25x
3.05x
7.45x
18. 18x18x
Compression ratio
Learning rate rewinding
Weight rewinding
Fine-tuning
Figure 8: The best achievable accuracy across retraining times by iteratively pruning.
18

---

Published as a conference paper at ICLR 2020
Unstructured ACCURACY versus SEARCH COST Tradeoff
One-shot
Iterative
ResNet-20 20 Unstructured, 1.25x
ResNet 20 Unstructured, 1.25x (iterative)
+1%
+1%
A Acruracy
Accuracy
0%
-1%
-1%
-2%
-2%
-35
-3%
10
67
95
125
153
182
10
67
96
125
153
182
Retraining epochs
Retraining epochs
ResNet-20 Unstructured, 1.56x
Net-Net20 Unstructured, 1.56x (iterative)
+1%
+1%
Arcuracy
Acouracy
06
0%
-1%
-1%
-2%
-2%
.3%
-3%
10
67
96
153
182
10
125
153
182
Retraining epochs
Retraining epochs
ResNet-20 20 Unstructured, 1.95x
ResNet 20 Unstructured, 1.95x (iterative)
+19
Arcuracy A Accuracy
Accuracy
0%
-1%
-1%
-2%
-2%
ResNett220
-35
-3%
10
07
96
125
153
182
10
67
125
153
182
Retraining epochs
Retraining epochs
ResNet-20 Unstructured, 2.44x
Net-Net20 Unstructured, 2.44x (iterative)
+1%
+1%
Accuracy
06
0%
-1%
-1%
-5%
-2%
3%
-3%
10
39
67
96
125
153
162
10
125
153
182
Retraining epochs
Retraining epochs
ResNet-20 Unstructured, 3.05x
ResNet 20 Unstructured, 3.05x (iterative)
+156
A Accuracy
Accuracy
0%
-1%
-1%
*
-2%
-2%
-356
-3%
10
67
96
125
153
182
10
125
153
182
Retraining epochs
Retraining epochs
RosNet-20 Unstructured, 3.82x
ResNeet-0 Unstructured, 3.81x (iterative)
+1%
+1%
Accuracy
Acourocy
06
0%
-1%
-1%
.5%
-2%
35%
3%
10
39
67
96
125
153
182
10
67
96
125
153
182
Retraining epochs
Retraining epochs
Learnnng aestnding
Low: LR wogld rocaading
Weight rewincing
Reinotinlsing
Fansluning
Figure 9: Accuracy curves across different networks and compressions using unstructured pruning.
19

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
ResNet-20 Unstructured, 4.77x
ResNet-20 Unstructured, 4.77x (iterative)
+1%
A Acruracy
Accurony
088
-1%
-1%
-2%
-2%
-5%
-3%
10
07
153
182
10
125
153
182
Retraining epochs
Retraiming epociss
ResNet-20 Unstructured, 5.96x
ResNet-22 20 Unstructured, 5.96 (iterative)
+15
+1%
Accuracy
Aocuracy
0%6
0%
-1%
-1%
-2%
-5%
ResNett220
-3%
-3%
10
39
67
96
125
153
182
10
67
96
125
153
182
Retraining epochs
Retraining epochs
RosNet-20 Unstructured, 7.45x
Nes-ett20 Unstructured, 7.45x (iterative)
+1%
A Acruracy
Aocuracy
0%%
-1%
-1%
.2%
-2%
-35
-396
07
153
182
125
153
182
Retraining epochs
Retraining epocias
ResNet-20 Unstructured, 9.31x
Ros Net-20 Unstructured, 9.31 x (iterative)
+1166
+1%
Accuracy
Avcuracy
06
0%
-1%
-1%
2%
-2%
-3%
-3%
10
39
67
96
125
153
152
10
39
67
96
125
153
182
Retraining epochs
Retraiming epochs
ResNet-56 Unstructured, 1.25x
ResNet-50 56 Unstructured, 1.25x (iterative)
+1%
+1%
Accuracy
Accurary
0%
0%
-1%
-1%
-2%
-2%
-5%
-3%
10
39
67
96
125
153
182
10
30
67
96
125
153
182
Retraining epochs
Retraining epochs
ResNet-56 Unstructured, 1.56x
ResNet-56 Unstructured, 1.56x (iterative)
+1%
ResNeet-6
Amuracy
Acouracy
0%
-1%
-1%
-5%
-2%
-5%
-3%
153
182
10
125
182
Retraining epochs
Retraining epochs
RosNet-56 Unstructured, 1.95
RosNot-56 Unstructurod, 1.95x (iterative)
+15
+1%
Accuracy
Accuracy
0%
0%
-1%
-1%
2%
.2%
-5%
10
39
67
96
125
153
182
10
30
67
96
125
153
182
Retraining epochs
Retraining epocits
Loaning aestuding
woght rewaading Reiutinlzing
Weight rewincing
Fiusttriig
20

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
Re Unstructured, 2.44x
ResNet-56 Unstructured, 2.44x (iterative)
+1%
A Arcurary
Accuracy
-1%
-1%
-2%
-2%
35
-3%
153
125
182
Retraining epochs ResNet-56 Unstructured, 3.05x
Retraining epochs
ResNet- 56 Unstructured, 3,05 (iterative)
+15
+1%
Arcuracy
Accuracy
0%
0%
-196
-1%
-2%
-2%
-3%
-3%
10
39
67
96
125
153
182
10
30
07
96
125
153
182
Retraining epochs
Retraining epochs
ResNet-56 Unstructured, 3.82x
ResNet-56 Unstructured, 3.81x (iterative)
+1%
À Arcurecy
Accuracy
0%
11%
-1%
*
-2%
-2%
35%
-3%
10
153
182
Retraining epochs
Retraining epocias
ResNet-56 Unstructured, 4.77x
ResNet-56 Unstructured, 4.77x (iterative)
+15
+1%
ResNet-56
A Accuracy
0%
Accurayy
0%
-1%
-1%
-2%
-2%
-5%6
-3%
10
39
67
96
125
153
182
10
30
07
96
125
153
182
Retraining epochs
Retraining epochs
RosNet-56 Unstructured, 5.96x
ResNet-56 Unstructured, 5.96x (iterative)
+1%
Arcumey
Accurocy
08
0%
11%
-1%
25
-2%
35%
-5%
10
67
153
182
10
125
153
182
Retraining epochs
Retraining epocias
ResNet-56 Unstructured, 7.45x
ResNet-56 Unstructured, 7.45x (iterative)
+1%
+1%
A Accurary
Apcuracy
0%6
0%
-1%
-1%
-2%
-2%
10
39
07
96
125
153
182
10
30
07
96
125
153
182
Retraining epochs
Retraining epochs
RosNet-56 Unstructured, 9.31
RosNet-56 Unstructured, 9.31x (iterative)
+1%
Accuray
Acouracy
08
0%
-1%
-1%
.5%
-2%
-5%6
.5%
10
67
153
182
10
125
153
182
Retraining epochs
Retraining epociss
Learnnng rate aestuding
Low-LR woght recading
Weight rewincing
Beiutinlzing
Pas-tarinn
21

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
ResNet-110 Unstructured, x.25x
ResNet-110 Unstructured, 1.25x (iterative)
+1%
+1%
A Arcurary
Accurary
-1%
-1%
-2%
-2%
35%
-3%
10
182
Retraining epochs ResNet-110 Unstructured, x
Retraining epochs
ResNet 110 Unstructured, 1.56x (iterative)
+15
+1%
Accuracy
Accuracy
0%
0%
-196
-1%
-2%
-2%
-3%
-3%
10
39
67
96
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retraining epochs
ResNet-110 Unstructured, x.95
ResNet-110 Unstructured, 1.95x (iterative)
+1%
À Arcurecy
Accuracy
0%
11%
-1%
-2%
-2%
35%
-3%
10
153
182
182
Retraining epochs
Retraining epochs
ResNet-110 Unstructured, 2.44x
ResNet-1100 Unstructured, 2.44x (iterative)
+15
+1%
ResNNe-1110
Accuracy
Accuracy
0%
0%
-1%
-1%
-2%
-2%
-5%6
-3%
10
39
67
95
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retraining epochs
RosNet-110 Unstructured, 3.05
ResNet-110 Unstructured, 3.05x (iterative)
+1%
Arcumey
Accuracy
08/
0%
-1%
-1%
.5%
-25
35%
-3%
10
153
182
182
Retraining epochs
Retraining epochs
ResNet-110 Unstructured, 3.82x
ResNet-1 Unstructured, 3.81x (iterative)
+156
+1%
Accurary
Accuracy
066
0%
-1%
-1%
-2%
-2%
-3%
10
39
07
96
125
153
182
10
30
67
95
125
153
182
Retraining epochs
Retraining epochs
RosNet-110 Unstructured, 4.77x
ResNet-110 Unstructured, 4.77x (itcrative)
+1%
Accurayy
Acouracy
08
0%
-1%
-1%
.5%
-2%
-5%
.3%
10
67
96
153
182
Retraining epochs
Retraining epochs
Learnnng tate aestuding
Low-LR woght rowading Beiotinlzing
Weight rewincing
Pas-taring
22

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
ResNet-110 Unstructured, 5.96x
RosNet-110 Unstructured, 5.96x (iterative)
+1%
A Aeruracy
Accuracy
0%
-1%
-1%
-2%
-2%
-3%
-3%
10
67
153
182
10
39
96
125
153
182
Retraining epochs
Retraining epochs
ResNet- Unstructure,, 7.45x
RosNet-1 Unstructured, 7.45x (iterative)
+1%
+ +1%
ResNet-
Accerray
0%
Apcuracy
0%
-1%
-186
-2%
-2%
-3%
-3%
10
39
67
96
125
153
182
10
30
67
96
125
153
182
Retraining epochs
Rotraining epochs
ResNet-110 Unsdructured, 9.31 x
ResNeet 110 Unstructured, 9.31 (iterative)
+1%
A Aeruracy
Aocurocy
0%
-1%
-1%
.2%
-2%
-35
-3%
07
153
182
10
39
125
153
182
Retraining epochs
Retraining epochu
ResNet-50 Unstructured, 1.25x
ResNet-50 Unstructured, 1.25x (iterative)
A Accuracy
0%
Accuracy
0%
25%
-2%
14%
.4%
Retraining epochs Re Unstructured, 1.56x
Retraining epochs
ResNet-50 Unstructured, 1.56x (iterative)
Acrurary
0%
Accuravy
0%
.2%
ResNet-50
4 -4%
-4%
9
63
63
76
90
Retraining epochs
Retraining epochs
ResNet-50 Unstructured, 1.95x
ResNet-50 Unstructured, 1.95x (iterative)
Accuracy
Accurucy
.2%
-2%
14%
9
50
63
90
36
50
63
76
90
Retraining epochs
Retraining epochs
ResNet-50 Unstructured, 2.44x
ResNet-50 Unstructured, 2. 2.44x (iterative)
Acrirary
0%
Accuraty
0%
-2%
-4%
-4%
Retraining epochs
Retraining epochs
Taaring rate vosiding
Lea-RR wright now mocnding Reintiniaing
Weight rowinding
Pine
23

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
ResNet-50 Unstructured, 3.05x
ResNet-50 Unstructured, x.05x (iterative)
A. Arcuracy
AAcuracyy
0%
22%
.2%
-4%
-4%
50
63
76
90
36
50
63
76
90
Retraining epochs
Retraining epochs
ResNet-50 Unstructured, 3.82x
ResNet-50 Unstructured, 3.81x (iterative)
Accurayy
0%
Avcuracy
-2%
-2%
4) -4%
-1%
9
63
Retraining epochs
Retraining epochs
ResNet-50 Unstructured, 4.77x
ResNet-50 Unstructured, x.77x (iterative)
W. Arcuracy
0%
Accuracy
0%
-4%
-4%
63
Retraining epochs
Retraining apochs
RosNet-50 Unstructured, 5.96
ResNet-50 Unstructured, 5.96x (iterative)
A Accuracy
Avcuracy
.25
-2%
-1%
-1%
9
22
50
63
78
90
22
36
63
76
90
Retraining epochs
Rotraining epochs
ResNet-50 Unstructured, 7.45x
ResNet-50 Unstructured, 7.45x (iterative)
A Arcuracy
0%
Acouracy
0%
25%
-2%
-4%
-4%
Retraining epochs
Retraining epochs
ResNet-50 Unstructured, 9.31 x
ResNet-50 Unstructured, 9.31 x (iterative)
3 Accuracy
0%
Avcurayy
0%
.5%
-2%
-4%
-1%
9
50
63
90
50
63
76
90
Retraining epochs
Retraining epochs
Loarnnng sewinding Weight rewincing
woght recadinn Beiutinlzing
Feusturing
24

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
GNMT Unstructured, 1.25x
GNMT Unstructured, 1.25x (iterative)
A.EUU
A BLEU
2,00
2.75
2.75
3.50
4.25
Retraining epochs
Retraining epochs
GNMT Unstructured, 1.56x
GNMT Unstructured, 1.56x (iterative)
A ALBU
BLEU
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.30
1.25
2.00
2.75
3.50
4.25
5.00
Retraining epochs
Rotraining opochs
GNMT Unstructured, 1.95x
GNMT Unstructured, 1.95x (iterative)
ALEU
BLEU
-6
1.25
2,00
2.75
3.50
4.25
1.25
2.00
2.75
3.50
1.25
Retraining epocis
Retraining epochs
GNMT Unstructured, 2.44x
GNMT Unstructured, 2.44x (iterative)
GNMT
A BLEU
BLEU
0.50
1,25
2,00
2.75
3.50
4.25
5.00
0.50
1.25
2.00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
GNMT Unstructured, 3.05x
GNMT Unstructured, 3.05x (iterative)
ALEU
A BLEU
-6
0.50
1.25
2,00
2.75
3.50
4.25
1.25
2.00
2.75
3.50
1.25
Retraining epocis
Retraining epochs
GNMT Unstructured, 3.82x
GNMT Unstructured, 3.82x (iterative)
A BLEUU
BLEU
0.50
1.25
2,00
2.75
3.50
4,25
5.00
0.50
1.25
2.00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
GNMT Unstructured, 4.77x
GNMT Unstructured, 4.77x (iterative)
BLRU
BLEU
-6
1.25
2.00
2.75
3.50
4.25
1.25
2,00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
Learning tate aestnding
Low-LR LR woght rewcading
Weight rewincing
Beiutinlzing
Pasetariin
25

---

Published as a conference paper at ICLR 2020
One-shot
Iterative
GNMT Unstructured, 5.96x
GNMT Unstructured, 5.96x (iterative)
BLLU
BLEU
0.50
1.25
2.00
2.75
3.50
4.25
5.00
0.50
1.25
2,00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
GNMT Unstructured, 7.45x
GNMT Unstructured, 7.45x (iterative)
GNMT
BLEU
BLEU
-6
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.50
1.25
2,00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
GNM'T Unstructured, 9.31x x
GNMT Unstructured, 9.31x (iterative)
A BLEU
BLEU
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.50
1.35
2,03
2.75
3.50
4.25
5.00
Retraining epochs
Retraining epochs
GNMT Unstructured, 11.64x
GNMT Unstructured, 11.64x (iterative)
BLEU
BLEU
-6
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.50
1.25
2.00
2.75
3.50
4.25
5.00
Retraining epochs
Retraining opochs
GNMT Unstructured, 14.56x
GNMT Unstructured, 14.56x (iterative)
A BLEU
BLEU
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.50
1.35
2.03
2.75
3.50
4.35
5.00
Retraining epochs
Retraining epochs
GNMT Unstructured, 18.18x
GNMI Unstructured, 18.18x (iterative)
A BLEU
BLEU
0.50
1.25
2,00
2.75
3.50
4.25
5.00
0.50
1.25
2.00
2.75
3.30
4.25
5.00
Retraining epochs
Retraining epochs
GNMT Unstructured, 22.73x
GNMT Unstructured, 22.73x (iterative)
BLEU
BLEU
0.50
1.25
2.00
2.75
3.50
4.25
5.00
0.50
1.35
2.00
2.75
3.30
4.25
5.00
Retraining epochs
Retraining epochs
26

---

Published as a conference paper at ICLR 2020
Structured ACCURACY versus SEARCH COST Tradeoff
Structured-A
Structured-B
ResNet-56 Structured A, 1.09x
ResNet-56 Structured B, 1.15x
+1%
+1%
Accurary
Aocuracy
0%
0%
-1%
-1%
-2%
(2015)
-2%
(2013)
-356
-3%
10
39
67
96
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retraining epuchs
ResNet-56 Structured-A, A, 1.18x
ResNet-56 Structured-B, 1.29
Accurary
0%5
Accuracy
00%
-1%
-1%
-2%
-2%
-3%
10
39
67
96
125
153
182
10
39
67
96
125
153
182
ResNet-56
Retraining epochs
Retraining epochs
ResNet-56 Structured-A A, 1.29
NesNet-56 Structured-D B, 1.43x
Arcusay
Avcuracy
-1%
-1%
25%
2%%
-3%
-3%
10
39
67
96
125
163
182
10
30
67
96
125
153
182
Retraining epochs
Retraiting epoclus
ResNet-56 Structured-A, 1.40x
ResNet-56 Structured-B, 1.57x
A Accurayy
0%
Accuravy
0%
-1%
-1%
-2%
-2%
-3%
10
39
67
96
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retraining epochs
RosNet-56 Structured-A, 1.51
RosNet-56 Structurod-B, 1.70x
Accuracy
066
Accuracy
0%
-1%
-1%
.5%
-2%
10
39
67
96
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retraining epochs
Tearing vesinding
Low-LR wright mocnwinng
Weight rewincing
Reintinlzing
Fansturing
Figure 10: Accuracy curves across different networks and compressions using structured pruning.
27

---

Published as a conference paper at ICLR 2020
Structured-A
Structured-B
ResNet-110 Structured-A, 1.02x
ResNet-110 Structured-B B, 1.45x
+1%
+1%
Armiracy
Accuracy
0%
0%
-1%
-1%
-2%
(2070)
-2%
(2013)
-3%
-3%
10
30
67
96
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retroining epochs
ResNet-110 Structurod-A, 1.0xx
ResNet-110 Structured-B, 2.09x
Arcuracy
096
Aucuraty
0%
-1%
-1%
2%
2%
-355
10
39
67
96
125
153
182
10
30
67
96
125
153
182
ResNet- 110
Retraining epochs
Retraiting epocls
ResNet- 110 Structured-A. A, 1.04x
ResNet-110 Structured-B B, 2.96
+155
+1%
Accunccy
0%
Accuracy
15%
-1%
.2%
-2%
-3%
10
39
67
96
125
153
182
10
30
67
96
125
153
182
Retraining epochs
Retraining epochs
ResNet-110 Structurod-A, 1.0xx
ResNet-110 Structured-B, 4.05x
Accuracy
096
Accuracy
0%
-15
-1%
.5%
-2%
10
39
67
96
125
153
182
10
39
67
96
125
153
182
Retraining epochs
Retraiting epochs
ResNot-1 10 Structured-A, 1.05x
ResNet-110 10 Structured-BB 5.46x
+155
Accuracy
065
Acouracy
0%
.1%
.1%
.2%
.2%
-355
10
19
67
96
125
163
182
10
30
67
96
125
153
182
Retraining epochs
Retraining epochs
Loaning cestnding
woight nocasding Beintinhbing
Weight mwincing
Pase-tuning
28

---

Published as a conference paper at ICLR 2020
Structured-A
Structured-B
ResNet-34 Structured A, 1.08x
ResNet- 34 Structured B, 1.12x
A Accuracy
Accuracy
25%
.2%
Liu (2119)
(2019)
9
22
36
30
63
76
90
22
36
50
63
76
90
Retraining epochs
Retraining epochs
ResNet-34 Struetured-A, 1.15x
ResNet-34 Structured-B B, 1.20
Accuracy
Accuracy
-2%
-2%
63
76
ResNet--34
Retraining epochs
Retraining epochs
ResNet-34 Structured-AA 1.20x
ResNet-34 Structured-B, 1.25x
0%
0%
Accuracy
A Accuracy
-1%
-1%
22
36
50
63
76
90
9
22
36
50
63
76
50
Retraining epochs
Retraining epuchs
ResNet-34 Structured-A, 1.24x
ResNet-34 Structured-B. 1.28x
0%
A Accuracy
4 Accuracy
2%
.2%
4%%
4%
Retraining epochs
Retraining epochs
ResNet-34 Structured-AN A, 1.26x
ResNet-34 Structured- B, 1.30
Aocuracy
Avcaruey
-2%
-2%
-1%
-4%
22
36
50
63
76
90
22
36
50
63
76
90
Retraining epochs
Retroining epochs
Learnnng tate zesinding
Low LR weight rewtading Beintiniiing
Weight mwincing
Passtuing
29

---

Published as a conference paper at ICLR 2020
lterative pruning FLOP speedup over original network
ResNet-56 hest-validated FT.PPs across densities
ResNet-110 best-validated FLOPs across densities
6,0X
Spoedup
Speedhp
6,00x
100x
4.00x
2,00x
2.44x
5,06x
3.81x Compression Ratio
9.31
Compression Ratio
ResNet-5t best-validated FLOPs across densities
Speodup
600x
Learning rale rewinding
100x
Weight rewinding
2.0x
Fine-tuning
1.56x
2.44x
3.81x
5.06x
9.31x
Compression Ratio
Figure 11: Speedup over original network for different retraining techniques and networks
Iterative pruning FLOP speedup over fine-tuning
ResNet-56 speedup of vall-aald FLOPs over liue-tainn ResNet-110 speedup of best-validated FLOPs over liue-tuiin
1.90X
1.00%
Speodhp
Speediu
1,30x
1.30x
1.00x
2000x
0.90X
1.50x
2.41x
3.8
0.00
1,50x
2.41x
3.1x
5,00x
9.80x
Compression Ratio
Compressoon: Ratio
ResNet-50 50 speedup of vall-valddated FLOPs over line line-tainn
1,30x
1.20%
Learning rate rewinding
Speedop
Weight rewinding
Fine-tuning tete-tuning
0.30X
1.50x
2.41x
3,80
Coupression Ratio
Figure 12: Speedup over original network for different retraining techniques and networks
F COMPRESSION RATIO VS FLOPS
In the main body the paper, we use the compression ratio as the metric denoting the EFFICIENCY of given neural network. However, the compression ratio does tell the full story: networks of different compression ratios can require different amounts of floating point operations (FLOPs) to perform inference. For instance, pruning a weight in the first convolutional layer of ResNet results in pruning more FLOPs than pruning weight in the last convolutional layer. Reduction in FLOPs can (but does not necessarily) result wall clock speedup (Baghdadi al., 2019). this appendix, we analyze the number of FLOPs required to perform inference on pruned networks acquired through fine-tuning and rewinding. Our methodology for one-shot pruning uses the same initial trained network for comparisons between all techniques, and prunes using the same pruning technique. This means that both networks have the exact same sparsity pattern, and therefore same number of FLOPs. For iterative pruning the networks diverge, meaning the FLOPs also differ, since global pruning.
Methodology. In this appendix, compare the FLOPs between different retraining techniques. We specifically consider iteratively pruned vision networks, where pruning weights earlier layers results in a larger reduction in FL.OPs than pruning weights in later layers. We compare using the
30

---

Published as a conference paper at ICLR 2020
networks as selected in the iterative subsection of Section 3, i.. the using the retraining time that results in the set of networks with the highest validation accuracy for each different compression ratio. Therefore the FLOPs reported in this section the FLOPs resulting from the most accurate network at a given compression ratio, not necessarily the minimum required FLOPs at that compression ratio. In Figure 11, plot the theoretical speedup
the original network i.e., the ratio of original FLOPs over pruned FLOPs. In Figure 12, we plot the theoretical speedup of each technique over fine-tuning i.e., the ratio of fine-tuning FLOPs that compression ratio the FLOPs of each other technique at that compression ratio.
Results. Both rewinding techniques find networks that require fewer FLOPs than those found by iterative pruning with standard fine-tuning. Due to the increased accuracy from rewinding, this results magnified decrease in INFERENCE-EFFICIENN
for rewinding compared to fine-tuning. For instance, a ResNet-50 pruned to maintain the same accuracy as the original network results in a 4.8x theoretical speedup from the original network with rewinding the learning rate, whereas similarly accurate network attained through fine-tuning has speedup of 1.7x.
31


Reviews:
[{"review_id": "r1l5tjbjtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper does an in-depth evaluation of the notion of rewinding pruned networks to the weights at a previous point in training, and then re-training the pruned network from then on. This is in comparison with fine-tuning a pruned network, where the retraining continues from the network's current weights. The authors focused on vision networks and unstructured magnitude pruning in their evaluation.\n\nThe paper is well-written and easy to follow, and the authors have done a good job of empirically comparing rewinding against fine-tuning in a number of different scenarios.\n\nMy main concern with the paper is that it seems too incremental to stand on its own. Rewinding is a notion that was already explored in the Lotter Ticket Hypothesis (Frankle et al., 2019), so this paper seems to be more of an extension of that work. The take-away message from this paper (stated by authors in their conclusion) is that practicioners \"should explore rewinding as an alternative to fine-tuning for neural network pruning\", but that's a case that was already made by the LTH paper. The current work certainly gives more weight to that claim, but I don't feel the contributions are strong enough on their own to justify a full conference publication.\n\nI encourage the authors to continue working on this, as it is very interesting and can be very useful. Some ideas to make the paper stronger:\n- Given that this is a purely empirical paper, it'd be better to not limit the experiments as much. Can you run on non-vision networks? What about mobile-net? Can you try different pruning techniques? etc.\n- How do the different methods compare in terms of accuracy/sparsity versus FLOPs?\n\nFinally, two minor comments to improve the writing:\n- First sentence of 3.1: s/that meet that the accuracy/that match the accuracy/\n- First sentence of 3.1: s/than fine-tuning can/compared to fine tuning", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1g3S9Nycr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Building on the results of [Frankle et al, 2019], this paper seeks to utilize rewinding as a core procedure in pruning neural networks, in combination with the usual fine-tuning procedures.  Specifically, [Frankle et al, 2019] demonstrate that it is possible to find sparse subnetworks, such that rewinding weights to their initial values and retraining from that initialization, yields test accuracy similar to the original network.  While this is already a form of pruning, the submitted paper explores a wider space of pruning procedures that utilize rewinding as a subroutine.\n\nThis wider framework includes the choice of rewind point (e.g. rewinding to partway through training rather than to initialization) and how to balance computation budget between rewinding (and retraining for an equivalent number of epochs) vs continuing to fine-tune a network.  Experiments cover this hyperparameter space, as well as the range of desired sparsity level (pruning amount).  Results show rewinding (to a point 30% - 60% into training) dominates any amount of fine-tuning, if moderate to high sparsity is desired.\n\nThe empirical study conducted by this paper is useful and complements the results previously reported in [Frankle et al, 2019].  However, the paper itself is light on novelty, as the core ideas were already established by [Frankle et al], and the application of them here is relatively straightforward.  The extensive experiments here add value to the conversation about the lottery ticket hypothesis, but are not otherwise ground-breaking.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skl8Xog0cB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nExtending the observations of Frankle et al. (2019, \"The Lottery Ticket Hypothesis\"), this paper examines \"rewinding\" as an alternative to fine-tuning in a typical network pruning process. After training to convergence for T iterations, the k% of weights with the smallest magnitude are pruned (set to zero). Typically, in fine-tuning, the remaining weights are continued to be trained for several more iterations at a small learning rate. With \"rewinding\", the remaining weights are reset to their values in iteration 0<=t<T and are trained for T-t iterations with the original learning rate schedule.\n\nExperiments with this method (and an iterative variant) show that (a) rewinding achieves 2-5x smaller networks matching the unpruned accuracy (with comprehensive tuning of the rewinding parameter t); (b) selection of rewinding iteration is important, but is flexible within a large range for higher sparsity pruning.\n\n*Rating*\nThe paper is very well written with good exposition, thorough notes and citations for all methodological choices,\nand an explicit statement of the limitations of the work.\n\nA few considerations relevant to the rating:\n(1) Novelty: The idea for rewinding is not novel, as acknowledged clearly in the paper. Frankle et al. (2019, \"Stabilizing the Lottery Ticket Hypothesis\") showed that rewinding to an early iteration of training (0<t<T) yielded better accuracy than rewinding to iteration t=0 for VGG-19 and ResNet-18 on CIFAR-10. However, Frankle et al. did not consider fine-tuning as it was not relevant to lottery ticket discovery. This submission studies the tradeoffs of rewinding vs. fine-tuning.\n\n(2) Thoroughness: The paper considers ResNet-20 and VGG-16 with CIFAR-10 and ResNet-50 with ImageNet. Conclusions would be strengthened with additional combinations of networks and datasets.\n\n(3) Acknowledged limitations: As noted, the paper doesn't consider any pruning criteria other than weight magnitude, nor does it consider structured pruning. The latter in particular is important for applications where prediction speed on commodity hardware is a limiting factor.\n\nAs it is, I think this paper a worthy (if limited) contribution to the understanding of network pruning.\n\n*Notes*\nTable 1/Figures *: note which dataset is used for each architecture\nFigures 2-3: It seems that many values are clipped by the legend range of +/- 0.5%. Consider showing the figure with a larger range or adding such a figure to the appendix.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers unanimously accepted this paper. 
Decision: Accept (Talk)