Network Pruning That Matters:  A Case Study on Retraining Variants
Authors: Duong Hoang Le
Binh-Son Hua
Venue: ICLR 2021 Poster (2021)
Keywords: Network Pruning

Abstract: Network pruning is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems. Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional fine-tuning technique in recovering the lost accuracy (Renda et al., 2020), but so far it is unclear what accounts for such performance. In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding. We find that the reason behind the success of learning rate rewinding is the usage of a large learning rate. Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e.g., the 1-cycle learning rate schedule (Smith et al., 2019). By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach). Our results emphasize the cruciality of the learning rate schedule in pruned network retraining - a detail often overlooked by practitioners during the implementation of network pruning. 

Published as a conference paper at ICLR 2021
NETWORK PRUNING THAT MATTERS:
A CASE STUDY ON RETRAINING VARIANTS
Duong H. Le VinAI Research, Vietnam
Binh-Son Hua
VinAI Research and VinUniversity, Vietnam
ABSTRACT
Network pruning is an effective method reduce the computational expense of wver-parameterized neural networks for deployment on low-resource systems. Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the tradi- tional tine-tunig technique in recovering the lost accuracy (Renda al.) (020), but so far it is unclear what accounts for such performance. In this work, we conduct ex- tensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding. We find that the reason behind the success of learning rate rewinding is the usage large learning rate. Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e.g., the I-cycle learning rate schedule (Smith & Topin 2019). By leveraging the right learning rate schedule in retraining, we demonstrate counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach). Our results emphasize the cruciality of the learning rate schedule in pruned network retraining a detail often overlooked by practioners during the implementation of network pruning.
INTRODUCTION
Training neural networks is an everyday task in the era of deep learning and artificial intelligence. Generally speaking, given data availability, large and cumbersome networks are often preferred as they have more capacity to exhibit good data generalization. In the literature, large networks are considered easier train than small ones (Neyshabur et al.) 2018) AAroa et al.] [2018] Novak eet al. 20188] Brutzkus & Globerson 2019). Thus, many breakthroughs in deep learning are strongly correlated to increasingly complex and parrpparamttrrized networks.
However, the use of large networks exacerbate the gap between research and practice since real-world applications usually require running neural networks in low-resource environments for numerous purposes: reducing memory, latency, energy consumption, etc. To adopt those networks to resource- constrained devices, network pruning (LeCun et al. 1990 Han et al. (2155) Li et al. 2016) is often exploited remove dispensable weights, filters and other structures from neural networks. The goal of pruning is to reduce overall computational cost and memory footprint without inducing significant drop in performance of the network.
A common approach to mitigating performance drop after pruning is retraining: we continue to train the pruned models for some more epochs. In this paper, we are interested in approaches based on learning rate schedules to control the retraining. A well-known practice is fine-tuning, which aims train the pruned model with a small fixed learning rate. More advanced learning rate schedules exist, which we generally refer to as retraining. The retraining step is critical part in implementing network pruning, but has been largely overlooked and tend to vary each implementation including differences in learning rate schedules, retraining budget, hyperparameter choices, etc.
Recently, Renda et al. (2020) proposed a -fte--of-the-ar technique for retraining pruned networks namely learning rate rewinding (LRW). Specifically, instead of fine-tuning the pruned networks with a fixed learning rate, usually the last learning rate from the original training schedule (Han et al. 2015] Liu et al. 2019), the authors suggested using the learning rate schedule from the previous epochs (i.e. rewinding). This seemingly subtle change in learning rate schedule led to an important result: LRW was shown to achieve comparable performance more complex and computationally expensive

---

Published as a conference paper at ICLR 2021
pruning algorithms while only utilizing simple norm-based pruning. Unfortunately, the authors did not provide the analysis to justify the improvement. In general, it is intriguing to understand the importance of a learning rate schedule and how it affects the final performance of pruned model.
In this work, we study the behavior of pruned networks under different retraining settings. We found that the efficacy from retraining with learning rate rewinding is rooted in the use of large learning rate, which helps pruned networks to converge faster after pruning. We demonstrate that the success of learning rate rewinding over fine-tuning is not exclusive to the learning rate schedule coupling with the original training process. Retraining with a large learning rate could outperform fine-tuning even with some modest retraining, e.g., for few epochs, and regardless of network compression ratio.
We argue that retraining is of paramount importance to regain the performance in network pruning and should not be overlooked when comparing two pruning algorithms. This is evidenced by our extensive experiments: (1) randomly pruned network can outperform methodically pruned network with only (hyper-pane parameerrs free) modifications of the learning rate schedule in retraining, and (2) simple baseline such as norm-based pruning can perform well as as other complex pruning methods by using a large learning rate restarting retraining schedule.
The contributions of our work are as follows.
We document a thorough experiment on learning rate schedule for the retraining step in network pruning with different pruning configurations;
We show that learning rate matters: pruned models retrained with a large learning rate consistently outperform those trained by conventional fine-tuning regardless of specific learning rate schedules;
We present a novel and counter-intuitive result achieved by solely applying large learning rate retraining: a randomly pruned network and simple norm-based pruned network can perform as well as networks obtained from more sophisticated pruning algorithms.
Given the significant impact of learning rate schedule in network pruning, we advocate the following practices: learning rate schedule should be considered critical part of retraining when designing pruning algorithms. Rigorous ablation studies with different retraining settings should be made for a fair comparison of pruning algorithms. To facilitate reproducibility, we would release our implementation upon publication.
2 PRELIMINARY AND METHODOLOGY
Pruning is a common method to produce compact and high performance neural networks from their original large and cumbersome counterparts. We can categorize pruning approaches into three classes: Pruning after training which consists of three steps: training the original network to convergence, prune redundant weights based on some criteria, and retrain the prumed model to regain the performance loss due to pruning (Li et al.) 2016)] Han et al.) [013 Luo al.) 2017 Ye et al. 2018) Wen et al.] 2016 He et al.] 2017); Pruning during training- we update the "pruning mask" while training the network from scratch, thus, allowing pruned neurons to be recovered (Zhu & Gupta
2017. Kusupati et al. 2020; Wortsman et al. 2019: Lin et al. 2020b He et al. 2019 2018); Prwning before training- Inspired by the Lottery Ticket Hypothesis (Frankle & Carbin| 2019), some recent
works try to find the sparsity mask at initialization and train the pruned network from scratch without
changing the mask (Lee et al., 2019
Tanaka et al. 2020 Wang al., 2020).
In this work, we are mainly concerned with the first category i.e. pruning after training which has the largest body of work our knowledge. Traditionally, the last step is referred to as ttnigg i.e., continue to train the pruned model with a small learning rate obtained from the last epoch of the original model. This seemly subtle step is often overlooked when designing pruning algorithms.
Particularly, we found that the implementation of previous pruning algorithms have many notable differences in their retraining step: some employed a small value of learning rate (e.g. 0.001 on ImageNet) to fine-tune the network (Molchanov et al., 2016 Li et al. 2016 Han et al. 2015) for small number of epochs, e.g., 20 epochs in the work by Li et al. (2016); ; some used a larger value of learning rate (0.01) with much longer retraining budgets, e.g., 60, 100 and 120 epochs respectively
on ImageNet (Zhuang et al. 2018) Gao et al. 2020 Li et al.] 2020);; You et al. (2019): [i et al. (2020) respectively utilized 1-cycle (Smith & Topin 2019) and cosine annealing learning rate schedule
2

---

Published as a conference paper at ICLR 2021
Pinetuming
Learzing Rate Restacing
Soord Leamingg Rote Restarting
Dyell Learting Rate Re Restarinng
0.10
0.00
0.10
0.0
Loaming Roti:
Looming Itet:
Lombin
0.00
0.00
00.00
0.06
0.04
D494
u64
0.00
20
60
1000
30
0140
33
(0
0.00
50
Epuchs
Eporta
Epredt
(a) FT
(b) LRW
(c) SLR
(d) CLR
Figure I: Learning rate with different schedules on CIFAR when retraining for 72 epochs. In (a).
the learning rate is fixed to the last learning rate of original training (i.e. 0.001). In (b). the learning rate is "rewound" to previous 72 epochs (which is 0.01). and is dropped to 0.001 after 32 epochs. In (c). after warming up the learning rate, we drop its value by the factor of 10x at 50% and 75% of remaining epochs. In (d), we warm up the learning rate from the lowest the highest value (of standard training) for the first few epochs, then decay the learning rate according to cosine function.
instead of conventional step-wise schedule. Despite such difference, the success of each pruning algorithm is only attributed to the pruning algorithm itself. This motivates us to ask the question: do details like learning rate schedule used for retraining matter?
In this section, we strive to understand the behavior of pruned models under different retraining configurations and how they impact the final performance. Specifically, we conduct experiments with different retraining schedules on simple baselines such as fi-orrm filters pruning (Li et al.] (2016) and magnitude-basee weights pruning (Han et al. 2015). We show that the efficacy of several pruning algorithms can be boosted by simply modifying the learning rate schedule. More importantly, the performance gain by retraining can be remarkable: the accuracy loss can drop to zero and in some cases better accuracy than baseline models can be achieved.
To analyze the effect of retraining pruned network, we based on learning rate rewinding (Renda iet al. 2020) and experiment with different retraining settings. Although in the previous work, Renda al. 2020) demonstrated the efficacy of learning rate rewinding across datasets and pruning criteria, there is a lack of understanding of the actual reason behind the success of this technique. Here we hypothesize that the initial pruned network is a suboptimal solution, staying in a local minima. Learning rate rewinding succeeds because it uses large learning rate to encourage the pruned networks to converge to another, supposingly better, local minima. Our experiment setups are as follows.
Retraining techniques. To verify this conjecture empirically, we conduct experiments with different learning rate schedules including learning rate rewinding (Renda et al. 2020 while varying pruning algorithms, network architectures and datasets. In this work, we consider the following retraining techniques:
1. TUNINT (FT) Fine-tuning tuning is the most common retraining techniques (Han et al. 2015 al., 2016 Liu al. 2019). In this approach, we continue train the pruned networks for epochs with the last (smallest) learning rate of original training.
LEARNING RATE REWINDING (LRW) Renda et al. (2020) propose reuse the learning rate schedule of the original training when retraining pruned networks. Specifically, when retraining for epochs, we reuse the learning rate schedule from the previous epochs, i.e., rewinding.
3. SCALED LEARNING RATE RESTARTING (SLR): In this approach. we employ the learning rate schedule that proportionally identical the standard training. For example, the learning rate is dropped by factor of 10x at 50% and 75% of retraining epochs on CIFAR, which is akin to original training learning rate adjustment. The original learning rate schedule can be found in Appendix A.
CYCLIC LEARNING RATE RESTARTING (CLR): Instead of using stepwise learning rate schedule as scaled learning rate restarting, we leverage the 1-cycle (Smith & Topin 2019), which is shown to give faster convergence speed than conventional approaches.
3

---

Published as a conference paper at ICLR 2021
ResNet-56 PFEC, 14% Sparsit
ResNes- 110 PFEC, 32% Sparsity
VGG-16 PFEC, 64% Sparsity
TRW
0.5
CLR
0.5
Acuzay
3orrra
0.0
0.0
0.5
0.5
15
100
125
150
50
13
100
125
150
Epodu
Epocta
Epozha
Figure 2: One-shot structured pruming on CIFAR-10 dataset using { norm filters pruning Li et al. 2016) while varying retraining budgets. As can be seen, learning rate schedule matters. Schedules that employ large learning rates (LRW, SLR, CLR) are significantly better than fine-tuning. Among them, CLR performs best in most cases.
Res 56 50% sparsity
Net-6e-56 90% sparsity
ResNet-34 PFEC, 11% Sparsity
FT
LBW
LBW
0.5
5 Aesuney
-0.5
-10
20
40
63
(a)
(b)
(c)
Figure 3: One-shot unstructured pruning on CIFAR-10 10 dataset using MWP (Han al. 2015 ((a)
and (b)) and structured pruning ImageNet with E -nnorm filters pruning (Li ct al. 2016) (c).
Note that for the last two strategies (SLR and CLR), we warmup the learning rate for 10% total retraining budget. For simplicity, we always use the largest learning rate of the original training for learning rate restarting. Specifically, the learning rate increased from the smallest learning rate of original training the largest one according to cosine function. The detailed learning rate schedule See also Appendix for the choice of warmup epochs.
of each technique is depicted in Figure
!
Pruning algorithms We consider the following dimensions in our experiments. For praning methods, we use fn-nrmm filters pruning (PFEC) (Li et al. 2016) and (global) magnitude- weights pruning (MWP) (Han et al.) (2015) and evaluate them the CIFAR-10 10, CIFAR- 100 and ImageNet dataset. We examine both variations of pruning namely one-shot pruning and iterative pruning when comparing the proposed retraining techniques. Furthermore, we also experiment the CLR schedule HRank (Lin et al. 2020a), Taylor Pruning (TP) (Molchanov et al. 2019) and Soft Filter Pruning (SFP) (He
al. 2018).
Our implementation and hyperparameters of é -norm filters pruning and magnitude weight pruning are based on the public implementation Li et al. (2019), which shown to obtain comparable results with the original works. For remaining algorithms, we use official implementations with hyperparameters specified according to their papers. The detailed configurations of training and fine-tuning is provided in the Appendix B for interested readers.
Evaluation. For CIFAR-10 and CIFAR- we run each experiment three times and report "mean + std". For ImageNet, we run each experiment once. These settings are kept consistently across architectures, pruning algorithms, retraining techniques, and ablation studies unless otherwise stated.
A CASE STUDY ON RETRAINING IN NETWORK PRUNING
3.1 RETRAINING COST AND PERFORMANCE TRADE-OFF
We first investigate the performance retraining techniques while varying the retraining budget. Figure 2 illustrates the results of f-norm filters pruning (PFEC) (Li et al. 2016) with different retraining techniques on CIFAR-10. 10. We can observe that, larger values of learning rate always attain higher performance than fine-tuming regardless of number of retraining epochs. Furthermore, with
4

---

Published as a conference paper at ICLR 2021
56 PFEC
ResNet-110, PFEC
BesNet-50 ImageNet
Accuracy
Accuary
Accuracy
0.2
0.4
0.6
0.8
Compression Ratio
Compression Ratio
Spuriit
Figure 4: Iterative pruning CIFAR- 10 dataset using & -nnrr filters pruning Li et al. (016)) ((a) and (b)) and on ImageNet using magnitude-basee weights pruning (Han et al. 2015) ((c))
some additional retraining budgets, e.g., 80 to 120 epochs compare 40 epochs in original works, we attain much higher compression ratio with almost no accuracy drop (see Figure S in Appendix).
Figure 3 reports the accuracy of ResNet-56 pruned with magnitude-based weights pruning (Han) et al. 2015) in one-shot manner. We study the impact of different retraining schedules on low and high sparsity i.e. 50% and 90% respectively. can be observed that under both setting learning rate restarting consistently outperforms fine-tuning across number of retraining epochs.
To verify the effectiveness of CLR on large-scale datasets, we conduct experiment using & -orm filters pruning with ResNet-34 on ImageNet in Figure e).. We find that CLR achieve lower performance than fine-tuning for low epochs. This expected phenomenon since (Huang et al. 2017) also found that learning rate restarting on ImageNet usually requires training for 45 epochs while only needs 40 epochs on CIFAR-10. When retraining with number of epochs higher than 45, CLR can reach higher accuracy than fine-tuning.
Thus, the value of LRW can be good heuristic to choose the learning rate for retraining after pruning. concrete example that we achieve significant gains with Taylor pruning when retraining for only 25 epochs as shown in Table g
3.2 MODEL SIZE AND PERFORMANCE TRADE-OFF
In this section, we investigate the trade-off of between model size and performance of compact models under a fixed retraining budgets. Renda et al. (2020) found that learning rate rewinding usually saturate at half of original training, thus, we perform on retraining for 80 epochs on CIFAR-10 and 45 epochs on ImageNet.
We first experiment with iterative l fn-nrrm filters pruning CIFAR-10 and report the results in Figure FigureAaa.b) [a.b). we can observe that SLR and CLR also perform comparable or better than LRW in this setting. We then conduct experiments with iterative unstructured pruning on ImageNet with ResNet-50 to verify the superior of CLR compared to fine-tuning in Figure(Fc). we find that MWP with CLR significantly outperform fine-tuning and even increase the accuracy of the network until 80% sparsity (of convolutional layers).
In our experiments. CLR usually reaches slightly higher accuracy than LRW and SLR. A possible explanation is that 1-cycle schedule gives better convergence speed than conventional schedules (Smith & Topin 2019). thus, bringing much better results when employing "low" retraining budgets.
3.3 EVALUATION WITH OTHER PRUNING ALGORRITHMS
We investigate whether the effectiveness of learning rate restarting holds with other pruning algorithms than simple norm- based pruning. As the performance of larger learning rate schedules such as LRW, SLR, and CLR are rather similar, we select CLR for use in this experiment. Particularly, we examine CLR with SFP (He et al. 2018 in this section..
Soft Filters Pruning (He et al. 2018). Previous sections demonstrate that the success of large learning rate hold for wide range of pruning algorithms, network architectures and datasets. However, we note that all examined pruning algorithms followed the reerrinng-erriiingggg.. paradigm, ie., it is necessary to retrain the pruned model. In this part, investigate regularizing where
5

---

Published as a conference paper at ICLR 2021
Table 1: Results (accuracy) for soft filters pruning (He al. 2018). The "w/o FT column demonstrate the performance of networks follow original work (do not retrain after final prune). "FT" column indicates results of networks when fine-tuning for 200 more epochs. "CLR-x" columns show accuracy of networks after retraining with CLR for more epochs. "%PF" indicates number of pruned filters.
Model
%PF
w/o FT
FT
CLR-50
CLR-100
CLR-200
CIFAR-10
30
92.74+0.28
92.85+0.21 0..94+0.26
92.57++0119
93.00+0.02 92.23++0.12
93.18 0.25 92.78 + 0.34
ResNet-56
40
91.95+0.229
92.19+0.16
CIFAR-10
ResNet-
40
92.25+0..55
0..2
92.3
92.79+00.88
92.91 + 0.41
CIFAR- 100
30
68.92 10.33
69.41
70.27-0..40
70.2410.43
70.29 0.46
ResNet-56
40
67 0.441
67.64+0.50
08.78+0.23
69.12 = 0.22
68.88 68.88+0.44
CIFAR-100
30
70.93
70.94110.58
71.77-10.60
71.7410.63
72.00 0.21
ResNet-1100
40
68.41 10026
69.04:10.41
70.22:10.14
70.05
70.34 + 0.24
gradually, softly prune the network during the course of training. In contrast to pruning-retraining the latter approach does not require further training after final pruning. That being said, we still make case study to verify if retraining does improve the performance of models pruned with this type of algorithms.
To make fair comparison between fine-tuning and no fine-tuning, randomly split the conventional training set of CIFAR- 10/CIFAR- 100 (including 50000 images) to train (90% images of total set) and val (10% remaining images) set and we then report the result of best-validation models on "standard" test set (including 5000 images).
We report the performance of pruned network without fine-tuning, with fine-tuning and with learning rate restarting (CLR) in Table We find that applying fine-tuning on these model usually give very fast convergences speed (roughly in first 20 epochs). However, fine-tuned models exhibit negligible gain then pruned models without fine-tuning. In contrast, CLR significantly improves the performance even with small retraining budgets. Interested readers can find our results with other pruning algorithms namely HRank (Lin et al. 2020a) and Taylor pruning (Molchanov et al.) 2019) in the Appendix C
3.4 SCALING ORIGINAL TRAINING
Liu et al. (019)) present empirical evidence that 3-stage pipeline of structured pruning inferior to training compact models from scratch if training budgets of compact models are scaled proportion to FLOPs ratio between original and pruned model. Following Evei et al., (2019): [ Gale et al. (2019), we conduct experiments to compare this strong baseline with E -norm filters pruning while employing CLR on CIFAR- 10, CIFAR- 100 and ImageNet. Specifically, we compare pruned networks retrained with fine-tuning and CLR mechanism with Scratch-B and Scratch-E(pochh which train pruned networks from scratch with the same computational budget and epochs respectively
When evaluating with CIFAR- 10 and CIFAR- 100, found that the validation accuracies of Scratch- B. Scratch-E, and CLR are very similar to each other, and therefore, the comparisons and ranking could be very sensitive to the variance of the accuracy. To reduce variance, employ the following training scheme. First, we re-run each experiment 5 times on CIFAR- 10 and CIFAR- 100 (more than previous experiments to reduce noise). Second, we randomly (and independently between each run) split the original training set of CIFAR-10 to training (90% images of total set) and validation set (10% images of total set) and report the performance of best validation models on original test set. Note that for ImageNet, we re-run the experiments 3 times. We found that for ImageNet, the variance negligible and thus we simply report the best validation accuracy.
For retraining, we adopt the same retraining budgets as in (Liu et al. 2019 Li et al. 2016) (40 and 20 epochs for CIFAR-10 10 and ImageNet respectively). Since the total number of retraining epochs is
'oor Scratch-E, we use 160 (original training) 40 (retraining) 200 epochs while training from scratch instead of 160 as in [Lu al. 2019

---

Published as a conference paper at ICLR 2021
Table 2: Results (accuracy) for mn-mrrm based filter pruning (Li al.
2016). Configurations of Model
and Pruned Model are both from the original paper (Li et al. 2016). The results of "Scratch-E" and
"Scratch-B" on ImageNet are taken directly from work of et 2019). Top- and second-ranked results are highlighted in bold blue and blue.
Dataset
Model
Unpruned
Version
Fine-tuning
Scratch-E
Scratch-B
CLR
VGG-16
+ 0.21
0.27
93.24 0.25
93.12 0.26
CIFAR-10
ResNet-56
92.32 + 0.32
92.18
92.60 +: 0.17
92.77
0.19
92.71 0.33
92.06 0.23
92.39 + 0.22
92.31
0.23
92.41 0.19
A
92.81 = 0.39
92.97 + 0.22
93.04 =0.04:00 0.18
93.08
0.38
ResNet-1100
92.91 +: 0.38
B
92.34 + 0.33
92.63 +: 0.70
93.20 # 0.25
93.03 0.31
VGGG-1
71.26 + 0.23
69.97 #: 0.32
71.37 +: 0.07
71.72 0.34
71.18 0.07
A
69.55 + 0.54
70.15 + 0.52
69.74 +: 0.57
70.04 0.30
CIFAR-100
ResNet-56
69.51 +: 0.25
69.53 0.28
69.64 + 0.24
69.91 0.53
70.06 0.35
70.71 +: 0.16
71.11 +: 0.05
70.61 +: 0.25
71.29 0.25
ResNet-1110
70.59 + 0.36
69.54 0.27
70.61 + 0.45
70.79
0.22
70.32 0.31
72.96
72.56
73.03
ImageNet
ResNet-34
73.30
72.50 0.08
72.29
72.91
73.14 0.03
limited, for ImageNet, we restart the learning rate to 0.01 (instead of 1.1 in the previous section) to guarantee the convergence of the models.
The results of pruned network via PFEC are shown in Table We can observe that for CIFAR-10, the results of fine-tuning and Scratch-B/E are aligned with the prior work Lf et al. (2019) while CLR achieve slightly better results than Scratch-E and comparable with Scratch-B. However, PFEC+CLR significantly outperform both fine-tuning and Scratch-B by large margin.
Discussion. Our empirical results suggest that practitioners should employ retraining with large learning rate schedules as alternative technique for fine-tuning to obtain compact models with better performance after network pruning. In our results, cyclic learning rate restarting (CLR) is slightly more efficient than scaled learning rate restarting (SLR) and learning rate rewinding (LRW).
INTERPLAY OF PRUNING ALGORIHHMS AND RETRAINING SCHEMES
Section Suggests that learning rate schedule significantly improve the performance of several pruning algorithms. We notice that there are notable differences between settings of implementation of network pruning especially in retraining phase. In this section, show that the difference between implementation could easily lead to misleading results and unfair comparisons between pruning algorithms.
4.1 STRONG BASELINE: L NORM FILTERS PRUNING WITH CLR
In this section, we demonstrate that even with same retraining budgets, utilizing simple CLR with ---or filters pruning can achive comparable or exceed the performance of more sophisticated saliency metrics without meticulous hyperparameters searching. The implementation details are as follows.
Our baseline For pruning, we adopt the pretrained of Torchvision and apply - Filters Pruning on these models. In our implementation, the number of removed filters in each block are approximately equal so that the final pruned models have similar number of parameters with the compared one. Thus, if any, this setting should favor other approaches involving laborious sensitive analysis. For retraining, we apply the CLR learning rate schedule while choosing the maximum value of learning rate according to learning rate rewinding. For pruning algorithms that only retrain for small number of epochs (e.g. epochs in case of Taylor Pruning) we also restart the learning rate value to the slightly higher value of 0.01.
Generative Adversarial Learning (GAL) Lin et al. 2019) suggest to leverage generative adver- sarial learning, which leams sparse soft mask in label-frea and end-to-end manner, effectively
7

---

Published as a conference paper at ICLR 2021
Table 3: Comparing the performance of pruned network via PFEC + CLR and GAL on ImageNet. The results of GAL are taken directly from original papers.
Model
Unpruned Top-1
Method
Param %
FLOPs %
Top-1
GAL-0.5 PFEC + CLR
17.2
43.0
71.95
17.6
75.26
ResNet-50
76.15
GAL-1
42.6
61.4
69.88
PFEC + CLR
43.1
73.81
Table 4: Comparing the performance pruned network via PFEC CLR and HRankPlus on ImageNet. The results of HRankPlus are taken directly from official Github repository.
Model
Unpruned Top-1
Method
Param %
FLOPs J %
Top-1
HRankPlus PFEC CLR
40.8
44.8
75.56
41.4
75.59
ResNet-50
76.15
HRRankPlus PFEC CLR
56.7
62.8
74.19
56.9
74.39
solve the optimization problem. The original work retrain the pruned network for 30 epochs with batch size 32 and initial learning rate of 0.01 (which is decayed by factor of 10 every 10 epochs). Thus, we use CLR with learning rate of 0.01 and batch size of 32 in experiments when comparing with GAL. Table 33 shows the results of PFEC + CLR when retrained under same settings with GAL. We that PFEC+CLR can create very strong baseline compare with GAL.
HRankPlus An extension of HRank (Lin et al. 2020a) which is better than HRank in both retraining efficiency and performance ? Inspired by the discovery that average rank of feature maps of filter is consistent, the authors suggested to iteratively remove low-rank feature maps that contain less information. In the original implementation, the authors retrain the pruned networks for 90 epochs with initial learning rate of 0.1 and also utilize label smoothing. For simplicity, we do not use label smoothing in our implementation. As shown in Table 12 PFEC+CLR also attain similar results with HRankPlus approach.
Additional results on CIFAR-10 and Discrimination-awal Channel Pruning (Zhuang et al. 2018). Taylor Pruning (Molchanov al. 2019), Provable Filters Pruning (Liebenwein al. 2020) reported in Appendix D for interested readers.
4.2 RANDOM PRUNING WITH LEARNING RATE RESTARTING
In last subsection, demonstrated that slight modification in retraining of simple -nomm Filters Pruning can exceed the perfomance of much more sophisticated pruning algorithms. In this subsection, investigate the interplay between pruning saliency metries and retraining configurations by comparing accuracy of randomly pruned networks with the original performance of methodically pruned networks.
We directly adopt the original implementation from the authors of Taylor Pruning, HRankPlus for comparisons. For PFEC and MWP we make use of the implementation of Liu et al. (2019). We selected these works because they do not utilize similar learning rate value LRW in the original implementation, i.e., the authors applied smaller value then the heuristic of LRW. The detailed training recipe of the original works can be found in Appendix E. To make the randomly pruned network have the same structure as the methodically pruned networks, i.e., equal number of filters per layer, propose to replace the importance score of each neuron estimated by these methods by a uniformly random score value.
For retraining randomly pruned networks, we adopt the same principle at described in Section 4.1 Table 33 presents novel results performance of random pruning CLR with corresponding results of considered pruning algorithms. It is worth pointing out that report the original results of all
httpps://git thtp:://ggittbuuuoo httpu//....///i///////aa/aaaaaaa
8

---

Published as a conference paper at ICLR 2021
Table Results of networks when applying random pruning and methodically pruning algorithms. "Original" column presents accuracy of pruned network reported in original papers. "R-CLR" presents the results of Random Pruning with CLR.
Dataset
Method
Model
Params ,
FI.PPs 4 %
Original
R-CLR
ResNet-56
70.0
74.1
92.32
92.40 0.16
ResNet-110
68.3
71.6
93.23
93.37 0.04
CIFAR-10
HRankPlus
DenseNet-40
61.9
59.9
93.66
93.71 0.05
VGG-16
87.3
78.6
98 98.10
93.06 + 0.07
ResNet-34
15.9
72.96 + 0.06
73.47 0.08
PFEC
ResNet-34
10.8
24.2
72.50 + 0.08
73.05 0.05
ImageNet
Taylor
ResNet- 50 72%
44.5
45.0
74.50
74.91
ResNet-50 81%
30.1
35.0
75.48
75.54
Pruning
ResNet-50 91%
11.4
20.0
76.43
75.93
methodical pruning are taken directly from their papers respectively. In our experiments, Random Pruning + CLR could surpass the performance of sophisticated saliency metrics even on large-scale and challenging dataset such as ImageNet with only minimal change in learning rate schedule and initial learning rate value.
Comprehensive results of random pruning with different compression ratio for 4--nrrm Filters Pruning and MWP is reported in Appendix E.2
These results suggest that retraining techniques, e.g., learning rate restarting and learning rate schedule, play pivotal role to final performance. Thus, in order to perform fair comparison of different methods, one should cautious of this seemingly subtle detail. While it is unclear to set up fair comparison between pruning algorithms that belong to different categories, e.g., pruning before/during/after training, iterative vs. oneshot pruning, we advocate standardizing (a set of) retraining configurations for each algorithm group thorough hyperparamters searching find the best configuration of each pruning method when evaluating different algorithms.
DISCUSSION AND CONCLUSION
In this work, we conducted extensive experiments to show that learning rates matter in achieving good performance of pruned neural networks. We concluded that compared to traditional fine-tuning, learning rate restarting in general an efficient way to retrain pruned networks to recover performance drop due to pruning. The success of learning rate rewinding is accounted by the of large learning rates in the retraining. We believe that these findings help raise awareness of proper use of learning rate schedule when desiging pruning algorithms, standardizing empirical experiments and allowing fair comparisons. Our takeaway message is:
Pruning algorithms should be compared in the same retraining configurations.
Our work is not without limitations. So far we investigated with hyperparameters identical to those of the original implementations of the pruning algorithms, and only experiment with different learning rate schedules. We also limited our experiments to the image classification task in computer vision. Considering more datasets and other domains is great research avenue for future work.
REFERENCES
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. arXiv preprint wrXiv:1802.06500 2018.
Alon Brutzkus and Amir Globerson. Why do larger models generalize better? a theoretical perspective via the xor problem. In International Conference Machine Learning, pp. 622-830. PMLR, 2019.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: 2019.
Making all tickets winners. arXiv preprint

---

Published as a conference paper at ICLR 2021
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint arXivv1902.00
Shanggian Gao, Feihu Huang, Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1899-1908, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808. 06866, 2018.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get for free. arXiv preprint arXiv:1004.00109 2017.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. arXiv preprint 20Xx::2002.03331, 2020.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pp. 598-559869 605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Bailin Li, Bowen Wu, Jiang Su, Guangrun Wang, and Liang Lin. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. arXiv preprint arXiv:2007.02491, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprini 1rXiv:1608.0877 2016.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2020.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1538, 2020a.
Shaohui Lin, Rongrong Ji, Chengian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2790-2799, 2019.
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020b.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066 2017.
10

---

Published as a conference paper at ICLR 2021
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1164-11272, 2019.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of ver-parametrization in generalization of neural networks. arXiv preprint arXiv: 12076, 2018.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.088760 2018.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02338 2020.
Leslie N Smith and Nicholay Topin. Super--onvergence:: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Hidenori Tanaka, Daniel Kumin, Daniel LK Yamins, and Surya Ganguli. Pruming neural networks without any data by iteratively conserving synaptic flow. arXiv preprint 2rXiv:2006.05467, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In Advances in Neural Information Processing Systems, pp. 2684-2694, 2019.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less -nn--ooo-o--s----foaiitiie assumption in channel pruning convolution layers. arXiv preprint 1rXiv:1802.00124 2018.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 2133-2144, 2019.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194-9203, 2018.
Michael Zhu and Suyog Gupta. To prume, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXi::1710.0189 2017.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discriminatin-awwan channel pruning for deep neural networks. Advances in Neural Information Processing Systems, pp. 875-886, 2018.
11

---

argmin Published as a conference paper at ICLR 2021
APPENDIX
A NETWORK PRUNING FORMULATION
Let us start with formal definition of the network pruning problem. Let IR. be the parameters of (large) neural network that needs to be pruned, D be the dataset for training. The parameters is updated to minimize the loss function L(:; D). Define sparsity mask E {0, indicating if
weight should be kept removed. Similar to Molchanov et al. (016), we can formulate the pruning
algorithms as finding the optimal sparsity mask such that:
m argmin L(" O m: D) DD)
(1)
m
where @ denotes the Hadamard (element-wise) product, 0" represents the (locally) optimal solution of 0. An alternative approach is minimizing the different between final output of two networks (Lin al. 2019
m
|FF(0" O m; - F(0"; D))l
m
where F:R + DDD is the function mapping input feature (i.e. images) to the final output (before softmax).
Theoretically speaking, the pruning formulation only takes the loss function into account, and there caveat that pruning can result in a set of weights that belong to a bad local minima, making the network generalize poorly. In practice, it is common to observe such issues from the performance drop the pruned model. In such cases, a fine-tuning process can be applied to further optimize the weights, mitigating the accuracy drop. However, what fine-tuning means, the weights might be only slightly adjusted without any guarantee about the final network performance.
B TRAINING CONFIGURATION
The implementation of -norm filters pruning (PFEC) Li al. (2016) and magnitude-based weights pruning (MWP) Han et al. (2015) are adopted from the work of Liu et al. (2019). The implementations of other algorithms are taken from the authors' official repositories.
For simplicity, we adopt Pytorch's pretrained models for ImageNet. The unpruned models (used for é ---ormm filters pruning) are trained with below configurations.
Table 6: Training configuration for unpruned models. To train CIFAR-10, we use Nesterov SGD with 0.9, batch size 64, weight decay 0.0001 for 160 epochs. To train ImageNet, Nesterov SGD with /3 = 0.9, batch size 32, weight decay 0.0001 for 90 epochs.
Dataset
Network
#Params.
Learning rate (1 training epoch)
Test accuracy
ResNet-56
0.85M
0.1
E (0)
CIFAR-10
a
0.01
EE [80, 120)
93.46 0.21%
ResNet-110
1.73M
0.001
te [120, 160)
ResNet-18
11.69M
69.76% top-1
0.1
t (0,30)
ImageNet
ResNet-34
21.8M
0.01
IE [30, 60)
73.30% top-1
0.001
LE (60,90)
ResNet-50
25.5M
76.15% top-1
B. ADDITIONAL RESULTS
In addition results in Figure 2
conduct experiments to examine impact of different retraining techniques to networks pruned with low and high compression ratios. The results are shown in Figure
12

---

Published as a conference paper at ICLR 2021
ResNet-56 PPEC, 49% Spansity
ResNet- PFEC, 63% Sparsity
VGG-16. 72% Spansity
0.5
Law
St.t
0.0
Accmey
-0.5
0.0
1.
100
125
Upoch
Epocas
Epochs
Figure 5: Results from pruning with high compression ratios for one-shot structured pruning CIFAR-10 dataset using & -nor filters pruning (Li al. 2016). With a proper learning rate schedule, is possible achieve almost accuracy drop while having much more compact models and using slightly more training budgets.
Table 7: Results (accuracy) for HRank ilters pruning (Lin et al. 2020a) on CIFAR-10. "Pruned + FTf" is the model pruned from the large model with original fine-tuning scheme that are reported. "Prune + FT and "Prune CLR" are results of our runs with original retraining and CLR respectively. Configurations of Model and Pruned Model are both from the original paper.
Model
Unpruned
Prune + FT f
Prune + FT
Prune + CLR
FLOPs | %
Params %
VGG-16 (BN)
93.93
92.34
91.97
92.53
65.3%
82.10%
ResNet-56
93.26
93.17
92.97
93.16
50.0%
42.4%
ResNet-110
93.50
94.23
93.84
93.90
37.9%
38.7%
DenseNet-40
94.81
94.24
94.10
94.10
40.8%
30.5%
C EVALUATING CLR WITH OTHER PRUNING ALGORITHMS
Taylor Pruning MMlchanov et al. 2019). In this method, the authors estimated the contributions of each filter to the final performance by utilizing the first- and second-order Taylor expansion and performing iterative pruning. In the original implementation, the authors fine tuned the network with a small learning rate (0.001) for 25 epochs on ImageNet. In our implementation, we opt to evaluate the accuracy of network retrained with learning rate restarting (CLR) with relatively small budget. Therefore, we keep the retraining budget the same and only increase the learning rate to 0.01.
Table shows the accuracy of ResNet-50 on ImageNet when pruning 72% of network weights. It can be seen that for a small retraining budget, employing learning rate restarting also gives significant gain in performance.
HRank (Lin et al., 2020a). Inspired by the discovery that average rank of feature maps of a filter consistent, the authors suggested to iteratively remove low-rank feature maps that contain less information. Particularly, we iteratively remove filters in each layer and then retrain the network for 30 epochs each time. We examine the performance of pruned models with HRank CIFAR while employing CLR at each fine-tuning step. Due to the heavily expensive computational cost for retraining, we only run each model once for HRank. the original version of this method, the authors used the learning rate of 0.01 and drop by 10x at epochs and 10. In our implementation, we restart the learning rate 0.1 and apply the CLR schedule.
The quantitative results are shown in Table 7 In our experiments, CLR also attains comparable or better results than proposed retraining methods in original work.
D STRONG BASELINES WITH EG -NORM FILTERS PRUNING
D.1 CIFAR-10
Furthermore, we show that by naively neglecting retraining configurations (i.e. budget), we can also reach state-of-the-art results CIFAR- 10 with PFEC et al. (2016). Specifically, prune two "standard" models namely ResNet-56 and ResNet-110 10 and retrain with CLR for 152 epochs, and then compare with sophisticated algorithms in Table
13

---

Published as a conference paper at ICLR 2021
Table 8:
Results (aceuracy) for Taylor filters pruning (Molchanov al. 2019) columns have same meaning with corresponding columns in Table
ImageNet. All
Model
Unpruned
Prune + FT
Prune + CLR
FLOPs %
Params %
Taylor-FO- FO----56%
71.69
72.51
67.2
66.8
TaylorFFO-B -O--N-2%
76.15
74.50
75.22
45.0
44.5
Taylor-FO-Bil F---O-------- 9N-81%
75.48
75.67
35.0
30.1
Table 9: Results of ResNet-56 and ResNet-110 on CIFAR-IO. The performance of other pruning algorithms are taken directly from original papers.
Model
Method
Unpruned
Params
FLOPs
Prune
Acc. 4
PFEC Li et al.
2016
93.04
14.1
27.6
93.06
-0,02
NISP Yu al.
2018
93.26
42.4
35.5
93.01
0.25
ResNet-56
HRank [.in
2020a
93.26
42.4
50.00
93.17
0.09
GAL. in al.
2019)
93.26
37.6
92.98
0.28
PFEC
+ CTR (ours)
93.21
48.7
93.29
-0.07
PFEC [i ct al.
2016)
93.53
32.6
38.7
93.30
-0.23
GAL-0.5 (Lin al.
(019
93.50
44.8
48.4
92.74
0.76
ResNet-110
HRank (Lin et al.
2020a)
93.50
59.2
58.2
93.36
0.14
PFEC+CLK (ours)
93.56
64.2
93.69
-0.13
D.2 IMAGENET
Taylor Pruning (Molchanoy et al. 2019) In this method, the authors estimated the contributions of each filter to the final performance by utilizing the first- and second-order Taylor expansion and performing iterative pruning.
Original work retrains pruned networks for 25 epochs with learning rate of 0.001 with the exception for Taylor-FO-BN-560 original works of Molchanov et al. (2019). most cases, PFEC+ CLR attains higher or comparable accuracy with Taylor Pruning.
Table [00 demonstrates the performance of PFEC+CLR compare with
Table 10: Comparing the performance of pruned network via PFEC + CLR and Taylor Pruning on ImageNet. The results of Taylor Pruning are taken directly from original papers.
Model
Method
Param %
FLOPs 1 %
Unpruned Top-1
Top-1
Taylor-F--BN-98 PFEC CLR
21.1
22.3
73.31
72.83
ResNet-34
22.5
73.30
73.01
Thylor-FO-BN-566
66.8
67.2
76.18
71.69
PFEC CLR
68.8
76.15
70.70
Taylor-FO-BN-72%%
14.5
45.0
76.18
74.50
ResNet-50
PFEC CLR
44.5
76.15
75.04
Taylor-FO-BN-8145
30.1
35.0
76.18
75.48
PFEC CL.
30.1
76.15
75.79
Taylor-FO-BN-91 PFEC CLR
11.4
20.0
76.18
76.43
12.5
76.15
76.37
Discrimination-awaree Channel Pruning (Zhuang et al. 2018) The authors introduce additional discrimination aware losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error.
14

---

Published as a conference paper at ICLR 2021
We the retraining budget of 60 epochs and set initial learning rate 0.01 same Zhuang et al. (2018). However, we employ the 1-cycle learning rate instead of stepwise learning rate as original work. The detailed comparison is presented in Table II
Comparing the performance of pruned network via PFEC + CLR and DCP on ImageNet. The results of DCP are taken directly from original papers.
Table 11:
Model
Param %
FLOPs %
Method
Unpruned Top-1
Top-1
28.1
27.1
DCP PFEC + CLR
69.64
69.21
31.9
69.76
69.31 = 0.06
ResNet-18
47.1
46.1
DCP PFEC + CLR
69.64
67.35
50.6
69.76
67.38
65.7
64.1
DCP PFEC + CLR
69.64
64.12
69.76
64.08
33.3
35.7
DCP PFEC + CLR
76.01
76.40
33.7
76.15
76.03
ResNet-50
51.4
55.5
DCP PFEC + CLR
76.01
74.95
51.5
76.15
75.16
65.9
71.1
DCP
76.01
72.75
66.1
PFEC + CLR
76.15
72.92
Provable Filters Pruning (Liehenwein et al. 2020)
This algorithm uses small batch of input data points to assign saliency score to each filter and constructs an importance sampling distribution where filters that highly affect the output sampled with correspondingly high probability.
Original implementation of Liebenwein et al. (020) retrain the pruned network on ImageNet for 90 epochs with standard learning rate schedule (drop learning rate by factor of 10 after 30 epochs) with batch size of 256 and weight decay of 0.0001. In our implementation, pruned networks are trained for epochs with maximum learning rate of 0.1 while keeping the same configurations for all other hyperparameters as original work.
Table 12: Comparing the performance of pruned network via PFEC CLR and Provable Filters Pruning (PFP) on ImageNet. The results of PFP are taken directly from original paper (Liebenwein
al. 2020
Model
Method
Param %
FLOPs %
Unpruned Top-1
Top-1
PFP (lowest top-1 err.)
18.0
10.8
76.13
75.91
PFEC CLR
18.1
76.15
76.56
ResNet-50
PFP (within
1.0% top-1)
44.0
30.1
76.13 76.15
75.21
PFEC CLR
44.3
75.38
E RANDOM PRUNING WITH CLR
E RETRAINING CONFIGURATION
We compare the configuration between original implementation of aforementioned pruning methods and corresponding random pruning Table I3 Note that for HRankPlus, the original implementation employs different values of weight decay for each model: 0.005, 0.006, 0.005, 0.002 for VGG-16, ResNet-56. ResNet-110 10, DenseNet-40 respectively. We found that these value is relatively higher than "conventional" values (0.0001) making the models hard to converge after restarting. Thus, use weight decay of 0.0005 through out experiments with HRankPlus. In all our experiments, other details such batch size, retraining budget are similar value of original implementation.
15

---

Published as a conference paper at ICLR 2021
Table 13: Training configurations of original pruning methods and random pruning.
Method
Model
Original
Random Pruning
ResNet-56
0.01
[0, 150)
ResNet-110
0.001
E [151, 225)
0.0001
E [226, 300)
DenseNet-40
Ahnir 0.001
HRankPlus
Oma 0.1
Omin 0.00001
0.01
E [0,50)
VGG-16
0.001
t [51, 100)
0.0001
EE [101, 150)
PFEC
ResNet-34
0.001
Ee [0, 20)
0.0001
Qmax 0.01
Qmii 0.000001
Taylor Pruning
ResNet-50
0.001
(0,
ResNet-56, CIFAR-10
ResNet-56.
CIFAR-100
ResNet- 110, CIFAR-10
ResiNet- CIFAR-100
DLa Bil
B47
AAcuracy
0.0
A Accunicy
Accuracy
A Acmmary
005
-0.5
-1.0
1.5
-1.6
1.50
1.25
1,50
1.75
1.50
Compressioo Ratio
Compression Ratin
Compression Ratio
Compression Ratio
(a) ResNet-56
(h) ResNet-56
(c) ResNet-110
(d) ResNet-110
CIFAR-10
CIFAR-100
CIFAR-IO
CIFAR- 100
Figure 6: One-shot structured pruning CIFAR-10 dataset using PFEC (Li et al. randomly filters pruning with different retraining schemes.
2016 and
E.2 RANDOM PRUNING WITH VARIOUS COMPRESSION RATIO
4-normm Filters Pruning Figure 6 illustrates the performance of networks pruned with PFEC and random pruning CIFAR-10 and CIFAR- 100 when retraining with 40 epochs the same setting used by Li et al. (201)). We can see that randomly pruned networks consistently achieve superior performance than methodically pruned networks (fine-tuned with standard learning rate schedule) in terms of accuracy. However, random pruning obtain lower accuracy than PFEC when using identical retraining techniques.
Magnitude-based Weights Pruning We extend the scope of experiment in Sec 4.2 to unstructured pruning and analyze performance MWP with CLR. Figure 7 represents the results of random pruning with CLR and methodically pruning while varying difference compression ratio in both iterative and oneshot pruning manner. Specifically, we retrain the trimmed network for 40 epochs. We observe that though Random Pruning CLR can achieve higher accuracy with low sparsity, the performance of randomly pruned network immensely reduced with the increasing of compression ratio.
16

---

Published as a conference paper at ICLR 2021
NesNett-56, CIFAR-10 (Oneshot)
ResNet-56, CIFAR-10 (Iterative)
1.0
1.0
CLR
FT
R-CLR
0.5
0.5
3 Accuracy
Accuracy
0.0
0.0
-0.5
-0.5
-1.0
-1.0
-1.5
-1.5
0.2
0.4
0.8
0.4
0.6
0.8
Compression Ratio
Compression Ratio
Figure 7: One-shot and Iterative unstructured pruning on CIFAR-10 10 dataset using MWP (Han et al. and randomly weights pruning with different retraining schemes.
2015)
ResNct-56 PFEC, 89% Sparsity
0.5
Accuracy
0.0
-0.5
Warmup 0%
Warmup 10% Warmup 15%
Eine-tuning
Warmup 5%
1.0
40
60
80
100
120
140
Epochs
Figure 8: 4-norm Filters Pruning (Li et al.
2016
CIFAR-10 with different number of warmup
epochs.
F PERFORMANCE OF CLR WITH DIFFERENT WARMUP EPOCHS
To avoid the laborious hyperparameters tuning process, simply warm up the learning rate for CLR and SLR for the first 10% budget of retraining. In this section, we investigate whether the performance of pruned network is sensitive to this hyperparameter.
Figure X presents the results of pruned networks obtained with fine-tuning and CLR while varying number of warmup epochs namely 0, 5, 10, 15% of total retraining. We observe that there is no significant diversity in results of CLR with different number of warmup epochs and all of them exceed performance of fine-tuning by large margin.
G TRAINING FROM SCRATCH WITH CLR
In this section, we conduct experiment where we train the unpruned models from scratch with CLR schedule. Then, we compare performance of networks pruned from these baseline with those pruned from models trained with conventional step wise learning rate schedule. Table 1 presents the accuracy of pruned ResNet-56 models on CIFAR-10 with different training and retraining recipes. use the budget 160 for training the baseline networks. For retraining the trimmed networks, we examine the budget of 40 and 56 epochs. We can that both CLR and SLR consistently exceed the performance of fine-tuning under these configurations.
17

---

Published as a conference paper at ICLR 2021
Table 14: Performance of ResNet-56 pruned from models trained with CLR and conventional step- wise learning rate schedule on CIFAR-IO. "Epoccs" column indicates the number of epochs for retraining trimmed networks. "Schedule" colum indicates the learning rate schedule used for training baseline (unpruned) networks. The best and second best methods are highlighted in bold blue and blue respectively.
Epochs
Schedule
Param
%
Baseline
Fine-tuning
SLR
CLR
Step-wise
13.7
93.15 + 0.36
92.81 + 0.49
92.93
0.15
93.14
0.37
CLR
13.7
93.45 + 0.17
93.06 + 0.20
93.03 0.16
93.15 0.11
Step-wise
34.9
93.15 + 0.36
92.25 + 0.35
92.83 0.16
92.81 0.32
CLR
34.9
93.45 + 0.17
92.50 + 0.07
92.63 = 0.05
93.03 0.16
Step-wise
13.7
93.15 + 0.36
92.81 + 0.41
92.86 0.28
93.22
0.37
CLR
13.7
93.45 + 0.17
93.04 + 0.16
93.22 0.05
93.37 0.25
Step-wise CLR
34.9
93.15 + 0.36
92.48 + 0.45
92.89 = 0.11
93.26 0.17
34.9
93.45 1 0.17
92.63 1 0.13
92.83 = 0.03
93.29 0.25
18


Reviews:
[{"review_id": "qxek1HUNtoe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper conducts extensive experiments to understand the reason behind the uncanny effectiveness of learning rate rewinding: the usage of a large learning rate.\n\n### pros\n* The paper, in general, is well-written, and the main message is very clear.\n* The paper identifies the reason behind the success of learning rate rewinding through several aspects: retraining cost, model size, and pruning algorithms. It provides guidelines alternative to fine-tuning for practitioners to obtain compact models with better performance after network pruning.\n* The observations of the random pruning are interesting and are aligned with the prior work [1].\n\n### cons\n1. The paper needs to improve its clarity.\n    * It is suggested to include the details for the scores (e.g. 1.12 x in the title) of the subfigures in Fig. 2. Right now it is unclear to me how to calculate the value and why it is meaningful to present the results under these values.\n    * Can you justify why it is necessary to include the learning rate warmup scheme for SLR and CLR, and why the paper only uses 10% of the total retraining budget? How much will the different fractions of the warmup epochs impact the retraining performance? An ablation study is required here.\n    * The observations for random pruning with learning rate restarting are interesting but more details are required. E.g., it is unclear to me the form of performing the random pruning. Is it layerwise random pruning (same sparsity per layer) or global-wise random pruning?\n2. Potential unfair comparison by using CLR. \n    * The paper investigates the impact of different learning rate schedules for the re-training, after pruning on the model pre-trained by the standard stage-wise learning rate schedule. This design choice is sufficient to provide some practical guidelines, but it may also blur the contribution: as CLR is quite different from the other learning rate schemes, it is natural to question if the performance gain is solely due to a better learning rate schedule (but not large learning rate). Can you also provide an ablation study in terms of using CLR for both the training from scratch and re-training, and then compare both the accuracy and the accuracy drop scores?\n    * The paper demonstrates the efficacy of CLR in terms of re-training the pruned model on the standard image classification benchmark. However, it is unclear to me if the same observations can be generalized to other CV tasks or even NLP tasks. It is encouraged to include some preliminary results to argue the generalization ability of the provided practical guidelines.\n\n### reference\n1. Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot, NeurIPS 2020.\n\n### post-rebuttal\nThe authors have addressed most of my concerns, thus I will increase my score from 5 to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "O1bANpi_hVi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary \nThis work focuses on evaluating different fine-tuning strategies after structured and unstructured pruning. The results show that high learning rate schedules (like cosine schedule) attain best performance in many different setting. With this high learning rate random (structured) pruning seems to work as well as the other pruning criteria. Overall the work has a strong coverage of experiments and the results could be helpful to the community. However, I think the work misses some important baselines and require a bit more work on writing. \n\n## Pros\n- A comprehensive coverage of different pruning algorithms is definitely a plus. Experiments are focusing more on the structured pruning methods, which I am not sure useful given the results of [1] (see cons below). \n\n- Authors seem to have a good knowledge of recent structured pruning methods, which makes the study and the experiments convincing/strong.\n\n## Cons\n- I don't think the following statement is true (at least many of the unstructured pruning methods: \"In most cases, pruning consists of three steps: training..prune..retrain\" The paper starts with this premise and ignores many of the other iterative pruning methods (which prune during training). Many of the best unstructured pruning methods are iterative GMP (https://arxiv.org/abs/1710.01878), DNW (https://arxiv.org/pdf/1906.00586.pdf), STR (https://arxiv.org/abs/2002.03231). \n\n- Only exception to the point above is the Table-1 (SFP); however in that table the authors miss an important baseline which is scaling the original training proportional to the Training+FT like it is done in [1, 2, 3]. This baseline should be added to the table and considered wherever possible. \n\n- \"Recently, Renda et al. (2020) proposed a state-of-the-art technique for retraining pruned networks namely learning rate rewinding\" I don't think this sentence is accurate. It's true authors claim SOTA, but I would argue they miss an important baseline in which the original training schedule of the iterative pruning algorithms are scaled according to the training budget as it is done in [1, 2, 3]. In my experience this method performs better than LRW. Therefore it would be nice to include this baseline whenever possible. \n\n- Does CLR results for l1-pruning exceed Scratch-B results of [1]. If yes, this is very important to mention/highlighy. Otherwise, I like to see a discussion about why we should care about structured pruning methods as the main focus of the work seems to be those. \n\n- What is the difference between PFEC and PFEC-B? And why do authors use this acronym? \"l1-structured\" might be a more appropriate choice. And what are the multipliers in Figure-2 (i.e. 1.12x, 1.45x...)? It would be better to use \"sparsity\". \n\n- \"A well-known practice is fine-tuning, which aims to train the pruned model with a small fixed learning rate.\" and \"More advanced learning rate schedules exist, which we generally refer to as retraining.\" I rather call of them fine-tuning or warm-restart, as all networks start with a pretrained network. This terminology would align better with the previous work. Then you call call constant-lr, lrw, clr, etc...\n\n## Minor Points\n- \"Here we hypothesize that the initial pruned network is a suboptimal solution, staying in a local minima.\" I found this statement a bit vague. Networks are usually not not converged after they are pruned (or even at the end of an ImageNet training). We also don't know whether with long enough training the small learning rate finetuning would get same good results or not. It would be nice to make this statement more precise. Maybe something like \"high learning rate would help find better minima faster.\" \n- \"as `1-norm filters pruning\" filter pruning. Structured pruning is used more often similarly \"weights pruning\" -> unstructured pruning\n\"For simplicity, we always use the largest learning rate of the original training for learning rate restarting...\" I didn't understand this statement. Is this for CLR?\n- \"For CLR and SLR, the learning rate is increased from the smallest learning rate of original training to the largest one according to cosine function\" probably the other way around Learning rate is decayed over time?\n\n- \"For ImageNet, we run each experiment once.\" It would be great to run few more to get more precise results before the final version. \n\n- \"under both setting learning rate restarting approaches consistently...\" -> under both settings lr-restart consistently...\n\n- Is Figure-3 a/b MWP? It would be nice to mention this in the title or caption.\n\n- \"Being that said,\" -> That being said\n\n- \"there are notable differents between\" ->  differences\n\n- \"for future works.\" -> for future work\n\n[1] RETHINKING THE VALUE OF NETWORK PRUNING, https://arxiv.org/pdf/1810.05270.pdf\n[2] The State of Sparsity in Deep Neural Networks, https://arxiv.org/abs/1902.09574\n[3] Rigging the Lottery: Making All Tickets Winners, https://arxiv.org/pdf/1911.11134.pdf\n\n## After Rebuttal\n- I thank authors for considering my suggestions. I increase my score to 5. Having a quick look (I am sorry that I didn't have more time) at the new results; most results on structured pruning seem to agree with [1]; with some improvements over baselines when CLR is used when training from scratch. Results on unstructured pruning seems minimal and focuses mostly on one-shot pruning; and furthermore the baseline suggested above (i.e. scaling the entire learning_rate schedule)  is not added to the iterative pruning results. Overall, I like the direction of the paper, but I think the motivation should be improved and results should be distilled.", "extra_scores": {}, "extra_text": {}}, {"review_id": "820tCAKilvN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors conducted a comprehensive set of experiments on choices of learning rate schedules for re-training/fine-tuning during iterative or after 1-shot pruning of deep convnets.  Empirically, they reported that high learning rate (LR) is particularly helpful in recovering generalization performance of the resultant sparse model.  The results are purely empirical, well-documented observations from well-designed experiments, which is of practical value in practice of network compression, and the consistent, somewhat surprising observation raises interesting questions.  \n\nNotably, this work has brought to attention an important but often overlooked aspect of network pruning: there exist complex interactions between the dynamics of optimization and sparsification, and as a consequence, it is only fair to compare two sparsification techniques when each of them are put in the _best_ optimization setup, respectively.  \n\nI have a few comments that I wish the authors would address here, discuss in revision or note for future work:\n\n(1) Why is large LR helpful in recovering the accuracy of sparse nets?  There is little information provided in these experimental results to shed light on this question.  There has been loss landscape studies of sparse nets during training (such as arxiv:1906.10732, arxiv:1912.05671)--perhaps these could be applied to study the problem.  If the high LR's role were to knock the solution out of bad local minima, then does adding noise to gradients or smaller batch size achieve similar effect at the initial phase of re-training?  \n\n(2) Given a fixed re-training flop budget, after a pruning operation on the network, both (a) weight value rewinding (as in the Lottery Ticket Hypothesis training), (b) re-training LR schedule (as in this work) might be potentially helpful.  How does weight value rewinding interact with LR?  \n\n(3) For the random pruning results in Sec. 4, do fine-grain unstructured pruning methods present the same results?  \n\n(4) Does the result generalize to transformer models?  What about optimizers?  Does Adam present a same story as SGDM? \n\nPage 5, line1 of the 3rd paragraph of Sec. 3.2: typo \"reachs\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "5ZDmrArjDtK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper analyzes the role of learning rate in re-training after pruning, building on previous findings that changing the learning rate schedule of re-training can result in higher accuracy than low-learning-rate fine-tuning. The paper proposes several learning rate schedules to compare, specifically a cyclic learning rate (gradually ramping up to and back down from the maximum learning rate schedule used during the original training phase) and a compressed version of the original learning rate schedule, and shows that these learning rate schedules outperform standard fine-tuning and also learning rate rewinding, showing that the findings of prior work come from using a higher learning rate in general and not any specific schedule. The paper than shows that choice of re-training learning rate schedule can have more impact on final accuracy than choice of saliency metric.\n\n# Strengths\n\nThe paper, fairly conclusively, finds the following novel results:\n- Re-training with a cyclic learning rate outperforms learning rate rewinding, seemingly leading to a new state-of-the-art re-training algorithm\n- Re-training with a scaled learning rate schedule attains similar accuracy to re-training with learning rate rewinding\n- Re-training a network that has already been pruned and trained with some scheme (e.g., Soft Filter Pruning, Taylor expansions) can reach higher accuracy with cyclic learning rate re-training than standard fine-tuning re-training.\n\nLess conclusively, though still with reasonable evidence, the paper finds:\n- The choice of learning rate schedule when re-training can have a higher impact on accuracy than the choice of saliency metric: specifically, at higher sparsities, re-training a randomly pruned netowrk with a cyclic learning rate schedule results in higher accuracy than re-training a magnitude pruned network with fine-tuning at a low learning rate.\n\n# Weaknesses\n\n- The evaluation of the paper focuses heavily on structured pruning. However, structured pruning can be an unreliable testbed for many of these techniques, as shown by [1]. The paper would benefit from discussion of [1] in relation to the structured pruning results -- for example, can similar accuracy be attained by training a randomly initialized pruned network with the same effective learning rate schedule? Without discussion or evaluation of baselines from [1], it's hard to know quite how to interpret the structured pruning results.\n- The findings about the interplay between pruning saliency metrics and re-training schedule (Section 4), while interesting, are only minimally validated. Specifically, it would be interesting to see full curves of accuracies for each of these techniques like the curves in Figure 4 (at least for weight-level pruning) with a plot showing the accuracy of ResNet-56 with \"Random Pruning + Fine-tuning\", \"Magnitude Pruning + Fine-tuning\", \"Random Pruning + CLR\", \"Magnitude Pruning + CLR\" across different sparsities, which would generate a lot more confidence in this result\n- The paper lacks specific hypotheses which are tested and validated/falsified. Specifically -- it's hard to know what conclusion to pull from Sections 3.1 and 3.2 other than that CLR > SLR >= LRW, and it's hard to know what conclusions to draw from Section 3.3.\n- The more conclusive findings of the paper, that a high learning rate is important for optimization of pruned networks and that cyclic learning rates improve on learning rate rewinding, are a relatively incremental contribution\n\n\n# Overall recommendation\n\n5: Weak reject\n\nI would be willing to raise this score if the authors address some of the weaknesses listed above. Specifically, if the authors can demonstrate that the results on MWP in Table 3 consistently generalize to other sparsities, or generate more confidence in the structured pruning findings (e.g., by showing that they result in higher accuracy than the trained-from-scratch structured pruned networks in [1]).\n\n# Other comments and suggestions\n\n- Minor typo in Figure 2 caption: \"LLRW\"\n- HRank in Section 3.3: if the results are not presented and discussed in the main body of the paper, it'd probably be better to move the entire discussion of HRank to the appendix\n- what does \"Params\" in the tables mean? I assume it means sparsity (i.e., percentage of parameters that are pruned away), but this is never made explicitly clear.\n- why is R-FT missing for MWP in Table 3?\n\n# References used in review:\n\n[1] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. \"Rethinking the value of network pruning\"\n\n\n# Update post author response:\n\nThanks to the authors for the response. The newly reported results (specifically, those of Appendix E.2) satisfactorily address my concerns about both the generalization of the pruning method v.s. re-training scheme results, both in terms of sparsity levels and unstructured/structured pruning (though the observation of the relationship between R-CLR and FT do not hold quite as strongly for unstructured pruning, they do hold at high enough sparsities to be interesting). I’ve raised my score to a 6 as a result.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper follows the observations of Renda et al. (2020) that the learning rate in the fine-tuning or retraining phase of neural network pruning is an under-considered component of the pruning process. Renda et al. (2020) argue for a technique that uses the learning rate schedule of the original training regime for fine-tuning. However, their work does not offer a hypothesis or an explanation for why this works.

This work instead offers more insight into why reusing the original learning rate is productive. Specifically, it shows that using high learning rates is the key component. To demonstrate this, the paper includes a study of using the original step-wise learning from the original training regimen, except accelerated for a given number of fine-tuning epochs. The paper also demonstrates that Cyclic Learning Rate Restarting (CLR) also provides an effective, if not better, learning rate schedule for fine-tuning.

As noted by the reviewers, the core observations and contributions of this work are modest, but are still a valuable addition to the literature in the pruning community.  

Having said that, there are some confounding issues with CLR. Specifically, that CLR itself may simply be a more effective learning rate schedule for training neural networks, independent of the particular application to fine-tuning (Reviewer 1). The revision includes an additional appendix that dispels some of this concern. However, indeed, the CLR does improve the base network performance for some configurations.

Broadly, the value proposition here is a thorough demonstration of learning rate schedules for fine-tuning with an overall take that comparisons between techniques need be more sensitive to this choice as previous work perhaps has not thoroughly considered alternative learning rates.


Decision: Accept (Poster)