Rethinking Again the Value of Network Pruning -- A Dynamical Isometry Perspective
Authors: Huan Wang
Can Qin
Yue Bai
Yun Fu
Venue: ICLR 2022 Submitted (2022)
Keywords: neural network pruning
dynamical isometry
model compression
filter pruning

Abstract: Several recent works questioned the value of inheriting weight in structured neural network pruning because they empirically found training from scratch can match or even outperform finetuning a pruned model. In this paper, we present evidences that this argument is actually \emph{inaccurate} because of using improperly small finetuning learning rates. With larger learning rates, our results consistently suggest pruning outperforms training from scratch on multiple networks (ResNets, VGG11) and datasets (MNIST, CIFAR10, ImageNet) over most pruning ratios. To deeply understand why finetuning learning rate holds such a critical role, we examine the theoretical reason behind through the lens of \emph{dynamical isometry}, a nice property of networks that can make the gradient signals preserve norm during propagation. Our results suggest that weight removal in pruning breaks dynamical isometry, \emph{which fundamentally answers for the performance gap between a large finetuning LR and~a small one}. Therefore, it is necessary to recover the dynamical isometry before finetuning. In this regard, we also present a regularization-based technique to do so, which is rather simple-to-implement yet effective in dynamical isometry recovery on modern residual convolutional neural networks.

Under review as a conference paper at ICLR 2022
RETHINKING AGAIN THE VALUE OF NETWORK PRUN- ING A DYNAMICAL ISOMETRY PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Several recent works questioned the value of inheriting weights in structured neu- ral network pruning because they empirically found training from scratch can match or even outperform finetuning a pruned model. In this paper, we present ev- idence that this argument is actually inaccurate because of using improperly small finetuning learning rates. With larger learning rates, our results consistently sug- gest pruning outperforms training from scratch on multiple networks (ResNets. VGG11) and datasets (MNIST, CIFARIO, 10, ImageNet) over most pruning ratios. To deeply understand why finetuning learning rate holds such critical role, we examine the theoretical reason behind through the lens of dynamical isometry, a nice property of networks that can make the gradient signals preserve norm during propagation. Our results suggest that weight removal in pruning breaks dynamical isometry, which fundamentally answers for the performance gap between large finetuming LR and small one. Therefore, it is necessary to recover the dynamical isometry before finetuning. In this regard, also present a regularization-bant based technique to do so, which is rather simple-to-implenee yet effective in dynamical isometry recovery on modern residual convolutional neural networks.
INTRODUCTION
Pruning is a time-honored methodology to reduce parameters in neural network without seriously compromising its performance (Reed, 1993; Sze et al., 2017). The prevailing pipeline of pruning comprises three steps: 1) pretraining: train dense model; 2) pruning: prune the dense model based on certain rules; 3) finetuning: retrain the pruned model to regain performance. Most existing research focuses on the second step, seeking the best criterion select unimportant weights incur as less performance degradation as possible. This 3-step pipeline has been practiced for more than 30 years (Mozer & Smolensky, 1989; LeCun et al., 1990) and is still extensively adopted in today's ' pruning methods (Sze et al., 2017).
to
These said, several recent works (Crowley et al., 2018; Liu et al., 2019) questioned the necessity of inheriting weights from pretrained model because they empirically found the small model trained from scratch can match (or sometimes outperform) the counterpart pruned from the pre-trained large model. This acutely challenges the past wisdom as well as our common belief about pruning. As far as we know, there is no formal response this critical conflict. A theoretical-Iewe understanding of this problem is even more elusive.
Meanwhile, the pruning community has been observing even more open questions. Specifically, (Renda et al., 2020; Le & Hua, 2021) found that the learning rate (LR) in finetuning holds a critical role in the final performance. A proper learning rate schedule (e.g., larger initial LR 10-2 vs. 10-3 with step-decay schedule) can improve the top-1 accuracy of pruned ResNet-34 model (He et al., 2016) by more than 1% on ImageNet (Deng et al., 2009). This discovery calls for more attention being paid to the finetuning step when comparing different pruning methods. Unfortunatcly, they did not present more theoretical insights to explain its occurrence. This also remains an open question in the community up to date.
In this paper, we will show these two open questions actually point to the same one. Specifically, we rerun the experiments of (Crowley et al., 2018; Liu et al., 2019) and find simply using larger fine- tuning LR (10-2 2 vs. 10-3 and decay it) can significantly improve the final performance. Compared to the improved pruning performance, training from scratch does not compete surpass pruning

---

Under review as a conference paper at ICLR 2022
anymore (see Tab. 1 and Tab. 2 on ImageNet). This observation invites many questions immediately: (1) Theoretical understanding: Why does this happen? What is the theoretical reason behind it? (2) Practical solution: If this is a problem, how to fix it? Can the understanding of this problem lead us to better pruning algorithms?
This paper will present answers to all these questions. The key tool employ to unveil the mys- teries is dynamical isometry (Saxe et al., 2014), which describes a kind of nice property in neural networks that are easy to optimize. We carefully design an explanatory experiment using linear MLP (multi-layer perceptron) network to demonstrate how finetuning LR affects the final perfor- mance by affecting dynamical isometry. In brief, we observe the finetuning process can recover dynamical isometry; a larger LR can help recover it faster (or better), hence the better final per- formance. The proposed explanation is validated by our empirical results and resonates with many empirical observations. Furthermore, by the explanation, we learn dynamical isometry recovery is rather imperative. To achieve so, we present a very simple regularization -ased method for prun- ing and show its effectiveness in recovering dynamical isometry modern residual convolutional neural networks (CNNs).
Contributions. (1) We empirically demonstrate the questioning about the value of inheriting weights in structured pruning in previous works is inaccurate and point out that the direct cause is improperly using small finetuning LR. Our finding justifies the value of inheriting weights in structured pruning. (2) On top of the empirical finding, more importantly, we present a theoretical explanation through examining the dynamical isometry of networks in pruning. This explanation is empirically validated by our carefully designed control experiments. (3) In addition to the theoreti- cal understanding, we also propose a regularization-batt Despite its brutal simplicity, it is shown effective to recover the broken dynamical isometry on mod- emn residual convolutional neural networks.
method for dynamical isometry recovery.
2 RELATED WORK
Conventional pruning. Pruning aims to remove many parameters possible in neural network meanwhile maintaining its performance. There are many ways to categorize pruning methods. The most popular two are grouping by pruning structure and methodology.
(1) In terms of pruning structure, pruning can be specified into unstructured pruning (Han et al., 2015: 2016) and structured pruning (Wen et al., 2016; Li et al., 2017; He et al., 2017). For the former, a single weight is the basic pruning element. Unstructured pruning can deliver a high compression ratio; whereas, without regularization, the pruned locations usually spread randomly the network, which is hard to exploit for acceleration. On the opposite, structured pruning introduces certain patterns the pruned locations, which benefit subsequent acceleration while cannot achieve as much compression. Choices between unstructured and structured pruning depend on specific application needs. For structured pruning, there are still many sub-groups (Mao et al., 2017). In the literature, without specific mention, structured pruning means filter pruning channel pruning. This paper focuses on structured (filter) pruning because the "no value of inheriting weights" " argument is mainly discussed in this context (Liu et al., 2019).
(2) In terms of pruning methodology (i.e., how to select unimportant weights to prune), pruning falls into two paradigms in general: importance-basee and penalty-based The former prunes weights based on some established importance criteria, such as magnitude (for unstructured pruning) (Han al., 2015; 2016) Lı-norm (for filter pruning) (Li al., 2017), saliency based on 2nd-order gradients (e.g., Hessian or Fisher) (LeCun et al., 1990; Hassibi & Stork, 1993; Theis et al., 2018: Wang et al., 2a; Singh & Alistarh, 2020). The latter adds penalty term to the objective function, drives unimportant weights towards zero, then removes those with the smallest magnitude. Note, the two groups are not starkly separated. Many methods take wisdom from both sides. For exam- ple, (Ding et al., 2018; 8; Wang et al., 2019b; 2021b) select unimportant weights by magnitude (akin to the first group) while also employing the regularization to penalize weights (akin to the second group). There is no conclusion about which paradigm is better, yet empirically, the state-of-the-ar pruning methods are closer to the second paradigm, i.e., deciding weights via training instead of some derived formulas. Although no theories have formally discussed the reason, we can take rough guess with the knowledge from this paper: Training can recover dynamical isometry, which is beneficial to subsequent finetuning.
2

---

Under review as a conference paper at ICLR 2022
For more comprehensive literature, we refer interested readers to several surveys: outdated one (Reed, 1993), some recent surveys of pruning alone (Gale et al., 2019; Blalock et al., 2020) or pruning sub-topic under the general umbrella of model compression and acceleration (Sze et al., 2017; Cheng et al., 201 8a;b; Deng et al., 2020).
Pruning at initialization (Pal). Recent years have seen several new pruning paradigms. The most prominent one is pruning at initialization. Different from the conventional pruning, which prunes a pretrained model, Pal methods prune a randomly initialized model. Existing Pal approaches mainly include (Lee et al., 2019; 2020; Wang et al., 2020; Frankle et al., 2021; Ramanujan et al., 2020) and the series of lottery ticket hypothesis (Frankle & Carbin, 2019; Frankle al., 2020). Interested readers may refer to (Wang et al., 2021a) for comprehensive summary about Pal.
This topic is relevant to this work mainly because one Pal paper (Lee et al., 2020) also examines pruning using the tool of dynamical isometry. The similarity between our paper and theirs is that we both employ dynamical isometry as a tool to examine the property of network pruning. However, our paper is significantly different from theirs in many axes: (1) Basic setting. The most obvious difference is that we focus on pruning a pretrained model while (Lee et al., 2020) focuses on pruning at initialization (Pal). They are two different tracks in pruning (as such, Pal methods typically do not compare with the methods of pruning pretrained models) and the latter was shown to consistently underperform the former (Frankle et al., 2021; Wang et al., 2021a). (2) Motivation. Despite the same tool (mean JSV), (Lee et al., 2020) uses it to select unimportant weights to prune (i.e., for a new pruning criterion), while we use it to analyze why finetuning LR has a significant impact on final performance. The role of finetuning LR in pruning is not mentioned at all in their paper. (3) Proposed technical method. (Lee et al., 2020) focuses on unstructured pruning, while we focuses on structured pruning. This further leads to fundamental difference when designing the dynamical isometry recovery (DIR) methods- In (Lee et al., 2020), their proposed method is use iterative optimization for approximated isometry (due to the irregular sparsity); while in our case, since the pruned filers can be completely removed from the network, one of our DIR method (OrthP) has closed-form solution and can achieve exact isometry. (4) Finally, in terms of empirical results, (Lee et al., 2020) only conducts experiments on MNIST (LeCun et al., 1998) and CIFAR (Krizhevsky, 2009), while we have extensive results the large-scale ImageNet dataset (Deng et al., 2009).
2.1 EMPIRICAL STUDY: LARGER FINETUNING LR IS CRITICAL
As far as we know, mainly two papers question the value of inheriting weights from pretrained model: (Crowley et al., 2018: Liu et al., 2019). Both papers draw two similar conclusions. (1) Inheriting weights from a pretrained model in pruning has no value, i.e., training from scratch the small model can match (or outperform sometimes) the counterpart pruned from a big pretrained model. (2) Given the fact of (1), what really matters in pruning may lie in the pruned architecture instead of the inherited weight values. As such, both papers propose to view pruning as a form of neural architecture search (Zoph & Le, 2017; Elsken al., 2019). In this section, we first reexamine the empirical studies in (Crowley et al., 2018; Liu et al., 2019) to show that the "no value of inheriting weights" argument is actually inaccurate owing improper finetuning LR schedules.
Reexamination of (Liu et al., 2019). Before presenting results, here are some important comparison setting changes worth particular attention: (1) In (Liu et al., 2019), they compare training from scratch with six pruning methods (five structured pruning methods (Li et al., 2017; Luo et al., 2017; Liu et al., 2017; He al., 2017; Huang& & Wang, 2018) and one unstructured pruning method (Han et al., 2015)). Here, we only focus on the LI-norm pruning (Li et al., 2017) on ImageNet. The main reason is that, Ln-norm pruning is well-known a very basic filter pruning method. If we can show it outperforms training from scratch already, it will be no surprise to see other more advanced pruning methods also outperform training from scratch. In this sense, -norm pruning is the most representative method here for our investigation. (2) In (Liu et al., 2019), they have two variants for the number of epochs in scratch training, "Scratch-E"" and "Scratch-B". For the former, different small models are trained for fixed number of epochs; for the latter, smaller models for more epochs to maintain the same computation budget (Scratch-B was shown to be better than Scratch-E in (Liu et al., 2019)). Also, they decay LR only to 10-3 following the official PyTorch ImageNet example'. Here, we simply train all the networks for the same number of epochs but
trained
3

---

Under review as a conference paper at ICLR 2022
Table 1: Top-1 accuracy comparison different implementations of the Lı-norm pruning (Li al., 2017) on ImageNet. We adopt the torchvision models unpruned models for fair comparison. ResNet-34-A speedup: 1.18x. ResNet-34-B speedup: 1.32x. The results of (Li et al., 2017) and (Liu et al., 2019) are directly cited from their papers. The best cases in our training from scratch and pruning are randomly repeated for times (# indicates stddev) to prevent random variation.
Implementation
Unpruned (%)
Pruned model Scratch (%)
(%)
Finetuning LR schedule
ResNet-34-A
(Not reported)
72.56
epochs, initial 10 fixed
(Li et al., 2017)
73.23
ResNet-34-B
(Not reported)
72.17
epochs, initial 10 fixed
ResNet-34-A
73.03
72.56
epochs. initial
lixed
(Liu al., 2019)
73.31
ResNet-34-B
72.91
72.29
epochs, initial 10 fixed
72.91
20 epochs, initial 10 fixed
72.94
90 epochs, initial 10-3 fixed
Our rerun
73.31
ResNet-34-A
73.51
73.88
90 epochs, initial 10 3. decay
78.92+t.08
90 epochs. initial 10 10-- decay
72.50
20 epochs, initial 10 10 fixed
72.58
90 epochs, initial 10 fixed
rerun
73.31
ResNet-34-B 73.16
73.61
90 epochs, initial 10 decay
73.62
90 epochs, initial 10 decay
Table 2: Top-1 accuracy comparison between scratch training ("Scratch") and L1 norm pruning (Li et al., 2017) on ImageNet. "PR" means pruning ratio. "We adopt the official torchvision mod- els unpruned models. "Finetuned-1" and "Finetuned-2" refers two finetuning LR schedules ("Finetuned-1", :": 90 epochs, initial 10-3, decay decay 45/ (6) Best results in bold, second best underlined, each averaged by 3 random runs.
"Finetuned-2": 90 epochs, initial 10-2,
Network
Params reduc. (%) FLOPS reduc. (%)
Scratch (%)
Pruned-Finetunee (%) Prumed-Fintuned-2 (%)
0
0
69.76*
0.1
9.56
9.58
70.15
70.43
70.48
0.3
28.32
28,18
68.9010.10
69.29
69.5410.09
ResNet-18
0.5
47.03
46.20
67.03
67.36_n.os
67.71
65.99
64.93
64..21
63.72 Lo.03
64.45 10.04
84.75
83.52
56.70.0.17
53.49=0010
85.80
0.95
89.51
88.03
51.83
44.46_0.11
49.99
0
0
73.31
0.1
9.84
9.92
73.53+0.10
73.86
74.04
0.3
29.15
29.26
72.50+0.17
73.11 00.00
73.31 to.0s
ResNet-34
0.3
48.41
48.12
71.27 10.03
71.71
71.85
67.95
67.63
68.69
68.90
69.33+0.04
87.26
86.97
62.0810.12
60.34 ...0
62.2610.066
0.95
92.16
91.69
57.21 1 +0.15
52.83
56.69
0
70.371
0.1
9.19
17.78
68.51+0.01
71.45-0.07
71.74+0.04
0.3
26.80
47.63
66.60
70.00
70.53
VGGI1
43.86
70.56
65.8540.13
67.34_0.05
68.0110.10
60.54
87.03
61.5640.00
62.14=n.09
63.1440.08
0.9
76.50
96.49
48.34
45.11=0.07
48.5211006
0.95
80.49
97.76
35.4710009
33.50_..10
38.47 18.13
ensure the epochs are abundant (120 epochs) and decay LR to a very small amount (10-5). These two changes are to make sure the networks are trained to full convergence. As we will show, one primary cause possibly leading (Liu et al., 2019) to an inaccurate conclusion is exactly that the pruned networks are not fully converged (see Tab. 1).
With the LR schedule changes, we rerun the experiments using the released code of (Liu et al., 2019). Results are presented in Tab. 1. In the implementations of (Liu et al., 2019), the finetuned model is outperformed by the scratch training one, hence their "no value of inheriting weights" argument. We also reproduce their settings (the two rows of "20 epochs, initial 10-3 fixed" in "Our rerun") for confirming their argument. However, the finetuning LR schedule "20 epochs, initial 10 10-3, 3, fixed" is actually sub-optimal; the network is not fully converged. Using the proper ones ("90 epochs, initial 10-3 decay" "90 epochs, initial 10-2 decay"), pruning outperforms training from

---

Under review as a conference paper at ICLR 2022
scratch for both ResNet-34-A and ResNet-34-B. (We note the pruned models even outperform the original models. This is probably because pruning reduces the network redundancy, thus curbing overfitting. This phenomenon is also widely observed in past pruning works (Han et al., 2016; Wen et al., 2016; He et al., 2017) especially under small pruning ratios as in Tab. 1.)
Tab. only presents two ResNet models and their speedups actually quite small. To see if the finetuning LR effect still holds across the full spectrum of pruning ratios and on other types of networks, we vary the pruning ratios from 0.1 to 0.95 and include experiments on VGG1 L_BN (Si- monyan & Zisserman, 2015).
Results are presented in Tab. 2. With a more proper finetune LR scheme (column "Pruned-Fintuned- 2" vs. larger the pruning ratio, the more of the improvement. Now, comparing the results of "Pruned- Fintuned-2" to those of "Scratch", we can see pruning outperforms scratch-training in most cases. Exceptions appear on ResNet-34/18 under extreme pruning ratios (90% and 95%). Despite them, we believe it is fair to say inheriting weights has value given the fact that 17/20 experiments in Tabs. 1 and 2 show pruning better than training from scratch, especially under the prun- ing ratios of practical interest (i.e., non-extreme pruning ratios). Retrospectively, (Liu et al., 2019) concluded oppositely because they faithfully re-implemented the L1-norm pruming method just ac- cording the description in the original paper (Li et al., 2017): fixed LR 10 20 epochs, which turns out far from optimal as we know now.
I"),, the performance can be improved significantly, A clear pattern is, the
Reexamination of (Crowley et al., 2018). Coincidentally, (Crowley al., 2018) adopted very similar finetuning LR scheme to (Liu et al., 2019): They finetuned the pruned network with the lowest LR (8 10-4, close to 10-3 in (Liu al., 2019)) during scratch training and also fixed. Like the empirical study above, we reproduce the experiments of (Crowley et al., 2018) and rerun them with a larger initial LR 21 and decay it during finetuning.
Detailed results are deferred to the Appendix (Tab. 10) due to the limited length here. We summarize the observation here- Exactly the same as the case in (Liu et al., 2019), when the proper finetuning LR is used, pruning actually outperforms the best scratch training scheme consistently.
Up to now, the results above have shown that the "no value of inheriting weights" argument in previous works is largely attributed to sup-optimal finetuning settings. A larger LR (e.g.. 10 can significantly improve the finetuning performance than small one (e.g., 10-3) 3). In fact, we are not the only one to discover this. Previous works (Renda et al., 2020; Le & Hua, 2021) also reported similar observation. Nevertheless, they do not link the phenomenon with the "value of inheriting weights" argument and do not conduct systematical empirical studies as we do here. More importantly, neither of them presented theoretical explanations about its occurrence next, we are about to bridge this gap. We present faithful theoretical explanation through the lens of dynamical isometry.
DYNAMICAL ISOMETRY IS THE KEY
3.1 PREREQUISITE: DYNAMICAL ISOMETRY
Dynamical isometry (DI) is studied under the topic of trainability of deep neural networks. was first brought up in (Saxe et al., 2014). Specifically, the dynamical isometry is defined as the singular values of the Jacobian matrix being around (Saxe et al., 2014). is easy to see that the networks with dynamical isometry is easy to train, because JSVs around 1 imply the gradient signals will not be amplified or attenuated seriously during propagation, preventing the network from gradient exploding or vanishing, which are well-known the main difficulties in deep network training (Glorot & Bengio, 2010; Sutskever et al., 2013). For linear networks, dynamical isometry can be achieved exactly by the orthogonal initialization proposed in (Saxe et al., 2014); while for neural networks with non-linearit (like ReLU (Nair & Hinton, 2010) and convolution, can only be approximated up to date (see Tab. 3).
Mean Jacobian singular values. Since dynamical isometry is measured by the Jacobian singular values (JSV's), we adopt the mean of Jacobian singular values (denoted by 5) as scalar metric for analysis. Specifically, for a Jacobian J E RCC Du (C stands for the output dimension, i.e., the number of classes, Dm, for the input dimension), apply singular value decomposition (Trefethen &
5

---

Under review as a conference paper at ICLR 2022
Table 3: JSVs (Jacobian singular values) of orthog- onal initialization (Saxe et al., 2014) on different types of neural networks on MNIST dataset. Note, only the linear MLP network can achieve dynamical isometry exactly (i.e., all the JSVs equal to 1).
M..---Linesr MLP-7-RoLU LeNot 5 Linear LeNee-5-ReLU
Mcan JSV
2
Network
Mcan JSV
Max JSV
Min JSV
0.0
0.2
0.4
0.6
0.8
Pruming ratio
MLP-7-Linear 1.000020.000 0000
1.0000
1.0000
Figure 1: Mean JSV (Eq. (1)) of pruned net- works w.r.. different pruning ratios on the MNIST dataset. Note, with larger pruning ratio, mean JSV hurt more.
MLP-7-ReI.U 1.2268 10.5519
3.2772
0.2282
LeNet-5-Linear 0.9983 1998310.0044
1.2330
0.7896
LeNet-5-ReLU 20.3731
3.6007
0.6151
Bau III, 1997) to it,
V S
Σ Ei.
where Σ is the singular value matrix and K =>min(C. min Din).
Pruning as poor initialization. We investigate how the pruning affects S. The results are shown in Fig. 1. As seen, the mean JSV is consistently damaged by pruning: and larger pruning ratio, more decrease of the mean JSV. This means, pruning actually servers as a very poor initialization scheme for the subsequent finetuning.
In stark contrast to the broad awareness that initialization is rather critical to neural network train- ing (Glorot & Bengio, 2010; Sutskever et al., 2013; Mishkin & Matas, 2016; Krähenbüühl et al., 2015; He et al., 2015), the initialization role of pruning has received negligible research attention, however. As far
know, no prior works have noted this issue when pruning a pretrained network or tried to recover the broken dynamical isometry before finetuning.
Simply put, our goal next to show it is this broken dynamical isometry that answers for the performance gap between LR 10 and 10 If dynamical isometry is fully recovered before finetuning, LR 10-2 vs. 10-3 should not cause significant performance difference anymore. To validate this explanation, we need method to fully recover dynamical isometry in filter pruning, as introduced next.
3.2 DYNAMICAL ISOMETRY RECOVERY IN FILTER PRUNING
In (Saxe et al., 2014), they propose weight orthogonalization scheme to achieve dynamical isom- etry for neural network initialization, namely, the initial weights are randomly sampled. Different from their case, here the initial weights are inherited from pretrained model by pruning. Therefore, we need to adapt it to our application.
For fully-connected layer parameterized by matrix Wo E J (for convolutional layer param- etcrized by a 4-d tensor of shape N XC*xXX it can be reshaped to matrix shape g xCHW) ) it reduces to matrix W of size R. XM,, (J1 S J, Ki K) after structured pruning. Then, apply the weight orthogonalization technique (Mezzadri, 2006) based on QR decomposition (Trefethen & Bau III, 1997) to W,
Q, R qrd(W); W = Q ( sign(diag/ (R))
where qrd(-) means QR decomposition; Q an orthogonal matrix of the same size as W (RR-i R is upper triangular matrix of size KKiK,, sign(-) is the sign function which returns the positive or negative sign of its argument; O represents the Hadamard (clement-wise) product aligned to the last axis (since and sign(diag(R)) share the same dimension the last axis).
As an orthogonalized version of W. W* recovers the dynamical isometry damaged by pruning. Therefore, propose to employ W instead of the original W as the initialization weights for later finetuning. We dub this weight orthogonalization method for pruned models as OrthP. With this method, fully recover broken dynamical isometry and continue analysis follows.

---

Under review as a conference paper at ICLR 2022
Table 4: Mean JSV of the first 10 epochs under different finetuning settings. Epoch 0 refers to the model just pruned, before any finetuning. Pruning ratio is 0.9. Note, with OrthP, the mean JSV is because OrthP can achieve exact isometry.
Epoch
2
5
10
LR=10-2 3, w/o OrthP 0.0004
0.6557
0.8946
1.0191
0,9826
1.0965
1.1253
1,2595
1.3298
1.2940
1.4238
LR=10 3, w/ OrthP
0.0004
0,0004
0.0006
0.0014
0.1103
0.2765
0.3501
0,4320
0.5167
0.7478
0.8501
LR=10 w/ OrthP
1.0000
1.2318
1.4144
1.4277
1.4017
1,4709
1.5171
1.5551
1.6082
1.6538
1.6648
LR=10 3, w/ OrthP
1.0000
1.5135
1.6630
1.7449
1.8250
1.8720
1.9193
1.9556
1.9943
2.0084
2.0409
Table 5: Test accuracy (%) of the first 10 epochs corresponding Tab. 4 under different finetuning settings. Epoch O refers to the model just pruned, before any finetuning. Pruning ratio is 0.9.
Epoch
3
4
5
6
10
LR=10 2, w/o OrthP
9.74
63.86
79.96
79.74
80.06
85.79
85,82
86.11
86.45
86.53
85,95
IR=10 3. w/o OrthP
9.74
9.74
9.74
12.09
21.74
27.95
33.55
35.92
49.19
65.50
69.90
IR=10 w/ OrthP
9.74
91.05
91.39
91.33
91.37
91.74
91.69
90.74
91.39
91.58
91.44
LR=10-4 33 w/ OrthP
9.74
90.81
91.59
91.77
91.85
92.04
92.12
92.22
92.12
92.33
92.25
3.3 ANALYSIS WITH MLP-7-LINEAR ON MNIST
Evaluated network. The network for analysis is a 7-1ayer linear MLP. We are aware that this toy network has little practical meaning, but it is very appropriate here for two reasons. First, as mentioned above, in our analysis we need method to recover DI exactly. Up to date, this can only be achieved on linear networks (see Tab. 3). Second, the linear MLP network is free from the intervention of modern CNN features (e.g. BN (loffe & Szegedy, 2015), residual (He et al., 2016).. By our observation, these features will make the problem complex and prevent us from seeing consistent results at the early analysis stage.
LR schedule setup. When we set different LR schedules with different initial LRs, we will (1) keep the total number of epochs the same, (2) keep the last LR the same, (3) intentionally use prolonged training epochs (like 90 or even 900 epochs on MNIST dataset. Typically, it only needs 30 epochs to reach convergence MNIST). All of these to ensure the networks are fully converged, helping us render a faithful conclusion. Step decay LR schedule is employed given its broad use.
Proposed explanation and its deducted hypotheses. We list the first 10-epoch mean JSV and test accuracy of pruning MI.P-7-Linear (at pruning ratio 0.9) under different finetuning setups in Tab. and Tab. 5, respectively. The following observations are worth our attention.
In Tab. 4, one important fact is that, the mean JSV can recover itself without any extra help during finetuning, regardless of different setups.
In Tab. 4, without OrthP, the mean JSV of 10-2 arises much fach,aattrr than that LR 10 By our analysis in the Appendix (Sec. D). larger mean JSV implies better dynamical isometry, which further implies easier optimization. Easier optimization finally leads to the better test accuracy of LR 10 against 10 3 when the training epoch is insufficient, as
shown in Tab. 5.
In Tab. 4, with OrthP, the broken dynamical isometry is exactly recovered (note at epoch O, the mean JSV is 1). Then the accuracy advantage of LR 10-2 over 10-3 is not significant anymore, e.g., in Tab. 5, at epoch 10, LR 10-2 is better than LR 10-3 by 16.05% without OrthP: while it is not better (actually worse) than LR 10-3 with OrthP.
Particularly note how the mean JSV trend in Tab, 4 correlates well with the test accuracy trend in Tab. 5. This promotes the plausible idea that it is dynamical isometry that answers for the accuracy gap fundamentally.
These observations inspire us to the following plausible explanation about how a larger finetuning LR can improve performance significantly:
larger finetuning LR helps the network update faster, thus the dynamical isometry (measured by mean JSV) recovers faster (and possibly better), which further leads to faster (and possibly better) optimization. For deep networks nowadays, better optimization typically implies better generaliza- tion, thus the larger finetuning LR eventually leads to the better test accuracy as we see.
7

---

Under review as a conference paper at ICLR 2022
Table 6: Summary of finetuning LR setups corresponding to the proposed hypotheses in Sec. 3.3.
Initial LR 10
Initial LR 10
For Hypothesis
epochs, Initial 10-2, decay 30/60
90 epochs, Initial 10-3, decay 45
For Hypothesis
900 epochs, Initial 10 10-2, decay 300/600
900 epochs, Initial 10- decay 450
For Hypothesis OrthP, 90 epochs, Initial 10-?, decay 30/60
OrthP, 90 epochs, Initial 10 10-3, decay 45
For Hypothesis
OrthP, 900 epochs, Initial 10 decay 300/600
OrthP, 900 epochs, Initial 10- decay 450
That is, a larger finetuning L.R shows performance advantage only if dynamical isometry is broken by pruning first. dynamical isometry is fully recovered before finetuning, a larger LR should not pose performance advantage anymore. The validation of this explanation can be specified into the following deducted hypotheses:
Hypothesis Given small number of epochs, mean JSV cannot be fully recovered by training, then the larger LR should show significant advantage over the smaller LR.
Hypothesis 2: With sufficient training epochs, mean JSV can be fully recovered by train- ing. Then the larger LR should have less advantage over the smaller LR.
we employ OrthP to exactly recover the mean JSV, given the small number of epochs again, the larger LR should have much less advantage now.
Hypothesis 3:
Hypothesis 4: If combine abundant epochs with OrthP, mean JSV will be recovered even completely, then the performance advantage of the larger LR over the smaller one should be even weaker.
Corresponding these four hypotheses, the eight finetuning LR settings are summarized in Tab. 15. The unpruned MLP model is trained with LR schedule "90 epochs, initial 10-2, decay 30/60". For filter pruning, we employ L1 Lı-norm pruning (Li et al., 2017) throughout this paper. Specifically, it sorts the neurons (or filters) by their L1 norms in ascending order and prunes those with the least norms by a predefined pruning ratio
The final accuracy results are shown in Tab. 7. We first analyze the results of pruning ratio 0.8 in Tab. 7. As seen, when finetuned for 90 epochs, LR 10-2 shows an advantage over LR 10-3 by 0.82% accuracy. It is tempting to draw conclusion based on this comparison that LR 10-2 is much better than LR 10-5 However, this is not the whole story:
With 900 epochs, LR 10 is greatly surpassed by LR 10-3 (91.64 vs. 92.54). The reason by our proposed explanation is that, with abundant epochs, the dynamical isometry can be recovered more completely, hence LR 10-2 does show advantages anymore over 10-3
When OrthP applied, LR 10 2 does not show significant advantages either, similar to the effect of increasing the number of training epochs. This is because that finetuning shares the same role of recovering dynamical isometry with OrthP. Just OrthP more effective since it is analytically targeting exact dynamical isometry.
When the best setting used (OrthP 900 epochs), LR 10-3 is slightly hetter than 10-2 Comparing "OrthP, 900 epochs" with "OrthP, 90 epochs", the gains are only marginal. This is because the dynamical isometry has already been fully recovered by OrthP, thus more training epochs do not show much value anymore.
different pruning ratio 0.9 also explored. Its results are line with those of pruning ratio 0.8 as shown in Tab. 7. In short, these empirical observations are fully in line with our expectations, justifying the validity of the proposed explanation.
Explaining LR effect on ImageNet. In Tab. 2. there is apparent trend that the larger pruning ratio, the more performance advantage LR 10-2 over 10-3 (this also appears on CIFARIO dataset, see Tab. 8). Using explanation, this phenomenon can also be explained now When the pruning ratio is greater, more dynamical isometry is damaged. LR 10-2 can find more use in these cases since it is faster/better to recover dynamical isometry, hence the more pronounced advantage.
closer look and more lessons. In the ResNet34 results above (Tab. 1) reported by (Li et al., 2017) and (Liu et al., 2019)), it is still easy to notice their finetuning L.R schedule may be sub-optimal given
8

---

Under review as a conference paper at ICLR 2022
Table Test accuracies (%) of the 4 hypotheses in Tab. 15 with MLP-7-Linear network on MNIST. Accuracy of unpruned model: 92.77%. Each setting is randomly run 5 times, mean accuracy and stddev reported. "Acc. gain" refers to the mean accuracy improvement of LR 10-2 over 10-3 Hyper-parameters: batch size 100, SGD optimizer, weight decay 0.0001, LR schedule: initial LR 0.01, decayed at epoch and 60 by factor 0.1, total epochs: 90. The LR schedule used for both scratch training and finetuning.
Pruning ratio 80%
Pruning ratio 90%
Finetuning setting
I.R 10-3
LR 10-4
gain
I.R 10
LR 10-2
Acc. gain
90 epochs
90.54.0.00
91.36
0.82
27.50
87.81
0.22
900 epochs
o2.54
10.44
0.90
90.44 10
87.83
2.61
OrthP, 90 epochs
92.77
92. 79 10.03
0.02
92.72.0..3
92.77 10.04
0.05
OrthP, 900 epochs
92.84.0.03
92.81
-0.03
92.86
92.79
-0.07
Scratch (Kaiming uniform)
92. 60.0.4
01. 48 11.48.0.22
Scratch (Orthogonal init)
92. 7620.0
92.76g0.01
their finetuning process (20 epochs) is obviously shorter than training from scratch (90 epochs). For Tab. 7, quite differently, the "90 epochs" finetuning setting appears nothing wrong (considering the unpruned network is trained for 90 epochs. finetuning for 90 epochs is not short; besides, the LR is also decayed to very small amount). However, the results of this setting actually lead us to partial conclusion that the larger LR is better than the smaller LR. This particular example shows how misleading the comparison results can be, even though they look perfectly fair, if we are not aware of the effect of dynamical isometry in structured pruning.
4 DYNAMICAL ISOMETRY RECOVERY (DIR)
By our analysis above, dynamical isometry recovery before finetuning is rather important. An effec- tive dynamical isometry recovery method is supposed to (1) close up the performance gap between finetuning LR 10-2 and 10-3 3,-3, (2) improve the final performance (otherwise, there is no point using it at all). As shown in Tab. 7, the proposed OrthP method for linear MLP networks can meet these two requirements; and since recovers DI exactly, it actually completely closes the performance gap between finetuning LR 10-2 2 and 10-3, Unfortunately, this method does not generalize to more practical non-linear convolutional neural networks, as shown Tab. 8, where using OrthP degrades the final performance. As remedy, in this section introduce very simple regularization-baat method that can work on practical CNNs.
Strong L2 regularization as drop-in remedy. In typical structured pruning algorithm, unim- portant filters are selected by some criterion. Then they are removed (zeroed out), followed by a finetuning process. We propose to apply super strong L2 regularization (e.g.. regularization factor equals to 1) to push the unimportant filters to rather close to zero first, before permanently removing them. This simple technique can readily work drop-in step into any structured pruning algorithm and it is very easy to implement on any deep learning framework.
In Appendix (Sec. E), we provide more explanation regarding how StrongReg is related to dynamical isometry.
Results in Tab. demonstrate the effectiveness of this simple technique on ResNet56. It improves the final pruning performance and also shrinks the performance gap between finetuning LR 10 and 10 (especially when the pruning ratio large), meeting the two requirements above. We are aware that this technique is preceded by several variants in the past literature (e.g., GReg-1 in (Wang et al., 2021b)), but they do not link their method with dynamical isometry recovery as we do here. Also, our technique is even simpler than theirs in (Wang et al., 2021b), To regularization is increased gradually to a very strong level, while here we use a fixed strong regularization, which probably is the simplest form of utilizing strong regularization.
RECTIFIED ARGUMENT ON VALUE OF PRUNING
When addressing concerns from the reviewers, we find a phenomenon against our "pruning has value" argument. Specifically, in Tab. 7, we provide the scratch training results with two random
9

---

Under review as a conference paper at ICLR 2022
Table 8:
Test accuracies (%) of pruning ResNet56 on CIFARIO. Unpruned accuracy: 93.78%. Each
setting is randomly run 3 times. "Acc. gain" means the accuracy gain of initial LR 10-2 over 10-3
Pruning ratio Sparstyy/peeedup
0.5
0.7
0.9
0.95
49.82%
70.57%
90.39%
95.19%/19
Finetuning LR schedule: 120 epochs, initial 10 ?, decay 60/90
Train from scratch
92.78
92. 1
28.36.0.20
84.60 14
L1 (Li et al., 2017)
=0.0r
92.26
88.71
84.63
L1 OrthP StrongReg
93.36
91.96
86.01
82.62=0.05
93.55 20.66
92.38
89.2440.16
85.90.10.19
Finetuning LR schedule: 120 epochs, initial 10
decay 80
L1 (Li et al., 2017)
93.12.0.10
91.77
87.57 20.09
83.10 12
StrongReg
93.44_0.06
91.96
88.69
85.45
Acc. gain. Acc.gain. (L1)
0.39
0.49
1.14
1.53
Acc. gain. (StrongReg)
0.11
0.42
0.55
0.45
initialization schemes, kaiming uniform and orthogonal initialization. The former is the default PyTorch initialization scheme for convolutional and linear layers?; the latter is proposed by Saxe et al. (2014) which can achieve exact isometry. The problem is, the best pruning results in Tab. 7 using OrthP are generally on par with the scratch training results using orthogonal initialization. Namely, pruning is shown no value therein. How should we respond to this observation against our prior claim and how does it affect our conclusions?
We present another set of results - We use MLP-7-ReLU as evaluation network instead of MLP-7- Linear above. Clearly, here we want to see if the non-linearity can make any difference. Results are shown in Tab. 9, where the dynamical isometry recovery method is changed from OrthP to StrongReg because OrthP is proposed for linear MLP networks while here the MLP is non-linear.
network Table 9: Test accuracies (%) of -norm pruning with MLP-7-ReLU network on MNIST. Accuracy
of unpruned model: 98. 16%. Each setting is randomly run 5 times, mean accuracy and stddev reported. Hyper-parameters: batch size 100, SGD optimizer, weight decay 0.0001, LR schedule: initial LR 0.01, decayed at epoch 30 and 60 by factor 0.1, total epochs: 90. The LR schedule used for both scratch training and finetuning.
Finetuning setting
Pruning ratio 80%
Pruning ratio 90%
LR 10 2, 90 epochs
96.75
94. 94.76g0.15
LR 10-2, StrongReg, 90 epochs Scratch (Kaiming uniform)
96.98 04
95. 95.1040.18
96.60 1-011
94..64
Scratch (Orthogonal init)
96.52+0.17
92.56
As seen, now, either using kaiming uniform or orthogonal initialization, pruning is consistently better than scratch training by a fair margin. The advantage is amplified when using StrongReg.
Then, worthy question here is: Why on the MLP-7-Linear, pruning is not better than orthog- onal initialization, while on MLP-7-ReLU, is?
This is actually straightforward to see if we see pruning as kind of initialization. As mentioned above, on linear MLP networks, orthogonal initialization is proven to be the optimal in the sense of exact isometry. That is, no other initialization can be better. Pruning is essentially also a kind of initialization for the subsequent fine-tuning, so not surprisingly, it cannot beat the (optimal) or- thogonal initialization, hence no value. However, when it comes to the MLP-7-ReLU network, orthogonal initialization is no longer optimal (as mentioned, there has been no method up to date that can achieve exact isometry for non-linear networks). Then, it is likely that pruning provides better initialization weights than orthogonal initialization or kaiming uniform. It just turns out that pruning really achieves this, and archives more with the help of StrongReg.
There are at least 3 key takeaways from the above toy but inspiring comparisons:
10

---

Under review as a conference paper at ICLR 2022
First, the comparison above shows that whether pruning has value seriously hinges on the network type, initialization scheme, and pruning ratios (and maybe more). Looking at pruning kind of initialization we do in this paper (and examine it through the lens of dynamical isometry) is actually pretty good perspective, from which many results that appear inconsistent random start to be explainable and logically consistent.
Second, pruning shows the value in the non-linear case, but no value in the linear case. On the whole, we believe it is fair to say pruning has value considering the non-linear case is more practical.
Third, more profoundly, the above results actually suggest, if dynamical isometry fully recovered, pruning will (probably) have no value indeed because it cannot beat the initial- ization scheme that can achieve exact isometry. Acquired with this knowledge, we update our prior "pruning has value" argument to more rigorous one: "pruning has the poten- tial be valuable if the random initialization of scratch training cannot achieve exact isometry". Practically, up to date, for non-linear networks (not to mention BN, residuals, convolution), there has been no such method that can achieve exact isometry. Thus, it is still likely for pruning to be valuable at present.
6 CONCLUSION
In this work, we present extensive empirical evidences to show the 'no value of inheriting weights" argument in prior works is inaccurate because of improper finetuning LR schedules. We further tap into dynamical isometry to explain why the finetuning LR has such a great impact on the final performance, through carefully designed control experiments with hypotheses. We show whether pruning has value seriously depends on the context (e.g., network type, random initialization scheme, pruning ratio). Looking at pruning as kind of initialization is a favorable perspective that can make seemingly inconsistent and random results become predictable and coherent. For practically amend- ing the broken dynamical isometry, we also present a rather simple regularization based technique that works effectively residual convolutional networks.
The finding of dynamical isometry in structured pruning in this paper justifies the value of inheriting weights, in line with the past research wisdom and our common beliefs. It also helps towards a better understanding of pruning and possibly can inspire more advanced pruning algorithms as dynamical isometry recovery has been shown worthy direction in the paper. In addition, the awareness of dynamical isometry in structured pruning can help us render more faithful conclusion when comparing different pruning methods.
REFERENCES
Davis Blalock, Jose Javier Gonzalez, Jonathan Frankle, and John V Guttag. What is the state of neural network pruning? In SysML, 2020.
Jian Cheng, Pei-song Wang, Gang Li, Qing-hao Hu, and Han-qing Lu. Recent advances in effi- cient computation of deep convolutional neural networks. Frontiers of Information Technology & Electronic Engineering, 19(1), 19(1):64-77, 2018a. 3
Yu Cheng. Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35 (1):126-136, 2018b. 3
Elliot J Crowley, Jack Turner, Amos Storkey, and Michael O'Boyle. A closer look at structured pruning for neural network compression. arXiv preprint arXiv:1810.04622 2018 1, 3, 14
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009. 1,3
Imagenet: A large-scale
Lei Deng, Guogi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4); 485-532, 2020.
11

---

Under review as a conference paper at ICLR 2022
Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. In AAAI, 2018. 2
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
JMLR, 20(55):1-22 2019. 3
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In ICML, 2020. 3
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? In ICLR, 2021. 3
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint
2019. 3
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010. 5,
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. NeurlPS, 2015. 3.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. In ICLR, 2016. 2, 5
B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NeurIPS, 1993. 2
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 1.7
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR, 2015.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017. 3,
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In ECCV, 2018.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Peta-dependent initializations of convolutional neural networks. arXiv preprint (tl::1511.668555 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 3
Duong H Le and Binh-Son Hua. Network pruning that matters: A case study on retraining variants. In ICLR, 2021. 1,5
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In NeurIPS, 1990. 1,2
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 ():22))22222222 1998. 3
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In /CLR, 2019.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. In ICLR, 2020. 3, 15
12

---

Under review as a conference paper at ICLR 2022
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017. 2, 8, 10, 14
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017. 3
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019. 2, 5, 8, 14, 15
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, 2017.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William Dally. Exploring the granularity of sparsity in convolutional neural networks. In CVPR Workshop, 2017. 2
Francesco Mezzadri. How to generate random matrices from the classical compact groups. arXiv
preprint math-ph/06090500, 2006.
Dmytro Mishkin and Jiri Matas. All you need good init. In ICLR, 2016.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from
network via relevance assessment. In NeurlPS, 1989.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's 's hidden in a randomly weighted neural network? In CVPR, 2020. 3
R. Reed. Pruning algorithms survey. IEEE Transactions on Neural Networks, 4(5):740-747, 1993. 3,3
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020. 1,5
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. In ICLR, 2014. 5, 6, 10,
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. In NeurIPS, 2020. 2
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In ICML, 2013. 5, 6
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: tutorial and survey. Proceedings of the IEEE, 105(12):229-2222) 2017. 1,3
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: (0l1:1180000777,, 2018. 2, 14
Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997. 5,
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning
in the kronecker- eigenbasis. JCML, 2019a.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020. 3
Huan Wang, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruning for efficient convnets via incremental regularization. In IJCNN, 2019b. 2
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Emerging paradigms of neural network pruning. arXiv preprint arXiv:2 064606 2021a.
13

---

Under review as a conference paper at ICLR 2022
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In ICLR, 2021b. 2,
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. NeurlPS, 2016. 2,
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
A RESULTS OF REEXAMINATION OF (CROWLEY ET AL., 2018)
Table 10: Test accuracy comparison between 2 pruning schemes and 4 scratch training schemes in (Crowley et al.. 2018). Network: WRN-40-2 (unpruned accuracy: 95.08%, params: 2.24M). Dataset: CIFAR-10. Results above the dashline are directly cited from (Crowley et al., 201 8); results below the dashline are from our reproducing (with the official code of (Crowley et al., 2018) at http://gittub.ooo/B.at wtw///aasaaaahh//yttrrreooo Wwtch/pytorch--pppee prrnnes for fair comparison). "Rerun" means rerun the code of (Crowley et al., 201 8) as is. "L.R 10-2" means we redo the finetuing for the pruned models in "Rerun" using our finetuning LR schedule (120 epochs, initial 10-2, decay 60/90). Finetuning is randomly repeated for 3 times, mean (stddev) accuracies reported. The main point here is that (Crowley al., 2018 1) draws the conclusion that scratch training is better than pruning because of an improper finetuning LR scheme. With the proper finetuning LR scheme, pruning is actually better than scratch training.
Method
Seratch?
500K params budget
params budget
1.5M params hudget
Params (M)
Acc. (%)
Params (M)
Acc. (%)
Params (M)
Acc. (%)
L Zy-morm pruning (Li et al., 2017)
x
0.51
90.86
1.02
92.61
1.52
93.63
Fisher pruning (Theis 2018)
0.52
92.59
1.02
1.52
94.51
Varying Depth
0.69
93.56
1.08
94.54
1.47
94.64
Varying Width
1.48
94.66
Varying Bottlenock
0.50
93.69
1.00
94,40
1.49
94.79
Fisher Scratch
0.52
93.72
1.02
94.65
1.52
94.86
L--horm proning (LT ct al., (
0.30
91.23
1.00
92.80
1:51
93.52
( norn pruning (1.i et al., 2017) (I. 10-3)
X
0.50
93.88 (0.10)
1.00
94.49 (0. (0.10)
1.51
94.92 (0.16)
Fisher pruning (Theis et al., 2018) (Rerun) Fisher pruning (Theis
0.52
92.17
93.57
1.48
94.67
2018) (1.R 10 10-
0.52
94.27 (0.09)
0.98
94.80 (0.02)
1.48
95,10 (0.13)
B TRAINING SETTING SUMMARY
There are three datasets in our experiments the paper: MNIST, CIFARIO, and ImageNet. Apart from some key settings stated in the paper, more detailed training setting summary is shown as Tab. 11. We use NVIDIA V100 GPUs for all our experiments.
Table 11: Training setting summary in finetuning. For the SGD solver, in the parentheses are the momentum and weight decay.
Dataset
MNIST
CIFARIO
ImageNet
Solver
SGD (0.9, Ie-4)
SGD (0.9, 5e-4)
SGD (0.9, le-4)
I.R schedule (Initial le-2)
Multi-step (decay 30/60)
Multi-step (decay 60/90)
Multi-step (decay 30/60/75
LR schedule (Initial lc-3)
Multi-step (decay 45)
Multi-step (decay 80)
Multi-step (decay 45/68)
Total epoch
90
120
90
Batch size
100
128 Random crop and horizontal tip Random crop and horizontal tlip
256
Data augmentation
None
A typical finetuning LR schedule looks like this in our paper: "90 epochs, initial 10 2 decay 30/60". Its meaning is: the total number of training epochs is 90; initial LR is 10-2; at epoch 30 and 60, LR multiplied by default factor 0.1. The others can be inferred likewise.
C WHY OUR REPRODUCED RESULTS ARE BETTER THAN (LIU ET AL., 2019)
In Tab. 1, we present the results of ResNet34 on ImageNet, pruned by the Lı-norm pruning method (Li al., 2017). Readers may be curious that why scratch-training results much
14

---

Under review as a conference paper at ICLR 2022
5.0
Moxs NV
Mon RSV
Mam ISV
Meus 28V
5.0
00000
2.5
60000
Im1 --001
2.5
01
1.0001
25
50
75
250
500
750
Ted occomnc
Toe suzumary
90
8
92.5
52.5
MMy
1-0001
1-6001
3-1 00)
0.1
I-001
90.0
90.0
1-101
50 Epuchs
75
250
500 Epoch
750
50
75
250
500
750
Epoch
Epoch
(a) 90 epochs
(b) 900 epochs
(c) OrthP, 90 epochs
(d) OrthP, 900 epochs
Figure 2: Mean JSV and test accuracy during finetuning with different setups. Note that with OrthP (c, d), mean JSV recovers faster; so does the test accuracy. The pruning ratio in this case is 0.9.
(This figure is best viewed in color).
higher than those of Rethinking (Liu et al., 2019) (by 0.4-0.6%). This has a historical reason, (Liu et al., 2019)' refers to the official PyTorch ImageNet example', where the LR is only decayed twice (from 1e-1 to le-2, then to le-3). However, by our empirical observation, le-3 is still not the lowest LR that the network converges. If the LR is decayed another time (to le-4), the top-1 accuracy can still bump by round 0.5-0.8% point, which is a significant improvement for ImageNet thus not negligible. Therefore, we choose to decay the LR further to le-4 and finally to le-5 to ensure the network is fully converged. This reiterates the comparison rule of our paper: comparing different methods in their best shape. Comparing scratch training to L₃ noom pruning before the finetumed network finally converges may appear fair (given the same number of epochs) but of less practical meaning and may hide the true picture.
D How TO Properly LOOK AT THE MEAN JSV METRIC FOR DYNAMICAL ISOMETRY
In the main paper, we use mean JSV as the metric measure dynamical isometry, for the following 3 specific reasons: (1) It was used by Lee et al. (2020), which analyzes the dynamical isometry for randomly initialized network. Given its success there, it very natural for us to also employ this metric for analyzing pretrained networks here. (2) We currently do not have better alternative, either. (3) In practice, we find mean JSV is informative (e.g. in Tab. 7, the mean JSV trend is well-correlated with the test accuracy trend), as long as we see it properly.
This section is meant to provide more background regarding how to look at the mean JSV for dy- namical isometry property. is not new invention of our paper but general practical guideline about the relationship between mean JSV and dynamical isometry in order to help readers better understand our paper.
DI (dynamical isometry) is defined by mean JSV close to in Saxe et al. (2014). Rigorously, in Saxe et al. (2014), DI describes the distribution of all JSVs. Mean JSV is only average sketch of the distribution. Nevertheless, this average approximation is accurate enough for analysis. In other words, if a network has mean JSV close to I, we can say this network has dynamical isometry.
Then, a non-trivial technical question is: When we deal with practical DNNs in the real world, how close is the so-called "close to 1 To our best knowledge, there is no outstanding theory to quantify this, so we resort to empirical analysis, specifically on the MLP-7-Linear network used in the main paper.
As seen, there is a clear trend in Tab. 12: larger pruning ratio, smaller mean JSV, and lower test accuracy (either before or after finetuning).
Particularly note the mean JSV range where the pruned network can be finetuned back to the original accuracy (92.77%), which is >= 0.0151. This means, for networks with mean ISV greater than
15

---

Under review as a conference paper at ICLR 2022
Table 12: Mean JSV and test accuracies (%) of MLP-7-Linear on MNIST under different pruning ratios. Each result is randomly run for 3 times. We report the mean accuracy and (std). "ft." is short for finetuning.
Pruning ratio
0.1
0.2
0.6
0.7
mean JSV
2.4987
1,7132
0.9993
0.5325
0
0,1180
0,0452
0.0151
0.0040
0.0004
Acc. before n.. 92.77
91.35
78.88
62.21
32.14
11.47
9.74
9.74
9.74
9,74
Acc. after ft. /
92.82 (0.05)
92.80
(0.04) 92.80 (0.01) 92.77 (0.01)
92.77 (0.02)
92.77 (0.00)
92.78 (0.02)
91.37 (0.03)
87,82 (0.03)
0.0151, in spite that their immediate accuracies (without finetuning) can be distinct (e.g., 91.35% at PR 0.1 vs. 9.74% at PR 0.7), intrinsically, they are equivalently potential after finetuning.
DI theory suggests that mean JSV equal to I is the best case. Then would ask, how about mean ISV equal to 2? 1.57 0.8? Are they really worse than the ideal value 1? Tab. 12 above shows not necessarily, because a network with mean JSV 0.2711 can reach comparable accuracy to the network with mean JSV 1. Only when the mean JSV is smaller than some threshold (in this case, 0.0151), it leads to an irrecoverable damage to the network optimization, which eventually results in lower generalization ability (i.e., test accuracy).
By this "trainable with equivalent potential" rule, if mean JSV lies in the range of >= 0.0151, we can regard it as "close to 1 because they can do just as well as 1.
Here, we only mention the lower bound, how about the upper bound? Can mean JSV 10, 000 also be regarded as "close to 1"? This is a good question worth further dedicated investigation. Yet, for now, we do not need to worry much about it, because in practice, a normally trained network rarely presents very large mean JSV by our empirical observation, but is quite likely to have very small mean JSV (0.0004 at pruning ratio 0.9 in Tab. 12 a concrete example).
E How STRONGREG IS RELATED TO DYNAMICAL ISOMETRY
The intuition behind StrongReg. Dynamical isometry describes nice state of the network that signals can propagate through it without serious magnitude explosion or attenuation. Since the weights in network are dependent on each other, removing some of them will definitely hurt the isometry because it is based upon all the weights. When we use a strong regularization to push these unimportant parameters zero, it explicitly makes the other parameters learn to not rely on them, that is, encouraging the gradients not to pass through these weights/neurons because they are going to be cleaned out. This way, when the unimportant parameters are physically removed, it will incur much less damage to the left parameters, thus maintaining the network dynamical isometry well.
StrongReg works by improving dynamical isometry. Here we apply StrongReg to pruning the MLP-7-Linear network at pruning ratio 0.8 and 0.9.
Table 13: Test accuracies (%) of applying StgongReg to pruning MLP-7-Linear network on MNIST.
Accuracy of unpruned model: 92.77%. Each setting is randomly run 5 times, mean accuracy and stddev reported. "Acc. gain" refers to the mean accuracy improvement of LR 10 2 over 10 3 Hapae-parammttrs: batch size 100, SGD optimizer, weight decay 0.0001, LR schedule: initial LR 0.01, decayed at epoch 30 and 60 by factor 0.1, total epochs: 90. The LR schedule is used for both scratch training and finetuning.
Pruning ratio 80%
Pruning ratio 90%
Finetuning setting
LR 10-3
LR 10-2
Acc. gain
LR 10-3
LR 10-2
Acc. gain
90 epochs OrthP, 90 epochs StrongReg, 90 epochs
90.54 10.02
91.36 91.0610.02
0.002
0.82
87.5 87.59
87.81 87.81.1..0
0.22
02.777.0.03
92.79...03 1..03
0.02
92.72.0.03
92.77
0.05
92.80
92.801o.02
0.00
92.48
92.52.1..44
0.04
As seen in Tab. 13, (1) similar to OrthP, StrongReg can rectify the test accuracy from the under- rated ones to 92.48 92.80, close to the best possible performance (around 92.77); (2) StrongReg can close the performance gap between LR 10-3 and 10-?, just like OrthP. In short, in terms of performance, StrongReg behaves very similarly to OrthP.
16

---

Under review as a conference paper at ICLR 2022
To further show StrongReg really works by maintaining dynamical isometry, like Tab, 4 and Tab. above, we list the first 10-epoch mean JSV and test accuracies (at PR 0.9) and get the following summarized results:
Table Mean JSV of the first 10 epochs under different finetuning settings. Epoch refers to the model just pruned, before any finetuning. Pruning ratio is 0.9. Note, with OrthP, the mean JSV is because OrthP can achieve exact isometry.
Epoch
10
TRR=10-2 w/o OrthP
0.0004
0.6557
0.8946
1.0191
0.9826
1.0965
1.1253
1,2595
1.3298
1,2940
1,4238
LR=10-3 w/o OrthP
0.0004
0.0004
0.0006
0.0014
0.1103
0.2765
0.3501
0,4320
0.5167
0.7478
0.8501
LR=10 OrthP
1.0000
1.2318
1,4144
1.4277
1.417
1.4709
1.5171
1.5551
1.6082
1.6538
1.6648
LR=10 OrthP
1.0000
1.5135
1.6630
1.7449
1.8250
1.8720
1.199
1,9566
1.9943
2.0084
2.0409
LR=10 2 SerongReg
3.0275
2.1538
2.0390
1.0191
1.9696
2.1011
1.9344
2,0810
2.0541
2.0563
2.0581
I.R=10 3 SerongReg
3.0275
2.9691
2.9514
2.9921
2.9869
3.0101
3.0448
3.0516
3.0555
3.0441
3.0244
Table 15: Test accuracy (%) of the first 10 epochs corresponding to Tab. 14 under different finetuning settings. Epoch 0 refers to the model just pruned, before any finetuning. Pruning ratio 0.9.
Epoch
4
8
9
10
LR=10 w/o OrthP
9.74
63.86
79.96
79.74
80.06
85.79
85.82
86.11
86.45
86.53
85.95
LR=10 w/o OrthP
9.74
9.74
9.74
12.09
21.74
27.95
33.55
35.92
49.19
65.50
69.90
LR=10 w/ OrthP
9.74
91.05
91.39
91.33
91.37
91.74
91.69
90.74
91.39
91.58
91.44
IR=10- w/ OrthP
9.74
90.81
91.59
91.77
91.85
92.04
92.12
92.22
92.12
92.33
92.25
IRR10 w/ StrongReg
89.13
91.68
91.11
91.37
91.84
91.47
90.11
91.32
90.78
91.55
91.29
LR=10- w/ StrongReg
89.13
92.03
91.84
92.13
92.05
91.85
92.00
92.02
92.09
92.08
92.08
(1) First, in Tab. 14, epoch 0, the mean JSV is around 3 (which can be considered close to the exact isometry 1, by our analysis in Sec. D) vs. 0.0004 when the dynamical isometry is not recovered. (2) Second, the test accuracy also has high starting point: epoch 1, using StrongReg achieves 91.68 (LR 10-2) and 92.03 (LR 10-3) vs. 63.86 (LR 10 10-2) and 9.74 (LR 10-3) when the dynamical isometry recovered.
Particularly note StrongReg can achieve similar mean JSV and test accuracy to OrthP, showing that StrongReg works by maintaining dynamical isometry, as OrthP does.
17


Reviews:
[{"review_id": "YMX00Wvatt", "reviewer": "Reviewer_YYie", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) **Nice analysis of previous papers**: It is commendable that this paper points out issues with two previous papers (Crowley et al 2018, Liu et al 2019) who make methodological errors by not tuning the learning rate for fine-tuning, and use the results of these erroneous experiments to make general statements about the efficacy of pruning (regardless of whether these statements are true or not). However from the perspective of these previous papers, it seems they had a fixed small fine-tuning budget of 20 epochs, and did not want to exceed this as we require number of fine-tuning epochs to be much smaller than the number of training epochs. While I do think that this requirement is arbitrary, it helps understand the claims of the previous papers.\n\n2) **Unfair experimental comparison**: In Table 1, it seems that the small Resnet models (A,B) trained from scratch are trained for 120 epochs, while the best prune-finetuned models are trained for 90 epochs. However also accounting for the training of the original large Resnet-34 model, the total number of training epochs are 90+90=180 epochs. Hence it seems that the prune+finetuned models are trained overall for a larger number of epochs, which may be a confounding factor which explains their better results. I would *strongly* suggest that the experiments be run such that the number of training epochs for \"training small model from scratch\" and \"large model training with pruning and finetuning\" be equalized for a fair comparison.      \n\n3) **Unclear why hypothesis must be true**: This paper rest on the hypothesis that training a large model followed by pruning and fine-tuning offers improved generalization benefits over training a small model from scratch. In principle it should be clear that both two methods can produce models with similar generalization given sufficiently strong optimizers, as the only difference between them is the initialization of the small model. Even in practice, we often are able to train models such that the train loss is close to zero (or sufficiently small), regardless of whether we train small models from scratch or fine-tune pruned models.  \nHence given that both methods solve the same overall optimization problem (i.e, minimizing loss of the small model), I am confused regarding why one would expect pruning + fine-tuning to generalize better than training from scratch, even in principle. For example, it is clear that given infinite epochs and a suitable (well-tuned) optimizer both methods should work equally well. Is the paper's hypothesis a claim about the speed of optimization, that pruning+fine-tuning is a fast way to obtain a small model? If so, how does this change after having accounted for the training of the dense model itself (which is usually slower to train)?\n\n4) **Incorrect metric for measuring Dynamical Isometry**: In equation 1, the paper uses mean singular values as a metric for assessing dynamical isometry. However if we require that all singular values are equal to one, then the metric to use would be the sum of (squared) deviations from unity of each singular value (\\sum_i (\\Sigma_{ii} - 1)^2). In such a metric, zero corresponds to dynamical isometry being achieved. However the metric of mean singular values has no such meaning as the standard deviation can be large. I am hence confused regarding why the paper chose to use the mean JSV metric, and if there is no strong reason for doing so, I would *strongly* recommend re-running experiments with the deviations-from-unity metric.\n\n5) **Relationship between mean JSVs and convergence unclear**: Hypotheses 1-4 and the boxed explanation talk about the \"recovery\" of mean JSVs and their relationship to network convergence. I am confused regarding why the behaviour of mean JSVs during or after training is deemed meaningful. I understand that too small or too large mean JSVs must be avoided at initialization as they can cause vanishing or exploding gradients, but can increasing mean JSVs from ~1 to ~5 (as in Fig 2) be deemed meaningful? It is thus not clear why mean JSV (or any measure of dynamical isometry) can be used to measure model convergence (ref. to boxed explanation in page 7). For example, if gradient-norm regularization is used, then the model would converge and yet have small JSVs. The only measure of model convergence is the gradient norm of the loss w.r.t. weights, and other measures are meaningless unless explicitly shown otherwise. ", "summary_of_the_paper": "This work takes a second look at the set of papers (Crowley et al 2018, Liu et al 2019) which claim that there is no generalization benefit offered by pruning, i.e., training a large network first and then pruning performs identically or worse than training the same pruned network architecture from scratch. This work claims that these previous works do not set learning rates during fine-tuning correctly while performing their experiments and hence these claims are invalid. The paper also offers an explanation for this phenomenon via dynamical isometery theory.", "summary_of_the_review": "This paper does not make a solid case for the hypothesis that \"pruning+finetuning produces models with better generalization than training small models from scratch\", as there are issues with both the experimental and the analytical studies. In the experimental setup, the comparisons are not performed fairly, while in the analytical parts use flawed metric to measure model convergence and dynamical isometry. Even fundamentally, it is unclear why the proposed hypothesis must be true in the first place. For all these reasons, I would recommend rejection.\n\nPost rebuttal update: The discussions did not change my opinion of the paper. However I appreciate the efforts taken by the authors to update the paper in light of reviewer suggestions, particularly to make the core argument more rigorous. Although I do not think the paper still succeeds at this, I am increasing my score slightly to reflect this change.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LdBwgvSpwzf", "reviewer": "Reviewer_nWUS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1. The empirical observation that inheriting pretrained weights is better than training a sparse network from scratch if the fine-tuning phase is carefully tuned.\n2. The empirical analysis on fine-tuning hyperparameters (LR, #epochs) might be useful to practitioners.\n\n# Weaknesses\n\n1. lack of novelty: connection to DI and pruning breaks DI is known [Lee et al 2020, analysed at initialization] and analysing for a pretrained network is straightforward and claiming it as a contribution is weak.\n\n2. The hypothesis on page 7 connecting larger LR in fine-tuning and DI is dubious and not backed by any theory or experiments. In fig.2 plots mean JSV increases and that does not mean improved DI. As DI theory suggests to have mean JSV around 1.\n\n3. The comparison against scratch does not seem fair. Usually, fine-tuning phase is much shorter compared to training from scratch but in this case fine-tuning is done for almost the same no of epochs as the scratch version. Is it possible that the scratch version could be tuned to improve the performance?\n\n4. In section 4, the L2 regularization of pruned weights is regarded as a method for DI recovery which in my opinion is unsubstantiated.", "summary_of_the_paper": "The paper argues pruning a pretrained network and finetuning is better than training a sparse network from scratch and tries to connect the fine-tuning phase properties to dynamical isometry (DI) literature. The empirical observation makes a case for inheriting the pretrained weights.", "summary_of_the_review": "There are many unsubstantiated claims (or weak statements) in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6sA-fgy87ZM", "reviewer": "Reviewer_bhQQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and organized. The effect of learning rate on fine-tuning stage of pruning is not thoroughly investigated and this paper provides an in-time and thorough study. My major concerns are as follows:\n\n- Pruning as poor initialization: the concept of initialization needs to be cleared. Does the pruning in Fig 1 refer to weights pruning for both networks? It might be true if it is weights pruning as different pruning ratios correspond to the same weights dimensions with different values. However, filter pruning results in different architectures and different pruning results are not different initializations with respect to the same architecture. \n\n- The dynamic isometry might explain the easiness of optimization, however, the relationship with the generalization is not quite clear. There might be other metrics such as flatness might show a similar trend as JSV. A comparison with other metrics for generalization could be more persuasive.\n\n- The analysis with MLP-7 and MNIST might be too simple for practical usage, and this method does not generalize to more practical non-linear convolutional neural networks. The authors propose strong regularization helps, however, no explanation on why this method helps and how it connects with previous analysis. Does this method work for MLP-7 and how is it connected to DI?", "summary_of_the_paper": "There are recent works questioning the value of inheriting weights in structured neural network pruning as it is empirically observed training from scratch can match or outperform finetuning a pruned model. This paper mainly includes three components: 1) the authors reinvestigated the problem and demonstrates that the conclusion is inaccurate because of improperly small finetuning learning rates. They show finetuning with pruned weights actually outperforms training from scratch, when larger learning rates and longer training epochs are adopted. 2)  the authors explored dynamical isometry (DI) to understand how finetuning LR affects the final performance. They show that weight pruning breaks dynamical isometry and finetuning can recover it and a larger LR can recover faster. 3) They proposed to fully recover dynamical isometry in fitler pruning before finetuning.\n", "summary_of_the_review": "The paper carefully studies recent thoughts on the meaning of pruning and made some rigorous investigations. Though some issues exists and DI may not be the ultimate solution to the question, the paper provides valuable thoughts for this filed, including fair experimental comparison and new explanations. Therefore I recommend to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-kEDu9CDHa2", "reviewer": "Reviewer_VvbL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1) This paper brings new insights for the role of the “weights” in the network pruning.\n2) This paper re-examines the established arguments with careful designed experiments over wide range of pruning ratios.\n3) The proposed technique is clearly motivated by empirical observations and is very effective on MLP networks under large pruning ratios.\n4) This paper is easy to follow.\n\nWeakness:\n\n1) Table 1,2 shows that fine-tuning a pruned network requires much more training time and tuning efforts to achieve a satisfying result. Although the author claims that OrthP can “complete” resolve the dynamical isometry issue, the pruned network converges still only after prolonged training epochs (90 and 900 epochs). Does that means the pruned networks weights are more difficult to optimize than random initialized ones, which contradicts the \"the value of weights\" claim?\n2) The application area of the proposed OrthP is limited to MLP (Table 6). Although the author proposed a Strong L2 Regularization for more complicated networks, this section seems to be “unfinished” — the connection with dynamical isometry seems a little bit weak to me, can the authors provides more elaborations? This seems to be more of practical interests.\n3) I have some confusions regarding the experiment designs and presented results (see the Questions section below). \n\nQuestions:\n\n1)  For Table 1: In “Our rerun”,  are “Scratch” results trained under 120 epochs with a decayed LR schedule? For Table 2, what are the hyper-parameters for the “Scratch” results, are they carefully tuned?\n2) Have the authors investigated the JSV of randomly initialized sparse networks? I am not sure if applying OrthP initialization or other orthogonal regularization methods on “Scratch” can also boost its performance. I believe this would be a fairer comparison to support \"the value of weights\" claim.\n3) Table 5: Can the authors also provide the “Scratch” results and their hyper-parameters? I am asking because the reasons stated in Weakness 1) — if the “Scratch” result is comparable to OrthP without using prolonged epochs, then this means even when the dynamical isometry is recovered, the pruning selected weights need more training time to optimize than randomly initialized ones. Also, in Table 2, it looks like “Scratch” results work well under large PR, which is exactly the case in Table 5.\n4) Related to 3), from Figure 2 and Table 5, it looks like OrthP takes the role of a large training epochs to recover dynamical isometry. It would be nice to see if OrthP works for non-prolonged epochs, e.g., < 90.", "summary_of_the_paper": "This paper challenges an existing argument that “inheriting the weights of the pruned network is not necessary for fine-tuning”. This paper provides empirical evidence that the previous experiments are not carried out with proper learning rates and training epochs to ensure the network convergence. \n\nThe authors conjecture that fine-tuning pruned network requires larger learning rates and longer training epochs because its dynamical isometry is broken. As a result, large learning rates (if training for shorter epochs) or long training epochs are required for the recovery of dynamic isometry. \n\nThen the authors propose OrthP to re-initialize the pruned weights to completely recover the dynamic isometry in simple MLP. With OrthP, the fine-tuning process is less sensitive to different learning rate and training epochs. \n", "summary_of_the_review": "I like this paper in that it thoroughly investigates existing arguments on the value of pruning weights. The perspective of OrthP is not that novel, but is simple and effective on MLP under large pruning ratios. \n\nMy major concern is in the experiments design and results part. I would consider raising my score if my questions can be addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper's primary contributions are:
* Contrary to previous claims, the authors empirically show that inheriting the weights after pruning can be beneficial when using *larger* fine-tuning learning rates than previously done.
* As an explanation, the authors provide suggestive results showing that pruning breaks dynamical isometry, which they claim explains why larger learning rates are needed.
* They propose a regularization-based technique to recover dynamical isometry on modern residual CNNs.

Generally, reviewers were positive about the ideas in the paper, however, even after the rebuttal 3/4 reviewers did not find the arguments were clear or strongly supported yet. One issue that came up several times is a request for more investigation of StrongReg+pruning. At this time, I have to recommend rejection, but I encourage the authors to follow up on the reviewers suggestions and submit to a future venue.
Decision: Reject