Revisiting Loss Modelling for Unstructured Pruning
Authors: César Laurent
Camille Ballas
Thomas George
Pascal Vincent
Nicolas Ballas
Venue:  (2021)
Keywords: Deep Learning
Network Pruning
Unstructured Pruning

Abstract: By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps, and systematically compare first and second order Taylor expansions. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.

Under review as a conference paper at ICLR 2021
REVISITING LOss MODELLING FOR UNSTRUCTURED PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps, and systematically compare first and second order Taylor expansions. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning the loss might not be a sufficient objective to design good pruning criteria.
INTRODUCTION
Neural networks are getting bigger, requiring more and more computational resources not only for training, but also when used for inference. However, resources are sometimes limited, especially on mobile devices and low-power chips. In unstructured pruning, the goal is to remove some parameters (i.e. setting them to zeros), while still maintaining good prediction performances. This fundamentally combinatorial optimization problem which is intractable even for small scale neural networks, and thus various heuristics have been developed prune the model either before training (Lee et al., 2019b; Wang et al., 2020), during training (Louizos et al., 2017; Molchanov et al., 2017; Ding et al., 2019), or in an iterative training/fine-tunin fashion (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Frankle & Carbin, 2018; Renda et al., 2020).
Early pruning work Optimal Brain Damage (OBD) (LeCun et al., 1990), and later Optimal Brain Surgeon (OBS) (Hassibi & Stork, 1993), proposed to estimate the importance of each parameter by approximating the effect of removing it, using the second order term of a Taylor expansion of the loss function around converged parameters. This type of approach involves computing the Hessian, which is challenging to compute since it scales quadratically with the number of parameters in the network. Several approximations have thus been explored in the literature (LeCun et al., 1990; Hassibi & Stork, 1993; Heskes, 2000; Zeng & Urtasun, 2019; Wang et al., 2019). However, statc-of- the-art unstructured pruning methods typically rely on Magnitude Pruning (MP) (Han al., 2015), simple and computationally cheap criterion based on weight magnitude, that works extremely well in practice (Renda et al., 2020).
This paper revisits linear and diagonal quadratic models of the local loss landscape for unstructured pruning. In particular, since these models are local approximations and thus assume that pruning steps correspond to small vectors in parameter space, we propose to investigate how this locality assumption affects their performance. Moreover, we show that the convergence assumption behind OBD and OBS, which is overlooked and violated in current methods, can be relaxed by maintaining the gradient term the quadratic model. Finally, to prevent having to compute second order information, we propose to compare diagonal quadratic models to simpler linear models.
While our empirical study demonstrates that pruning criteria based on linear and quadratic loss models are good at preserving the training loss, it also shows that this benefit does not necessarily transfer to better networks after fine-tuning, suggesting that preserving the loss might not be the best objective optimize for. Our contributions be summarized as follows:

---

Under review as a conference paper at ICLR 2021
1. We present pruning criteria based both linear and diagonal quadratic models of the loss, and show how they compare at preserving training loss compared to OBD and MP.
2. We study two strategies to better enforce locality in the pruning steps, pruning in several stages and regularising the step size, and show how they improve the quality of the criteria.
3. We show that using pruning criteria that are better at preserving the loss does not necessarily transfer to better fine-tuned networks, raising questions about the adequacy of such criteria.
BACKGROUND: UNSTRUCTURED PRUNING
2.1 UNSTRUCTURED PRUNING PROBLEM FORMULATION
For given architecture, neural networks are family of functions fo X - y from an input space X' to an output space y, where 0 E IR is the vector that contains all the parameters of the network. Neural networks are usually trained by seeking parameters 0 that minimize the empirical risk (
Σ, e (fo (2,) t.) of a loss function & on a training dataset D = { t.)s composed of N (example, target) pairs.
The goal of unstructured pruning is to find a step A to add to the current parameters 0 such that || A0 o = D, i.c. the parameter vector after pruning is desired sparsity x E [0. 1]. While doing the performance of the pruned network should be maintained, differ much from (()). Unstructured pruning thus amounts to the following minimization problem:
C(O - A0) should not
AC A0) def C(( + A0) - (())) s.t. 10 + A0l = (1 11--8 K)
minimize 20
(1)
Directly solving this problem would require evaluating C(O + Ao) for all possible values of A0, which is prohibitively expensive, so one needs to rely heuristics find good solutions.
2.2
OPTIMAL BRAIN DAMAGE CRITERION
Optimal Brain Damage (OBD) (LeCun et al., 1990) proposes to use quadratic modelling of C(O + A0), leading to the following approximation of AL A0):
aL(O)*
O (0, 20)
م 2 Ao H(0)
00
where H()) is the Hessian of L(O). H()) being intractable, even for small-scale networks, its Generalized Gauss-Newton approximation G(0) (Schraudolph, 2002) is used in practice, as detailed in Appendix A. Then, two more approximations are made: first, it assumes the training of the network has converged, thus the gradient of the loss wrt 0 is O. which makes the linear term vanish. Then, it neglects the interactions between parameters, which corresponds a diagonal approximation of G()), leading to the following model:
1 GaL (0)
1 I G (0)0 8k 2
ACOBD (0,)0)) ~
LBD
(3)
LBBD is the saliency of each parameter, estimating how much the loss will change if that parameter Sk pruned, so if AQ, OO. Parameters can thus be ranked by order of importance, and the ones with the smallest saliencies (i.e. the least influence on the loss) are pruned, while the ones with the biggest saliencies are kept unchanged. This can be interpreted as finding and applying binary mask m (0, 1)2 to the parameters such that f + A0 ) m, where @ is the element-wise product.
2.3 MAGNITUDE PRUNING CRITERION
Magnitude Pruning (MP) (Han et al., 2015), is popular pruning criterion in which the saliency is simply based on the norm of the parameter:
MP S
0²
Despite its simplicity, MP works extremely well in practice (Gale et al., 2019), and is used in current state-of-the-art methods (Renda al., 2020). We use global MP as baseline in all our experiments.
Although LeCun et al. (1990) uses H(0) in the equations of OBD, it is actually G(Ø) which was used in practice (LeCun, 2007).
2

---

Under review as a conference paper at ICLR 2021
2.4 OPTIMAL BRAIN SURGEON
Optimal Brain Surgeon (OBS) (Hassibi & Stork, 1993) also relies on the quadratic model in Equation solve the minimization problem given in Equation 1, but the Lagrangian formulation to include the constraint to the solution of the minimization problem. Since OBS requires to compute the inverse of HO),, several approximations have been explored in the literature, including diagonal, as in the original OBS, Kronecker- facttred (Martens & Grosse, 2015) as in ML-Prune (Zeng & Urtasun, 2019), or diagonal, but in an Kronecker-factoret Eigenbasis (George al., 2018), as EigenDamage (Wang et al., 2019). While we use OBD in our demonstrations and experimental setup, everything presented in this paper can also be used OBS-based methods. We leave that for future work.
REVISITING Loss MODELLING FOR UNSTRUCTURED PRUNING
In this work, we investigate linear and diagonal quadratic models of the loss function and their performance when used for pruning neural networks. In our empirical study, we aim at answering the following questions:
1. How do criteria based weight magnitude, linear quadratic models compare at preserving training loss (i.e. at solving the minimization problem in Equation 1)?
2. How does the locality assumption behind criteria based linear and quadratic models affect their performances?
3. Do pruning criteria that are better at preserving the loss lead better fine-tuned networks?
We now describe the linear and quadratic models we use, as well as the strategies to enforce locality of the pruning steps.
3.1 LINEAR AND QUADRATIC MODELS
In current training strategies, regularization techniques such as early stopping dropout (Srivastava al., 2014) are often used to counteract overfitting. these setups, there is no reason assume that the training has converged, implying that the lincar term in the Taylor expansion should not be neglected. Thus, one can build a pruning criterion similar to OBD that includes the gradient term in the quadratic model from Equation 2, leading to the following saliencies:?
OC(0)"
1 I GeL (0)A0
OC(0) 00
1
AC (0, (0,) ~
A + 2
QM 8
O2 + - Gza
(5)
a0
2
Recall the constraint A0 E (-0,,, 0}. hence the saliencies. This criterion generalizes OBD for networks that are not convergence, and provides similar saliencies for networks that have converged.
To avoid the computational cost associated with computing second order information, which is prohibitive for large scale neural networks, one can use a simpler linear model (LM) instead of a quadratic to approximate AL A0), leading to the following approximation and saliencies:
OC(()) 00
aC(O) 00
LLLM 20)
A0
LM Sk
0
The saliencies of the linear model are very related to the criterion used in Single-shot Network Pruning (Lee et al., 2019b), as demonstrated by Wang et al. (2020).
3.2 ENFORCING LOCALITY
One important point keep in mind is that linear and quadratic models (whether diagonal not) are local approximations, and are generally only faithful in a small neighbourhood of the current parameters. Explicitly showing the terms that are neglected, have:
AC A0) L (0, A0) O(|A0|))) A QM (0, (0,00) O(I B
Note that this idea has been explored for OBS (Singh & Alistarh, 2020), well for structured prun- ing (Molchanov al., 2019)

---

Under review as a conference paper at ICLR 2021
So when approximating AC with A we neglect the terms in ((140)))) and when approximat-
ing AC with ACQM we neglect the terms in O [ Both approximations are thus only valid in small neighbourhood of 0, and are extremely likely to be wrong when | is large. We list here different tricks to prevent this from happening.
Performing the Pruning in several Stages || la can be large when a large portion of the parameters is pruned once. An easy fix typically used mitigate this issue is perform the pruning in several stages, re-estimating the model at each stage. The number of stages, which we denote by æ, is typically overlooked (e.g. both Zeng& & Urtasun (2019) and Wang al. (2019) use only 6 stages of pruning). Our experiments, in upcoming Section 5, show that it has drastic impact on the performances. Note that, without fine-tuning phases between the different pruning stages, this strategy violates the convergence assumption behind OBD and OBS, since after the first stage of pruning the network is no more at convergence.
The sparsity at each stage can be increased following either a linear schedule. where each step prunes the same number of parameter, or an exponential schedule, where the number of parameters pruned at each stage gets smaller and smaller. The later is typically used in the literature (Zeng & Urtasun, 2019; Wang et al., 2019; Frankle & Carbin, 2018; Renda et al., 2020). We compare them in Section 5.
Constraining the Step Size As is often done when using quadratic models (e.g. Nocedal & Wright (2006), one can penalize the model when it decides to take steps that are too large, in order to stay in a region where we can trust the model. This can be done by simply adding the norm penalty ||0 1| to the saliencies computed by any criterion (Equations 3, 5 or 6), where À is a hyper-parameter that controls the strength of the constraint: a small value of À leaves the saliencics unchanged, and a large value of À transforms the pruning criterion into MP (Equation 4).
Other Considerations ||0||2 la can be large if 0 is large itself. This is dependent on the training procedure of the network, but can be casily mitigated by constraining the norm of the weights, which can be done using La regularisation or weight decay. Since nowadays weight decay is almost systematically used by default when training networks (e.g. He et al. (2016b); Xie et al. (2017); Devlin et al. (2018)), we do not investigate this further.
4 METHODOLOGY
We follow the main recommendations from Blalock et al. (2020). For fair comparison between criteria, all experiments are from our own PyTorch (Paszke et al., 2017) re-implementation, and ran on V100 GPUs. We use 5 different random seeds, and both mean and standard deviations are reported. We experiment with a MLP on MNIST, and with both VGGI (Simonyan & Zisserman, 2014) and a prevaccivation residual network 18 (He et al., 2016b) on CIFAR1O (Krizhevsky et al., 2009), to have variability in architectures, while using networks with good performance to number of parameters ratio. We further validate our findings on ImageNet (Deng et al., 2009) using a residual network 50 (He et al., 201 Although MNIST is not considered a good benchmark for pruning (Blalock et al., 2020), it can still be used to compare the ability of different criteria to solve the minimization problem in Equation 1. See Appendix B for details about splits, data augmentation and hyper-parameters.
Pruning Framework Algorithm presents the pruning framework used in this work: we first train the network, then perform several stages of pruning, and finally perform a single phase of fine-tuning, using the same hyper- parameterssss as for the original training. Global pruning is used for all the criteria. Note that because of their convergence assumption, OBD and OBS advocate for fine-tuning after each stage of pruning. Since LM and QM are not based on this assumption, they should perform well in this proposed framework. While the fine tuning-phase would require lyper-parameters optimisation, Renda et al. (2020) showed that using the same ones as for the original training usually leads to good results. The hyper-parameters used in our experiments are provided Appendix
Performance Metries The performances of the pruning criteria are measured using two metrics: First, we use A A0) [C + A0)- [()),, which is the quantity that the pruning criteria
4

---

Under review as a conference paper at ICLR 2021
Algorithm Pruning Framework
Require: Network So with IRP, dataset D, number of pruning iterations x, and sparsity s.
1: So Training(So. D)
2: Ko f-
3: 1
4: for = 1 to т do
5:
K; f:-1 + (K-KO) or R4 Ki-1 + (R
Compute sparsity for iteration i
6: s Saliencies( (foriiieesfoom D)
D Compute saliencies (Equation 3, 4, 5 or 6).
7: m m|argsort(s)] K; D].
Mask the parameters with smallest saliencies.
8: feom + Training(fo..r ffom. D)
Optional fine-tuning
9: return foom:
are designed to minimize (recall Equation 1). Second, we use the validation error gap before/after fine-tuning, which is the metric ultimately care about when designing pruning methods.
PERFORMANCES BEFORE FINE-TUNING
We evaluate the impact of enforcing locality in the LM, QM and OBS criteria. For each criterion, Figure 1 reports ( AO) as function of X, for different number of pruning stages т,, using the exponential pruning schedule, and Figure 5 in Appendix show the same results for the linear pruning schedule. A typical usage of these criteria would be with regularisation strength À 0 and number of pruning stages æ Rs MP, the baseline, which invariant to both and æ, also reported in dashed black. For reference, the networks reached validation error rate before pruning of 1.47 + 0.04 % for the MLP, 10.16 + 0.29 for VGGI and 4.87 + 0.04 % for the PreActResNet18.
Model
Quathal Mode
molimel Brai
0430 080
OUR-00 (UU-000
10t0
10
10
10th
10
10
101
10
10th
10
10a3
10th
10
101
101
100
10th
10
10-3
10
10th
161
101
10
(a) MLP on MNIST with 98.8% sparsity.
UnsarModd
Quadratic Modd
Ostimal Brain Damage
1428 600
040 A0)
(b) VGGI] I on CIFARIO with 95.6% sparsity.
30
10
10th
107
30"
10
10
10
30
307
3
10a
30
10
10a 107
301
167
(c) PreActResNett8 on CIFARIO with 95.6% sparsity.
Figure 1: AC A0) for different number of pruning stages constraint strength, using either (left) LM, (middle) QM or (right) OBD criteria. MP, which is invariant to and to the number of pruning stages, is displayed in dashed black. The curves the mean and the error bars the standard deviation over random seeds. OBD with क and diverged for all of the seeds. Increasing the number of pruning stages drastically reduces ((, A0). A X > can also help improving performances. Figure Appendix contains the same plots, but displaying the validation gap before/after pruning.
function of X, the step size
5

---

Under review as a conference paper at ICLR 2021
5.1 IMPACT OF THE ASSUMPTIONS BEHIND THE DIFFERENT CRITERIA
Locality Assumption Figure 1 shows that increasing the number of pruning stages drastically reduce AC(O, A0) when using LM, QM and OBS criteria. It demonstrates the importance of applying local steps when pruning. Constraining the steps size through 2 |0x112 can also reduce AC 20), on CIFARI0 in particular. The trend, however, is less pronounced on MNIST. We hypothesize that it is due to the pruning step size: the MLP contains 260k parameters, vs 9.7M for VGGI1, so the number parameters pruned at each stage VGGI1 is still large, even with π 140. This translates to a bigger 2 that needs to be controlled by the regularisation constraint.
Convergence Assumption When performing the pruning in several stages, we also observe that LM and QM can reach better performances than OBD. Without retraining phases between pruning stages, we violate the convergence assumption of OBD. This is however not the case for LM and QM, since they are not based this assumption. Note that OBD still works reasonably well VGGII. This could be be related to the depth of VGGII: VGG11 is deeper than the MLP, but not equipped with residual connections like the 8
5.2 LOSS-PRESERVATION CAPABILITIES OF THE DIFFERENT CRITERIA
Table contains the best AC(O, Δ0) for each of the networks and pruning criteria. Our main observation is that the criteria that model the loss (LM and QM in particular) are better at preserving the loss than MP. Similarly to Table 1, Table 3 in Appendix contains the best validation error gap before/after pruming, where
observe similar tendencies.
Table 1:
Summary of the best AC AO) across values of À for different networks and pruning criteria, with ग = 140. QM achieves better loss poss-rreeervation than other criteria. OBD performs worse than QM, since violate its convergence assumption when pruning in several stages.
AC A0)
Network
MP
OBD
LM
QM
MI.P on MNIST
2.02 + 0.10
1.83
0.11
1.17
0.03
1.05 +
VGG11 on CIFAR10
1.84 + 0.44
0.89
+ 0.24
0.90 + 0.21
0.86 + 0.22
PreActResNet18 on CIFAR10
2.23
+ 0.14
1.95 + 0.46
1.36 + 0.18
1.22 + 0.31
5.3 LINEAR VS EXPONENTIAL PRUNING SCHEDULE
Figure 2 compares the impact of | and reports the training error gap when pruning VGG1 on CIFARIO 10 in several stages, using either the linear or the exponential pruning schedule. We also compare against one-shot pruning, reference. The exponential schedule allows to maintain more constant | | throughout the pruning procedure, which limits the maximum size of | 22, and thus better satisfies the locality assumption.
10a
10P
Dxpanetix Cre-shuc
10
AL88.00
100
103
121
Spardit
Figure 2: Linear vs exponential schedule using QM on VGGII. Left: AC A0) vs sparsity, zoomed the end. Markers denote the 14 pruning stages. Middle: || each stage. Right: Same as Figure 1, but comparing exponential (solid) and linear (dotted) schedules at 95.6% sparsity, with æ (14, 140). We get smaller || per pruning stage when using exponential instead of linear schedule, resulting in smaller AC A0). It is advantageous to use that schedule when the pruning budged limited, i.e. when small. This advantage vanishes for larger values of π.

---

Under review as a conference paper at ICLR 2021
6 PERFORMANCES AFTER FINE-TUNING
We now fine-tune the pruned networks using the same hyper-paraaae param-prraeeterss and number of epochs than for the original training. Table 2 shows the validation error gap between the non-pruned networks and the pruned networks after fine-tuning, for all considered criteria. LM performs better than MP on both the MLP and VGG11 (0.5% difference), but all criteria perform similarly on the PreActResNet These results are consistent with the observations of Blalock et al. (2020). As reference, global random pruning resulted in validation error rate of 47.18 + 6.8 % for the MLP, and resulted in non-retrainable networks on CIFAR1O (with 90 % error rate).
Table Best validation error gap of the fine-tuned networks (lower is better), for different pruning criteria, across values of X and LM is better than MP the MLP and VGG11. All the methods reach similar levels performance on the N8
Gap of Validation Error (%)
Network
MP
OBD
LM
QM
MLP MNIST
2,4 0.3
2.0 0.1
1,9 + 0.3
1.9 = 0.2
VGGII CIFARIO
0,2 0.2
00.1
-0.3 0.1
-0.1 0.1
PreActResNet1$ on CIFARIO
0.2 0.2
0.2 0.2
0.1
0.2 = 0.2
6.1 CORRELATION BETWERN LOSS-PRESERVATTIO AND PERFORMANCES AFTER FINE-TUNING
An important observation is that the hyper- ppperpprraettrs and π that give the best performing criteria in terms of AC(O, A0) in Table are not the same as the ones that give the best performing criteria after fine-tuning in Table 2. We display in Figure 3 scatter plots of all the experiments we ran, to observe how well loss pss-preseevvatin correlates with performance after fine-tuning:
MP
0BO
LM
QM
MNIST
VEG11 on CIFAR10
ProActResNet11
4.0
0.8
2.5
0.6
Vald. Ereer (h)) afterf: f-t
Valid Emrr (%) atteri-t
Volid Emar (%) afterf-t.
0.4
1.5
3.0
0.2
0.0
1.0
-0.2
2.0
0.0
0.00.8
p= 0.27
0.20
0.6
10
AL18, AB) betore fine-tuning
22 28) before tuningg
AU/B, 48) betore fine tuning
Figure 3: Gap of validation enror after fine-tuning as a function of AC A0). Each point is one random seed,
experiment, i.c. coefficient computed on all the data points. Except for the ML. on MNIST, there only a weak correlation between ( A0) and the gap of validation after fine-tuning. Thus, the performance after pruning cannot be explained solely by the loss-preserving abilities of the pruning criteria.
and one X. p is the Spearman's rank correlation
Quite surprisingly, although we are able to obtain networks with smaller A A0). and thus better performing networks right after pruming, the performances after fine-tuning do not correlate significantly with the gap. Except for the MLP on MNIST, whose Spearman's ''s rank correlation coefficient is 0.67, there is only weak correlation between AL A0) and the validation error gap after fine-tuning (p = 0.27 for VGGII and م = 0.20 for PreActResNet1:3 8). Figure in Appendix contains the same scatter plots, but showing L(O @ m) after fine-tuning instead of the validation error gap, and similar trends can be observed. Figure 11, also in Appendix. shows similar scatter plots, but for different sparsity levels VGG11. Finally, Figure in Appendix contains the same scatter plots but displaying the validation error gap before fine-tuning versus the validation error gap before fine-tuning.
7

---

Under review as a conference paper at ICLR 2021
To verify that these observations not due to specific choice of fine-tuning hyper- paper--arameters. we perform show in Figure 15 in Appendix C.3 the fine-tuning curves of networks pruned using MP and our best QM criteria. We observe that, except for MNIST, the difference training loss right after pruning disappears after only epoch of fine-tuning, erasing the advantage of QM over MP.
grid search and report similar results in Appendix C.2. Also, we
6.2 DISCUSSION
These results highlight an important issue: minimizing (C A0), no matter what model is used, might be used to design better pruning criteria, but it does not necessarily transfer to a better pruning method when fine-tuning is involved. The performance after fine-tuning cannot be explained solely by the local loss psss-preserving abilities of the criteria, and other mechanisms might be at play. Thus, the effect of fine-tuning should also be taken into account when designing pruning criteria.
For instance, Lee et al. (2019a) and Wang et al. (2020) proposed different heuristics to take into account gradient propagation in the context of foresight pruning, i.e. pruning untrained networks right after initialisation. Wang et al. (2020) argues that minimizing A ((A A ) in that context makes little sense, since the network is producing random predictions. In Appendix C.4 we compare our results to two pruning methods based on preserving the gradient flow, GraSP (Wang et al., 2020) and SynFlow (Tanaka et al., 2020), and show that they comply with our observations above.
Finally, several recent articles looking further into the impact of various pruning criteria subsequent training or fine-tuning (Lubana Dick, 2020; Evei et al., 2020; Frankle et al., 2020).
7 SCALING UP TO IMAGENET
To investigate whether our observations also hold on larger datasets, we perform similar experiments with LM, QM and OBD on the ResNet50 on ImageNet. Before pruning, the network reached 76.4 % validation accuracy. Figure presents results at 70 % sparsity, in similar fashion Figure I and Figure 3. We observe a similar trend: The best pessepreeriiig models are not necessarily the best models after fine-tuning. See Appendix B for the detailed experimental setting, and see Figure 17 for results at 90 % sparsity.
n=1
100
MP
Linear Model
Quadratic Model
ResNet50 on ImageNet
107
80*
75.5
04.0
01.00
Valid Acturecy (5) efter
75.0
10
30a
10
10
10
10
101
0
10
10
10:1
107
101
74.5
Optimal Brain Damage
74.0
1.0F
40 Valid. Accuracy (9k before f.it.
20
10
1.0
Figure 4: Same as Figure and Figure 3, for the ResNet50 ImageNet, with sparsity of 70 %. Increasing the number of pruning stages and constraining the step size reduce A ( A0). However, the best-loss preserving criteria, which maximize the validation accuracy right after pruning, do not produce better networks after fine-tuning. They perform similarly if their validation accuracy after pruning is 20%. Criteria that outperform MP right after pruning do not achieve better performance after fine-tuning.
8 CONCLUSION
In this paper, we revisited loss modelling for unstructured pruning. We showed that keeping the gradient term in the diagonal quadratic model allows to relax the convergence assumption behind
8

---

Under review as a conference paper at ICLR 2021
OBS and OBD. We also showed the importance of locality when using loss models for pruning: increasing the number of pruning stages and constraining the step size are two improvements that produce better lossppregg pruning criteria and that should be added to the recommendation list of Blalock et al. (2020). Finally we observed that the loss right after pruning does not always correlate with the performances after fine-tuning, suggesting that better loss before fine-tuning is not solely responsible for the performances after fine-tuning. Thus, future research should focus on ways to model the actual effect of subsequent fine-tuning when designing pruning criteria.
REFERENCES
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.03033, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition. Ieee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint TrXiv:I8100 04000, 2018.
Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et al. Global sparse momentum sgd for pruning very deep neural networks. NeurIPS, 2019.
Utku Evei, Yani A Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. arXiv preprint arXiv:2010.03533 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09577 2019.
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in kronecker factored eigenbasis. In NIPS, 2018
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.02677 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NIPS. 2015.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NIPS, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016b.
Tom Heskes. On "natural" learning and pruning in multilayered perceptrons. Neural Computation, 2000.

---

Under review as a conference paper at ICLR 2021
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015. URL v37/ioffe15, html.
mlr press/
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. Who afraid of convex optimization? NIPS Workshop on Efficient Learning, 2007. nonconvex. pdf.
URL https::////// / cs nyu. wwtpe//aaattaaaaaaaeeeeeeeeeeeee http:/ttaaaseeeeenneneeeeeeeeeee wwweeeuunnn000000 20771207-nonorveer
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In NIPS, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2224 1998.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks initialization. arXiv preprint 190.::090000600 2019a.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr SNIP: Single- Shot Network Pruning based on Connection Sensitivity. In ICLR, 2019b. URL hww.t:tt/////e http:::///pppeeeiiiieeo eww. net. forum? id== B1 B1VZgjAcYX.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv:1712.01312, 2017.
Ekdeep Singh Lubana and Robert P. Dick. A gradient flow framework for analyzing network pruning, 2020.
James Martens and Roger Grosse. Optimizing neural networks with cronecker-factoree approximate curvature. In ICML, 2015.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Veto, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020. URL https: /ttp://///eereeieeeeom wew.iiewweeettt. forum? f JONKA.
Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent.
Neural computation, 2002.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1 1556.1566,.6..,,,, 2014.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. arXiv preprint arXiv:2004.14332 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overlitting. JMLR, 2014.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05440 054667, 2020.
10

---

Under review as a conference paper at ICLR 2021
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker- fackored eigenbasis. arXiv preprint arXiv:1905.05994. 2019.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020. URL hwwph:tt///ooot https://openreview.. fertfform?id- SKgsACVKPH.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Wenyuan Zeng and Raquel Urtasun. MIPrume: Multi-layer pruning for automated neural network compression, 2019. URL
id=rl g522RcKm
APPENDIX
A GENERALIZED GAUSS-NEWTON
Having to compute H()) is obvious drawback of quadratic models, and thus a common first step to approximate H()) using the Generalized Gauss-Newte approximation (Schraudolph, 2002):
H(0) 0 (1:)
1
050 (2:)
de (u,
8 so (xI)s
e (u,
(8)
N
00
00
Oue
002
(=fo(=.)
G( Generalized Gauss-Newton
R00
(
(9)
where K is the number of outputs of the network. G(0) has the advantage of being casier compute and is also positive semi-definite by construction.
B DETAILS ON THE EXPERIMENTAL. SETUP
B.1 SETUP
Datasets We use the MNIST dataset (LeCun et al., 1998), and hold-out 10000 examples randomly sampled from the training set for validation. We also use CIFAR10 (Krizhevsky et al., 2009). where the last 5000 examples of the training set are used for validation, and we apply standard data augmentation (random cropping and flipping, as in He et al. (2016b)) during training phases. For ImageNet (Deng et al., 2009), we follow the experimental setting Goyal al. (2017).
Network Architectures On MNIST, we use a MLP of dimensions 784-300-100-10, with Tanh activation functions. On CIFARIO, we use both: a VGGI1 (Simonyan & Zisserman, 2014), equipped with ReLUs (Nair & Hinton, 2010), but no Batch Normalisation (Toffe & Szegedy, 2015); and the PreActResNer18. which the 18-layer pre- activatiiioo variant of residual networks (He al., 2016b). MLP leverages Glorot & Bengio (2010) as initialization while the the weights of VGG11 and NreActResNer188 are initialized following He et al. (2015), and the biases are initialized to On ImageNet (Deng et al., 2009), we use ResNet-50 (He et al., 201 60) with Batch Normalization, and follow the initialization strategy described in (Goyal et al., 2017).
B.2 EXPERIMENTS
For the MNIST and CIFAR10 1 experiments, the network is first trained for fixed number of epochs, using early stopping on the validation set to select the best performing network. aaper-paraeetes used for training are selected via grid search (before even considering pruning). Then prune large fraction of the parameters. For OBD, LM and QM, we randomly select, at each iteration of pruning, 1000 examples (10 mini-batches) from the training to compute the gradients and second

---

Under review as a conference paper at ICLR 2021
order terms of the models." Finally, retrain the network using exactly the same syyper-arammtent as for the initial training.
For ImageNet, uses the exact same Syype--prameee than Goyal et al. (2017).
MLP on MNIST momentum factor of 0.9, 12 regularisation of 0.0005 and mini-batch size of 100. We prune 98.85% of the parameters.
We train the network for 400 epochs, using SGD with learning rate of 0.01,
VGG11 on CIFARIO We train the network for 300 epoch, using SGD with a learning rate of 0.01, momentum factor of 0.9, 12 regularisation of 0.0005 and mini-batch size of 100. The learning rate divided by 10 every 60 epochs. We prune 95.6% of the parameters.
PreActResNet18 on CIFARIO We train the network for 200 epochs, using SGD with a learning rate of 0.1, momentum factor of 0.9, regularisation of 0.0005 and mini-batch size of 100. The learning rate is divided by 10 every 70 epochs. We prune 95.6% of the parameters.
ResNet50 on ImageNet For ImageNet, train ResNet50 using V V100 GPUs. The total mini- batch size 256, and we train our baseline network for 90 epochs. The learning rate schedule is identical to Goyal al. (2017): linear warm-up in the first epochs and decay by factor of 10 at epochs 30, 60 and 80. We then prune 70% of the parameters. After pruning, we fine-tune the models for 90 epochs using learning following syper-parameter values: π E {1, 100), {le 1e -e-1_, 0, 10, }. 1600 examples are used compute the first and second order terms of the linear and quadratic models.
of 1e For LM, QM, OBD, investigates the
C SUPPLEMENTARY RESULTS
C.I PERFORMANCES BEFORE FINE-TUNING
Validation error Table Table 3 is the same Table 1, but containing the best validation error gap before/after pruning instead of AC( A0). We can observe similar trend as in Table 1: LM and QM give better performances than MP, and OBD performs poorly, since the convergence assumption not respected.
Table 3: Best validation error gap before/after pruning for different networks and pruning criteria.
Gap of Validation Error (%)
Network
MP
OBD 16.35 : 0.77
LM
QM
MLP on MNIST
72.09 3.72
64.89
5.74
15.22 : 0.62
VGG11 on CIFARIO
56.19 + 17.9
18.84 + 5.54
5.89 + 1.52
5.92 + 2.14
PreActResNet18 on CIFARIO
74.13 + 4.59
49.08 + 8.18
26.79 + 8.61
21.48 + 5.96
Linear pruning schedule Figure 5 contains the same experiments than Figure 5, but using the linear schedule instead of the exponential one. There is drastic difference in performances: One need roughly 10x more stages of pruning with the linear schedule to reach the training gap of the exponential schedule.
C.2 PERFORMANCES AFTER FINE-TUNING
Validation error figures Figures and 7 contain the same experiments than Figure 5, but displaying the validation eтror gap, for linear and exponential schedules, respectively. For completeness, Figure shows the validation error gap before fine-tuning.
Using 1 1000 examples the whole training set made no difference in our experiments. Using less examples started to degrade the performances, which concord with the observations of Lee al. (2019h)
12

---

Under review as a conference paper at ICLR 2021
-3-
Quadie =- 100)
100
Unear
Optimal frain Damage
06
Bun 66
D00
(a) MLP MNIST with 98.8% sparsity.
Optimal Damage
8130 600
15
20
10
10
100
107
107
0.6
10
10
10
10
107
367
107
12
10
10
10
10
10
107
(b) V on CIFARIO with 95.6% sparsity.
10
Linest Model
Quadratic Model
101
Brain
001
AUI0 201
040
(c) PreActResNed18 on CIFARIO with 95.6% sparsity.
Figure 5: Same as Figure 1, but using equally spaced pruning steps. Note the difference in number of pruning iterations.
+
Enorros
50
8 %
10
(a) MLP MNIST with 98.8% sparsity.
mif Erver
aat trierto:
25
vafe baror
75
50
80
50
28
(b) VGG11 on CIFARIO with 95.6% sparsity.
Opšima Damage
73
VI Emor
2
MM Emor
50
8
50
30
3.5
10-
100
18
(c) PreActResNetl8 CIFAR10 with 95.6% sparsity.
Figure 6: Same as Figure 1, but displaying the validation error gap before fine-tuning. With proper number of pruning stages and step size regularization, LM and QM can produce pruned networks that are drastically better than the ones pruned using MP.
Validation gap before and after fine-tuning Figure is the same as Figure 3, but showing the validation error gap after fine-tuning function of the validation error gap before fine-tuning. As for Figure 3, we do not observe much correlation between the validation error before and after the fine-tuning.
13

---

Under review as a conference paper at ICLR 2021
4.0
Liner Model
RuadroticMod:
Oztimal Brain Damage
3.55
3.0
20
12
20
10
15
10
10
10
10
10
10
15
101
101
10
10
10
10
10
30
109
(a) ML.P MNIST with 98.8% sparsity.
Cundrasi
Octinal Cemage
S
0.00
6,0
6,00
16
(b) VGGI1 on CIFARIO with 95.6% sparsity.
vilei 011e 1.C (94)
af errr (S)
Ludd eme
10
1.7
(c) PreActResNet18 CIFARIO with 95.6% sparsity.
Figure 7: Same as Figure 1, but displaying the validation error gap after fine-tuning.
200
1000
3.5
3.5
3.5
2.5-
20-1
12
10
10
10
107
10
12
10
16
10
10
10
167
10
(a) on MNIST with 98.8% sparsity.
Jrear Vedel
Quadracc Model
Cetimo Brain Cerrgge
10t4
10
10
10a1
107
10
12"
12-4
20
10
10
12
10th
10
20
12
10
(b) VGG11 CIFARIO with 95.6% sparsity.
Inee
Quadratic
Optimal Bran Damage
1. (56)
vate erie
1-4 (S0)
vole
13-3
13-2
10
10
16
162
13-2
15-2
1044
10F
10
102
15-3
10-2
304
11F
165
10
(c) PreActResNett8 on CIFAR1O with 95.6% sparsity.
Figure 8: Same as Figure 7, but using equally spaced pruning steps. Note the difference in number of pruning stages.
Training loss after fine-tuning Figure 10 the same Figure 3 but showing C(O O m) after fine- tuning as a function of AC A0). It has a similar trend as Figure 3: there is not much correlation between the loss before and after fine-tuning, except on MNIST.
Different sparsity levels Figure 11 shows the performances of different criteria VGG11 1 CIFARIO, for different sparsity levels. When the sparsity is low (89.3 %), the network has enough capacity return to its original performances after fine-tuning. When the sparsity is too high (98.6 %)
14

---

Under review as a conference paper at ICLR 2021
MLP on MNIIS
VEG11 on CIFAR10
ProActRes/ee118 en GIFAR1O
0.8
0.6
Volld Errer aftert.t. (16)
2.5
3.5
0.4
Vaid trrer afterfee
0.2
3.0
Vvlld Errer
2.5
1.0
2.0
1.5
0.0
-0.0
20
40
80
20
40
20
Valld. before
Valid. Eror before
Valld. (%)
Figure 9: Same as Figure 3, but showing the validation error gap after fine-tuning function of the validation eтror gap before fine-tuning. Networks with drastically different performance before fine-tuning can still produce similar performances after fine-tuning.
MIP MNIST
VGG11 on CIFAR10
PreActResNeL1e on CIFAR1O
0.2.99
p
0.03
0.02
119
L
0.1
0.01
0.08
0.01
0.006
10
fine-tuning
fine-tuning
aell fine-tuning
Figure 10: Same as Figure 3, but showing L(O @ m) after fine-tuning as a function of AC(O, A0). Except for the MLP on MNIST, there is only a weak correlation between A AO) and C(O 2 m) after fine-tuning.
then all criteria produce networks with random predictions. There might be sweet spot in between, one would require more powerful model to verify this supposition.
QM
89.3 (%)
95.6 (%) Sparsity
98.6 (%)
p= 0.13
p
0.20
p=
-0.28
6. Valld. Eroo 180 aftertz T-t.
a Valid. Error (80 after 1-L.
3-
a Valld. Error 180 after f-t.
3-
1
AL 46) before fine-tuning
A110 58) before fine-tuning
AL(O, Ad) before fine-tuning
Figure 11: Same as Figure 3, but for different sparsity levels on the VGGI] on CIFARIO. When the sparsity is low, the network has enough capacity to return to its original performances after fine-tuning. When the sparsity is too high, then all criteria produce networks with random predictions.
optimisation Figure 13 shows the impact of hhpehh-yaammnee parameter optimization for the fine-tuning phase. We performed grid search with three different learning rate (0.1, 0.01, 0.03)
15

---

Under review as a conference paper at ICLR 2021
89.3 (1) Sparsity
0.2
0.0
-0.2
-0.4
Volld
-0.6
0.06
0.1
0.2
0.3 0.4
AL 48) betore fine-tuning
Figure 12: Same as Figure 11 (left), but zoomed on smaller values of AC A0).
and three different 1-regularisation (0. 5c-4, 5e-5). All 9 sets of baper-parameters were tested on LM, QM and MP on different random seeds. In this set of experiments, we used X E {0, 0.01, 0.1,1) and æ E {14, 1403. Optimizing hyper- pppee-prraeetess for fine-tuning can lead to better performance after fine-tuning, but does not increases the correlation between the performances after fine-tuning and AC, A0). The lack of correlation can thus not explained by bad fine-tuning typer-parameters.
ProActResNet18 on CIFAR10
ProA.tResNet18 on CIFAR10
125
1.25
029
1.00
Vole. Ernor 15) alturt4t 1-t
elter
0.50
0.50
0.5
0.25
0.00
0.00
0=40
-025
-025
0.7
2110, 601 before fine-tuning
A110. 68) before fine-tuning
Figure 13: Left: Using the hyper-parameters for fine-tuning the of the original training. Right: Performing hyper-parameters optimisation for the fine-tuning. This figure shows that optimizing the hyper-parameters for fine-tuning can improve the performances of the network after pruning. However, it reduces the correlation between the performances after fine-tuning and A AC, A0). The lack of correlation can thus not be explained by poor fine-tuning hyper-parameters.
MP
0BD
LM
QM
PreActResNet18
0.8
1.50
2.4
0.6
1.25
Veldd (80)
Valld Error attrr-t (26)
Valld Error attrr.tt (96)
0.4
1.00
0.2
0.75
0.50
1.0
0.3 0.95
1.05 1.1 1.5 1.2533
0.5
0.6 0.7 0.8
0.
1.3
0.8
0.9
13
61 A69 before fine-tuning
A/ d110.000 Adl before fine-tuning
8/ ddll before fine-turing
Figure 14: Same as Figure 3, but zooming on the best performing networks in terms of AC(O, A0).
16

---

Under review as a conference paper at ICLR 2021
C.3 FINE-TUNING CURVES
To investigate whether one of the networks is suffering from optimization issues during fine-tuning, show in Figure 15 the fine-tuning curves of networks pruned using MP and our best QM criteria. We observe that, except for MNIST, the difference in training loss right after pruning disappears after only one epoch of fine-tuning, erasing the advantage of QM over MP.
MLP NIIST
VC011 CITAR10
Prokctiaalee00 OPAR12
HF ON
10
ml
MP
MF
19
04
0M
1.0
10a
10
10P
10
10
12
19a
10
Ecochs
Etechs
Ecochs
Figure 15: Fine-tuning losses (dotted is training, solid is validation) of networks pruned using MP and QM criteria. All the curves are the average over the 5 seeds. We not show the standard deviation for clarity. Left: MIP, middle: VGG11 and right: PreActResNet18 Except for MNIST, the difference in loss right after pruning (i.c. at epoch 0) disappears after one epoch of fine-tuning.
C.4 RESULTS USING GRASP AND SYNFLOW
We compare our results with two additional pruning methods that focus on preserving the flow of the gradient in the network instead of preserving the loss: GraSP (Wang et al., 2020), data- dependant method, and SynFlow (Tanaka et al., 2020), a data-agnostic one. Both methods were design to be applied at initialisation,
investigate here their use trained networks. We use E {1, 100, 1000), and added our proposed step size constraint À to the pruning criteria as well.
Figure 16 shows the scatter plot of the preservation of the loss vs the performance after fine-tuning. Similarly to what we observed before, there is no clear evidence that better preserving the loss lead to better performance after fine-tuning.
MP
O60
LM
Syntlow
Gra
VGG11 on CIFAR1O
2.5
PreAciResNeL18 CIFAR10
0.75
Valid Eror (%) after 1.t.
2.0
0.50
0.25
0.00
1.0
-0.25
Velid
0.5
-0.75
-1.00
-0.5
10"
30"
10"
AL(8, 48) tuning
AL50, 0.8) fine-tuning
Figure 16: Same Figure 3 showing GraSP and SynFlow VGG11 (left) and the PreActResnet18 (right). This Figure shows that one can observe a large AC A0) and yet obtain very good performance after fine-tuning. This is especially true in the case of GraSP for VGGIL. Furthermore, we can observe similar behaviour on PreActResNet18 where two different methods can lead to similar performance after fine-tuning while having completely different A A0): GraSP with AC AO) z 10th has fine-tuning performance similar to MP with AC A0) < 10¹
C.5 RESULTS ON IMAGENET
Figure 17 is the same as Figure 4, but with 90 & sparsity. At that sparsity level, the validation accuracy right after pruning is close random for all the pruning criteria. There is however quite big variation in performances after fine-tuning: at equal performance before fine-tuning, some models achieve 70 % validation accuracy after fine-tuning, while others only reach 60 %.
17

---

Under review as a conference paper at ICLR 2021
MP
0BD
LM
QM
Uneer Model
Quadratic Model
ResNet50 ImageNet
70
23.0 201
Volid Accuracy (96) afterr 1.2
68
10
107
10
10
Optimal Brain Damage
64
62
0.0
0.5
1.0
1.5
2.0 2.5
Valid, Accuracy (36) before (-A.
Figure 17: Same Figure 4, with 90 sparsity. Increasing the number of pruning stages and constraining the step size reduce AC(O, A0). However, the best-loss preserving criteria, which maximize the validation accuracy right after pruning, do not produce better networks after fine-tuning.
18


Reviews:
[{"review_id": "wZrzZC7c-de", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors study the use of loss-modeling to maintain model quality when inducing unstructured sparsity in deep neural networks. They study a range of different approximations and modifications that can help improve the quality of the approximation (taking local steps, avoid large changes in weight magnitude, avoiding assumptions about convergence). The authors conduct a thorough empirical investigation that yields practical observations for the design of future pruning techniques.\n\nPros:\n\nThe paper is well written and well organized and the empirical investigations are well done. The observations made by the authors are interesting and practically useful for the development of future pruning techniques. Namely,\n1. That including first order terms in loss approximations can relax the convergence assumption behind some existing loss modeling approaches to enable more flexible application of the pruning algorithm.\n2. The quality of the local loss approximation can be improved by taking a series of smaller pruning steps.\n3. That loss-preservation does not necessarily translate into accuracy preservation.\n\nCons:\n\nIt would be nice to see experiments in domains other than computer vision. For example, language modeling with RNNs or Transformers. Results at a wider range of sparsity levels for ImageNet would also have been useful, as it seems possible that these techniques could perform differently for high sparsity (>90%) than they do for moderate sparsity (e.g., the 70% sparsity reported in Figure 4).\n\nComments:\n\nAnother ICLR 2021 submission is highly relevant to your investigation: https://openreview.net/forum?id=rumv7QmLUue. Their theoretical/empirical results appear to corroborate your conclusions that loss preservation is not necessarily the best metric to optimize for when you care about accuracy preservation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mkMz61ZDztH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Although the paper is covering an interesting topic, much of what's in the paper can be found in other works, and there's not a lot of novelty to the insights, nor a large breadth of experiments to justify it as a survey paper.\n- The linear and quadratic loss functions are not new.\n- The enforcing locality part is essentially why the pruning strength annealing schemes exist, this insight is not new and can be found in Zhu&Gupta, or in Bayesian settings like in the Molchanov paper, they suggest annealing from a Bayesian statistical perspective\n- The fact that this leads to multiple stages of pruning to be a good idea, is also known in the literature this paper cites.\n- The most meat of the paper is in the 'survey' part of it, investigating the results... but this section feels lacking since there are not a lot of experiments, and the insights of e.g. post-finetuning can be largely found in e.g. Blalock et al. I'm missing deeper insights/analysis here. What is the reason for this? How do we remedy this? What are the characteristics of networks that lead to this behavior? \n- It would have been great to see a lot more insight/experiments on this topic. The authors throw up a lot of hypothesis and suggestions/ideas throughout the paper, but don't back them up. E.g. If the ||Theta|| term is better of to be constant throughout the pruning procedure... can we somehow make an annealing scheme that keeps ||Theta|| small and constant throughout the pruning process, and show that that works well? This can be proven/shown somehow. \n\nI do think the paper is well written; and I encourage the authors to look further into this topic and come up with more novel insights/results and methods to improve pruning\n\nOther things/questions/suggestions:\n- In formulations (1), (2) and (5), (6). Why are the absolute brackets necessary? Especially for models that have not converged, why would you want to stay close to the original loss, as opposed to just decreasing the overall loss of the model? \n- For most of the discussion in section 3.2, the authors talk about the norm of the delta theta squared being large. But this largeness is relative to the 0th and first order term which the authors glance over. Under 'other considerations' for example, if the weights theta are large, the gradients likely follow suit. Thus the absolute magnitude of the weights might not matter, as it's the relative size of this to the gradient terms that should be considered. \n- Constraining the step size in section 3.2. Interestingly, if you take the assumptions that for each layer, the hessian of the loss w.r.t. the output is sqrt(lambda/2), and your input distributions to a layer are normalized, you end up with the weaker norm penalty. This is a crude approximation of the second order term, which would give this method a bit more of a theoretical foundation than just a regularization term.\n- 5.1 Convergence assumption. I don't get this part, both the OBD method, and the linear and quadratic loss terms depend on the locality, so all will also depend on the amount of steps taken. For OBD, as long as you recalculate the Gauss-Newton matrix, I don't see why this method is different when not doing fine-tuning. \n- 5.1 Convergence assumption. The result cited in appendix A is a very well-know result. How could this link explain the OBD performance on the VGG network? 'Could' is not strong enough to make it into a paper. \n\nSmall tidbits:\n3. Do pruning criteria better at preserving the loss lead to better fine-tuned networks? <- this sentence doesn't flow nicely. I would add a 'that' so you have do pruning criteria that are better ...", "extra_scores": {}, "extra_text": {}}, {"review_id": "d1QycbMMAq4", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- This paper conducted a detailed study on how does the loss modeling affects the final performance of the pruned model. The authors first provided a unified view of various pruning algorithms (e.g., Magnitude Pruning, SNIP, OBD, and OBS), which can be categorized into three classes: weight magnitude, linear and quadratic models of the loss function. In the experiments, the authors seek to answer the questions: 1) how well do each criterion preserve the loss; 2) how does the locality assumption affect the final performance; and 3) how does the loss relate to the final performance? Empirical, the authors found that the quadratic model preserves the loss the best, as expected. Also, the loss after pruning seems not strongly correlated with the performance after fine-tuning.\n\nOverall:\n\nThis paper is well-written and easy to follow. The authors did a great job of unifying the analysis of several pruning algorithms. More importantly, revisiting the loss modeling of network pruning is interesting, and it might invoke further research efforts in better understanding the pruning techniques developed in the past and also inspire researchers in designing improved pruning algorithms. However, I still have the following questions:\n\n- The authors show that the loss after pruning does not correlate strongly with the accuracy after fine-tuning. In Figure 3, the change in the loss ranges from 0 to 5. Can you show the plot with a smaller range of the change in loss, e.g., 0~0.5? I believe a large change in loss means that the pruning results are very close to random, so the comparisons in this regime may not be meaningful.\n\n- For testing the locality assumption, you introduce an L2 penalty on the changes. To me, this is more like a weighted combination of the original pruning criteria and the magnitude pruning criteria. Why not using some other techniques, such as backtracking line search for determining the pruning ratio at each iteration? \n\n- In equation (5), why do we need to take the absolute value? I think preserving the loss is only meaningful when the network is converged. If the model is not converged, then it would be preferable to prune those weights whose removal will decrease the loss. In this sense, the sign of the loss change should not be ignored. \n\n- The third plot in the second row of Figure 1 shows that OBD with more iterations has a larger change in the loss. Do you have any explanation for this?\n\nRating:\n- I vote for a weak acceptance due to the above reasons. I believe the studied topic in this paper is important and impactful for the pruning community. In the meantime, it would be great if the author can propose some hypotheses on this phenomenon.  To me, preserving the loss is a way to enforce the pruned network to stay close to the original solution, and in this regime, it's easy for the optimization algorithm to find a good enough solution. I will raise my rating if the authors can address my concerns well during the rebuttal period.\n\n==========================After rebuttal=================================\n\nThanks very much for your efforts to address my concerns. I kept my score unchanged. I agree with most of the responses, except for the response to \"L2 penalty, backtracking line search for determining the pruning ratio\". This paper is not proposing a practical algorithm but a revisiting, so I don't think the computational cost is a bottleneck in preventing you from using more advanced methods to get more robust conclusions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mqtc21Q3HyK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops two modified version of Optimal Brain Damage (OBD) criteria, namely LM and QM (linear and quadratic model) to measure the importance/saliency of model weights. It then compares these three together with Magnitude Pruning (MP), and show that these among four criteria: 1. for the first three, using iterative pruning to enforce locality of the gradient calculation is important, 2. the best method for training loss before fine-tuning does not necessarily lead to best validation accuracy after fine-tuning. \n\nThe paper's empirical investigation is valuable and appreciated. It helps me understand OBD more throughly and the assumptions behind it. It is also useful to know that using iterative pruning can improve these gradient approximation-based methods because of locality.\n\n1. My primary concern is the experiments does not seem to lead to a useful guidance for future practice. The paper does conclude for the first three using iterative pruning is useful, but these three criteria are rarely used nowadays and MP is the mainstream, and the simplest method. The paper also didn't conclude which of the four criteria is in general best and recommended. From table 2 it seems to be LM but the paper did not conclude this way. This is also possibly due to that the experiments are not run extensively on different datasets and architectures. \n\n2. The paper compared the training loss before fine-tuning, and validation acc. after fine-tuning. But I think validation acc. before fine-tuning is also a quantity worth investigating. \n\n3. More importantly, the paper shows the training loss (before fine-tuning) and valid acc. (after fine-tuning) are not necessarily correlated, but did not give explanation on why this could be the case through experiments, or give useful suggestions to achieve a good valid acc. after fine-tuning.\n\nOverall I appreciate the empirical study but I suggest conducting the experiments on more datasets and architectures, and extract a useful conclusion to guide future practices.\n\n+++++++++++++++++\n\nI appreciate the clarified messages of the paper, and would like to see them emphasized more clearly in the next version of the paper. But due to the limited experimental scale on ImageNet (added in rebuttal, and in my understanding, it only verifies one of the multiple observations mentioned in the paper), I'm still leaning on rejection. I updated my score from 4 to 5.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a systematic breakdown and evaluation of several assumptions and algorithmic choices for pruning algorithms. As covered in the reviews, the evaluation and its conclusion offers a timely contribution to the broader community.

In particular, this paper uncovers the observation that precisely modeling the loss (and hence minimizing the drop in loss after pruning) may not in fact yield improvements in pruning. This is an important observation as the community continues to propose new techniques with the justification that their improved performance results from improved loss modeling.

A significant concern on the part of the reviewers is the limited practical prescription offered by the paper. Specifically, the paper does not propose a new algorithm. It also doesn’t necessarily identify why this interesting phenomenon emerges.  For example, to the latter, it doesn't articulate what features of the network or loss landscape is indicative of this property.

Ultimately, the decision for this paper is very challenging given the reviews. Whether or not a phenomena is interesting is an inherently subjective consideration. Moreover, without a clear technical prescription or path forward that can be evaluated on its merits, the reviews fall into two categories of either 1) those that --- by my estimation --- felt personally inspired by the work and 2) those that could not intuit the impact of the observation.  

A significant complication is that the narrative of the paper includes claims around addressing locality and convergence which, if not read with the understanding that contributions here are simply a synthesis of current work, appear as claims to novelty (when these techniques have no or limited novelty). This is a source of contention in at least one review.

Given this partition, my recommendation is Reject.

For future versions of this paper, I recommend that the authors narrow the claimed contributions to exclusively focus on the final observation that modeling the loss may not be as important as thought. 

The work in this paper on developing the ideas around convergence and locality can, instead, be cast as efforts to provide best available baselines for the topline claim.  I believe these changes will eliminate a significant source of distraction, enabling readers (and reviewers) to avoid any attempt to evaluate the novelty of  the locality and convergence narratives, which have indeed been considered in other work in various ways.

An additional step that I highly recommend for this paper to unambiguously clear the bar is to identify with what the performance of pruning does correlate. Appendix C.4 provides an evaluation of two recent gradient preservation methods. Unfortunately, the paper did not present if, instead, the preservation of the gradient correlated with additional performance. 

In essence, the paper need not solve the mystery by providing a SoTA algorithm that exploits the right features of the problem for pruning. However, it would be valuable to provide a roadmap for future directions along with an articulation of the challenges down those directions.

Decision: Reject