A Gradient Flow Framework For Analyzing Network Pruning
Authors: Ekdeep Singh Lubana
Robert P. Dick
Venue: ICLR 2021 Spotlight (2021)
Keywords: Network pruning
Gradient flow
Early pruning

Abstract: Recent network pruning methods focus on pruning models early-on in training. To estimate the impact of removing a parameter, these methods use importance measures that were originally designed to prune trained models. Despite lacking justification for their use early-on in training, such measures result in surprisingly low accuracy loss. To better explain this behavior, we develop a general framework that uses gradient flow to unify state-of-the-art importance measures through the norm of model parameters. We use this framework to determine the relationship between pruning measures and evolution of model parameters, establishing several results related to pruning models early-on in training: (i) magnitude-based pruning removes parameters that contribute least to reduction in loss, resulting in models that converge faster than magnitude-agnostic methods; (ii) loss-preservation based pruning preserves first-order model evolution dynamics and its use is therefore justified for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gradient norm via pruning can produce poorly performing models. We validate our claims on several VGG-13, MobileNet-V1, and ResNet-56 models trained on CIFAR-10/CIFAR-100.

Published as a conference paper at ICLR 2021
A GRADIENT FLOW FRAMEWORK FOR ANALYZING NETWORK PRUNING
Ekdeep Singh Lubana & Robert P. Dick EECS Department University of Michigan Ann Arbor, MI 48105, USA (eslubana, dickrp) @umich. edu
ABSTRACT
Recent network pruning methods focus on pruning models early-on in training. To estimate the impact of removing parameter, these methods use importance measures that were originally designed to prune trained models. Despite lacking justification for their early-on in training, such measures result in surprisingly low accuracy loss. To better explain this behavior, we develop a general framework that uses gradient flow unify state-of-the-ee importance measures through the norm of model parameters. We use this framework to determine the relationship between pruning measures and evolution of model parameters. establishing several results related to pruning models early-on in training: (i) magnitude- based pruning removes parameters that contribute least to reduction in loss, resulting in mod- els that converge faster than magnitude-agnosst based pruning preserves first-order model evolution dynamics and its use is there- fore justified for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gra- dient norm via pruning produce poorly performing models. We validate our claims on several VGG- 13, MobileNet-V. VI, and ResNet-56 models trained on CUFAR-TO/CIFAR-10M
methods; (ii) poss-preservation
INTRODUCTION
The use of Deep Neural Networks (DNNs) in intelligent edge systems has been enabled by extensive research model compression. "Pruning" techniques are commonly used to remove "unimportant" filters to either preserve or promote specific, desirable model properties. Most pruning methods were originally designed to compress trained models, with the goal of reducing inference costs only. For example, Li et al. (2017); He et al. (2018) proposed to remove filters with small f1/82 norm, thus ensuring minimal change in model output. Molchanov al. (2017; 2019); Theis et al. (2018) proposed to preserve the loss of a model, generally using Taylor expansions around a filter's parameters estimate change in loss function of its removal.
Recent works focus on pruning models at initialization (Lee et al. (2019, 2020)) or after minimal training (You al. (2020)), thus enabling reduction in both inference and training costs. To estimate the impact of removing parameter, these methods use the same importance measures as designed for pruning trained models. Since such measures focus on preserving model outputs or loss, Wang et al. (2020) argue they are not well-motivated for pruning models early-on in training. However, in this paper, we demonstrate that if the relationship between importance measures used for pruning trained models and the evolution of model parameters is established, their use early-on in training can be better justified.
In particular, we employ gradient flow to develop general framework that relates state-of-the art importance measures used in network pruning through the norm of model parameters. This framework establishes the relationship between regularly used importance measures and the evolution of a model's 's parameters, thus demonstrating why measures designed to prune trained models also
gradient flow refers gradient descent with infinitesimal learning rate; Equation 6 for short primer.

---

Published as a conference paper at ICLR 2021
perform well early-on in training. More generally, framework enables better understanding of what properties make a parameter dispensable according to a particular importance measure. Our findings follow. (i) Magnitude-basee pruning measures remove parameters that contribute least to reduction in loss. This enables magnitude-base pruned models to achieve faster convergence than magnindee-gnnotic measures. (ii) Loss-preservee
based measures remove parameters with the least tendency to change, thus preserving first-order model evolution dynamics. This shows the use of pessepreservation is justified for pruning models early-on training as well. (iii) Gradient-norm based pruning is linearly related to second-order model evolution dynamics. Increasing gradient norm via pruning for even slightly trained models can permanently damage earlier layers, producing poorly performing architectures. This behavior is result of aggressively pruning filters that maximally increase model loss. We validate our claims on several VGG-13, MobileNet-Vi and ResNet-56 models trained on CIFAR-10 and CIFAR-100.
2 RELATED WORK
Several pruning frameworks define importance measures to estimate the impact of removing parameter. Most popular importance measures are based on parameter magnitude (Li et al. (2017); He
et al. (2018), Liu et al. (2017)) or loss preservation (Molchanov et al. (2019, 2017); Theis et al. (2018); Gao et al. (2019)). Recent works show that using these measures, models pruned at initialization (Lee et al. (2019); Wang et al. (2020); Hayou et al. (2021)) or after minimal training You et al. (2020)) achieve final performance similar to the original networks. Since measures for pruning trained models motivated by output loss preservation, Wang al. (2020) argue they may not be well suited for pruning models early-on in training. They thus propose GraSP, a measure that promotes preservation of parameters that increase the gradient norm.
Despite its success, the foundations of network pruning are not well understood. Recent work has shown that good "subnetworks" that achieve similar performance to the original network exist within both trained (Ye et al. (2020)) and untrained models (Frankle & Carbin (2019); Malach et al. (2020); Pensia al. (2020)). These works thus prove networks can be pruned without loss in performance, but do not indicate how network should be pruned, i.e, which importance measures are preferable. In fact, Liu et al. (2019) show reinitializing pruned models before retraining rarely affects their performance, indicating the consequential differences among importance measures are in the properties of architectures they produce. Since different importance measures perform differently (see Appendix E), analyzing popular measures to determine which model properties they tend to preserve can reveal which measures lead to better- seaterrperforming architectures.
From an implementation standpoint, pruming approaches can be placed in two categories. The first, structured pruning (Li et al. (2017); He et al. (2018); Liu et al. (2017); Molchanov et al. (2019, 2017); Gao et al. (2019)), removes entire filters, thus preserving structural regularity and directly improving execution efficiency on commodity hardware platforms. The second, unstructured pruning (Han et al. (2016b); LeCun al. (1990); Hassibi & Stork (1993)), is more fine-grained, operating at the level of individual parameters instead of filters. Unstructured pruning has recently been used to reduce computational complexity as well, but requires specially designed hardware (Han et al. (2016a)) or software (Elsen et al. (2020)) that are capable of accelerating sparse operations. By clarifying benefits and pitfalls of popular importance measures, our work aims to ensure practitioners are better able to make informed choices for reducing DNN training/inference expenditure via network pruning. Thus, while results in this paper are applicable in both structured and unstructured settings, our experimental evaluation primarily focuses on structured pruning early-on in training. Results on unstructured pruning are relegated to Appendix H.
PRELIMINARIES: CLASSES OF STANDARD IMPORTANCE MEASURES
In this section, we review the most successful classes of importance measures for network pruning. These measures will be our focus in subsequent sections. We use bold symbols to denote vectors and italicize scalar variables. Consider a model that is parameterized as (()) at time t. We denote the gradient of the loss with respect to model parameters at time g( (t)),, the Hessian as H (t))),, and the model loss as (()).)). A general model parameter is denoted as (tt). The importance of a set of parameters , (1) is denoted ((),
2

---

Published as a conference paper at ICLR 2021
Magnitude-based measures: Both f1 norm (Li et al. (2017)) and (2 norm (He et al. (2018)) have been successfully used as magnitude-focee focused importance measures and generally perform equally well. Due to its differentiability, f2 norm can be analyzed using gradient flow and will be our focus in the following sections.
I())(t)) (t)l Σ (O,(t))? O
poss-preservation based measures: These measures generally use first-order Taylor decomposi- tion to determine the impact removing a set of parameters has on model loss. Most recent methods (Molchanov et al. (2019; 2017); Ding et al. (2019); Theis et al. (2018)) for pruning trained models are variants of this method, often using additional heuristics improve their performance.
L (t 0, L (t) = - (t)
The equation above implies that the loss of pruned model higher (lower) than the original model if parameters with negative (positive) value for e (t)g (t)) are removed. Thus, for preserving model loss, the following importance score should be used.
(()) = 6 ())
Increase in gradient-norm based measures: Wang al. (2020) argue pos-preseevvatio based methods are not well-motivated for pruning models early-on in training. They thus propose GraSP, an importance measure that prunes parameters whose removal increases the gradient norm and can enable fast convergence for pruned model.
[|g (@(t) - (2) e llg (o(+))l (t)((()))))
The above equation implies that the gradient norm of a pruned model is higher than the original model if parameters with negative value for (t)))))))) are removed. This results in the following importance score.
(t), ()))))())))))()
(5)
As mentioned before, these importance measures were introduced for pruning trained models (except for GraSP), but are also used for pruning models early-on in training. In the following sections, we revisit the original goals for these measures, establish their relationship with evolution of model parameters over time, and provide clear justifications for their use carly-on in training.
GRADIENT FLOW AND NETWORK PRUNING
Gradient flow, or gradient descent with infinitesimal learning rate, is continuous-time version of gradient descent. The evolution over time of model parameters, gradient, and loss under gradient flow can be described as follows.
a0(t) g(O(t)):
(Parameters over time)
Øt
@g(@(t))
(Gradient over time)
H(@(())g(O(t);
Øt
OL(() = llg(
(Loss over time)
Øt
Recall that standard importance measures based on loss posserreservatio (Equation 3) or increase in gradient-nome (Equation 5) are derived using first-order Taylor series approximation, making them exactly valid under the continuous scenario of gradient flow. This indicates analyzing the evolution of model parameters via gradient flow can provide useful insights into the relationships between different importance measures. To this end, we use gradient flow to develop a general framework that relates different classes of importance measures through the norm of model parameters. As we develop this framework, we explain the reasons why importance measures defined for pruning trained models also highly effective when used for pruning early-on in training. In Appendix B we further extend this framework to models trained using stochastic gradient descent, showing that up first-order approximation, the observations developed here are valid under SGD training too.
3

---

Published as a conference paper at ICLR 2021
Magnitude
Proposed
Loss-pres.
Test
Train
100
95
Accuracy
90
80
80
65
70
50
50
100
150
0
50
100
150
Epoch
Epoch
Figure I: Train/test accuracy curves for pruned ResNet-56 models on CIFAR-10 (left) and CIFAR- 100 (right) over 25 rounds. Models pruned using magnitudebassed pruning (Magniude)), the proposed extension to loss preservation (Proposed), and loss loss-preservation based pruning (Loss-pres.) Magnitude- pruning converges fastest, followed by the proposed measure. Curves for other models and number of rounds are shown in Appendix F
4.1 GRADIENT FLOW AND MAGNITUDE-BAN PRUNING
We first analyze the evolution of the f2 norm of model parameters, a magnitude-based pruning measure, function of time under gradient flow. For model initialized @ denote parameters at time I as (T)) Under this notation, the distance from initialization can be related to model loss follows:
2
(0 0
(a)
(T)) (00) =
dt
g(O
S T
(l(((tt)
(b) T
aL(t)
dt T (L(O) L(T))
Ōt
T - < [(0) - I(T):
where (a) follows from the application of triangle inequality for norms and the Cauchy-Schwar inequality and (b) follows from Equation 6, The inequality above a modification a previously known result by Nagarajan & Kolter (2017), who show that change in model parameters, measured by distance from initialization, is bounded by the ratio of loss at initialization to the norm of the gradient at time T. Frequently used initialization techniques (Glorot Bengio (2010); He et al. (2015)). This reduces the impact of initialization, making |O(T)) ( =
zero-centered with small variance
(T))) In fact, calculating the importance of filters in model using /2 norm versus using distance from initialization, we find the two measures have an average correlation of 0.994 across training epochs, and generally prune identical parameters. We now use Equation 7 to relate magnitude-bane pruning with model loss any given time. Specifically, reorganizing Equation 7 yields
1 | e -100) L(O) 1 |
L(T) S L(0)
(8)
Based on this analysis, make the following observation.
Observation The larger the magnitude of parameters at a particular instant, the smaller the model loss at that instant will be. If these large-magnitude parameters are preserved while pruning (instead of smaller ones), the pruned model's 's loss decreases faster.
Equation 8 shows that the norm of model parameters bounds model loss at any given time. Thus, by preserving large-magu mnggemman parameters, magnitude- than seggnittud-agnnstte techniques. For example, later show that loss posserreservttoon based pruning removes the most slowly changing parameters, regardless of their magnitude. Thus magnitude- based pruned models will generally converge faster than loss- possprreservation based pruned models.
pruning enables faster reduction in loss
To verify this analysis also holds under the SGD training used in practice, we train several VGG-13, MobileNet-V V1, and ResNet-56 models CIFAR-10 and CIFAR-100. Our pruning setup starts with
4

---

Published as a conference paper at ICLR 2021
Table Accuracy of pruned models on CIFAR-IO/CIEAR-ION (over 3 seeds; base model accuracies are reported in parentheses; std. dev. < 0.3 for all experiments; best results are in bold, second best are underlined). Magnitude-ban based pruning (Mag.) and the proposed extension to loss preservation (Proposed: Zo.ee, (0)()))),))) (0:())) consistently outperform plain loss-preservation based
pruning (Loss). With more rounds, the proposed measure outperforms Magnitude-basee pruning too.
Pruning Rounds
round
rounds
rounds
CIFAR-10 10
pruned
Mag.
Loss
Proposed
Mag.
Loss
Proposed
Mag.
Loss
Proposed
VGG 13 (93.1)
75%
92.05
92.01
92.13
92.32
91.43
92.29
92.06
91.53
92.45
MohileNerr (92.3)
75%
91.71
91.17
91.73
91.76
90.99
91.52
90.96
91.77
ResNet-56 (93.1)
60%
91.11
91,09
91.54
80.80
91.39
91.88
91.95
91.47
92.17
CIFAR-100 100
% pruned
Mag.
Loss
Proposed
Mag.
Loss
Proposed
Mag.
Loss
Proposed
VGG-13 (69.6)
65%
67.89
68.61
69.01
69.31
67.88
68.25
68.31
68.11
68.93
MobileNet (69.1)
65%
68.01
67.16
68.52
68.33
67.21
68.41
67.58
67.31
68.35
ResNet-56 56 (71.0)
50%
66.92
66.88
67.10
68.04
67.11
67.70
68.75
67.74
68.62
randomly initialized models and uses a prune-and-train framework, where each round of pruning involves epoch of training followed by pruning. A target amount of pruning (e.g., 75% filters) divided evenly over a given number of pruning rounds. Throughout training, we use small temperature value (= 5) to ensure smooth changes to model parameters. We provide results for 1, 5, and 25 rounds for all models and datasets. The results after round, where pruning is single-shot, demonstrate that our claims are general and not an artifact of allowing the model compensate for its lost parameters; the results after and 25 rounds, where pruning is distributed over number of rounds, demonstrate that our claims also hold when models compensate for lost parameters.
The results are shown in Table 1 Magsitude-based pruning consistently performs better than loss- preservation based pruning. Furthermore, train/test convergence for magnitude-based pruned models is faster than that for loss coss-preservation based pruned models, as shown in Figure 1, These results validate our claim that magnitude-basee pruning results in faster converging, better- setter-performing models when pruning early-on in training.
Extending existing importance measures: fundamental understanding of existing importance measures can be exploited to extend and design new measures for pruming models early-on in training.
For example, we modify loss-pue pesseeeesrvvaion based pruning remove magiitue parameters that do not affect model loss using the following importance measure: Eg,e, (:(()) (0:(1)) J.
Using (t)g (0.( to preserve loss, this measure retains training progress up to the current instant; using (t)l. this measure is biased towards removing small- maggitue parameters and should produce a model that converges faster than loss poss-preservation alone. These expected properties are demonstrated in Table I| and Figure 1 the proposed measure consistently outperforms loss- preservation based pruning and often outperforms magnitude- rounds are used (see Table 1); train/test convergence rate for models pruned using this measure are better than those for loss- preseervevvvnnn pruning, and competitive to those of magnitude-bast pruning (see Figure 1).
pruning, especially when more
4.2 GRADIENT FLOW AND LOSS-PRESERVATION BASED PRUNING
Loss- poss-preservation based pruning methods use first-order Taylor decomposition to determine the impact of pruning model loss (see Equation 3). We show that magnitude-basel and loss- preservation based pruning measures are related by an order of time-derivative.
a (1((t)) 20 (0(1)
Øt
Øt
(9)
20²
where (a) follows from Equation 6 This implies that
@ ( 2 (1
(10)
Øt
Based on Equation 10, the following observations be made.
5

---

Published as a conference paper at ICLR 2021
1.0
1.0
1.0
0.6-
Comrectionn Hask Distance
Concaction Mark Dirtance
0.4
Mask Distance
Correlation
0.11
Hask Distance
Gomelation
0.5
0.2-
0.5
0.25
0.5
0.0
0
0-yy
50
100
150
100
150
150
Epoch
Epoch
Epoch
(a) VGG-13.
(b) MobileNet-V1.
(c) ResNect-6.
Figure 2: Correlation between Ao and loss pess-preservation based importance (see Equation 3) at every 10th epoch. Also plotted is the distance between pruning masks (target ratio: 20% filters), as used by You et al. (2020) to decide when to prune a model. As the distance between pruning masks over consecutive epochs reduces, |o becomes more correlated with oss-preseervttin importance. Results for a similar experiment on Tiny-ImageNet are shown in Appendix I and in an unstructured pruning setting are shown in Section H.I
Observation 2: Up to a constant, the magnitude of time-derivative of norm of model parameters (the score for magnitude-bast based pruning) is equal to the importance measure used for loss-preservation Equution 3). Further, loss-preservation corresponds to removal of the slowest changing parameters ?
Equation 10 implies that loss-preservation based pruning preserves more than model loss: It also pre- serves the first-order dynamics of a model's evolution. This resuli demonstrates that loxs-preservation based importance is in fact a well-motivateed measure for pruning models early-on in training. This explains why loss pess-preservation based pruning has been successfully used for pruning both trained (Molchanov et al. (2017; 2019)) and untrained (Lee et al. (2019)) models. We also draw another important conclusion pertaining to pruning of trained models: parameters that continue to change more important to retain for preserving model loss, while parameters that have the least tendency to change are dispensable. We highlight the relationship in Equation 10 was also noticed by Tanaka al. (2020), but they do not study its implications in detail.
Observation 3: Due to their closely related nature, when used with additional heuristics, magnitude- based importance measures preserve loss.
Pruning methods often use additional heuristics with existing importance measures to improve their efficacy. E.g., You al. (2020) recently showed that models become amenable to pruning early-on in training. To determine how much to train before pruning. they create a binary mask to represent filters that have low importance and are thus marked for pruning. filter's importance defined using the magnitude of its corresponding BatchNorm- parameter. If change in this binary mask is minimal over a predefined number of consecutive epochs, training is stopped and the model is pruned.
Due to the related nature of magnitude- based and Joss-preservation based measures, as shown in Equation 10, we expect use of additional heuristics can unknowingly extend an importance measure to preserve model properties that it was not intentionally designed to target. To demonstrate this concretely, we analyze the implications of the "train until minimal change" heuristic by You et al., as explained above. In particular, consider a certain BatchNorm-scale parameter that has small magnitude and is thus marked for pruning. Its magnitude must continue to remain small to ensure the binary mask does not change. This implies the change in over an epoch, or Ao, must be small. These two constraints can be assembled in a single measure: la Ao The contimuous-time at As shown in Equation 9 and Equation 10, lo do is
per-iteration version of this product is |o Oo the mathematical equivalent of poseprresevvation based importance (Equation 3). Therefore, when applied per iteration, removing parameters with small magnitude, under the additional heuristic of minimal change, converts magritude-based pruning into a loss-preservation strategy. If parameters do not change much over several iterations, this argument can further extend to change over an epoch. To confirm this, we train VGG-13, 13, MobileNet-Vil and ResNet-56 models on CIFAR- 100. record the distance between pruning masks for consecutive epochs, and calculate correlation between posserreservation based importance and la Ag). As shown in Figure 2, as the mask distance reduces,
at
This observation also implies loss preservation's importance measure does not depend magnitude of parameters directly. We thus call loss-preservation magnitude-agnostic technique.

---

Published as a conference paper at ICLR 2021
Correlation 0.85
Correlation 0.88
Correlation 0.72
0.04
0.5
HHO)
0.00
-0.5
-0.4
-0.04
-0.01
0.01
-0.03
0.00
0.03
-0.004
0.000
0.004
g(0)
0
0, + g(o)
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
versus O (O) for ResNet-56 models trained on CIFAR- 100. Plots are shown for filters (a) at initialization, (b) after 40 epochs training, and (c) after complete (160 epochs) training. The correlation is averaged over seeds and plots are for seed. As shown, the measures are highly correlated throughout model training, indicating gradient-non norm increase may severely affect model loss if a partially completely trained model is pruned using ( (O)()) (()).. Plots for other models are shown Appendix G Results of this experiment an unstructured pruning setting are shown in Section H.l
Figure 3: O7
the correlation between the importance measures increases. This confirms that due the intimate relationship between magnitude-basee pruning and loss- loss-preservation based pruning, when used with additional heuristics, magnitude-basee pruning extends to preserve model loss as well.
4.3 GRADIENT FLOW AND GRADIENT-NOwE BASED PRUNING
Having demonstrated the relationship between the first-order time derivative of the norm of model parameters and loss-preservation based pruning, we consider the second-order time derivative of the norm of model parameters. We demonstrate its connection with GraSP (Wang et al. (2020)), method designed for use early-on training increase gradient-norm using pruning.
8 (0) 0t2
a (10(112)
a (OTT(t)) (t)g((())))
(a)
Ot
Ot
Ot
= 2 g (0(1) (t(t) + (T(())
0g (g)))
(1I)
at
bb) (Ig(0 +*((H((())(()))))
where (a) follows from Equation 9 and (b) follows from Equation This implies that
8 10(1) (IL((())) @ ((H(((()))()))))
(12)
012
Equation 12 shows the second-order time derivative of the norm of model parameters linearly related to the importance measure for increase in gradient-norm (Equation 5). In particular, parameters with negative O (())))) (t)(t) reduce the "acceleration" at which a model approaches its final solution. Thus, removing them can speedup optimization.
We now use the analysis above to demonstrate limitations to increase of gradient-nome norm using pruning.
Observation 4: Increasing gradient-norm via pruning removes parameters that maximally increase model loss.
Extending the "acceleration" analogy, recall that if an object has increasingly negative velocity, its acceleration will be negative as well. As shown before in Equation 9 the velocity of norm of model parameters is constant multiple of the first-order Taylor decomposition for model loss around current parameter values. Thus, the parameter with the most negative value for O ()) (()) is likely to also have large, negative value for (O()))(()))())))
From Equation 2 we observe that removing parameters with negative O p (0g O (1)) increases model loss with respect the original
7

---

Published as a conference paper at ICLR 2021
Table 2: Accuracy of models pruned using different variants of gradient-norm based pruning on CIFAR- 10/CO/////RROOOO (over 3 seeds; base model accuracies are reported in parentheses; std. dev. < 0.3 for all models, except for GraSP (T=1) with 5/25 rounds (std. dev. < 2.1); best results are in bold, second best are underlined). Gradient-norm preservation ( ([GraSP (T=1))) outperforms both GraSP with large temperature (T=200) and without temperature (T=I).
Pruning Rounds
I round
5 rounds
25 rounds
CIFAR- 10
pruned (%)
GraSP (T-200)
GraSP (T-1)
[GraSP (T=1)
GraSP (T=200)
GroSP (T=1)
[GraSP] (T=1)
GraSP (T=200)
GraSP (T=1)
GraSP (T=1)
VGG- 13 (93.1)
75%
91.62
91.32
91.92
90.46
83.77
91.83
89,57
77.12
91.75
MobileNet (92.3)
75%
89.46
90.03
91.03
86.88
80.95
90.89
80.37
76.06
91.01
Re (93.1)
60%
91.01
90.81
91.21
90.44
80.15
91.25
86.23
10.00
91.63
C10
pruned
GraSP
GraSP (T=1)
GrrSPP (T=1)
GraSP
GraSP (T=1)
GraSP (T=1)
GraSP
GraSP (T=1)
[GSPP]
(%)
(T=200)
(T=200)
(T=200)
(T=1)
VGG-13 (69.6)
65%
68.51
65.79
68.64
66.40
$4.56
68.07
65.18
42.83
68.13
MobileNee (69.1)
65%.
64.52
64.79
67.17
63.02
53.35
67.28
59.14
46.41
67.69
ResNet-56 (71.0)
50%
67.09
67.02
67.15
66.97
52.98
67.01
57.55
1.00
68.02
model. This indicates T parameters that maximally increase model loss.
may in fact increase the gradient norm by removing
To test this claim, we plot @ (())))))((())))))) alongside (())(((()) for filters of VGG-13, MobileNet-VI, and ResNet-56 models trained CIFAR-100 at various points in training and consistently find them to be highly correlated (see Figure 3). This strong correlation confirms that using O p (()) as an importance measure for network pruning increases gradient norm by removing parameters that maximally increase model loss.
Wang al. (2020) remark in their work that it is possible that GraSP may increase the gradient-norm by increasing model loss. We provide evidence and rationale illuminating this remark: we show why preserving loss and increasing gradient-norm are antithetical. To avoid this behavior, Wang al. propose to use large temperature value (200) before calculating the Hessian-gradient product. We now demonstrate the pitfalls of this solution and propose a more robust approach.
Observation 5: Preserving gradient-now norm maintains second-order model evolution dynamics and results in perfer--erforming models than increasing gradient-nore norm.
Equation 12 shows the measure 6 (L)H(O(())) ((()) affects a model's second-order evolution dy- namics. The analysis in Observation 4 shows 6 (())(((()) and O (O)g(((() are highly correlated. These results together imply that preserving gradient-norm ( p ())))))))()))))))) should preserve both second-order model evolution dynamics and model loss. Since the correlation with loss ceess--reservatiin based importance is not perfect, this strategy is only an approximation of loss-preservation. However, it is capable of avoiding increase in model loss due to pruning.
To demonstrate the efficacy of gradient-norm preservation and the effects of increase in model loss on GraSP, we prune VGG-13, MobileNet-Vi V1, and ResNet-56 models trained on CIFAR-10 10 and CIFAR-100. The results are shown in Table 2 and lead to the following conclusions. (i) When single round of pruning is performed, the accuracy of GraSP is essentially independent of temperature. This implies that using temperature may be unnecessary if the model is close to initialization. (ii) In a few epochs of training. when reduction in loss can be attributed to training of carlier layers (Raghu et al. (2017)), GraSP without large temperature chooses to prune earlier layers aggressively (see Section G.2 for layer-wise pruning ratios). This permanently damages the model and thus the accuracy of low-temperature GraSP decreases substantially with increasing training epochs. (iii) At high temperatures, the performance of pruned models is more robust to the number of rounds and pruning of earlier layers is reduced. However, reduction in accuracy is still significant. (iv) These behaviors are mitigated by using the proposed alternative: gradient- nrom preservation (t) gg( @ (t) Since this measure preserves the gradient-norm and is correlated to loss- ress-preeervaion. focuses on ensuring the model's 's second-order training dynamics and loss remain the same, consistently resulting in the best performance.
8

---

Published as a conference paper at ICLR 2021
5 CONCLUSION
In this paper, we revisit importance measures designed for pruning trained models to better justify their use early-on in training. Developing a general framework that relates these measures through the norm of model parameters, we analyze what properties of parameter make it more dispensable according to each measure. This enables us to show that from the lens model evolution, use of magnitude- and poss-preservation based measures is well-justified early-on in training. More specifically, by preserving parameters that enable fast convergence, magnitude- based pruning generally outperforms mngnitude-agnostee methods. By removing parameters that have the least tendency to change, loss- preservation based pruning preserves first-order model evolution dynamics and is well-justified for pruning models early-on in training. We also explore implications of the intimate relationship pruning and loss- coss-preservation based pruning. demonstrating that one can evolve into the other training proceeds. Finally, we analyze gradient-norm based pruning and show that it is linearly related to second-order model evolution dynamics. Due to this relationship, we find that increasing gradient- norm via pruning corresponds to removing parameters that maximally increase model loss. Since such parameters are concentrated in the initial layers early-on on in training, this method can permanently damage a model's initial layers and undermine its ability to learn from later training. To mitigate this problem, we show the more robust approach is to prune parameters that preserve gradient norm, thus preserving a model's second-order evolution dynamics while pruning. In conclusion, our work shows the use of an importance measure for pruning models early-on in training difficult to justify unless the measure's 's relationship with the evolution of a model's s parameters over time is established. More generally, we believe importance measures that specifically focus pruning early-on should be directly motivated by a model's 's evolution dynamics.
between magnitude-basse
6 ACKNOWLEDGEMENTS
We thank Puja Trivedi, Daniel Kunin, Hidenori Tanaka, and Chaoqi Wang for several helpful discussions during the course of this project. We also thank Puja Trivedi and Karan Desai for providing useful comments in the drafting of this paper.
REFERENCES
X. Ding, G. Ding, X. Zhou, Y. Guo, J. Han, and Liu. Global Sparse Momentum SGD for Pruning Very Deep Neural Networks. In Proc. Adv. in Neural Information Processing Systems, 2019.
E. Elsen, M. Dukhan, T. Gale, and K. Simonyan. Fast Sparse ConvNets. In Proc. Int. Conf. on Computer Vision and Pattern Recognition, 2020.
J. Frankle and M. Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.
In Proc. Int. Conf. Learning Representations, 2019.
W. Gao, Y. Liu, C. Wang, and S. Oh. Rate Distortion For Model Compression: From Theory To Practice. In Proc. Ini. Conf. on Machine Learning, 2019.
X. Glorot and Y. Bengio. Understanding the Difficulty of Training Deep Feedforward Neural Networks. In Proc. Int. Conf. on Artificial Intelligecee and Statistics, 2010.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and W. J. Dally. EIE: Efficient Inference Engine on Compressed Deep Neural Network. Proc. Int. Symp. on Computer Architecture, 2016a.
Han, J. Pool, Tran, and W. Dally. Learning Both Weights and Connections For Efficient Neural Networks, In Proc. Adv. in Neural Information Processing Systems, 2016b.
B. Hassibi and D. G. Stork. Second Order Derivatives For Network Pruning: Optimal Brain Surgeon. In Proc. Adv. in Neural Information Processing Systems, 1993.
S. Hayou, J. Ton, A. Doucet, and Y. W. Teh. Robust Pruming at Initialization. In Proc. Int. Conf. on Learning Representations, 2021.

---

Published as a conference paper at ICLR 2021
K. He, X. Zhang, S. Ren, and J. Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proc. Int. Conf. on Computer Vision, 2015.
Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft Filter Pruning For Accelerating Deep Convolutional Neural Networks. In Proc. Int. Joint Conf. on Artificial Intelligence, 2018.
G. Hinton, O. Vinyals, and J. Dean, Distilling the Knowledge in a Neural Network. In NeurIPS Deep Learning and Representation Learning Workshop, 2015.
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal Brain Damage. In Proc. Adv. in Neural Information Processing Systems, 1990.
N. Lee, T. Ajanthan, and P. Torr. SNIP: SNIP: Single-Shot Network Pruning Based on Connection Sensitivity. In Proc. Int. Conf. Learning Representations, 2019.
N. Lee, T. Ajanthan, Gould, and P. Torr. A Signal Propagation Perspective For Pruning Neural Networks at Initialization. In Proc. Int. Conf. on Learning Representations, 2020.
H. Li, A. Kadav, L. Durdanovic, H. Samet, and H. P. Graf. Pruning Filters For Efficient ConvNets. In Proc. Int. Conf. on Learning Representations, 2017.
Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning Efficient Convolutional Networks Through Network Slimming. In Proc. Int. Conf. on Computer Vision, 2017.
Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the Value of Network Pruning. In Proc. Int. Conf. on Learning Representations, 2019.
E. Malach, G. Yehudai, S. Shalev-shwartz, and O. Shamir. Proving the Lottery Ticket Hypothesis: Pruning is All You Need. Proc. Int. Conf. on Machine Learning, 2020.
Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning Convolutional Neural Networks For Resource Efficient Inference. In Proc. Int. Conf. on Learning Representations, 2017.
Molchanov, A. Mallya, Tyree, I. Frosio, and Kautz. Importance Estimation For Neural Network Pruning. In Proc. Int. Conf. on Computer Vision and Pattern Recognition, 2019.
V. Nagarajan and 7. Kolter. Generalization in Deep Networks: The Role of Distance from Initializa- tion. In Workshop on Deep Learning: Bridging Theory and Practice, 2017.
A. Pensia, S. Rajput, A. Nagle, H. Vishwakarma, and D. Papailiopoulos. Optimal Lottery Tickets via Subset Sum: Logarithmic PaeeeParramttiizzationnn is Sufficient. In Proc. Adv. in Neural Information Processing Systems, 2020.
M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis For Deep Learning Dynamics and Interpretability. Proc. Ady. in Neural Information Processing Systems, 2017.
H. Tanaka, D. Kunin, D. Yamins, and S. Ganguli. Pruning Neural Networks Without Any Data by Iteratively Conserving Synaptic Flow. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2020.
L. Theis, I. Korshunova, A. Tejani, and F. Huszár. Faster Gaze Prediction with Dense Networks and Fisher Pruning. In Proc. Euro. Conf. on Computer Vision, 2018
C. Wang, G. Zhang, and R. Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. Proc. Int. Conf. on Learning Representations, 2020.
M. Ye, C. Gong, L. Nie, D. Zhou, A. Klivans, and Q. Liu. Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection. In Proc. Int. Conf. on Machine Learning, 2020.
H. You, C. Li. P. Xu, Y. Fu. Y. Wang, X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin. Drawing Barly-Bird Tickets: Towards More Efficient Training of Deep Networks. In Proc. Int. Conf. on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2021
A
ORGANIZATION
The appendix is organized follows:
Appendix B
Extends our gradient flow framework towards SGD based training.
Appendix C
Details the setup for training base models.
Appendix D
Details the setup for training pruned models.
Appendix
Random/Uniform pruning results CIFAR-100.
Appendix F:
Train/Test curves for magnitude-basee based and loss poss-preservation based pruned
models.
Appendix G More Results Gradient-norm Based Pruning-
Section G.I Scatter plots demonstrating correlation between (()
and p (t)g(((())
Section G.2 Provides layer-wise pruning ratios for models pruned using gradient-norm based pruning variants.
Section G.3;
Train/Test curves for gradient-norm based variants used the main paper.
Appendix H Provides empirical verification for observations 2-4 in unstructured pruning setup.
Appendix Provides empirical verification for observations 2-4 on -iny-emaeeNet.
B EXTENDING THE FRAMEWORK TO SGD TRAINING
The main paper focuses using gradient flow to establish the relationship between frequently used importance measures for network pruning and evolution of model parameters. In practice, however, deep neural networks are trained using SGD (stochastic gradient descent) its variants (e.g., SGD with momentum). This section serves to demonstrate that up to a first order approximation, our observations based upon gradient flow theoretically valid for SGD training 1oo. Our analysis based on the fact that when an importance measure is employed, most pruning frameworks stop model training and use several mini-batches of training data determine an approximation for gradient/Hessian. This enables us to use an expectation operation over mini-batches of data, making theoretical analysis tractable.
B. NOTATIONS AND SGD UPDATES
We first establish new notations for this section. For completeness, notations used in the main paper are recapped as well:
Notations:
(t)) denotes model parameters at time t. 7) denotes the learning rate at which stochastic gradient descent trains a model. (t); X&) denotes the estimate of gradient of loss and H(O X,) denotes the estimate of Hessian of loss with respect model parameters time calculated using the tth mini-batch (X₂). g( O (t)) denotes the full-batch gradient of loss and H (t)(t)) denotes Hessian of loss with respect model parameters time t.
Under stochastic gradient descent, a random mini-batch of data is sampled amongst all mini-batches without replacement. For example, mini-batch X; sampled, gradient approxi- mation based on its data elements will be used to update model parameters at the current iteration. Therefore, the model parameters evolve as follows:
SGD Update:
( - e() m)(((()) X.).
(13)
11

---

Published as a conference paper at ICLR 2021
B.2 CHANGE IN NORM MODEL PARAMETERS: A (
We first use SGD to analyze change in norm of model parameters, denoted as A |(((t)l Assume mini-batch X; is used update model parameters at the current iteration.
|(0(1) - |0(()) - ,d((()); - 10
_10(1]1 2n
X,) + x [0(01)1
(14)
22 0() X,) 0
-2, -2) X;),
where we ignored terms with higher order products of learning rate 77. Smaller the learning rate, the better this approximation will be.
As mentioned before, to compute importance estimates for network pruning, most pruning frameworks stop the training process and compute estimates for gradient/Hessian using several mini-batches of take expectation over the mini-batch sampling process. For example,
data. To account for this, input data is denoted as X, under expectation, Equation 14 changes as follows:
Ex,xx
[|)(())
Ex,xx (OO(t))
21 - (d()(t); X,))
2n 02) g()(t)).
(15)
(0))
Ex
= 20(1))
As the main paper shows (see Equation 9), under gradient flow, the norm of model parameters evolves as follows:
a 1 (((
a
(16)
Therefore, using Equation 15 and Equation 16 it be evidently seen that up first-order approximation, the expected rate at which norm of model parameters evolves under SGD is exactly the same as that under gradient flow. In the main paper, this relationship to conclude Observations 2 and 3. Thus, our analysis in this section indicates Observations 2, are in fact valid approximations under SGD training as well.
B.3 CHANGE IN CHANGE OF NORM OF MODEL PARAMETERS: 2 (0
We now determine the expected rate at which change norm of model parameters itself changes under SGD. To this end, we will calculate A [0())]2 2 = A(A [ (t)
Equation 14 shows that when mini-batch X; is used take an optimization step, up to a first-order approximation, the change in norm of model parameters (A [[)(t) is as follows:
A ||)(t)] = 2n (
(17)
After the optimization step based on mini-batch X, has been completed, the model parameters change from (t) to (tt) nd( (();; X,). Assume that a mini-batch X, is sampled for the next optimization step-i.e., the next gradient estimate is d (+(t) - nn X.); X,). To relate this new gradient estimate with the previous one, use first-order Taylor approximation for model gradient based on mini-batel Xy:
d (t))) 7d (0(I; - (I))))))
(18)
12

---

Published as a conference paper at ICLR 2021
Note that in the above equation, the Hessian estimate is based mini-batch X, only. Using this relationship, the change in norm of model parameters can be approximated as follows:
A [|@(t) (t)); x.) 2 = 2n (o(t) - ( x0)0
2n 2n (0(1) (0000000000000000000000000000000
+ 2=2210000;KKk(11000:))0----)0
(19)
where we ignore terms of the order of (,7²). This above result can now be used to calculate
A @ (t) our desired quantity.
|0(1)) - (t((t) -)())))) |0
+ (2
= =j (()
+ + d ( (((()))))
(20)
We again use the fact that importance estimates for network pruning are estimated by stopping
model training and computing gradient/Hessian estimates over several mini-batches of data. We
denote this through an expectation over input data X, where our random variables are the mini-
batches X,, and Xy. As mini-batches are sampled independently, we use the result that expectation of product of their functions can be independently evaluated: i.c., (xx,x,,x(((X)))(X Ex,x Ex, [X(xxx())))
(XX))..
Ex,,X
A
O (1)
2n Ex,, X (d)())
21²² Ex, (00)"((00)
22n/ Ex,x,~xX [((0(0:x)''(((00xx)))
= -n2 (() - (O((:
+ 201² [/)0(()) ) Exx
+ +2n Ex,ex (0)) Ex,x (d
= 22 O(1) (g(O((]] -(0)))))((()))---)) - 2n (())))())(((
2n g(O(())
2n (a00(0))"*++
(21)
A |0()) 2
Ex,Xxxxx
= 2 (()))))))))))())))))))))))))))))
(22)
As shown in the main paper (see Equation 12). under gradient flow, the rate at which change in norm of model parameters changes is as follows:
a (
2
(23)
0+2
As can be evidently seen in Equation 22 and Equation 23, up to first-order approximation, the expected rate which change of norm of model parameters changes under SGD is exactly the same
13

---

Published as a conference paper at ICLR 2021
that under gradient flow. In the main paper, use this relationship to conclude Observations 4 and 5. Thus, our analysis this section indicates Observations 4, 5 are in fact valid approximations under SGD training well.
Overall, in this section, we showed that the relationships between commonly used measures for network pruning and their expected impact model parameters is the same for both SGD training and gradient flow, up to a first-order approximation.
C TRAINING BASE MODELS
We analyze models trained on CIFAR-10 and CIFAR-100 datasets. All models are trained with the exact same setup. In all our experiments, we average results across 3 seeds.
Since pruning has a highly practical motivation, we argue analyzing a variety of models that capture different architecture classes important. To this end, use VGG-13 vamilla CNN), MobileNet- V1 (a low-redundancy CNN designed specifically for efficient computation), and ResNet-56 (a residual model analyze effects of pruning under skip connections). The final setup follows:
Optimizer: SGD,
Momentum: 0.9,
Weight decay: 0.0001,
Learning rate schedule: (0.1, 0.01,
Number of epochs for each learning rate: (80, 40, 40),
Batch Size: 128.
D TRAINING PRUNED MODELS
The general setup for training pruned models is the
that for training base models. However, for fair comparison, if the prune-an and-train framework takes n rounds, we subtract epochs from the number of epochs alloted the highest learning rate. This ensures same amount of training budget for all models and pruning strategies. Note that even if this subtraction is not performed, the results do vary much, as the model already converges (see Appendix For train/test curves). The final setup is as follows.
Optimizer: SGD,
Momentum: 0.9,
Weight decay: 0.0001,
Learning rate schedule: (0.1, 0.01, 0.001).
Number of epochs for each learning rate: (80 number of pruning rounds, 40, 40),
Batch size: 128.
Since we prune models in a structured manner, our target pruning ratios are chosen terms of the number of filters to be pruned. A rough translation in terms of parameters follows:
VGG-13: 75% filters (~>94% parameters) for CIFAR-10; 65% filters (~89% parameters) for CIFAR-100,
MobileNet- V1: 75% filters (~-94% parameters) for CIFAR-10; 65% filters (~89% parame- ters) for CIFAR-100
ResNet-56: 60% filters (~88% parameters) for CIFAR-10; 50% filters (~83% parameters) for CIFAR-100.
The percentage of pruned parameters is much larger than the percentage of pruned filters because generally filters from deeper layers are pruned more heavily. These filters form the bulk of a model's parameters, thus resulting in high parameter percentages.
14

---

Published as a conference paper at ICLR 2021
E RANDOM/UNIFORM PRUNING
In this section, we compare various pruning measures with random and uniform pruning under the same experimental setup as used in the main paper (see (see Appendix D). For random pruning, we assign a random score between 0-1, with equal probability, to each filter. For uniform pruning, we decide a target amount of percentage pruning and remove that much percentage filters from all layers. For example, if 50% of the model is to be pruned, we remove half the filters from each layer randomly.
If which importance measure is used for pruning a model in structured pruning settings is not an important choice, both random and uniform pruning schemes should perform competitively with standardly used importance measures for network pruning. However, as we show in Table 3, random and uniform pruning perform much worse than the standardly used metrics for pruning.
Table 3: Reduction in accuracy for CIFAR-100 models pruned randomly, uniformly, and using the importance measures analyzed in the main paper. Test accuracies of base models are reported in the table. When pruning is distributed over several rounds, it often leads to small amounts pruning in given round. Since only integer number of filters can be pruned, small ratio will result in no pruning at layer with small number of filters and other layers will be pruned more to compensate for the same. While this issue does not arise in VGG-13 13 and MobileNet-Vi V1 models, for ResNet-56. which has several layers with only 16 filters per layer, often very minimal amounts of pruning takes place. Due to this, later layers are pruned more aggressively than earlier layers, resulting in poor performance in both 5 and 25 rounds of random/uniform pruning for ResNet-56.
VGG-13
Base Model
Random
Uniform
Mag. Based
Loss Based
Proposed Extemsion
GraSP (T=200)
GraSP (T=1)
GGrSPP (T=1)
(65% pruned)
round rounds 25 rounds
69.62
-5.71
-5.60
-1.73
-1.01
-0.61
-1.11
-388
-0.96
-.57
-1.74
-.22
1.53
4.55
-3.19
-1.31
-0.51
-0.69
-4.44
26.79
-1.48
MobileNet-Vi (65% pruned)
V1
Buse Model
Random
Uniform
Mag. Based
Loss Based
Proposed Extension
GraSP (T=200)
GraSP (T=1)
GraSPP
(T=1)
round
69.10
-6.54
-9.71
-1.09
-1.94
-0.58
-4.8
4.31
-1.93
5 sounds
-6.55
-4.66
-0.77
-1.89
-0.69
-6.08
-15.35
-1.82
25 rounds
-5.59
-3.56
-1.52
-1.79
-0.75
-9.6
22.61
-1.41
ResNet-56 (50% pruned)
Base Model
Random
Uniform
Mag. Based
Loss Basd
Proposed Extension
GraSP (T=200)
GraSP (I=1)
IrrSPP
(I=1)
round 5 rounds
71.01
-4.00
-3.42
-4.09
-4.13
-3.91
-3.92
-3.99
-3.86
-9.05f
-7.58"
-2.97
-3.90
-3.31
-4.04
-18.03
-4.00
25 rounds
-27.541
-70.01
-2.26
-3.27
-2.39
-13.46
-70.01
-2.99
F TRAIN/TEST PLOTS
This section further demonstrates that magnitude based pruned models converge faster than loss- preservation based pruned models. For magnitude-based pruning, the importance is defined as $2 norm of a filter; for loss-preservation. the importance is defined variant of SNIP (Lee al. (2019)) applied to entire filter. The plots show train/test curves for VGG-13, MobileNet- V1, and ResNet-56 models for 1, 5, and 25 pruning rounds each. Also plotted train/test curves for the proposed extension to loss prssereesrvvtioon which intentionally biases loss preseereesovtooon towards removing small magnitude parameters (see Section 4.1 for more details).
15

---

Published as a conference paper at ICLR 2021
F.1
VGG-13
Magnitude
Proposed
Loss-pres.
Test
Train
100
100
Accuracy
90
60
80
20
70
0
50
100
150
50
100
150
Epoch
Epoch
Figure 4: VGG-13: round of pruning. CIFAR-10 (left); CIFAR-100 (right).
Magnitude
Proposed
Loss-pres.
Test
Train
100
100
Accuracy
80
90
80
40
70
100
150
50
100
150
Epoch
Epoch
Figure 5: VGG-13: 5 rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
Magnitude
Proposed
Loss-pres.
Test
Train
100
100
Accuracy
90
60
80
20
70
0
50
100
150
50
100
150
Epoch
Epoch
Figure 6: VGG-13: 25 rounds of pruning. CIFAR-10 (left); CIFAR- 100 (right).
F.2
MOBILENET-VI
Magnitude
Proposed
Loss-pres.
Test
Train
100
80
Accuracy
90
60
40
80
20
70
100
100
150
Epoch
Epoch
Figure 7: MobileNet-Vil V1: 1 round of pruning. CIFAR-10 (left); CIFAR-100 (right).
16

---

Published as a conference paper at ICLR 2021
Magnitude
Proposed
Loss-pres.
Test
Train
100
80
Accuracy
90
60
40
80
20
70
0
50
100
150
0
50
100
150
Epoch
Epoch
Figure 8: MobileNet-VI: rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
Magnitude
Proposed
Loss-pres.
Test
Train
100
80
Accuracy
90
60
40
80
20
70
50
100
150
0
100
150
Epoch
Epoch
Figure 9: MobileNet-VI: 25 rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
F.3
RESNET-56
Magnitude
Proposed
Loss-pres.
Test
Train
100
95
Accuracy
90
80
80
65
70
50
0
50
100
150
0
50
100
150
Epoch
Epoch
Figure 10: ResNet-56: round of pruning. CIFAR-10 (left); CIFAR- 100 (right).
Magnitude
Proposed
Loss-pres.
Test
Train
100 1
95
Accuracy
90
80
80
65
70
50
100
100
150
Epoch
Epoch
Figure 11: ResNet-56: 5 rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
17

---

Published as a conference paper at ICLR 2021
Magnitude
Proposed
Loss-pres.
Test
Train
100
95
Accuracy
90
80
80
65
70
50
100
150
0
50
100
150
Epoch
Epoch
Figure 12: ResNet-56: 25 rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
G MORE RESULTS ON GRADIENT-NORM BASED PRUNING
This section provides further results on gradient-norm based pruning. The general implementation of these measures requires the calculation of Hessian-grann -raadinet products. Similar the implementation by Wang et al. (2020), we define a constant amount of memory that stores randomly selected samples for all classes. For the original GraSP variant that increases gradient norm, we follow the original implementation and use temperature of 200 during the calculation of Hessian-gradies product. This involves division of the model output (i.e., the logits vector) by a constant (= 200) before using softmax for classification (see Hinton et al. (2015) for further details). For the GraSP variant without large temperature, this temperature value is brought down to 1. Finally, for the gradient norm preservation measure, the GraSP variant without large temperature is used alongside absolute value operator to remove parameters that least change gradient norm (i.e., least affect second-order model evolution dynamies).
G.1 SCATTER PLOTS FOR (()H((())))((()))) vs. (t)g(((()
This subsection provides scatter plots demonstrating the highly correlated nature of OZ ((())((()))((((()))))))))))))))) and @ (+)) The correlation is averaged over 3 seeds and plots are for seed. As can be seen in the plots, the measures are highly correlated throughout model training, indicating gradient-norm increase may severely affect model loss if partially trained or completely trained model is pruned using O ())))))))(((()).
Correlation 0.954
Correlation 0.950
Correlation 0.78
HiO) g(o)
2.5
30
0.0-
-00
-2.5
-0.04
-0.10
0.25
-0.08
0.00 0
0.08
-0.003
0.000
0.003
0. g())
e,
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 13: of (tt)) versus O (t) (t)) for VGG-13 models trained on CIFAR- 100. Plots are shown for filters (a) at initialization, (b) after 40 epochs training, and (c) after complete (160 epochs) training.
18

---

Published as a conference paper at ICLR 2021
Correlation 0.954
Correlation 0,951
Correlation 0.82
HHO) g
3.5
0.03
0.0-
0.00
-0.8
-3.5
-0.03
-0.03
-0.01
0.01
-0.06
0.00
0.06
-0.004
0.000
0.004
0 g())
o
0, + g(0)
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 14: O (()))))))(()) versus O (t)) for MobileNet-VI models trained on CIFAR-100. Plots are shown for filters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
Correlation
Correlation
Correlation
0.04
0.5
(HO)
0.0
0.0
0.00
-0.5-
-0.4
-0.04
-0.01
0.00
0.01
-0.03
0.00
0.03
-0.004
0.000
0.004
0 g(0)
0 g(0)
0 g(0)
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 15: ()()) (())) versus 0 (gg (())) for ResNet-56 models trained on CIFAR- 100. Plots are shown for filters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
G.2 LAYER-WISE PRUNING RATIOS
In Table 2 we showed that few epochs of training, when reduction in loss can be attributed to training of earlier layers (Raghu et (2017)), GraSP without large temperature chooses to prune earlier layers aggressively. This failure mode is an inherent tendency of GraSP, since GraSP assigns least importance to parameters whose removal maximally increases model loss (see Observation 5 in Section 4.3).
To further elaborate on this comment, first provide layer-wise pruning ratios of models pruned using different variants gradient-norm based pruning-i.e.. GraSP with large temperature (GraSP (T=200)). Gradient norm preservation (IGraSPI), and GraSP without large temperature (GraSP (T=1)). Figure 16 provides results for VGG-13 models; Figure 17 provides results for MobileNet-V V1 models; and Figure 18, Figure 19, and Figure 20 provide results for block-1, block-2, and block-3 of ResNet-56 models, respectively. For sequential architectures (VGG-13 and MobileNet-V1), we can vividly see earlier layers are pruned aggressively. For ResNet-56, module, the first convolutional layer is pruned lot more. This holds well with our claim that GraSP removes parameters which maximally increase the model loss, for removing parameters from the first convolutional layer in residual module will completely block signal propagation for that module. The remaining pruning budget can then be used to remove parameters from the first convolutional layer of other residual modules, instead of focusing on the Layer-2 of the same module. Note that this behavior becomes more explicit as several rounds of pruning are used and invariably earlier Layer- 1's in earlier blocks are pruned more.
find that in any given residual
Overall, across all models, GraSP without large temperature prunes earlier layers very aggressively. This permanently damages the model and thus the accuracy of low-temperature GraSP is much lower than other variants of gradient-norm based pruning. Even for GraSP with large temperature, when more rounds of pruning used, of large temperature is unable to satisfactorily avoid aggressive pruning in earlier layers (c.g., see Figure 17c). Only Gradient norm preservation is able to avoid this failure mode, and does so across all models and settings.
19

---

Published as a conference paper at ICLR 2021
From more theoretical standpoint, behavior. In particular, Equation shows model loss (L())) over time reduces proportion to negative of the gradient norm. For example, if model has N layers and (t), g(),(tt) denote
again employ gradient flow to better understand this
parameters, gradient of the n layer, then the model loss reduces at the following rate:
aT(() Ōt
(g()())))
Σ O n=1
(24)
This implies, at any given time, if a layer has a higher gradient norm, it contributes more to reduction in loss. This makes parameters of that layer an appropriate candidate for removal by GraSP.
We plot the layer-wise gradient norm of different models trained on CIFAR-100 over time (see Figure 21). As can be evidently seen, gradient norm is highest for the earlier layers in the beginning. Later in training, especially after the learning rate is dropped (80 epoch). it is difficult to discern which part of the model has higher gradient norm. Overall, this experiment provides further evidence that early-on in training, reduction in model loss can be majorly attributed to the optimization of parameters in the earlier layers, as noted by Raghu al. (2017) through the use of SVCCA. This leads to aggressive pruning of earlier layers when GraSP is used as importance measure. Gradient norm preservation, however, is able to avoid this failure mode.
100
100
N temaing
20
Titr Semantee
00
Nitlrr Tamasnn
05-
20
20
Layer trdex
Layer Trols
Layer V10EX
(a) 1 round.
(b) 5 rounds.
(c) 25 rounds.
Figure 16: Layer-wise pruning ratios for VGG-13 with a target ratio 65% pruning. Results are provided for pruning over (a) 1 round, (b) 5 rounds, and (c) 25 rounds. When the model is allowed to train even slightly, GraSP without temperature prunes earlier layers very aggressively.
Grdff TT2001
Gridf ITAC
Crcft TT200
Drift (TD
Codf T1000) 100
STter Semaning
Tite Remaning
Flise Remaning
62
40
40
(a) round.
(b) rounds.
(c) 25 rounds.
Figure 17: Layer-wise pruning ratios for MobileNet-VI with target ratio of 65% pruning. Results are provided for pruning over (a) round, (b) rounds, and (c) 25 rounds. When the model is allowed to train even slightly, GraSP without temperature prunes earlier layers very aggressively.
20

---

Published as a conference paper at ICLR 2021
Redy
Oeth
CredP mXX30
Kiddy
100
Sraft m7-001
NTia Remanng
60
60
40
20L:1215541222272
(a) Block I.
(b) Block 2.
(c) Block
Figure 18: Layer-wise pruning ratios for (a) Block 1, (b) Block 2, and (c) Block of ResNet-56. with a target ratio of 50% pruning over round. For (a) Block I, even layer indices are Layer-I of a residual module and odd layer indices are Layer-2 of residual module; for (b) Block 2 and (c) Block 3, index 0 is Layer-I, I, index module; thereafter, for (b) Block 2 and (c) Block odd layer indices are Layer-1 of residual module and even layer indices are Layer-2 of a residual module. This figure shows that when the model is initialization, GraSP with and without large temperature have only slight bias towards pruning earlier layers more than deeper layers.
Layer-2, and index 2 is the learned shortcut layer of the first residual
100
MAMP +21
100
100
SHAPTHAA
Soose
The Ramaning
Ther namanna
Ramaning
60-
40
40
40
1215041566111
(a) Block 1.
(b) Block 2.
(c) Block 3.
Figure 19: Layer-wise pruning ratios for (a) Block 1, (b) Block 2, and (c) Block 3 of ResNet-56, with a target ratio of 50% pruning over 5 rounds. For (a) Block 1, even layer indices are Layer-1 of a residual module and odd layer indices are Layer-2 of residual module; for (b) Block 2 and (c) Block 3, index 0 is Layer-1, index I is Layer-2, and index 2 is the learned shortcut layer of the first residual module: thereafter. for (b) Block 2 and (c) Block odd layer indices are Layer-1 1 of a residual module and even layer indices are Layer-2 of residual module. This figure shows that when the model is allowed train even slightly, GraSP without large temperature prunes Layer- of residual modules in all blocks aggressively, with clearly higher amount of pruning in the earlier blocks. GraSP with large temperature prunes all earlier layers aggressively.
20- 20-
Filee famanns
40
Flll Ramantin
85-
60
65
20-
trdex
Liyer hoox
Layer hobo
(a) Block 1.
(b) Block 2.
(c) Block
Figure 20: Layer-wise pruning ratios for (a) Block 1, (b) Block 2, and (c) Block 3 of ResNet-56, with
a target ratio of 50% pruning over 25 rounds. For (a) Block 1, even layer indices are Layer-I of residual module and odd layer indices are Layer-2 of residual module; for (b) Block 2 and (c) Block 3, index is Layer-1, index 1 Layer-2, and index 2 the learned shortcut layer of the first residual module; thereafter, for (b) Block 2 and (c) Block 3 odd layer indices are Layer-1 of residual module and even layer indices are Layer-2 allowed train even slightly. GraSP without temperature prunes Layer- I's of residual modules in all blocks aggressively, with a clearly higher amount of pruning in the earlier blocks. GraSP with large temperature prunes all earlier layers aggressively.
residual module. This figure shows that when the model is
21

---

Published as a conference paper at ICLR 2021
0.84
Cem,3
25
Com,0
0.7-
Gmr
Cmm,1
Com,2
Com,1
0.6
Cmm,3
2.-
Cow,3
Cmr,1
Cor,3
Godem
0.5-
Cmm,3
Conr.
Com,5
15-
Cor,3
Cowr5
Con,
Conr 3
Comr
10-
Con,33 9
Con,3
Comr_0
05-
[om,11
Cony
30
70
110
120
kpoch
Epoch
(a) VGG-13.
(b) MobileNet-V1.
Lom
30
323
320
320
(c) ResNet-56: Block 1.
(d) ResNet-56: Block 2.
(c) ResNet-56: Block 3.
Figure 21: Layer-wise gradient norm for (a) VGG-13, (b) MobileNet-VI, (c) Block of ResNet- 56, (d) Block 2 of ResNet-56, and (e) Block 3 of ResNet-56 models trained CIFAR-100, For MobileNet-Vi V1, number of pointwise convolutional filters in a layer and number of depthwise convolutional filters the following layer need be the same. Due to this, we only use pointwise convolutional filters to prune model and thus this figure visualizes the gradient norm for pointwise convolutional filters only. For ResNet-56 plots, bold lines indicate the first convolutional layer (Layer-1) 1) in residual module and dotted lines indicate the second convolutional layer (Layer-2) in that same module. Early-on in training, we generally find gradient norm much higher for earlier layers and these layers are invariably the ones pruned more by GraSP (see layer-wise pruning ratios Figure 16 for VGG-13 models; Figure 17 for MobileNet-Vi models; and Figure 18, Figure and Figure 20 for ResNet-56 models, respectively).
G.3 TRAIN/TEST CURVES FOR GRADIENT-NORM BASED PRUNING VARIANTS
This subsection provides train/test curves for different variants of gradient-norm based pruning methods considered in this paper. These curves demonstrate the large temperature GraSP variant does not improve convergence rate substantially, if ever, in comparison to gradient-nomn -orm preservation. On the other hand, does result in significant performance loss when the model is even slightly trained.
G.3.1 VGG-13
GraSP (T=1)
|GraSP] (T=1)
GraSP (T=200)
Test
Train
100 4
100
Accuracy
90
80
80
60
70
40
100
150
50
100
150
Epoch
Epoch
Figure 22: VGG-13: 1 round of pruning. CIFAR-10 (left); CIFAR-100 (right).
22

---

Published as a conference paper at ICLR 2021
GrasP (T=1)
[GraSP| (T=1)
GrasP (T=200)
Test
Train
100
100
Accuracy
90
80
80
60
70
40
0
50
100
150
0
50
100
150
Epoch
Epoch
Figure 23: VGG-13: rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
GraSP (T=1)
[GraSP] (T=1)
GraSP (T=200)
Test
Train
100
100
Accuracy
90
80
80
60
70
40
0
50
100
150
0
50
100
150
Epoch
Epoch
Figure 24: VGG-13: rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
G.3.2
MOBILENET-VI V1
GraSP (T=1)
|GraSP| (T=1)
GraSP (T=200)
Test
Train
100
100
Accuracy
90
80
80
60
70
40
50
100
150
0
50
100
150
Epoch
Epoch
Figure 25: MobileNet-VI: round of pruning. CIFAR-10 (left); CIFAR-100 (right).
GraSP (T=1)
|rraSP| (T=1)
GraSP (T=200)
Test
Train
100 4
100
Accuracy
90
80
80
60
70
1.
40
100
150
50
100
150
Epoch
Epoch
Figure 26: MobileNet-Vis V1: rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
23

---

Published as a conference paper at ICLR 2021
GrasP (T=1)
[Gra5P] (T=1)
GrasP (T=200)
Test
Train
100
100
Accuracy
90
80
80
60
70
40
0
50
100
150
0
50
100
150
Epoch
Epoch
Figure 27: MobileNet-VI: 25 rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
G.3.3
RESNET-56
GraSP (T=1)
|GraSP] (T=1)
GraSP (T=200)
Test
Train
100
100
Accuracy
90
80
80
60
70
40
50
100
150
0
50
100
150
Epoch
Epoch
Figure 28: ResNet-56: round of pruning. CIFAR-10 (left); CIFAR-100 (right).
GraSP (T=1)
|GraSP| (T=1)
GrasP (T=200)
Test
Train
100 4
100
Accuracy
90
80
70
50
100
150
0
50
100
150
Epoch
Epoch
Figure 29: ResNet-56: 5 rounds of pruning. CIFAR-10 (left); CIFAR-100 (right).
GraSP (T=1)
GraaSP| (T=1)
GraSP (T=200)
Test
Train
100
100
Accuracy
90
80
80
60
70
40
0
50
100
150
0
50
100
150
Epoch
Epoch
Figure 30: ResNet-56: 25 rounds of pruning. CIFAR-I0 (left); CIFAR- 100 (right). GraSP (T=1) is not shown because it achieves random performance (10% for CIFAR- 10 and 1% for CIFAR-100).
24

---

Published as a conference paper at ICLR 2021
H RESULTS ON UNSTRUCTURED PRUNING
The experiments in this paper focus structured pruning, which enables acceleration commodity hardware resources without requirement of specially designed hardware/software, However, note that theoretical analysis is independent of the pruning setup used. Thus, revisit some of our experiments under an unstructured pruning setup, hence demonstrating the empirical generality of results.
H.1 OBSERVATIONS 2/3
In Observation (see Section 4.2), we show that loss poss-rreservation based pruning techniques tend to remove filters with minimal movement in their magnitude. This leads us Observation 3 (see Section 4.2), which shows that with the added heuristic of "train until minimal change" (You et al. (2020)), where magnitude-based pruning removes parameters with small magnitude and minimal change, magnitude-baset pruning implicitly performs loss-preservation as well. The main paper verifies this claim at a filter-level granularity (structured pruning) for models trained on CIFAR-100 (see Figure 2).
In Figure 31, show that even when analyzed finer granularity of parameters (unstructured pruning) in convolutional filters, our claim continues to hold well. In particular, we show that as training continues and change in parameters reduces, 10,4,p becomes highly correlated with loss- preservation based importance (O)(((()))) This demonstrates that our observations relating magnitude-basee pruning and loss-preservation pruning continue to hold well an unstructured pruning setup too.
Comeiition Mask Distance
Cormettion
0.6
Mask Distance
Distance
0.5
0.4
0.2
0.2
Commection Movi Dedwnce
0
50
100
150
0
50
100
150
50
100
150
Epoch
Epoch
Epoch
(a) VGG-13.
(b) MobileNet-V1.
(c) ResNet-56.
Figure 31: Correlation between 00,40p and loss-preservation based importance (()((t)(((())) at every 10th epoch for models trained on CIFAR-100. Also plotted is the distance between pruning masks (target ratio: 20% parameters), similar to masks used by You et al. (2020) decide when to prune model. In contrast to You et al., these masks function on parameter level granularity, as used unstructured pruning. We note that as the distance between pruning masks over consecutive epochs reduces, Ap becomes more correlated with foss-preservation importance.
H.2 OBSERVATION 4
Observation 4 indicates gradient-norm based pruning (i.e., GraSP Wang et al. (2020)) removes parameters which maximally increase model loss. The main paper confirms this claim a filter-level granularity (structured pruning) for CIFAR- 100 models (see Figure 3), showing that there exists and
high correlation between (())))()
for this particular setting.
This subsection generalizes this claim to an unstructured pruning setup by providing scatter plots demonstrating the highly correlated nature of (t)) ure 32) for VGG-13 models, Figure 33 for MobileNet-Vi V1 models, and Figure 34 for ResNet-56 models). contrast with Section G.I where plots are visualized on a filter-level granularity, the following plots are visualized on a parameter-level granularity, as expected in an unstructured pruning setup. As can be seen in the plots, the measures are highly correlated throughout model training, indicating gradient- norm increase may severely affect model loss if partially trained or completely trained model is pruned using (()(())(((())))))))))))) (())))((()))))))))))))( (t)).. This demonstrates that our observations for gradient-nomm based pruning continue to be valid in unstructured pruning setup well.
and (t()) (t)) (see Fig-
25

---

Published as a conference paper at ICLR 2021
Correlation 0.956
Correlation 0.819
Correlation 0.837
HH0) 9.
30
3e
3e 6
-3e-4
-3e-4
-3-6-
-ee5
5e-5
-ee4
1e-4
-4e-6
4e-6
0g(
6 gi
x gl
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
for VGG-13 13 models trained on CIFAR-IOW 100. Plots are shown for parameters (a) at initialization, (b) after 40 epochs training, and (c) after complete (160 epochs) training.
Figure 32: O p ((((()) versus
Correlation 0.914
Correlation 0.967
Correlation 0.785
gio)
5e-2
4e-1
30-3
0-
0-
-5e-2
-4-1
-1e:3
1e:3
99-3
9e-3
-4e-4
4e
0% g(@)
0 gio)
e gle))
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 33: ()))))))))) versus (p)())(O(t) for MobileNet-V V1 models trained on
CIFAR-100. Plots are shown for parameters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
Correlation 0.745
Correlation 0.894
Correlation 0.742
Se-2
Se-2
1e-2-
(HO)
1e.22
-5e-2
.ee2
-1e-3
1e-3
-1e-3
16-3
-4e-4
4e-4
- -g())
O
g(0)
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
for ResNet-56 models trained on CIFAR- 100. Plots are shown for parameters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
Figure 34:
versus
RESULTS ON TINY-IMAGENET
To further verify our claims, we repeat some our experiments on Tiny-Imaged and confirm that our claims indeed generalize to the same.
1.1 OBSERVATIONS 2/3
In Observation 2 (see Section 4.2), we show that loss-preservation based pruning techniques tend to remove filters with minimal movement in their magnitude. This leads us to Observation (see Section 4.2), which shows that with the added heuristic of "train until minimal change" (You al. (2020)), where magnitude- based pruning removes parameters with small magnitude and minimal change, magnitude-basee pruning implicitly performs loss-preservation well. While the main paper
26

---

Published as a conference paper at ICLR 2021
verifies this claim on CIFAR-100 100 (see Figure 2), in this section replicate this behavior Tiny- ImageNet. As shown in Figure 35, we indeed find that lower the magnitude and movement between BatchNorm scale parameters over subsequent epochs, the higher the correlation between magnitude- based pruning and loss-preservation based pruning is. This demonstrates that our observations relating magnitude-based pruning and oss-preservattin pruning generalize larger scale dataset too.
110
1.0
1.0
0.4
Mark Distance
Correlation
0.3
0.15-
0
50
100
150
0
50
100
150
50
100
150
Epoch
Epoch
(a) VGG-13.
(b) MobileNet-VI.
(c) ResNet-56.
Figure 35: Correlation between |o (proxy for magnitude- pruning with the added "train until minimal change" heuristic) and poss-preservation based importance (see Equation 3) at every 10th epoch. Also plotted the distance between pruning masks (target ratio: 20% filters), as used by You et al. (2020) to decide when to prune model. As the distance between pruning masks over consecutive epochs reduces, |o becomes more correlated with loss-preservation importance.
1.2 OBSERVATION 4
Observation 4 indicates gradient-norm based pruning (i.e., GraSP Wang et al. (2020)) removes parameters which maximally increase model loss. The main paper confirms this claim for CIFAR-100 models (see Figure 3), showing that there exists high correlation between O (L) (()))((())) and O (t)g(O()) (t)) for this particular setting. Here, we replicate this experiment and confirm our models (see Figure 36 for VGG-13 models, Figure 37 for MobileNet- VI models, and Figure 38 for ResNet-56 models).
observation
Correlation 0.988
Correlation 0.989
Correlation 0.989
HHG) g(
1.5
0.0
-1.5
-2
-0.06
0.00
0.10
-0.07
0.00
0.07
-0.05
0.05
0.15
gi
e
gie:
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 36: (() versus - (t))) for VGG-13 models trained on Tiny- ImageNet. Plots are shown for filters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
27

---

Published as a conference paper at ICLR 2021
Correlation 0.812
Correlation 0.792
Correlation 0.847
0.5
HiG)
0.5
-0.8
-0.03
-0.01
0.01
-0.02
0.00
0.02
-0.03
0.03
o g(0)
gie:
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 37: T (()))))))((((( versus @ (t))( for MobileNet-VI models trained on Tiny-Iman Imageen. Plots are shown for filters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
Correlation 0.986
Correlation
Correlation
5
0.1
0-
0.0-
-5
-0.1
-5
-0.05
0.00
0.d5
-0.1
0.0
0.1
-0.005
0.000
0.dos
0 gio)
0 gie)
(a) Initialization.
(b) 40 epochs.
(c) 160 epochs.
Figure 38: 6 (t) ()))))) versus O ( for ResNet-56 models trained on Tiny- ImageNet. Plots are shown for filters (a) at initialization, (b) after 40 epochs of training, and (c) after complete (160 epochs) training.
28


Reviews:
[{"review_id": "UT4XYNMXeI7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors study proposed importance metrics for pruning neurons/channels in deep neural networks and analyze what properties of parameters are favored by each approach by studying the relationship between model parameters, gradients, 2nd order derivatives and loss. Through this analysis they develop a rich understanding of the consequences of different pruning criteria and use their understanding to propose modifications to existing techniques that produce higher quality models across different settings.\n\nPros:\n\nThe framework used by the authors is clear and easy to understand but also very general. The authors’ mix of empirical results and theoretical analysis makes a convincing case for the accuracy of their observations. The authors go beyond observation and analysis and use their insights to design new approaches to pruning that outperform existing techniques. The paper is well written and well organized.\n\nCons:\n\nThis paper has few limitations. The main limitation is that all experiments were conducted on relatively small datasets (CIFAR). Given that is has been shown that some techniques in model compression produce state-of-the-art results on small tasks but fail on larger models and datasets [1, 2], I’d encourage their authors to further validate their insights on a larger dataset (i.e., ImageNet).\n\nComments:\n\nI found that the authors waited a long time to explain the term “gradient flow”, which was important in sections 1-3 but not fully detailed until the start of section 4. On page 1 the authors say in parenthesis that gradient flow is “gradient descent with infinitesimal learning rate”, but I found this explanation was not clear. The second sentence of section 4 “the evolution over time of model parameters, gradient, and loss” was much more clear. I’d encourage the authors to potentially work some of these details earlier into the text.\n\nReferences:\n1. https://arxiv.org/abs/1902.09574\n2. https://arxiv.org/abs/2003.03033\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S_xhcwa9775", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a detailed analysis on pruning heuristics, and its applications to early pruning. It thoroughly analyzed magnitude-based pruning, loss-preservation based pruning, and gradient-norm based pruning. The paper demonstrated the results on CIFAR-10 and CIFAR-100 datasets. it's very timely research to guide the audience which heuristic is better. My major concern is the novelty over existing pruning heuristics, since the techniques have all been proposed before. The other concern is the evaluation and the scale of the dataset. Given the results in table 2 different by less than a percent, and Cifar training is very noisy, it's hard to tell the difference. Just like the Lottery Ticket hypothesis works on Cifar but does not work on ImageNet, different pruning heuristics needs to be verified on the large scale ImageNet dataset in order to be convincing. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fpj2dqMqygP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper studies different families of pruning criteria and their impact on training dynamics (especially early training). Stemming from the observations, authors provide improvements to the 1st and 2nd order saliency methods. \n\n## Pros\n- Authors provide simple and useful explanations to various pruning criteria that are based on the Taylor approximation of the loss functions.\n- Even the authors don't mention this in the contributions, they propose some improved versions of existing criteria. For example the updated taylor score with $\\theta^2g(\\theta)$ or absolute valued GrasP. This is great and it might worth focusing on these criteria further providing further evidence on their usefulness. Currently, they seem a bit arbitrary. For example, why not third power $\\theta^3g(\\theta)$ or additive biasing of magnitude $(g(\\theta)+c)*\\theta$. I recommend authors to run their versions in unstructured setting too.\n\n## Cons \n- Authors choose to focus on structured pruning since resulting networks are dense and acceleration is straight-forward. However,  they miss an important work on structured pruning [1]. This relatively well-known work shows that pruned (structured) networks can be trained to full accuracy from scratch. In other words, their value lies on doing some kind of an architecture search over layer widths. The motivation of the work needs to be revisited in the light of these results. Since we can retrain pruned networks from scratch, it probably doesn't matter which neuron we choose and therefore which criteria is better. Unstructured pruning doesn't have this training from scratch issue, and I recommend authors to at least include and maybe shift the focus to unstructured pruning. \n- \"but requires specially designed hardware (Han et al. (2016a)) or software (Elsen et al. (2020)). While results in this paper are applicable in both settings, our experimental evaluation focuses on structured pruning due to its higher relevance to practitioners.\" All neural networks require special hardware if you want to accelerate them. I think a better motivation here is to point out to the difficulties at accelerating sparse operations and limited availability/support for such operations in existing frameworks. And I am not sure how useful structured pruning algorithms are given the results of [1].\n- \"The larger the magnitude of parameters at a particular instant, the smaller the model loss at that instant will be.\" This is likely to be true in simple settings, however it is not a sufficient condition; especially for networks with batch norm. You can arbitrarily scale neurons if there is a batch-norm and you can come-up with arbitrary ordering if needed. I recommend re-phrasing this observation and/or stating the assumptions better (I don't remember seeing any assumption on the network itself). How the regularization or gradient noise will effect this statement? \n- \"Thus, the parameter with the most negative value for Θ(t)g(Θ(t)) is likely to also have a large, negative value for Θ(t)H(Θ(t))g(Θ(t))\" This is not clear to me. Assume 1d case where Θ(t)= -1; g(Θ(t))=2; H(Θ(t))=-1 -> Θ(t)g(Θ(t))=-2; Θ(t)H(Θ(t))g(Θ(t))=2. I can see the correlation in the figure but it doesn't seem like an obvious thing. Maybe because hessian don't have many negative eigenvalues?\n\n## Rating\nI found the results and analysis interesting, however motivation needs to be updated. The work would also benefit from including unstructured pruning experiments. \n\n## Minor Points\n- \"Recent works focus on pruning models at initialization (Frankle & Carbin (2019);...\" Lottery Ticket paper prunes after training and show existence of some initializations that achieve good performance.. \n-  Equations 6/7 $\\frac{dL}{dt}= ||g(\\theta)||^2$ assuming gradient descent shouldn't be a learning rate?\n- \"...than magnitude-agnostic techniques.\" Which methods are these? As far as I see, all methods use magnitude information in their formulas directly or indirectly.\n- In Table:1, I recommend authors to bold both scores if they lie within the std of each other; so that we can identify which improvements are significant.\n- It would be nice to show how the temperature parameter is used for GrasP\n\n[1] https://arxiv.org/abs/1810.05270\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1DY7M6s1lU2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper contributes to explaining why saliency measures used for pruning trained models may (or may not) also be effective for pruning untrained or minimally trained models, by developing the relationship between those saliency measures and different forms of the norm of model parameters based on the evolution of model parameters via gradient flow (basically derivatives w.r.t. time). This result leads to several interesting interpretations that could shed some light on on-going efforts to understand recent methods of pruning early-on (e.g., pruning at initialization or after minimal training) and potential extensions to existing saliency measures. The idea of employing gradient flow is novel for its purpose and seems to be accurately executed.\n\nThe main concern is that there is a gap between the flow model and the actual optimization method used in this work (SGD with momentum), or more generally standard optimization methods for deep learning. In this regard, the claim of “evolution dynamics” seems a bit exaggerated and remains as theoretical; experiments are strictly speaking not entirely valid to support it either. (minor) Related work is written as if pruning is only done via saliency-based methods (e.g., “pruning frameworks generally define importance measures”) without taking into account various others such as optimization based methods employing sparsity inducing penalty terms. On a different but related note, the writing becomes a bit loose when it comes to referencing existing methods; it is worth correcting them and clarifying the scope/focus of this work.\n\nFurther questions:\n- Why do you study structured pruning *only*? The provided reasons (“unstructured pruning requires specially designed hardwares or softwares” or “higher relevance to practitioners”) don’t seem valid enough if the purpose really lies in analyzing. Can you provide any results for unstructured pruning?\n- Can you provide evidence to support the claim “GraSP without large temperature chooses to prune earlier layers aggressively” (besides Raghu et al. 2017)?\n- Based on Tables 1 and 2 the proposed extension to loss-preservation method works the best, while the differences across different methods seem a bit marginal. Is my understanding correct?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a broad framework for unifying various pruning approaches and performs detailed analyses to make recommendations about the settings in which various approaches may be most useful. Reviewers were generally excited by the framework and analyses, but had some concerns regarding scale and the paper's focus on structured pruning. The authors included new experiments however, which mostly addressed reviewer concerns. Overall, I think is a strong paper which will likely be provide needed grounding for pruning frameworks and recommend acceptance. 
Decision: Accept (Spotlight)