Neural Pruning via Growing Regularization
Authors: Huan Wang
Can Qin
Yulun Zhang
Yun Fu
Venue: ICLR 2021 Poster (2021)
Keywords: model compression
deep neural network pruning
Hessian matrix
regularization

Abstract: Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring. (1) The former topic is newly brought up in this work, which we find critical to the pruning performance while receives little research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed. (2) The growing penalty scheme also brings us an approach to exploit the Hessian information for more accurate pruning without knowing their specific values, thus not bothered by the common Hessian approximation problems. Empirically, the proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning. Their effectiveness is demonstrated with modern deep neural networks on the CIFAR and ImageNet datasets, achieving competitive results compared to many state-of-the-art algorithms. Our code and trained models are publicly available at https://github.com/mingsun-tse/regularization-pruning.

Published as a conference paper at ICLR 2021
NEURAL PRUNING VIA GROWING REGULARIZATION
Huan Wang, Can Qin, Yulun Zhang Yun Fu
Northeastern University, Boston, MA, USA {wang. huan, gin.caa@norttttt @nootheessttrnncccmommmm edu, yulun100e gmail.mmall....mm.mm com, yunfuêece neu. edu
ABSTRACT
Regularization has long been utilized learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application grows large gradually to tackle two central problems of pruning: pruning sched- ule and weight importance scoring. (1) The former topic is newly brought up this work, which we find critical to the pruning performance while receives lit- tle research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed. (2) The growing penalty scheme also brings formation for more accurate pruning without knowing their specific values, thus not bothered by the common Hessian approximation problems. Empirically, the proposed algorithms are easy to implement and scalable to large datasets and net- works in both structured and unstructured praning. Their effectiveness is demon- strated with modern deep neural networks on the CIFAR and ImageNet datasets, achieving competitive results compared many wate-of-the-- art algorithms. Our code and trained models are publicly available at https://github.. http://tttuub.commmmgmnnn seerregularization--ruing
new scenario where the regularization
approach to exploit the Hessian in-
INTRODUCTION
As deep neural networks advance in recent years LeCun et al. (2015); Schmidhuber (2015), their remarkable effectiveness comes cost of rising storage, memory footprint, computing resources and energy consumption Cheng et al. (2017); Deng et al. (2020). Neural network pruning Han et al. (2015; 2016); Li et al. (2017); Wen et al. (2016); He et al. (2017); Gale et al. (2019) is deemed as a promising force to alleviate this problem. Since its early debut Mozer & Smolensky (1989); Reed (1993), the central problem neural network pruning has been (arguably) how to choose weights to discard, i.e., the weight importance scoring problem LeCun et al. (1990); Hassibi & Stork (1993); Molchanov et al. (2017b; 2019); Wang et al. (2019a); He et al. (2020).
The approaches to the scoring problem generally fall into two groups: importance- based and regularization-bass Reed (1993). The former focuses on directly proposing certain theoretically sound importance criterion so that pruning process is typically one-shot. In contrast, Eegularization-basee approaches typically select unimportant weights through training with a penalty term Han et al. (2015); Wen al. (2016); Liu et al. (2017). However, the penalty strength is usually maintained in a small regime to avoid dam- aging the model expressivity. Whereas, large penalty strength can be helpful, specifically in two aspects. (1) A large penalty can push unimportant weights rather close to zero, then the pruming later barely hurts the performance even if the simple weight magnitude is adopted as criterion. (2) It is well-known that different weights of neural network lic on the regions with different local quadratic structures, i.e., Hessian information. Many methods try to tap into this to build a more accurate scoring LeCun et al. (1990); Hassibi & Stork (1993); Wang et al. (2019a); Singh & Al- istarh (2020). However, for deep networks, is especially hard to estimate Hessian. Sometimes, even the computing itself can be intractable without resorting to proper approximation Wang et al. (2019a). On this problem, we ask: it possible to exploit the Hessian information without knowing
prune the unimportant weights once for all. Thus, the
"Corresponding author.

---

Published as a conference paper at ICLR 2021
their specific values? This is the second scenario where growing regularization can help. We will show under a growing regularization, the weight magnitude will naturally separate because of their different underlying local quadratic structure, therein can pick the unimportant weights more faithfully even using the simple magnitude-based criterion. Corresponding to these two aspects, will present two algorithms based on a growing I-2 regularization paradigm, in which the first highlights a better pruning schedule' and the second explores a better pruning criterion.
Our contributions. (I) We propose a simple yet effective growing regularization scheme, which can help transfer the model expressivity to the remaining part during pruning. The encouraging performance inspires us that the pruning schedule may be as critical the weight importance eni- terion and deserve more research attention. (2) We further adopt growing regularization to exploit Hessian implicitly, without knowing their specific values. The method can help choose the unim- portant weights more faithfully with theoretically sound basis. In this regard, our paper is the first to show the connection between magnitude- based pruning and Hessian- based pruning, pointing out that the latter can be turned into the first one through our proposed growing regularization scheme. (3) The proposed two algorithms are easy to implement and scalable to large-scale datasets and net- works. We show their effectiveness compared with many state-of-the-arr Especially, the methods can work seamlessly for both filter pruning and unstructured pruning.
RELATED WORK
Regularization- pruning. The first group of relevant works is those applying regularization to learn sparsity. The most famous probably is to use Le L1 regularization Louizos et al. (2018); Liu et al. (2017); Ye et al. (2018) due to their sparsity-inducing nature. In addition, the common L2 regularization is also explored for approximated sparsity Han et al. (2015; 2016). The early papers focus more on unstructured pruning, which is beneficial to model compression yet not to acceleration. For structured pruning in favor of acceleration, Group-wise Brain Damage Lebedev & Lempitsky (2016) and SSL Wen et al. (2016) propose to use Group LASSO Yuan & Lin (2006) to learn regular sparsity, where the penalty strength is still kept in small scale because the penalty is uniformly applied to all the weights. To resolve this, Ding et al. (2018) and Wang et al. (2019c) propose to employ different penalty factors for different weights, enabling large regularization.
Importance-based pruning. Importance-based pruning tries to establish certain advanced impor- tance criteria that can reflect the true relative importance among weights as faithfully as possible. The pruned weights are usually decided immediately by some proposed formula instead of by train- ing (although the whole pruning process can involve training, c.g., iterative pruning). The most widely used criterion is the magnitude- based:: weight absolute value for unstructured pruningHan et al. (2015; 2016) or L Lo-oo for structured pruning Li et al. (2017). This heuristic criterion was proposed a long time ago Reed (1993) and has been argued to be inaccurate. In this respect, improvement mainly comes from using Hessian information to obtain a more accurate approxima- tion of the increased loss when a weight is removed LeCun et al. (1990); Hassibi & Stork (1993). Hessian is intractable to compute for large networks, so some methods (e.g., EigenDamage Wang al. (2019a), WoodFisher Singh & Alistarh (2020)) employ cheap approximation (such as K-FAC Fisher Martens & Grosse (2015)) to make the 2nd-order criteria tractable on deep networks.
Note that, there is no hard boundary between the importance- Many papers present their schemes in the combination of the two Ding et al. (2018); Wang et al. method focuses more
and regularization-bast based.
(2019c), The difference mainly lies in their emphasis: Regularization-bass on an advanced penalty scheme so that the subsequent pruning criterion can be simple; while the importance- based one focus more on an advanced importance criterion itself. Mcanwhile, regular- ization paradigm always involves iterative training, while the importance- based can be one-shot Le- Cun et al. (1990); Hassibi & Stork (1993); Wang et al. (2019a) (no training for picking weights to prune) or involve iterative training Molchanov et al. (2017b; 2019); Ding al. (2019a;b).
Other model compression methods. Apart from pruning, there are also many other model com- pression approaches, e.g., quantization Courbariaux & Bengio (2016); Courbariaux et al. (2016); Rastegari et al. (2016), knowledge distillation Buciluă et al. (2006); Hinton et al. (2014), low-
By pruning schedule, we mean the way to remove weights (e.g., removing all weights in single step multi-steps), not the training schedule such learning rate settings, etc.
2

---

Published as a conference paper at ICLR 2021
rank decomposition Denton al. (2014); Jaderberg et al. (2014); Lebedev et al. (2014); Zhang et al. (2015), and efficient architecture design or search Howard et al. (2017); Sandler et al. (2018); Howard et al. (2019): Zhang et al. (2018); Tan & Le (2019); Zoph & Le (2017); Elsken et al. (2019). They are orthogonal to network pruning and can work with the proposed methods to compress more.
PROPOSED METHOD
3.1 PROBLEM FORMULATION
Pruning can be formulated as a transformation T(*) that takes a pretrained big model w as input and output small model W1. typically followed by a fine-tuning process F(*), which gives us the final output W2 = F. We do not focus on F(x) since it is simply a standard neural network training process, but focus on the process of W1 T(w). The effect of pruning can be further specified into two sub- tub-transformations: (1) M = Ti (w), which obtains a binary mask vector M that decides which weights will be removed; (2) (ww).. which adjusts the values of remaining weights. That is,
W1 Tww) = Ti (w) @ T2(w)= M @ (2(w).
(1)
For one-shot pruning, there is no iterative training at Ti- It depends a specific algorithm to decide whether to adjust the remaining weights. For example, OBD LeCun et al. (1990) and L1- norm pruning Li et al. (2017) do not adjust the kept weights (i.e., T2 is the identity function) while OBS Hassibi & Stork (1993) does. For learning-basse pruning, both T₁ and T2 involve iterative training and the kept weights will always be adjusted.
In the following, we will present our algorithms in the filter pruning scenario since we mainly focus on model acceleration instead of compression in this work. Nevertheless, the methodology can seamlessly translate to the unstructured pruning case. The difference lies in how we define the weight group: For filter pruning, a 4-d tensor convolutional filter (or 2-d tensor for fully-connected layers) is regarded as weight group, while for unstructured pruning, a single weight makes group.
3.2 PRUNING SCHEDULE: GREG-1
Our first method (GReg-I) is a variant of Lı-norm pruning Li et al. (2017). obtains the mask M by L1 Li-oorm sorting but adjusts the kept weights via regularization. Specifically, given pre-trained model w and layer pruning ratio Ti. we sort the filters by L1 and set the mask to zero for those with the least norms. Then, unlike Li et al. (2017) which removes the unimportant weights immediately (i.e., one-shot fashion), we impose a growing L2 penalty to drive them to zero first:
X, d + 8A, {j MD] = 0},
(2)
where Xy is the penalty factor for j-th weight; dA is the granularity in which we add up the penalty. Clearly, smaller 5A means this regularization process smoother. Besides, XÚ is only updated every K2 iterations, which is a buffer time to let the network adapt to the new regularization. This algorithm is to explore whether the way we remove them (i.e., pruning schedule) leads to a difference given the same weights to prune. Simple as it is, the scheme can bring significant accuracy gains especially under large pruning ratio (Tab. 1). Note that, we intentionally set 5A the same for all the unimportant weights to keep the core idea simple. Natural extensions of using different penalty factors for different weights (such as those in Ding al. (2018); Wang et al. (2019c)) may be worth exploring but out of the scope of this work.
When Xy reaches a pre-set ceiling т, we terminate the training and prune those with the least L1- norms, then fine-tune. Notably, the pruning will barely hurt the accuracy since the unimportant weights have been compressed to typically less than 1 the magnitude of remaining weights.
1000
3.3 IMPORTANCE CRITERION: GREG-2
Our second algorithm is to further take advantage of the growing regularization scheme, not for pruning schedule but scoring. The training of neural networks prone to overfitting, so regulariza- tion is normally employed. L2 regularization (or referred to as weight decay) is a standard technique for deep network training. Given a dataset DD model parameters w, the total loss will typically be
E(w, D) = L(w, D) + 2 All
(3)
3

---

Published as a conference paper at ICLR 2021
where L is the task loss function. When the training converges, there should be
Oc
Xw Ow, = 0,
(4)
where Wi indicates the i-th weight at its local minimum. Eq. (4) shows that, for each specific weight element, its equilibrium position is determined by two forces: loss gradient (i.e., guidance from the task) and regularization gradient (i.e., guidance from our prior). Our idea is slightly increase the À to break the equilibrium and see how it results in a new one. A general impression is: If À goes a little higher, the penalty force will drive the weights further towards origin and it will not stop unless proper loss gradient comes to halt it and then new equilibrium is reached at uē Considering different weights have different scales, we define a ratio r; = / /w[ to describe how much the weight magnitude changes after increasing the penalty factor. Our interest lies in how the ri differs from one another and how it relates to the underlying Hessian information.
Deep neural networks are well-known paraammtttter and highly non-convex. To obtain fea- sible analysis, we adopt a local quadratic approximation of the loss function based on Taylor se- ries expansion Strang (1991) following common practices LeCun et al. (1990); Hassibi & Stork (1993); Wang et al. (2019a). Then when the model is converged, the error & can be described by the converged weights w and the underlying Hessian matrix H (note H is p.s.d. since the model is converged). After increasing the penalty À by ox, the new converged weights can be proved to be
w (H 5A HH"
(5)
where I stands for the identity matrix. Here we meet with the common problem of estimating Hessian and its inverse, which are well-known to be intractable for deep neural networks. We explore two simplified cases to help us move forward.
(1) H is diagonal, which is a common simplification for Hessian LeCun et al. (1990), implying that the weights are independent of each other. For = with second derivative hig- With Le penalty increased by 5A (6) > 0), the new converged weights can be proved to be
hig
1
oi hi + dd W., = +T
(6)
dbtha +1'
where r, E [0, 1) since h, 2 and 8A > 0. As seen, larger hüs results in larger r; (closer to 1), meaning that the weight is relatively less moved towards the origin. Our second algorithm pri- marily builds upon this finding, which implies when we add a penalty perturhation to the converged network, the way that different weights respond can reflect their underlying Hessian information.
(2) In practice, we know H is rarely diagonal. How the dependency among weights affects the finding above is of interest. To have closed form of inverse Hessian in Eq. (5), we explore the 2-d case, namely, w = wy ). H = (112 kzz) ). A = ( hhs h12 )- The new converged weights can h22+8x)- be amalytically solved below, where the approximation equality is because that aA is rather small,
(hhn + Kndd - ) + 81h12g
1
[hinbea + kn h
(7)
102
H
(huihe + h22dA - his)ws + 6xh12w
[ (hhsh22 + h22dA - 1112) )w
(h11122 + h118d h12)) r2
(h11h22 + h226d -.
(8)
As seen, h1 > h22 also leads to rı > r2, in line with the finding above. The existence of weight dependency (i.e., the h12) actually does not affect the conclusion since it is included in both ratios.
These theoretical analyses show us that when the penalty is increased at the same pace, because of different local curvature structures, the weights actually respond differently weights with larger curvature will be less moved. As such, the magnitude discrepancy among weights will be magnified as À grows. Ultimately, the weights will naturally separate (see Fig. for an empirical validation). When the discrepancy is large enough, even the simple L1 -norm can make an accurate criterion. Notably, the whole process happens itself with the uniformly rising L2 penalty, no need to know the Hessian values, thus not bothered by any issue arising from Hessian approximation in relevant prior arts LeCun al. (1990); Hassibi & Stork (1993); Wang et al. (2019a), Singh & Alistarh (2020).
In terms of the specific algorithm, all the penalty factor is increased at the same pace,
Aj= Xy + 5A, for all j.
(9)
4

---

Published as a conference paper at ICLR 2021
Algorithm GReg-1 and GReg-2 Algorithms
Input: Pre-trained model w, pruming ratio for 1-th layer rı, 1~ L, original weight decay 7.
2: Input: Regularization ceiling т, ceiling for picking 7', interval Ka, Ks, granularity 6'.
3: Init: Iteration i 0. X = for all filter j. Set kept filter indexes Sf for each layer
4: Init: Set pruned filter indexes Sp by L -norm sorting, set S/. to full set, for each layer I.
5: while Xy S J E S/ do
6: if i % K, = 0 then
7:
if S = Ø and X3 > - E Sy then
8:
Set Sp by Lı-norm scoring, St as the complementary set of Sp, for each layer I.
9:
end if
10:
A = Xy + 5d for j E Sp. X3- = -y for j E S for each layer I
11:
end if
12:
Weight update by stochastic gradient descent (where the regularization is enforced). i
14:
end while
Train for another K, iterations stabilize. Then prune by Lı-norms and get model W1.
16:
Fine-tune W1 to regain accuracy.
17:
Output: Pruned model W2.
When X, reaches some ceiling r', the magnitude gap turns large enough to let L -nor do scor- ing faithfully. After this, the procedures are similar those in GReg-1: X for the unimportant weights are further increased. One extra step is to bring back the kept weights to the normal mag- nitude. Although they are the "survivors" during the previous competition under a large penalty, their expressivity are also hurt. To be exact, we adopt negative penalty factor for the kept weights to encourage them to recover. When the X for unimportant weights reaches the threshold 7 (akin to that of GReg-1), 1), the training is terminated. L1 -pruning is conducted and then fine-tune to regain accuracy. To this end, the proposed two algorithms can summarized in A Algorithm
Pruning ratios. We employ pre-specified pruning ratios in this work to keep the core method neat (see Appendix for more discussion). Exploring layer-wise sensitivity is out of the scope of this work, but clearly any method that finds more proper pruning ratios can readily work with our approaches.
Discussion: differences from IncReg. Although our work shares general spirit of growing reg- ularization with IncReg Wang et al. (2019c;b), our work is actually starkly different from theirs in many axes:
Motivation. The motivations for using the growing regularization are different. Wang et al. (2019c;b) adopt growing regularization to select the unimportant weights by train- ing. Namely, they focus the importance criterion problem. In contrast, we use growing regularization to investigate the pruning schedule problem (for GReg-I) or exploit the un- derlying Hessian information (for GReg-2). The importance criterion is simply L Lorm..
Algorithm design. Wang et al. (2019c;b) assign different regularization factors to different weight groups based on their relative importance, while we assign them with the same fac- tors. For GReg-1, 1, this may not be substantial difference, while for GReg-2, the difference is fundamental because the theoretical analysis of GReg-2 (Sec. 3.3) relies on the fact that regularization factors are kept the same for different weights.
Theoretical analysis. The algorithm in Wang et al. (2019c;b) is generally heuristic-based, while our work provides rigorous theoretical analyses (Sec. 3.3) support the proposed algorithm GReg-2.
Empirical performance. Both our methods are significantly better than Wang et al. (2019c;b) the large-scale ImageNet dataset, which will be shown in the experiment section (Tab. 3).
Discussion: other regularization forms. The proposed methods in this work adopts L2 regulariza- tion. Here we discuss the possibility to generalize the method to other regularization forms (L1 and Lo). (1) For GReg- it can be easily generalized to other regularization forms like L1. For GReg-2, 2, since the theoretical basis in Sec. 3.3 relies on the local quadratic approximation, Lg regularization meets this requirement while L, does not. Therefore, GReg-2 cannot be (easily) generalized to the
5

---

Published as a conference paper at ICLR 2021
Comparison between pruning schedules: one-shot pruning vs. proposed GReg-1. Each setting is randomly run for times, mean and std accuracies reported.
Table 1:
ResNet56 + CIFAR1O: Baseline accuracy 93.36%,
#Params: 0,8530M, FLOPs: 0.1255G
Pruning ratio r (%)
50
70
90
92.5
95
Sparsity (%) / Speedup
49.82/1.99x
70.57/3.59x
90.39/11.41 x
93,43/11.76x
95. 19
Acc. (%, +one-ho)
92.97
91.88+0.09
87.34
87.31
82.79
Acc. (%, GReg-1, ours)
93.06-0.08
92.28 92.23.0.211
89.49
88.39
85.97+0.16
Acc. gain (%)
0.09
0.35
2.15
1.08
3.18
VGG19 + CIFAR100: Baseline accuracy 74.02%, #Params: 20.0812M:
FLOPP: 0.3982G
Pruning ratio r (%)
50
60
70
80
90
Sparsity (%) / Speedup
74.87/3.60x
84.00/5.41
90.98/8.84x
95.95/17.30x
98.96/44.22x
Acc. (%, +one--hhot)
71.49=0.14
70.27
66.05
61.59+0.03
51.3640.11
Acc. (%, GReg-1, ours)
71.50 0.11
70.33 1.
67.35
63. 63.55 1.2.9
57.09
Acc. gain (%)
0.01
0.06
1.30
1.96
5.73
L regularization as far as we can see currently. (2) For Lo regularization, it is well-known NP-hard. In practice, it is typically converted to the I,1 regularization case, which we just discussed.
4 EXPERIMENTAL RESULTS
Datasets and networks. We first conduct analyses on the CIFARIO/100 datasets Krizhevsky (2009) with ResNet56 He et al. (2016)/GG19 Simonyan & Zisserman (2015). Then we evaluate our methods on the large-scale ImageNet dataset Deng et al. (2009) with ResNet34 and 50 He et al. (2016). For CIFAR datasets, we train our baseline models with accuracies comparable to those in the original papers. For ImageNet, we take the official PyTorch Paszke et al. (2019) pre-trained models as baseline to maintain comparability with other methods.
Training settings. To control the irrelevant factors as we can, for comparison methods that release their pruning ratios, we will adopt their ratios; otherwise, will use specified ones. We com- pare the speedup (measured by FLOPs reduction) since we mainly target model acceleration rather than compression. Detailed training settings (e.g., hyper-parameters and layer pruning ratios) are summarized in the Appendix.
4.1 RESNET56/VGGIJ ON CIFAR- 10/100
Pruning schedule: GReg-1. First, we explore the effect of different pruning schedules on the performance of pruming. Specifically, conduct two sets of experiments for comparison: (1) prune by L1 -nomr sorting and fine-tune Li et al. (2017) (shorted as "Lo+ono-sho"";; (2) employ the proposed growing regularization scheme ("GReg-1")) and fine-tune. We a uniform pruning ratio scheme here: Pruning ratio is the same for all 1-th conv layer (the first layer is not pruned following common practice Gale al. (2019)). For ResNet56, since has the residual addition restriction, we only prune the first conv layer in a block as previous works do Li et al. (2017), For comprehensive comparisons, the pruning ratios vary in a large spectrum, covering acceleration ratios from around 2x to 44x. Note that we do not intend to obtain the best performance here but systematically explore the effect of different pruning schedules, so we employ relatively simple settings (e.g., the uniform pruning ratios). For fair comparisons, the fine-tuning scheme (e.g., number of epochs, learning rate schedule, etc.) is the same for different methods. Therefore, the key comparison here to see which method can deliver better base model before fine-tuning.
The results are shown in Tab. We have the following observations: (1) On the whole, the proposed GReg-1 consistently outperforms L sone-hott. It is important reiterate that the two settings have exactly the same pruned weights, so the only difference how they are removed. The accuracy gaps show that apart from importance scoring, pruning schedule is also critical factor. In the Appendix D, we present more results to demonstrate this finding actually is general, not merely limited to the case of Iı-norm criterion. The proposed iallaariiaatiobssaeed pruning schedule is consistently more favorable than the one-shot counterpart. (2) The larger pruning ratio, the more

---

Published as a conference paper at ICLR 2021
(a) ResNet56
(b) VGG19
(c) ResNet34
(d) ResNer50
(a) Iter
(b) Iter 5,000
(c) Iter 10,000
(d) Iter 15,000
Figure 1: Row 1: Illustration of weight separation as L2 penalty grows. Row 2: Normalized filter
L -nor over iterations for ResNet50 layer2.3.comy (please see the Appendix for VGG19 plots).
Table 2: Comparison of different methods on the CIFARIO and CIFARI00 datasets.
Mcthod
Network/Datasct Basc acc. (%)
Pruned acc. (%)
Acc. drop Speedup
CP He et al. (2017)
92.80
91.80
1.00
2,00
AMC He ct al. (2018b)
92.80
91.90
0.90
2,00x
SFP Hc ct al. (2018a)
ResNet56/CIFAR10
93.59
93.36
0.23
2.11x
AFP Ding al. (2018)
93.93
92.94
0.99
2,56x
C-SGD Ding et al. (2019a)
93.39
93.44
-0.05
2.55x
GReg-1 (ours)
93.36
93.18
0.18
2.55x
GReg-2 (ours)
93.36
93.36
0,00
2.55x
Kron-OBD Wang al. (2019a)
73.34
60.70
12.64
5.73x
Kron-OBS Wang et al. (2019a)
73.34
60.66
12.68
6.09x
EigenDamage Wang et al. (2019a) VGGI9/CIFARI00
73.34
65.18
8.16
8.80x
GReg-1 (ours)
74.02
67.55
6.67
8.84x
GReg-2 (ours)
74.02
67.75
6.47
8.84x
pronounced of the gain. This is reasonable since when more weights are pruned, the network cannot recover by its inherent plasticity Mittal et al. (2018), then the regularization-base helpful because it helps the model transfer its expressive power the remaining part. When the pruning ratio is relatively small (such as ResNe156, r = 50%), the plasticity of the model is enough to heal, so the benefit from GReg-1 is less significant compared with the one-shot counterpart.
way is more
Importance criterion: GReg-2. Here we empirically validate our finding in Sec. 3.3, that is, with uniformly rising La penalty, the weights should naturally separate. We claim, if hn h22- there should be r1 T2, where T1 an ,2 wa
da (the mark indicating the local minimum
is omitted here for readability). ri > r2 leads to ai wia namely, ri 16 ai This shows that, after the L2 penalty grows little, the new magnitude ratio of weight I over weight 2 will be magnified hi fu22 (W1, w2 are positive in the analysis here, while the conclusion still holds if either of them is negative). In Fig. (Row 1), we plot the standard deviation (divided by the means for normalization since the magnitude varies over iterations) of filter Ln-norms as the regularization grows. As seen, the normalized -nom stddev grows larger and larger as À grows. This phenomenon consistently appears across different models and datasets. To figuratively understand how the increasing penalty affects the relative magnitude over time, in Fig. 1 (Row 2), we plot the relative I, -noms (divided by the max T, -norm for normalization) at different iterations. As shown, it is hard to tell which filters are really important by the initial filter magnitude (Iter 0), but under a large penalty later, their discrepancy turns more and more obvious and finally it is very easy identify which filters more important. Since the magnitude gap is large, the simple Li-norm can make sufficiently faithful criterion.
CIFAR benchmarks. Finally, compare the proposed algorithms with existing methods the CIFAR datasets (Tab. 2). Here we adopt non-uniform pruning ratios (see the Appendix for specific numbers) for the best accuracy-FLOPP trade-off. On CIFAR10, compared with AMC He et al.

---

Published as a conference paper at ICLR 2021
Table 3: Acceleration comparison ImageNet. FLOPP: ResNet34: 3.66G, ResNet50: 4.09G.
Method
Network
Base top-1 (%)
Pruned top-1 (%)
Top-1 drop
Speedup
Li (pruned-Bi B) Li al. (2017)
73.23
72.17
1.06
1.32x
Taylor-FO Molchanov al. (2019)
73.31
72.83
0.48
1.29
ResNet34
GReg-1 (ours)
73.31
73.54
-0.23
1.32x
GReg-2 (ours)
73.31
73.61
-0.30
1.32x
ProvableFP Liebenwein et al. (2020)
76.13
75.21
0.92
1.43x
ResNer50
GReg-1 (ours)
76.13
76.27
-0.14
1.49x
"OFFP Ding et al. (20196)
75.34
75.63
-0.99
1.49x
ResNet50
GReg-1 (ours)
75.40
76.13
-0.73
1.49x
IncReg Wang et al. (2019b)
75.60
72.47
3.13
2.00x
SFP He et (2018a)
76.15
74.61
1.54
1.72x
HRank Lin et al. (2020a)
76.15
74.98
1.17
1.78x
Taylor-FO Molchanov et al. (2019)
76.18
74.50
1.68
1.82x
Factorized Li et al. (2019)
ResNet50
76.15
74.55
1.60
2.33x
DCP Zhuang
(2018)
76.01
74.95
1.06
2.25x
CCP-AC Peng et al. (2019)
76.15
75.32
0.83
2.18x
GReg-I (ours)
76.13
75.16
0.97
2.31x
GReg-2 (ours)
76.13
75.36
0.77
2.31X
CCSGDD30 Ding al. (2019h)
75.34
74.34
0.80
2,6x
AOPP Ding et al. (2019h)
ResNet50
75.34
75.11
0.23
2.31
GReg-2 (ours)"
75.40
75.22
0.18
2.31
LFPC He et al. (2020)
76.15
74,46
1.69
2.55x
GReg-1 (ours)
ResNet50
76.13
74.85
1.28
2,56x
GReg-2 (ours)
76.13
74.93
1.20
2.56x
IncReg Wang et al. (2019b)
75.60
71.07
4.53
3,00x
Taylor-FO Molchanov et al. (2019) ResNer50
76.18
71.69
4.49
3.05 x
GReg-1 (ours)
76.13
73.75
2.38
3.06
GReg-2 (ours)
76.13
73.90
2.23
3.06
Since the base models of C-SGD and AOFP have a much lower accuracy than ours, for fair comparison, we
train our base models with similar accuracy.
Table 4: Compression comparison on ImageNet with ResNet50. #Parameters: 25.56M.
Method
Base top- (%)
Pruned top- (%)
Top-1 drop Sparsity (%)
GSM Ding et al. (2019c)
75.72
74.30
1.42
80.00
Variational Dropout Molchanov et al. (2017a)
76.69
75.28
1.41
80.00
DPF Lin al. (2020b)
75.95
74.55
1.40
82.60
WoodFisher Singh & Alistarh (2020)
75.98
75.20
0.78
82.70
GReg-1 (ours)
76.
75.45
82.70
GReg-2 (ours)
76.13
75.27
0.86
82.70
(2018b), though it adopts better layer-wise pruning ratios via ceinforceeentt-eamming
algorithms
can still deliver more favorable performance using sub-optimal human-specified ratios. AFP Ding al. (2018) is another work exploring large regularization, while they do not adopt the growing scheme as we do. Its performance is also less favorable on CIFARIO as shown in the table, Although our methods perform a little worse than C-SGD Ding et (2019a) on CIFARIO. ImageNet dataset, we will show our methods are significantly better than C-SGD.
the large-scale
Notably, on CIFARI00, Kron-OBD/OBS (an extension by Wang al. (2019a) of the original OBD/OBS from unstructured pruning to structured pruning) are believed to be more accurate than In-norm in terms of capturing relative weight importance LeCun (1993); Wang et al. (2019a). Yet, they are significantly outperformed by our GReg-1 1 based on the simple -nom scoring. This may inspire us that an average pruning schedule (like the one-shot fashion) can offset the gain from more advanced importance scoring scheme.
(1990): Hassibi & Stork
4.2 RESNET34/50 ON IMAGENET
Then we evaluate our methods on the standard large-scale ImageNet benchmarks with ResNets He et al. (2016). We refer the official PyTorch ImageNet training example" to make the imple-
8

---

Published as a conference paper at ICLR 2021
mentation (such data augmentation, weight decay, momentum, etc.) is standard. Please refer to the summarized training setting in the Appendix for details.
The results shown in Tab. 3. Methods with similar speedup are grouped together for casy com- parison. In general, our method achieves comparable or better performance across various speedups on ResNet34 and 50. Concretely, (1) On both ResNet34 and 50, when the speedup is small (less than 2x), only our methods (and AOFP Ding et al. (2019b) for ResNet50) can even improve the top-1 accuracy. This phenomenon is broadly found by previous works Wen et al. (2016); Wang al. (2018); He et al. (2017) but mainly on small datasets like CIFAR, while we make it on the much challenging ImageNet benchmark. (2) Similar to the results on CIFAR (Tab. 1), when the speedup is larger, the udvantage of our method is more obvious. For example, ours GReg-2 only outperforms Taylor-FO Molchanov et al. (2019) by 0.86% top-1 accuracy at the 2x setting, while ~ 3x, 2 2 is better by 2.21% % top-1 accuracy. (3) Many methods work on the weight importance crite- rion problem, including some very recent ones (ProvableFP Liehenwein et al. (2020), LFPC He al. (2020)). Yet as shown, our simple variant of L1 -nom pruning can still be strong competitor in terms of accuracy-FLOPs trade-off. This reiterates one of our key ideas this work that the pruning schedule may be as important as weight importance scoring and worth more research attention.
Unstructured pruning. Although we mainly target filter pruning in this work, the proposed meth- ods actually can be applied to unstructured pruning effectively. In Tab, 4, we present the results of unstructured pruning on ResNe150. WoodFisher Singh & Alistarh (2020) is the state-of-the-ard Hessian-based unstructured pruning approach. Notably, without any Hessian approximation, GReg-2 can achieve comparable performance with it (better absolute accuracy, yet slightly worse accuracy drop). Besides, the simple magnitude pruning variant GReg-I delivers more favorable result, implying that a better pruning schedule also matters in the unstructured pruning case.
CONCLUSION
Regularization long deemed as a sparsity-learning tool in neural network pruning, which usually works in the small strength regime. In this work, we present two algorithms that exploit regulariza- tion in a new fashion that the penalty factor is uniformly raised to large level. Two central problems regarding deep neural pruning are tackled by the proposed methods, pruning schedule and weight importance criterion. The proposed approaches rely on few impractical assumptions, have a sound theoretical basis, and are scalable large datasets and networks. Apart from the methodology it- self, the encouraging results on CIFAR and ImageNet also justify our general ideas in this paper: (1) In addition to weight importance scoring, pruning schedule is another pivotal factor in deep neural pruning which may deserve more research attention.
(2) Without any Hessian approximation, we can still tap into its power for pruning with the help of growing L2 regularization.
ACKNOWLEDGEEMENTS
The work is supported by the National Science Foundation Award ECCS-191682 and the U.S. Army Research Office Award 1911NF-11--1-067..
REFERENCES
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii, Model compression. In SIGKDD, 2006. 2
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09228) 2017.
M. Courbariaux and Y. Bengio. BinaryNet: Training deep neural networks with weights and activa- tions constrained +1 or -1. arXiv preprint arXiv:1602.028344 2016. 2
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or 1. arXiv preprint 16Xiv:1602.028800 2016. 2

---

Published as a conference paper at ICLR 2021
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey, Proceedings of the IEEE, 108(4): 485-532, 2020.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NewrIPS, 2014. 3
Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. In AAAI, 2018. 2, 7.8
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very
deep convolutional networks with complicated structure. In CVPR, 2019a. 2, 7,8 8
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive enn width optimization. In ICML, 2019b. 2, 8, 9, 13
Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, al. Global sparse momentum sgd for pruning very deep neural networks. In NeurIPS, 2019c. 8
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. JMLR, 20(55):1-21 1-21, 2019. 3
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: N0Xiv::9022099744 2019. 1,
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. NeurIPS, 2015. 1,2
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. 1,2
B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NewrlPS, 1993. 2, 3, 4,
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 6,8
Yang He, Guoliang Kang, Xuanyi Dong. Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In ISCAI, 2018a. 7,8
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. CVPR, 2020. 1,8,
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net-
works. In ICCV, 2017. 1, 7,7, 7, 13
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. AMC: Automl for model compression and acceleration on mobile devices. In ECCV, 2018b, 8b. 7
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In
NeurlPS Workshop, 2014. 2
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint /905, 022244 2019. 3
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint veXiv:1704.0480 2017. 3
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014.
10

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 6
V. Lebedev and V. Lempitsky. Fast convnets using group-wise brain damage. In CVPR, 2016. 2
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv: 11126554 2014.
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In NeurlPS, 1990. 1, 3, 13, 14, 15
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436, 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017. 2, 3, 8, 14
Tuanhui Li, Baoyuan Wu, Yujuu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convo- lutional neural networks via factorized convolutional filters. In CVPR, 2019. 8
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In ICLR, 2020. 8,
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In CVPR, 2020a.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In ICLR, 2020b. 8
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017. 1,2
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019, 14
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. In ICLR, 2018. 2
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In /CML, 2015. 2
Deepak Mittal, Shweta Bhardwaj, Mitesh M Khapra, and Balaraman Ravindran. Recovering from random pruning: On the plasticity of deep convolutional neural networks. In WACV, 2018. 7
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML., 2017a. 8
Molchanov, S. Tyree, and Karras. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017b. 1,2
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, 2019. 2, 8,
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. In NeurlPS, 1989.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019. 6
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In ICML. 2019. 8
11

---

Published as a conference paper at ICLR 2021
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
R. Reed. Pruning algorithms survey. IEEE Transactions Neural Networks, 4(5):740-747, 1993. 1,2 2
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. In NeurIPS, 2020. 1,2, 8,
Gilbert Strang. Calculus. Wellesley-Cambridge Press, 1991.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the samecker-factored cigenbasis. ICML, 2019a. 1, 2, 4, 8
Huan Wang, Qiming Zhang, Yuehai Wang, and Haoji Hu. Structured probabilistic pruning for convolutional neural network acceleration. In BMVC, 2018.
Huan Wang, Xinyi Hu, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruning for efficient convolutional neural networks via incremental regularization. IEEE Journal of Selected Topics in Signal Processing, 14(4):775-788, 2019b. 8
Huan Wang, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruning for efficient convnets via incremental regularization. In IJCNN, 2019c. 2,
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NeurlPS, 2016. 1, 2,
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller- norm feoommllss--ee www---lees----neeeeee informative assumption in channel pruming of convolution layers. In ICLR, 2018.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, 68(1):49-67, 2006. 2
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. In CVPR, 2015.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. 3
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awen aware channel pruning for deep neural networks. In NeurlPS, 2018.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A. EXPERIMENTAL SETTING DETAILS
Training setting summary. About the networks evaluated, we intentionally avoid AlexNet and VGG on the ImageNet benchmark because the single- bragc- architecture is no longer representa- tive of the modern deep network architectures with residuals (but still keep VGG19 on the CIFAR analysis to make sure the findings are not limited to one specific architecture). Apart from some key settings stated the paper, a more detailed training setting summary is shown as Tab. 5.
Table 5: Training setting summary. For the SGD solver, the parentheses are the momentum and weight decay. For ImageNet, batch size 64 is used for pruning instead of the standard 256, which is because we want to save the training time.
Dataset
CIFAR
ImageNet
Solver
SGD (0.9, 5c-4)
SGD (0.9, 1c-4)
LR policy (prune)
Fixed (Ie-3)
LR policy (linetune)
step (0:1e-2, 60:1e-3, 90:le-4) Multi- step (0:1e-2, 30:1e-3, 60:1e-4, 75:1e-5)
Total epoch (finctune)
120
90
Batch size (prune)
256
64
Batch size (finctunc)
256
Pruning ratios. Although several recent methods Ding et al. (2019b); Singh & Alistarh (2020) can automatically decide pruning ratios, in this paper we opt to consider pruning independent with the pruning ratio choosing. The main consideration is that pruning ratio is broadly believed to reflect the redundancy of different layers LeCun et al. (1990); Wen et al. (2016); He et al. (2017), which an inherent characteristic of the model, thus should not be coupled with the subsequent pruning algorithms.
Before we list the specific pruning ratios, we explain how we them. (1) For ResNet, if has N stages, we will use list of N floats to represent its pruning ratios for the N stages. For example, ResNet56 has stages in conv layers, then "[0, 0.5, 0.5, 0.5]" means "for the first stage (which also the first conv layer), the pruning ratio is 0; the other three stages have pruning ratio of 0.5". Besides, since we do not prune the last conv in a residual block, which means for two-layer residual block (for ResNet56), 50), we only prune the first and second layers. (2) For VGG19, we use the following pruning ratio setting. For example, "[0:0, 1-9:0.3, 10-15:0.517 means "for the first layer (index starting from O), the pruning ratio is 0; for layer to 9, the pruning ratio is 0.3; for layer 10 to 15, the pruning ratio is 0.5"
only prune the first layer; for three-layer bottleneck block (for ResNet34 and
With these, the specific pruning ratio for each of our experiments in the paper are listed in Tab, 6. We do not have strong rules them, except one, which is setting the pruning ratios of higher stages smaller, because the FL.OPs of higher layers are relatively smaller (due to the fact that the spatial feature map sizes are smaller) and we are targeting more acceleration than compression. Of course, this scheme only is quite crude, yet as our results (Tab. 3 and 4) show, even with these crude settings, the performances are still competitive.
B PROOF OF EQ. 5
When quadratic function converges at w with Hessian matrix H. it can be formulated as E (w H(w w") + C,
(10)
where C' is constant. Now new function is made by increasing the La penalty by small amount 5A, namely.
E + 5A Tw.
(11)
Let the new converged values be w then similar to Eq. 10, & can be formulated as
E (w w FH( w
Ć, where A H 6AI.
(12)
13

---

Published as a conference paper at ICLR 2021
Table 6: Pruning ratio summary.
Dataset
Network
Speedup
Pruned top-1 accuracy (%)
Pruning ratio
CIFARTO
ResNer56
2.55x
93.36
[0. 0.75, 0.75, 0.32, 0] [1-15:0.7]
CIFAR 100
VGG19
8.84
67.56
ImageNet
ResNet33
1.32x
73.44
[0, 0.50, 0.60, 0.40, OJ"
ImageNet
ResNet50
1.49
76.24
[0, 0.30, 0.30, 0.30, 0.14, 0]
ImageNet
ResNet50
2.31x
75.16
[0, 0.60, 0.60, 0.60, 0.21,
ImageNet
ResNet50
2.56x
74.75
[0, 0.74, 0.74, 0.60, 0.21, 0]
ImageNet
ResNet50
3.06
73.50
[0, 0.68, 0.68, 0.68, 0.50,
* In addition to the pruning ratios, several layers are skipped, following the setting of T1 (pruned-B) Li al. (2017). Specifically, we refer to the implementation of Liu et al. (2019) https///gihuh..om/fiicc
1-nor-pruninng
Meanwhile, combine Eq. 10 and Eq. 11, we can obtain
= (w -w")" H(( w") 8Aw Iw
(13)
Compare Eq. 13 with Eq. 12, we have
(H + 6XI) Hw* = w'= 6AI) 'Hw".
(14)
C PROOF OF EQ. 7
h11
+ 8A
h12
H-1
h22 + 0. -h12
-h12
(15)
h12
h22 + 82ʃ
H
h1 + 88A
Therefore, w
H Hw'W"
Hww
10 H
H
wi
huz 5A
-112
{ h1
hi2
wi
102
w
H
-112
hn + 82
{
hi2
hz2
w
(16)
(lunh22 + kn1A Nig 8Xhr2g
H
(hur/z + luzdd hiz)wr + 8hh1gwi
D GREG-1 + OBD
In Sec. 4.1, we show when pruning the same weights, GReg-I is significantly better than the one- shot counterpart, where the pruned weights are selected by the Ln-norm criterion. Here we conduct the same comparison just with different pruning criterion introduced in OBD LeCun et al. (1990). OBD is also an one-shot pruning method, using a Hessian-based criterion which is believed to be more advanced than -nomm.
Results are shown in Tab. 7. As seen, using this more advanced importance criterion, our pruning scheme based growing regularization is still consistently better than the one-shot counterpart. Besides, it is also verified here that better pruning schedule can bring more accuracy gain when the speedup larger.
E FILTER L -I-NORM CHANGE OF VGG19
In Fig. (Row 2), we plot the filter L, norm change over time for ResNet50 on ImageNet. Here we plot case of VGG19 on CIFAR 100 to show the weight separation phenomenon under growing regularization is general one across different datasets and networks.
F
AND SENSITIVITY ANALYSIS
There are five introduced values in our methods: interval K,, K,, granularity 5A. Their settings are summarized in Tab. 8. Among them, the ceilings
regularization ceiling 7, ceiling for picking 7',
14

---

Published as a conference paper at ICLR 2021
Table 7: Comparison between pruning schedules: one-shot pruning vs. our proposed GReg-1 using the Hessian-based criterion introduced in OBD LeCun et al. (I990). Each setting is randomly run for 3 times, mean and std accuracies reported. We vary the global pruning ratio from 0.7 0.95 so as to cover the major speedup spectrum of interest. Same as Tab. 1, the pruned weights here are exactly the same for the two methods under each speedup ratio. The finetuning processes (number of epochs, LR schedules, etc.) are also the same to keep fair comparison.
ResNet56 CIFARIO: Baseline accuracy 93.36% #Params: 0.8530M, FLOPs: 0.1255G
Speedup
2.15x
3.00x
4.86x
5.80x
6.87
Acc. (%, OBD)
92.90 (0.05)
91.90 (0.04)
89.82 (0.11)
88,56 (0.11)
86.90 (0.03)
Acc. (%, Ours)
92.94 (0.12)
92.27 (0.14)
90.37 (0.17)
89.78 (0.06)
88.69 (0.06)
Ace. gain (%)
0.04
0.37
0.55
1.22
1.79
VGG19 + CIFARI00: Baseline accuracy 74.02%, #Params: 20.0812M, FLOPs:
0.3982G
Speedup
1.92x
2,96x
5,89
7.69x
11,75x
Acc. (%, OBD)
72.68 (0.08)
70.42 (0.16)
62.54 (0.13)
59.18 (0.32)
54.19 (0.57)
Ace. (%, Ours)
73.08 (0.11)
71.30 (0.28)
65.83 (0.13)
62.87 (0.20)
59.53 (0.10)
Acc. gain (%)
0.40
0.88
3.29
3.69
5.34
A MHLhh Myy
(a) Iter O
(b) Iter 10,000
(c) Iter 20,000
(d) Iter 30,000
Figure 2: Normalized filter L norm over iterations for VGG19 layer3.
1 is set to make sure the unimportant weights are pushed down enough (as stated in the main paper, normally after the regularization training, their magnitudes are too small to cause significant accuracy degradation if they are completely removed). 7' 0.01 is set generally for the same goal as r. but since it is applied to all the weight (not jnstjust the unimportant ones), we only expect it to be moderately large (thus smaller than T) so that the important and unim- portant can be differentiated with clear boundary. For the dA, we use very small regularization granularity 5A, which our theoretical analysis is based on. We set its value to Ic-4 for GReg-1 I and lc-5 for GReg-2 with reference the original weight decay value 5 x 10 (for CIFAR models) and 10-4 (for ImageNet models). Note that, these values come from our methods per se, not directly related to datasets and networks, thus invariant them. This is why we can employ the same set- ting of these three hyper-parameted in all our experiments, freeing practitioners from heavy tuning when dealing with different networks or datasets.
are set through validation:
Table 8: Hyper-parameters of our methods.
Notation
Default value (CIFAR) Default value (ImageNet)
8A
GReg- I: Ie-4, GReg-2: lc-5
1
7'
0.01
K.. 5k iterations
10 iterations
iterations
K,
40k iterations
A little bit of change is for Ku, Kg. Both are generally to let the network have enough time to converge to the new equilibrium. Generally, we prefer large update intervals, yet we also need to consider the time complexity: Too large of them will bring too many iterations, which may be unnecessary. Among them, K, is less important since it is to stabilize the large regularization (T = 1). We introduce it simply to make sure the training is fully converged. Therefore, the possibly more sensitive byper-parameter is the Kú, (set to 5 for ImageNet and 10 for CIFAR). Here we will show the performance is insensitive to the varying Kg, As shown in Tab. 9, the peak performance appears around Ke 15 for ResNet56 and K, = 10 for VGG19. We simply adopt 10 for uniform setting in our paper. We did not heavily tune these hyper-paramente
yet as seen, they work pretty well across different networks and datasets. Notably, even for the worst cases in Tab. 9 (in
15

---

Published as a conference paper at ICLR 2021
blue color), they are still significantly better than those of the "L=ooneshot"" scheme, demonstrating the robustness of the proposed algorithm.
Table 9: Sensitivity analysis of K2 on CIFAR10/100 datasets with the proposed GReg-I algorithm.
Ku 10 is the default setting. Pruning ratio 90% (ResNet56) and 70% (VGG19) explored here. Experiments are randomly run for times with mean accuracy and standard deviation reported. The best is highlighted with bold and the worst is highlighted with blue color.
K
5
10
15
20
+o shot
Acc. (%, ResNet56)
89.40.0.04
89.38
89.1940.24
89.69 05
89.6240.18
87.31
Ace. (%, VGG19) 67.22
67.32
67. 07.355.0.15
67 v0.06
66.93+a.22
66.05 +0.04
G MORE RESULTS OF PRUNING SCHEDULE COMPARISON
In Tab. show using Lı-norm sorting, our proposed GReg-1 consistently surpass the shot schedule even pruning the same weights. Here we ask more general question: Can the benefits from a regularization-based schedule consistently appear, agnostic the weight importance scoring criterion? This question is important because it will show if the gain from a better pruning schedule only a bonus concurrent with the L1 criterion or a really universal phenomenon. Since there are literally so many weight importance criteria, we cannot ablate them one by one. Nevertheless, given a pre-trained model and pruning ratio r, no matter what criterion, its role is select filter subset. For example, there are 100 filters layer and 0.5, then they are at most (300) importance criteria in theory for this layer. We can simply randomly pick a subset of filters (which corresponds certain criterion, albeit unknown) and compare the one-shot way with regularization- based way on the subset. Based on this idea, we conduct five random runs on the ResNet56 and VGG19 to explore this. The pruning ratio is chosen as 90% for ResNet56 and 70% for VGG19 because under this ratio the compression (or acceleration) ratio is about 10 times, neither too large nor too small (where the network can heal itself regardless of pruning methods).
The results shown in Tab. 10. Here is sanity check: Compared with Tab. the mean accuracy of pruning randomly picked filters should be less than pruning those picked by L -norm, confirmed by 86.85% 87.34% for ResNet56 and 65.04% 66.05% for VGG19. As seen, in each run, the regularization-- based way also significantly surpasses its one-shot counterpart. Although five random runs still too few given the exploding potential combinations, yet shown by the curacy standard deviations, the results are stable and thus qualified to support our finding that the regularization-base pruning schedule is better the one-shot counterpart.
Table 10: Comparison between pruning schedules: one-shot GReg-1. Pruning ratio is 90% for ResNet56 and 70% for VGG19. In each run, the weights to prune are picked randomly before the training starts.
ResNet56 + CIFARIO
Run
Run #2
Run #3
Run #4
Run #5
Med
Acc. (%, one-shot)
87.57
87.00
86.27
86.75
86.67
86.85
Acc. (%, GReg-1, ours)
89.26
88.98
88.78
89.42
88.96
89.08
VGG19 CIFAR100
Run #1
Run #2
Run #3
Run #4
Run #5
Mean
Acc. (%, one-shot) Acc. (%, GReg-1, ours)
64.56
65.06
65.07
65.05
65.48
60.04
66.63
66.57
66.80
66.80
67.16
16


Reviews:
[{"review_id": "twxK8DYsy4y", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose regularization-based pruning methods with the penalty factors uniformly increased over the training session. The first algorithm (GReg-1) sorts the filters by L1-norm and only applies the increasing regularization to the “unimportant” filters; the second one (GReg-2) applies the increasing regularization to all the filters. The experiments are very extensive and convincing to support the claimed contributions.\n\nStrengths:\n1.The idea of utilizing Hessian without knowing its values is interesting to me. Theoretical analysis in Sec. 3.3 looks sound, which should be the main theoretical contribution of this work.\n\n2.Empirical performances are promising, especially the ImageNet one in Tab. 3 and 4. \n\n3.Good performance. On CIFAR-10 and ImageNet, they evaluate the proposed methods with popular deep neural networks, reporting encouraging performances.\n\n4.The methods are easy to implement with current deep learning tools and can work in both the structure pruning and unstructured pruning cases, which is a plus. \n\n5.The authors present theory analyses to show this will cause the filters to separate owing to their different underlying Hessian structure, thus achieving “exploiting the Hessian information without knowing their specific values”. \n\n6.Generally, this paper is well-written with sound proofs for their formulas, although there are some small problems (see the weakness below) that the authors may want to resolve.\n\nWeakness:\n1.In Sec. 3.3, it says h11 > h22 leads to r1 > r2, where r is defined by the new magnitude over the old one. But in Fig. 1, the plot is the normalized magnitude stddev. How these two are related to each other is not so straight to me. Could the authors give more explanation about this since it is said in the paper that Fig.1 is an “empirical validation” for the analysis in Sec. 3.3?\n\n2.Some typos and small glitches: \n“importance-based one focus more” -> focuses\nAppendix: In the last sentence of A.1 “our results (Tab. R3 and R4)” (I didn’t find R3, R4). Please clarify it.\nAppendix: Footnote in Tab. 6, the references of [7] [54] seem pointing to nowhere.\n\n===============================After response ====================== The authors have addressed all my concerns.  I would like to keep my initial score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jHvpZGpspHa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper deals with regularisation based model compression for DNNs focusing on filter pruning. In particular they focus on growing regularisation during training instead of using a high regularisation parameter at the beginning. Growing regularisation for unimportant weights and negative penalty for important weights help them in improving the expressive power of the important weights.\n\n+ve\n\n- The paper is easy to follow and the authors make it quite clear about the problems related to filter pruning that they are tackling in this work.\n\n- The experiments conducted are quite elaborate and the authors have provided detailed description about the experimental setup, hyper-parameters tuning, and training strategy.\n\nConcerns\n\n- The core idea of growing the regularisation parameter is not quite novel. The paper “Structured pruning for efficient convnets via incremental regularization“ by Wang et. al. also uses different learning rates for different set of parameters as well as they also grow the regularisation during training.\n\n- In table 1, it would good to see a competitive method other than one-shot pruning because it is well established (this paper also mentions it correctly) that pruning a large number of filters in one-shot usually results in significant drop in accuracy. Comparing with iterative pruning methods like “Importance estimation for neural network pruning” by Molchanov et.al. would make the comparison fair especially for higher pruning percentages.\n\n- While comparing GReg1 & GReg2 with other methods in Table 2&3, it would be good to see comparison with other regularisation based methods like “Structured pruning for efficient convnets via incremental regularization“ by Wang et. al. \n\n- In Table 2, the authors should include comparison with Ding et.al. for VGG as well. It seems that for ResNet56 the performance of GReg-1 and GReg-2 is quite comparable to Ding et.al.  \n- The paper needs thorough proof reading as there are a lot of grammatical mistakes.\n\n- Lastly, the style files used by the authors doesn’t match with the ICLR2021 style files. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PHjmCFBg75Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores how the basic L2 regularization can be exploited in a growing fashion for better deep network pruning. The authors proposed two algorithms in this work: (1) The first (called GReg-1) is a variant of the L1-norm based filter pruning method [Ref1]. The important/unimportant filters are decided by their L1-norms. Later the unimportant ones are forced to zero through the proposed rising penalty scheme. (2) The second algortihm (called Greg-2) imposes the rising L2 regularization on all the filters. It is theoretically shown in the paper that this makes the parameters to separate to different degrees according to their local curvatures (ie, Hessian values). The method takes advantage of this by driving the weights into two groups with stark magnitude difference and then prunes by the simple L1-norm criterion.  The two methods are demonstrated effective on CIFAR10/100 and ImageNet benchmarks in the comparison with many state-of-the-art methods. \n\nSignificance:\nThe paper focuses on deep neural network pruning, which is of interest to both the academia and industry. The empirical results look significant when compared with related works.\n\nClarity and quality:\nThe writing of paper is clear. Important formulas are proved. Related works are properly discussed.\n\nOriginality:\nThe growing regularization method looks novel to me. It further consists of two branches. The first one is simple but the results are inspiring. The second one has a good theoretical basis.\n\nPros:\nThe regularization process introduced in GReg-1 allows the network to adapt and recover before some weights are eventually removed. This idea is easy to understand and the investigation of the effect of regularization pruning schedule is insteresting. It is quite surprising that when pruning the same weights, regularization schedule can be quite better than the one-shot manner.\n\nThe proposed strategy and its theory basis of GReg-2 seem novel to me. It presents a new way to exploit the Hessian information like no other works before. Although the analyses are two simplified cases, the intuition is straight and also verified empirically on modern deep networks.\n\nThe empirical study looks strong on CIFAR10/100 and the large scale ImageNet benchmark. \n\nIn addition to structured pruning (filter pruning), they also evaluate their methods in the unstructured pruning case and also achieve good performance compared with the recent pruning paper based on advanced Hessian-approximation.\n\nCons:\nThe two methods work pretty well on ImageNet from table 3 and 4, yet one small concern is they are outperformed by C-SGD on CIFAR10 in table 2. Is there more explanation and analysis for this?\n\nCan the proposed methods be generalized to other regularization forms like L1 and L0 [Ref2]? This should be discussed properly in the paper.\n\nOther questions:\nHow the layer-wise pruning ratios are decided in table 3 and 4 for the proposed methods?\n\nFor the Greg-1, it shows pruning the same weights with different schedule schemes can lead to quite different results. One concurrent paper [Ref3] in this venue has a similar finding (different pruning methods with different performances turn out pruning very similar weights). How is your work related with theirs?\n\n[Ref1] Pruning filters for efficient convnets. In ICLR, 2017.\n[Ref2] Learning sparse neural networks through l0 regularization. In ICLR, 2018¬¬¬\n[Ref3] Rethinking the Pruning Criteria for Convolutional Neural Network, Anonymous, ICLR 2021 submission: https://openreview.net/forum?id=ZD7Ll4pAw7C\n\nIn all, I think it’s a good paper and rate as ‘accept’. If most of my concerns and questions are addressed in the rebuttal, I would like to upgrade my rating score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6rRos5i0yO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new pruning scenario using regularization to better prune the network. The scenario has two-component, the first one proposes a new pruning schedule that does not directly remove the neurons that need to prune from the network. It removes the neurons by adding an L2 regularization and makes the neurons that need to remove gradually decrease to zero. The second one gives the importance score to the neurons. It uses the L2 regularization and studies how the coefficient \\lambda of the regularization term can influence the weight change to derive the neuron's importance in the neuron network. By perturbing the penalty term to the converged network, the algorithm can get the Hessian information to score the neurons but uses less time than calculating the Hessian. The paper also shows many empirical results on various benchmarks to show their advantages when using the new schedule and scoring criterion during the pruning process. The result shows that their method can get better at a fast speed.\n\nPros:\nI think the paper is interesting, both two steps make sense to me.  For the importance criterion statement, the paper also gives the theoretical analysis. Plus, the paper in good writing and easy to understand.\n\nCons:\nCurrently, GReg-1 is based on L1-norm pruning. It would be interesting if the author can show the more empirical result of applying GReg-1 on other pruning methods that directly removes the neurons. It can better support your statement that the pruning schedule is important.\n\n\n===============================After response ======================\nThanks for stressing my concern, the additional experiment makes the empirical result more convincing. Overall I think this is a good paper, I prefer to keep my score and rate it as accept.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel pruning algorithm for neural networks, gently regularizing the weights away (through weight decay) and using Hessian information instead of simple magnitude. All in all an idea that is simple and effective, and could be of interest to a large audience. 

AC
Decision: Accept (Poster)