Rethinking the Pruning Criteria for Convolutional Neural Network
Authors: Zhongzhan Huang
Wenqi Shao
Xinjiang Wang
Liang Lin
Ping Luo
Venue: NeurIPS 2021 Poster (2021)
Keywords: Pruning Criteria
Similarity
Applicability
Convolutional Filter
Weight Distribution

Abstract: Channel pruning is a popular technique for compressing convolutional neural networks (CNNs), where various pruning criteria have been proposed to remove the redundant filters. From our comprehensive experiments, we found two blind spots of pruning criteria: (1) Similarity: There are some strong similarities among several primary pruning criteria that are widely cited and compared. According to these criteria, the ranks of filters’ Importance Score are almost identical, resulting in similar pruned structures. (2) Applicability: The filters' Importance Score measured by some pruning criteria are too close to distinguish the network redundancy well. In this paper, we analyze the above blind spots on different types of pruning criteria with layer-wise pruning or global pruning. We also break some stereotypes, such as that the results of $\ell_1$ and $\ell_2$ pruning are not always similar. These analyses are based on the empirical experiments and our assumption (Convolutional Weight Distribution Assumption) that the well-trained convolutional filters in each layer approximately follow a Gaussian-alike distribution. This assumption has been verified through systematic and extensive statistical tests.

Rethinking the Pruning Criteria for Convolutional Neural Network
Zhongzhan Huang!
Wenqi Shao Xinjiang Wang' Liang Lin"
Ping Luo 4x
'Sun Yat-Sen University, "The Chinese University of Hong Kong,
Sensee Time Research," 'The University of Hong Kong
Abstract
Channel pruning is popular technique for compressing convolutional neural networks (CNNs), where various pruning criteria have been proposed to remove the redundant filters. From our comprehensive experiments,
found two blind spots of pruning criteria: (1) Similarity: There are some strong similarities among several primary pruning criteria that are widely cited and compared. According these criteria, the ranks of filters' Importance Score are almost identical, resulting similar pruned structures. (2) Applicability: The filters' Importance Score measured by some pruning criteria are too close to distinguish the network redundancy well. In this paper, we analyze the above blind spots on different types of pruning criteria with layer-war wise pruning or global pruning. We also break some stereotypes, such as that the results of & and &g pruning are not always similar. These analyses are based on the empirical experiments and our assumption (Convolutional Weight Distribution Assumption) that the well-trained convolutional filters in each layer approximately follow Gaussian-alike distribution. This assumption has been verified through systematic and extensive statistical tests.
Introduction
Pruning 1 2 trained neural network is commonly seen in network compression. In partieular, for CNNs, channel pruning refers to the pruning of the filters in the convolutional layers. There are several critical factors for channel pruning. Procedures. One-shot method [5]: Train network from scratch; Use certain criterion to calculate filters' Importance Score, and prune the filters which have small Importance Score; After additional training, the pruned network can recover its accuracy to some extent. Iterative method [1 6 7 Unlike One-shot methods, they prune and fine-tune a network alternately. Criteria. The filters' Importance Score can be definded by given criterion. From different ideas, many types of pruning criteria have been proposed, such as Norm-based [5]. Activation-based 8 g Importance- based [10 IT] BN-based [12 and on. Strategy. Layer-wise pruning: In each layer, we can sort and prune the filters, which have small Importance Score measured by a given criterion. Global pruning: Different from layer-wiss wise pruning, global pruning [2 13 sort
the filters from all the layers through their Importance Score and prune them.
In this work, we conduct our investigation on a variety of pruning criteria. As of the simplest and most effective channel pruning criteria, & pruning (5) is widely used in practice. The core idea of this criterion is sort the norm of filters in one layer and then prune the filters with small l norm. Similarly, there is l2 pruning which instead leverages the Eg norm 6 & and &g can be seen as the criteria which use absolute Importance Score of filters. Through the study of the distribution of norm, 4 demonstrates that these criteria should satisfy two conditions:
(1) the variance of the norm of the filters cannot be too small; (2) the minimum norm of the filters should be small enough. Since
"Coresponding author: pluo. lhi @ gmail.com; t co-first author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1:
An example to illustrate the phenomenon that different criteria may select the similar sequence of filters for pruning. Taking VGGI6 (3rd Cony) and ResNet18 (12th Conv) on Norm-based criteria as examples. The pruned filters' index (the ranks of filters' Importance Score) are almost the same, which lead to the similar pruned structures.
Criteria
Model
Pruned Filters' Index (Top 8)
Model
Pruned Filters' Index (Top 8)
f1
88
[111, 212, 61, 68. 152, 45]
VGGI6
[102, 28, 9, 88, 66, 109, 86, 45]
ResNet18
61, 171, 42, 243, 129]
VGG16
[102, 28, 88, 9, 109, 66, 86, 45]
GM
ResNet18
[111, 212, 33, 61, 68, 45, 171, 421
VGG16
[102, 28, 9, 88, 109, 66, 45, 86)
Fermat
ResNet18
[111, 33, 61, 45, 171, 42, 68]
VGG16
[102, 28, 88, 9, 109, 66, 45, 86]
VGG16
VGG16
WRN28-10 10
Canoistan rwet MM817 Wa 3002
Cencii mear
HaNE 52802 WT 00008
Frequency
Frequency
Frequency
Conokriccal laye wwt.mo 000
5 Convkkteed lyye vert wat D037
BockG-land cord
12-
Conokiiccal
Concoitionl mebi 6.7
MaKIs
marn 2000 DORR:
TWee USIES
9015
50
05
10
0.0
0.2
5000 020 012
8.00
3.25
0.50
0.75
1.00
335
359
1.75
2.00
BN-Y
Taylor 82
Eg norm
Figure 1: Visualization of Applicability problem. i.e, the histograms of the Importance Score measured by different types of pruning criteria (like BN_>, Taylor lg and Eg norm). The Importance Score in each layer are close enough, which implies that is hard for these criteria to distinguish redundant filters well in layer-wise pruing.
these two conditions do not always hold, a new criterion considering the relative Importance Score of the filters is proposed 4 Since this criterion uses the Fermat point (i.e., geometric median [14), we call this method Fermat. Due to the high calculation cost of Fermat point, [4] further relaxed the Fermat and then introduced another criterion denotes as GM. To illustrate each of the pruning criteria, let Fij E Nokk represent the jيh filter of the 7th convolutional layer, where N; is the number of input channels for jth layer and & denotes the kernel size of the convolutional filter. In 7th layer, there are No+1 filters. For each criteria, details are shown in Table 2 where F denotes the Fermat point of Fe, in Euclidean space. These four pruning criteria are called Norm-based pruning in this paper as they utilize norm in their design.
Previous works 15
18I. including the criteria mentioned above, the main concerns commonly consist of (a) How much the model was compressed: (b) How much performance was restored; (c) The inference efficiency of the pruned network and (d) The cost of finding the pruned network. However, few works diseussed the following two blind spots about the pruning criteria:
3
16
(1) Similarity: What are the actual differences among these pruning criteria? Taking the VGG16 and R on ImageNet as an example, we show the ranks of filters' Importance Score under different criteria in T It is obvious that they have almost the same sequence, leading to similar pruned structures. In this situation, the criteria used absolute Importance Score of filters (L, and the criteria used relative Importance Score of filters (Fermat, GM) may not be significantly different.
Table 2: Norm-based pruning criteria.
Criterion
Details of Importance Score
& 5
Fiy
&
Faylla
Fermat 4
[F - Figllz
GM 4
Σ
||Fa
- Figllz
(2) Applicability: What is the applicability of these pruning criteria to prune the CNNs? There is a toy example w.r.t. lg criterion. If the Eg norm of the filters in one layer are 0.9, 0.8, 0.4 and 0.01, according to smaller-norm-leer sonter--oom--ess--ine -es--inforraatie assumption 11). it's apparent that we should prune the last filter. However, if the norm are close, such as 0.91, 0.92, 0.93, 0.92, it is hard to determine which filter should be pruned even though the first one is the smallest. In Fig. D we demonstrate some real examples, i.e. the visualization of Applicability problem under different networks and criteria.
In this paper, we provide comprehensive observations and in-depth analysis of these two blind spots. Before that, in Section 2 we propose an assumption about the parameters distribution of CNNs, called Convolution Weight Distribution Assumption (CWDA), and use it as a theoretical tool to analyze the two blind spots. We explore the Similarity and Applicability problem of pruning criteria in the following order: (1) based criteria (layer-wise pruning) in Section 3 (2) Other types

---

of criteria (layer-wise pruning) in Section (3) and different types of criteria (global pruning) in Section [5] Last but not least, we provide further discussion on: the conditions for CWDA to be satished, (ii) how our findings help the community in Section 6 In order to focus on the pruning criteria, all the pruning experiments are based on the relatively simple pruning procedure, Le., one-shot method.
The main contributions of this work are two-fold:
(1) We analyze the Applicability problem and the Similarity of different types pruning criteria. These two blind spots can guide and motivate researchers to design more reasonable criteria. We also break some stereotypes. such as that the results of & and lz pruning are not always similar.
(2) We propose and verify an assumption called CWDA, which reveals that the well-trained convo- lutional filters approximately follow a Gaussian-alike distribution. Using CWDA, we succeeded in explaining the multiple observations about these two blind spots theoretically.
2 Weight Distribution Assumption
In this section, we propose and verify assumption about the parameters distribution of the convolutional filters.
(Convolution Weight Distribution Assumption) Let Fij E &Nxxx be the jth well-trained filter of the hh convolutional layer. In genera such a distribution:
? in 2th layer, Fig (j = 1,2, ....... Ni+1) are i.i.d and follow
Fig N(0, Eingg + e + Eieee)):
(1)
where Eg bblckk diag(K1, K2, KN:) block diagonal matrix and the diagonal elements of black are 0. is small constant. The values of the off-block-die fifbbloc--diagnall elements are 0 and K2 xk² xx²
Re
= 1, 2, ..., N₁. Eingg = diag(a (1, 02 aN, xkxk) k) is a diagonal matrix and the elements of diag are close enough.
This assumption is based on the observation shown in the Fig. To estimate Eding + E block we use the correlation matrix F FFT where F e ( xkx denotes all the parameters in 2th layer. Tak- ing convolutional layer of ResNet18 trained on ImageNet as example, find that F FF is a block diagonal matrix. Specifically, each block is a 622 x 122 matrix and the off-diagonal elements are close to O. We visualize the jth filter F.y Nexk in 7th layer in Fig. ((c), and this phenomenon reveals that the parameters in the same channel of Fiy tend to be linearly correlated, and the parameters of any two different channels (yellow and green channel in Fig, [ c)) in F., only have low linear correlation.
RoaNa13
2.1 Statistical test for CWDA
100
In fact, CWDA is not easy to be verified, e,g., for ResNet164 trained on Cifar100, the number of filters in the first stage is only 16, which is too small to be used to estimate the statistics in CWDA accurately. Thus, We consider verifying four necessary conditions of CWDA:
k
20:0
(8) Figure 2: (a-b) Visualization of FFT in ResNet- trained on ImageNet dataset. More experiments can be found in Appendix N These experiments are based on torchvison model zoo (20). which can guarantee the generality and reproducibility. (c) A convolutional filter. & is the kernel size and N; denotes the number of input channels.
(b)
(1) Gaussian. Whether the weights of F.y ap- proximately follows Gaussian-alike distribu- tion; (2) Variance. Whether the variance of the diagonal elements of Ediag are small enough; (3) Mean. Whether the mean of weights of Fij is close to 0. (4) The magnitude of e. Whether e small enough.
The results of the tests are shown in Appendix P where we consider a variety of factors for the statistical tests, including different network structure, optimizer, regularization, initialization, dataset,
In Section 6 make further discussion and analysis on the conditions for CWDA to be satisfied.

---

training strategy, and other tasks in computer vision (e.g., semantic segmentation, detection and so on). The test results show that CWDA has great generality for CNNs.
3 About the Norm-based criteria
We start from the criteria in Table which are widely cited and compared 21
22
23
24
25
3.1 Similarity
In this section, we further verify the observation that the Norm-based pruning criteria in Table 2 are highly similar from two perspectives. Empirically, we conducted large amount of experiments on image classification to investigate the similarities. Theoretically, we rigorously prove the similarities of the criteria in Table 2 in layer-wise pruning under CWDA.
we show the test accuracy of the ResNet56 after pruning and fine-tuning under different pruning ratios and datasets. The test accuracy curves of differ- ent pruning criteria at different stages are very close under different pruning ratios. This phe- nomenon implies that those pruned networks us- ing different Norm-based criteria are very simi- lar, and there are strong similarities among these pruning criteria. The experiments about other commonly used configs of pruning ratio can be we show the Spearman's 's rank correlation coefficien (Sp) between different pruning criteria. The Sp in most convolutional layers are more than 0.9, which means the network structures are almost
Empirical Analysis. (1) In Fig. 3
Beshlet5 CIPAR10
Restie156 on CIFAR100
R350
R200
Figure 3: Test accuracy of the ResNet56 on CI- FAR10/100 while using different pruning ratios. "LI pruned" and "LI tuned" denote the test accu- racy of the ResNet56 after & pruning and fine- tuning, respectively. ratio is 0.5, we prune 50% filters in all layers.
found in Appendix (2) Fig. 4
the same after pruning. Note that the Sp in transition layer relatively small, and the transition layer refers to the layer where the dimensions of the filter change, like the layer between stage I and stage 2 of a ResNet. The reason for this phenomenon may be that the layers in these areas are sensitive. It is interesting but will not greatly impact the structural similarity of the whole pruned network. The similar observations are shown in Fig. 2 in [16], Fig. 6 and Fig. 10 in (5).
VGGI6-CIFARIO
CI
VGGi66 Nel
Speannon's carrelation
Spearmen's corretations
Speamman's correlation
1200
12:0
Index layer'
Index layer
Index of layer
ResNet56 CIFAR
ResNet56- CIFAR 100
Ne Image
Sperment's correltioon
Spoarmsn'' comrelation
Spounoon's corretation
Bel
Index of layer
Index of layer
Index of l'ayeer
Figure 4: Spearman's 's rank correlation coefficient (Sp) between different pruning criteria on several networks and datasets (more experiments can be found in Appendix R).
*Sp is nonparametric measurement of ranking correlation, and it assesses how well the relationship between two variables can be described using monotonic function, iee., filters ranking sequence in the same layer under two criteria in this paper.

---

Theoretical Analysis. Besides the experimental verification, the similarities via using layer-wise pruming among the criteria Table can also be proved theoretically in this section. Let C₁ and C2 be two pruning criteria to calculate the Imporiance Score for all convolutional filters in one layer. If they can produce the similar ranks of Importance Score, we define that C1 and C2 are approximately monotonic to each other and use C _ C2 to represent this relationship. In Section 3.1 we use the Sp to describe this relationship but it's hard to be analyzed theoretically. Therefore, we focus on a stronger condition. Let X (21, 22, .... 2x.) and Y
(y1, 32, .... M&.) be two given sequenced we X/E(X) and Y Y //E(Y) This operation does not
first normalize their magniuude, i.e., let change the ranking sequence of the elements of X and Y, because E(X) and IE() are constants, (X) are small enough, then
i... X Y X = Y. After that, if both Var(X/Y2 and Var (Y/X)) the Sp between X and Y close to 1, where X X/Y- (Ti/Mi, "T, yit) The reason is that in these situations, the ratio X/Y and Y Y/X will be close to two constants a, b. For any 1 1 i i k, G 2 a Di and ÿi b- Fe. So, ab R and a, b A O. Therefore, there exists an approximately monotonic mapping from f to T. (linear function), which makes the Sp between X and Y close to 1. With this basic fact, we propose the Theorem [ which implies that many Norm-based pruning criteria produces almost the same ranks of Importance Score.
Theorem 1. Let dimenssion random variable X meet CWDA, and the pair of criteria (C1, C2) one of (fi, l2), (f2, Fermat) or (Fermat, GM), we have
(()( (1( Varx
( (2(())
)
E B(n), where (x) denotes (XX)/B(C((() and (x denotes Cx(X)//((2(X)), B(n) denotes the upper bound of left-hand side and when is large enough, B(n) + O.
max Varx
(2)
Proof. (See Appendix C
In specific, for th convolutional layer of CNN, since F.j E j 1,,,...N meet CWDA and the dimension n is generally large, we can obtain & _ l2, lz _ Fermat and Fermat GM according to Theorem [ Therefore, have & _ fg _ Fermat = GM, which verifies the strong
similarities among the criteria shown in Table 2
3.2 Applicability
In this section, we analyze the Applicability problem of the Norm-based criteria. In Fig. (Right), we know that there are some cases where the values of Importance Score measured by & criterion are very close (e.g., the distribution looks sharp), which make & criterion cannot distinguish the redundant filters well. It's related to the variance of Importance Score. 4 argue that small norm deviation (the values of variance of Importance Score are small) makes it difficult to find an appropriate threshold to select filters to prune. However, even if the values of the variance are large, still cannot guarantee to solve this problem. Since the magnitude of these Importance Score may be much greater than the values the variance, we can use the mean of Importance Score to represent their magnitude. Therefore, we consider using relative variance (C(,(C(FA) to describe the Applicability problem. Let ([A((F)) > O and
Var,[C(FA) = Var(C(FA)/E((((F))
where is a given pruning criterion and FA denotes the filters in layer A. The criterion C for layer A has Applicability problem when Var,[C((F,) is close to O. Then we introduce the Proposition ] to provide the estimation of the mean and variance w.r.t. different criteria when the CWDA hold: Proposition 1. If the convolutional filters FA in layer A meet CWDA, then we have following estimations:
Criterion
Mean
Variance (1- (1-3)oāda
((FA)
42/To V ( (A)
(a(FA)
02/2
Fermat(Fx)
I dAth y (A)
0
"Since X is not random variables here, B(X) and Var(X) denote the average value Σi= xi/ lk and the sample variance Dk (z: E(X))/(k 1), respectively.
5

---

where dA and A denote the dimension of FA and the variance of the weights in layer A, respectively.
Proof. (See Appendix A
Based on the Proposition T we further provide the theoretical analysis for each criteria:
(i) For (F From Proposition!!
! we can obtain that
a /l/V/22 da dA =
(62(al =
(4)
where g(da) r( I( / (/A is monotonically increasing function w.r. dA. From Eq. 4.. Var, [(2(FA) depend on and dA. When σд is small or dA is large enough, Var, [((Fa] tends be 0.
(ii) For Fermat (FA). From the proof in Appendix D we know that the Fermat point F of FA and the origin 0 approximately coincide. From Table 1 the mean and variance of (er) are the same as (FA)ss in Proposition | Hence, a similar conclusion can be obtained for Fermat criterion. i.e., the Imporiance Score tends to be identical and it's hard to distinguish the network redundancy well when σл is small or d is large enough.
F Falla 110 Falle FA 22. Therefore,
(iii) For 4 FA)) Intuitively, the & criterion should have the same conclusion as the E criterion. However, given the Proposition]
we can obtain that !
Var, Var,|(((A) = (oddd)/)dd/////o////// c(()) aA.
(5)
where <(x) < constant w.r.t æ. Note that (ar,(& only depend on σA, but not the dimension n. Moreover, for the common network structures, like VGG, ResNet shown in Fig. (b) and (d), the dimension of the filters are usually large enough. Therefore, compared with L,, & criterion is relatively not prone to have Applicability problems, unless the is very small.
Leparobee prasing
Eaperarise praskng
Laserwke pmzing
Logerwise prasing
Globel percing
Glchal prasing
MSRTN
Resheed6
(3) Recheess 14t
Convolatinal layer
convolutioal
Index convolutional
f. EXCm
nomE
Taglor
4: TTayoer
.NN
BN.5
EEtropy
APPZ
Figure 5: The Similarity and Applicability problem for different types of pruning criteria in layer-wise or global pruning.
About other types of pruning criteria
In this section, study the Similarity and Applicability problem in other types of pruning criteria through numerical experiments, such as Activation- pruning [8] 91, Importance- based prun- ing [10 [T] and BN-based pruning [12]. For each type, choose two representative criteria and we
call them: (1) Norm-based: I, and f2; (2) Importance-based: Taylor & and Taylor l2 [10 26]:
(3) BN-based: BN_ BN_ 3 12]: (4) Activation-hased: Entropy 9 and APoZ. 8 The details
of these criteria can be found in Appendix[X K
The Similarity for different types of pruning criteria. In Fig. (a-d), show the Sp between different types of pruning criteria, and only the Sp greater than 0.7 are shown because if Sp < 0.7, means that there is no strong similarity between two criteria in the current layer.
'Tee empirical result for slimming training 22 is shown in Appendix

---

According to the Sp shown in Fig. 5 (a-d), we obtain the following observations: (I) As verified in Section 3.1 & and Ea can maintain a strong similarity in each layer; (2) In the layers shown in Fig. S (a) and Fig. 5 (d), the Sp between most different pruning criteria are not large in these layers, which indicates that these criteria have great differences in the redundancy measurement of convolutional filters. This may lead to phenomenon that one criterion considers a convolutional filter to be important, while another considers it redundant. We find a specific example which is shown in Appendix J (3) Intuitively, the same type of criteria should be similar. However, Fig. 5 (b) and Fig. (c) show that the Sp between Taylor & and Taylor E2 is not large, but Taylor &g has strong similarity with both two Norm- based criteria. Moreover, the Sp between BN 2 and each Norm- based criteria exceeds 0.9, but it is not large in other layers (Fig. (a) and Fig. (d)). These phenomena are worthy of further study.
VCG0E OPAR320
VOSS in CroPEte
Resivebe EPPR200
PocPA-09 and PARICe
of
g
into conviltiinnl
Doddd typtt
1800e
Fiee coecldos Syprr
Discnas
Figure 6: The magnitude of the Importance Score measured by l and le criteria.
The Applicability for different types of pruning criteria. According to the analysis in Section 3.2 the Applicability problem depends on the mean and variance of the Importance Score. Fig. 5 shows the result of the Importance Score measured by different pruning criteria cach layer of VGG16. Due to the difference in the magnitude of Importance Score for different criteria, for the convenience of visualization, the value greater than 1 is represented by 1.
g--)
First, we analyze the Norm-based criteria. In most layers, the relative variance Var, [fg] is much smaller than that of [ar,[(]], which means that the & pruning has Applicability problem in VGG16, while the é does not. This consistent with our conclusion Section (.2 Next, for the Activation- based criteria, the relative variance Var, is large in each layer, which means that these two Activation- based criteria can distinguish the network redundancy well from their measured filters' Importance Score. However, for the Importance-based and BN-based criteria, their relative variance Var, are close to O. According to Section 3.2, these criteria have Applicability problem, especially in the deeper layers (e.g., from 6th layer to the last layer).
5 About global pruning
Compared with layer-wise pruning, global prun- ing more widely (27 Io [2] used in the cur- rent research of channel pruning. Therefore, in this section we may also analyze the Similarity and Applicability problem of global pruning.
Value
Value 6
Valu of Fj
Vinua Fg
Applicability while using global pruning. In fact, for global pruming, both & and & cri-
Stentasd driisten
Our
teria not prone to Applicability problems. From Proposition I we show that the esti-
mations for the mean of Importance Score in
layer A for & and & are a day/ and
Figure 7: The global pruning simulation for the unpruned network with only two layers.
respectiely. - resperiiyy Since J and d are quite different, shown in Fig. (b)
and (d), hence the variance of the Importance
Score may be large in this situation. Fig. (a) and (c) show such kind of difference of the magnitude on different convolutional layers. In addition, from our estimations in Fig. (c), this inconsistent magnitude can be explained for another common problem in practical applications of global pruning:
the ResNet is easily pruned off. As shown in Fig. (c), we take ResNet56 as an example. Since the Importance Score in first stage is much smaller than the Importance Score in the deeper layer,
7

---

global pruning will give priority to prune the convolutional filters of the first stage. For problem, we suggest that some normalization tricks should be implemented or a protection mechanism should be established, e.g., mechanism which can ensure that each layer has at least a certain number of
convolutional filters that will not be pruned. Unlike some previous works [3] 28 29], which make suggestions from qualitative observation, we provide a quantitative view to illustrate that these tricks are necessary.
Similarity while using global pruning. In Fig. 5 (e-1), we show the similarity of different types of pruning criteria using global pruning on VGG16 and ResNet56. Comparing to the results from the layer-wise pruning shown in Fig. Fig.35 (a-d), we can find that the similarities of most pruning criteria are quite different in global pruning. In addition, the same criteria may have different results for different network structures in global pruning, e.g., in Fig. (e), we can find lg = Taylor l2 and BN, = Ez, but this observation does not hold in Fig. [5 (f). particular, different from the result about ResNet56 in Fig. (f), the similarity between & and le is not as strong as the one in the layer-wise case. This phenomenon is counter intuitive.
To understand this phenomenon, we first consider about simple case, i.e., the unpruned network has only two convolutional layers (layer A and layer B). The filters in these two layers are FA ( A...., FX) and FB (F) 17 ..... Pm)). ....,
According to CWDA, for 1 S i S n and 1 S S m, FA and FB can follow (O, LāA and (0, Lg respectively. Next, we show Sp between Importance Score measured by l, and l2 pruning in different dimension ratio d O and оB in Fig. (). Moreover, to analyze this phenomenon concisely, we draw some scatter plots as shown in Fig. 7 (a-d), where the coordinates of each point are given by (value of 4, value of f2). The set of the points consisting of the filters in layer A is called group-A. A. Then we introduce the Proposition 2. Proposition 2. If the convolutional filters F' in layer A meet CWDA, then ((A(( and E[(2(FA)/&{(F)) only depend on their dimension d
Proof. (See Appendix A.
Now we analyze the simple case under different situations:
(1) For dA/dB = 1. If 03 = oz. in fact, it's the same situation as layer-wise pruning. From Theorem we know that group-A and group-B coincide and approximately lie on the same line, resulting & = lg. If á * ob. group-A and group- B lie on two lines, respectively. However, these two lines have the same slope based on Proposition 2 as shown in Fig. 7 (a). For these reasons, we have E _ l2 when d
(2) For da/dg + 1. In Fig. 7 (b-d), there are three main situations about the position relationship between group-A and group-B. B. In Fig. (b). according to Theorem I the points in group-A and group-B B are monotonic respectively. Moreover, their Importance Score measured by & and l2 do not overlap, which make é, and E are approximately monotonic overall. Thus, Ei = l2. However, for Fig. F (c-d), the Sp is small since the points in these two group are not monotonic (the Importance Score measured by Ei or f2 has large overlap). From Proposition [ and the da/2 (Appendix Da. these two situations can be described as:
approximation 1 (( 1/T 4
dA
o AdA R dBdB or DAVdA dANDOBVVEe
(6)
where d F dg. Through Eq. (6) can obtain the two red lines shown in Fig. (f-h). It can be seen that the area surrounded by these two red lines is consistent with the area where the Sp is relatively small, which means our analysis is reasonable. Based on the above analysis, conditions about E _ & in global pruning for two convolutional layers as shown in Table
summarize the
Next, we go back to the the situation about real neural networks in Fig. 5 (c-f). (1) For ResNet56. As shown in Fig 6 (d), the dimen- sions of the filters in each stage are almost the same. From Table ( (1), the pruning results after l and l2 pruning in each stage are similar. And, the magnitudes of the Importance Score in each stage are very different, since Table 3 (2), we can obtain that & = l2 for ResNet56.
Table 3: The conditions about & _ E2 in global pruning for two layers (layer A and layer B)
da da?
R
dag
/dn, 2
4 = E2?
VAA
(1)
(2)
x
x
X
(3)
x
/
(4)
X
7
X
(2) For VGG16. As shown in different dimensions but similar Importance Score measured by fi lz, such as "layer 2" and "layer
compared with ResNet56, VGG16 has some layers with
8

---

8" for lg criterion in 6 (a). explain why the result ? and l pruning not similar Fig. (e) for VGGI6. In Appendix [ more experiments show that we can increase the Sp in global pruning by ignoring part of these pairs of layers, which support our analysis.
(a). From Table B (3-4), these pairs of layers make the Sp small, which
6 Discussion
6.1 Why CWDA sometimes does not hold?
CWDA may not always hold. As shown in Appendix P small number of convolutional filters may not pass all statistical tests. In this section, we try to analyze this phenomenon.
(1) The network not trained well enough. The distribution of parameters should be discussed only when the network is trained well. If the network does not converge, it is easy to construct scenario which does not satisfy CWDA, e.g., for a network with uniform initialization, when is only
be trained for a few epochs, the distribution of parameters may be still close to uniform distribution. At this time, the distribution obviously does not satisfy CWDA. specific example in Appendix [
(2) The number of filters is insufficient. In Appendix P the layers that can not pass the statistical tests are almost those whose position is in the front of the network. A common characteristic of these layers is that they have few filters, which may not estimate statistics well. Taking the second convolutional layer (64 filters) in VGG16 on CIFARIO as an example, first, the filters in this layer can not pass all the statistical tests. And then the Sp in this transition layer is relatively small, as shown in Fig. 4 However, in Fig. [ we change the number of filters in this layer from 64 to 128 or 256. After that, the Sp increases significantly, and the filters can pass all the statistical tests when the number of filters is 256. These observations suggest that the number of filters major factor for CWDA to be hold.
VGG16-2 Conv (64)
VGG16-2nd Conv (128)
VGG16-22d Cony (256)
Spearnan 0o conrelationn
Spearmnn's comelation Eation
Pass all the
Pass all the statisiccal tests?
Passs all the statistical tests?
statistical tosts?
Index of layer
Index layer
Index of layer
Figure 8: The Sp between different pruning criteria on VGG16 (CIFARIO). The number of filters in the second convolutional layers is changed from 64 256. The filters in this layer can pass all the statistical tests when the number of filters is 256.
6.2 How our findings help the community?
(1) We propose an assumption about the parameters distribution of the CNNs called CWDA, which is an effective theoretical tool for analyzing convolutional filter. In this paper, CWDA is successfully used to explain many phenomena in the Similarity and Applicability of pruning criteria. In addition, it also explains why the ResNet is easily pruned off in global pruning. In Section [. since CWDA can pass statistical tests in various situations, it can be expected that can also be used as an effective and concise analysis tool for other CNNs-related areas, not just pruning area.
[2I]]
(2) In this paper, we study the Similarity and Applicability problem about pruning criteria, which can guide and motivate the researchers to design more reasonable criteria. For Applicability problem, we suggest that, intuitively, it reasonable that the Importance Score should be distinguishable for the proposed novel criteria. For Similarity, used for ensemble learning to enhance their pruning performance (23]. In this case, the similarity analysis between criteria in this paper is important, because highly similar criteria cannot bring gains to ensemble learning.
more and more criteria are proposed, these criteria can be
(3) In pruning area, l and E2 are usually regarded the same pruning criteria, which is intuitive. In layer-wise pruning, we do prove that the and l2 pruning are almost the same. However, in global pruning, the pruning results by these two criteria are sometimes very different. In addition, compared

---

with & criterion, &g criterion is prone to Applicability problems. These counter-intuitive phenomena enlighten that we can't just rely on intuition when analyzing problems.
Table 4: The random pruning results of VGGNet with different criteria which have the Applicability problem. The VGG 16 and VGG19 are trained on CIFARIO0 The unpruned baseline accuracy of VGG16 and VGG19 are 72.99 and 73.42, respectively.
max (r= (i)
mean (r=10%
(r=20%)
max (1=20%
mean (r=20)
71.41
72.65
71.75
1.24
71.01
72,47
71.32
|46
VGG19 VGG16
Taylor &
0.67
Taylor &g
71,87
72.37
71.91
0.5
71.66
72.27
71.65
0.61
BN,
0.55
BNg
71.15
72.58
71.43
1.43
71.06
72.11
71.87
1.03
Taylor Ea
71.67
73.04
72.23
1.37
71.6
72,98
72.24
1.38
Taylor da
72.12
72.99
72.28
0.87
72.04
72.83
72.54
0.79
BN,
72.01
73.23
72.25
1.22
71.98
72.32
72.12
0.34
BNg
72.25
73.23
72.41
0.98
72.04
72.65
72.33
0.61
(4) Similar to the setting in Fig. we can explore the effect of pruning filters with similar Importance Score the performance. First, find that the criteria (62, Taylor f1. Taylor lz, BN, and BNg) for VGGNet can cause the Applicability problem in most layers (Fig. 55. As such, we randomly select 10% 20% filters be pruned by the uniform distribution U 1] in each layer, and the selective filters will be in similar Importance Score. Finally, we finetune the pruned model (there are 20 random repeated experiments). A denotes the difference between acc. min acc.) Since their Importance Score are very similar, when the network is pruned and finetuned, it can be expected that the performance should be similar in these repeated experiments. However, from the results in the above table, although the Importance Score of the pruned filters is very close, we still get pruning results with very different results (e.g. the A of VGG16 lq more than 1). It means that these criteria may not really represent the importance of convolutional filters. Therefore, it necessary to re-evaluate the correctness the existing pruning criteria.
and min (i.e.
Acknowledgments. Z. Huang gratefully acknowledges the technical and writing support from Mingfu Liang (Northwestern University), Senwei Liang (Purdue University) and Wei He (Nanyang Technological University). Moreover, he sincerely thanks Mingfu Liang for offering his self- purchasing GPUs and Qinyi Cai (NetBase, Inc.) for checking part of the proof in this paper. This work was supported in part by the General Research Fund of Hong Kong No.27208722 the National Key R&D Program of China under Grant No. 2020AAA0009700,, the National Science Foundation of China under Grant No.61836012 and 61876224, the National High Level Talents Special Support Plan (Ten Thousand Talents Program), and GD-NSF (no.2017A330312000
References
[1] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[2] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pages (64-171. 1993.
[3] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 15 2015.
[4] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruming via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4340-4349, 2019.
[5] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608.0:660.0877700 2016.
[6] Yang He, Guoliang Kang, Xuanyi Dong. Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 06866, 2018.
[7] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019.
10

---

[8] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint XXi::6077.03560 2016.
[9] Jian-Ha han Luo and Jianxin Wu. An antropy-based pruning method for enn compression. arXiv preprint arXiv: (tti:/70000 706.05791, 2017.
[10] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint
1,04444010 2016.
[11] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11224-112277 11272, 2019.
[12] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference on Computer Vision, pages 2736-2744, 2017.
[13] Wei He, Meiqing Wu, Mingfu Liang, and Siew-Kei Lam. Cap: Context-aware pruning for semantic segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 960- 009,969,2222 2020.
[14] Michael B Cohen, Yin Tat Lee, Gary Miller, Jakub Pachocki, and Aaron Sidford. Geometric median in nearly linear time. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 9-21. ACM, 2016.
[15] Jian-Hao Luo. Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5066, 2017.
[16] Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et al. Global sparse momentum sgd for pruning very deep neural networks. In Advances in Neural Information Processing Systems, pages 6379-6391, 2019.
[17] Xin Dong. Shangyu Chen, and Sinno Pan. Learning prune deep neural networks via layer- wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pages 4857-4867, 2017.
[18] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In International Conference on Learning Representations, 2020.
[19] Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smalle-noomlles--nnnmmaiie assumption in channel pruning of convolution layers. International Conference on Learning Representations, 2018.
[20] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
[21] Zechun Liu, Xiangyu Zhang, Zhiqiang Shen, Zhe Li, Yichen Wei, Swang-Ting Cheng, and Jian Sun. Joint multi-dimensee pruning. arXiv preprint arXiv:2005. 08931, 2020.
[22] Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2018-807,, 2020.
[23] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019-2018, 2020.
11

---

[24] Yuchen Liu, David Wentzlaff, and SY Kung. Rethinking glass-diccrmminatie based enn channel pruning. arXiv preprint arXiv:2004.14499 14492, 2020.
[25] Bailin Li, Bowen Wu, Jiang Su, Guangrun Wang, and Liang Lin. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. arXiv preprint arXiv:2007.0249/1 2020.
[26] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[27] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1810.05270 2018.
[28] Ting-We Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1518-1528, 2020.
[29] Wenxiao Wang, Cong Fu, Jishun Guo, Deng Cai, and Xiaofei He. Cop: Customized deep model compression via regularized correlation-based filter-level pruning. arXiv preprint 1906, 10337, 2019.
[30] Gavin E Crooks. Survey of simple, continuous, univariate probability distributions. Technical report, Technical report, Lawrence Berkeley National Lab, 2013., 2012.
[31] Rodrigo R Pescim, Clarice GB Demétrio, Gauss M Cordeiro, Edwin MM Ortega, and Mariana R Urbano. The beta generalized half-normal distribution. Computational statistics data analysis, 54(4):945-95) 957, 2010.
[32] RL Graham. Applications of the fkg inequality and its relatives. In Mathematical Programming The State of the Art, pages 1 115-131. Springer, 1983.
[33] Lars Hormander. The analysis of partial differential operators. Springer, 1983.
[34] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[35] Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari Morcos. Luck matters: Understanding training dynamics of deep relu networks. arXiv preprint arXiv: 13405, 2019.
[36] I Bellido and Emile Fiesler. Do backpropagation trained neural networks have normal weight distributions? In International Conference on Artificial Neural Networks, pages 772-775. Springer, 1993.
[37] Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. PhD thesis, University of Toronto, 1995.
[38] Jinwook Go, Byungjoon Baek, and Chulhee Lee. Analyzing weight distribution of feedforward neural networks and efficient weight initialization. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 840-849. Springer, 2004.
[39] Hubert W Lilliefors. On the kolmogorov -miinoo test for normality with mean and variance unknown. Journal of the American statistical Association, 62(318):399-40, 1967.
[40] Bradley Efron. Student's t-test under symmetry conditions. Journal of the American Statistical Association, 64(328): 278-13022 1969.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
June 2016.
[42] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147, 2013.
12

---

[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[44] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimizaaiion, 30(4):838-855 1992.
[45] Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:144..590 2014.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[47] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1505,00055 2015.
[48] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[49] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12((ul):12112215, 2011.
[50] Senwei Liang, Yuehaw Khoo, and Haizhao Yang. Drop-activation: Implicit parameter reduction and harmonic regularization. arXiv preprint arXiv:1811.0583 2018.
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages 630-645. Springer, 2016.
[52] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019.
[53] Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
165.v:160..07146, 2016.
[54] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1210, 2012.
[55] Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint (rXiv:/708.0455: 04552, 2017.
[56] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[57] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference on Computer Vision, pages 0023-6032, 2019.
[58] Jie Hu, Li Shen, and Gang Sun. Squeerre-nnd-excitaaiin networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 7132-7141, 2018
[59] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[60] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[61] Zhongzhan Huang, Senwei Liang, Mingfu Liang, and Haizhao Yang. Dianet: Dense-and- implicit attention network. arXiv preprint arXiv: 2019.
13

---

[62] HyunJae Lee, Hyo-Eun Kim, and Hyeonseob Nam. Srm: A style-based recalibration module for convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1854-18623 1862, 2019.
[63] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256, 2010.
[64] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)211-252, 2015.
[65] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3- 2018.
[66] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11))2278-2324, 1998.
[67] Senwei Liang, Zhongzhan Huang, Mingfu Liang, and Haizhao Yang. Instance enhancement batch normalization: an adaptive regulator of batch noise. arXiv preprint 1rXiv: 04000, 2019.
[68] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 132.6113,, 2013.
[69] Xiang Li, Xiaolin Hu, and Jian Yang. Spatial group-wise enhance: Improving semantic feature learning in convolutional networks. arXiv preprint 190iv: 0.666,, 2019.
[70] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(11):2481-2 1-245,, 2017.
[71] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.
[72] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[73] Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2970-2979, 2017.
[74] Sebastian Lutz, Konstantinos Amplianitis, and Aljosa Smolic. Alphagan: Generative adversarial networks for natural image matting. arXiv preprint arXiv:1807.10080 2018.
[75] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[76] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1115/55111441.1111111111 2015.
[77] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608, 03983, 2016.
14


Reviews:
[{"review_id": "y9cZJWJWjt6", "reviewer": "Reviewer_Y2y2", "summary": "The paper attempts to study and compare different pruning \"criteria\".\nThe paper addresses an interesting question, trying to evaluate if there is a need for a more complex pruning criteria. However the paper can be better organized. And some of the concerts are hard to follow. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The work is original and provides an attempt to evaluate the validity of new and more complex pruning criteria\nQuality: The paper attempts to prove empirical study along with their theoretical/mathematical proof which adds more significance to work. However, the paper organization needs additional work. Especially the introduction part, where the main concepts and problems are introduced. \nClarity: The paper is hard to follow. The main concepts are purely introduced and paper is hard to follow. \"Convolution Weight Distribution Assumption\" were not well introduced and hard to follow. It would be helpful if more details be provided on the existing pruning criteria. Provided grouping is too generic and not well organized.\nSignificance: The work attempts to provide a tool for more unified evaluation of the pruning methods, which was missing in the current methodology. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "uTEzQQrKDRo", "reviewer": "Reviewer_6Th5", "summary": "Paper studies the problem of selecting a criterion for ranking based structured pruning. Particularly, authors observe that norm based criteria show quite similar behavior. Authors proposed a similarity and an applicability properties of 8 considered criteria. Additionally paper proposes a Convolutional Weight Distribution Assumption. Observations made in the paper will be helpful to the future work on structured pruning.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) Paper studies an important problem of comparing and understanding various pruning criteria. \n2) Selected number of criteria is quite inclusive and contains norm based criteria, gradient based criteria, and activation based. \n3) Proposed properties of the pruning criterion is quite interesting and novel. For example Similarity is a good starting point for developing new pruning criterion in the future. Applicability attempts to estimate certainty of different criteria.\n4) Comparison between layer-wise and global pruning is on point and provides different observations. Many pruning papers consider only one setting and it makes hard to compare them, but authors used the same framework and provide insights that same criteria behave differently. \n\nWeaknesses:\n1) Paper lacks some conclusions about how to interpret Similarity, Applicability and CWDA can help to understand what criteria is the best out of all considered. It might also seem that a criterion with random score assignment might score the highest in similarity and applicability (can authors comment on the Random pruning as it is an important baseline for any pruning method?). Would be interesting to see Fig 5 for that and CWDA. \n2) CWDA depends on regularization techniques applied during training. For example weight decay will push weights to be Gaussian, Batch Normalization will de-correlate filters, Drop-out will correlate them more. Could authors comment on how those settings will affect the study and if their settings were varied during the study.\n3) Applicability assumes that ideal criterion will clearly distribute filters with ranks (deviation of the scores will be high). However, the hypothesis might be wrong as regularization will significantly affect it. For example if drop-out is used then it is known that the layer will be more robust to pruning (doesn't matter what we remove because it prevents co-adaptation). In this case the ideal criterion will have a low Applicability but will be still relevant. Because of this fact, the Applicability property might be not correct even for the graph in Figure 5. \n\nFor rebuttal I am looking in understanding the following:\nAnswer to points 1-3. Analysis of random pruning will be very interesting.\nCan authors finetune some of the model they get from pruning to verify that provided analysis doesn't break after the model is trained (for CWDA at least). Also we will see if similarity is important, as a different choice for only 1 neuron might make a huge difference. \nComment on the limitations (what authors see as a limitation for this work) as well.\n\nOriginality - Novel, didn't see the same study in other works\nQuality - well written\nClarity - great\nSignificance - will be interesting to benchmark new applications\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation seem to be in the criteria during pruning and do not reflect (or not shown) any outcome of the fine-tuning these models. For example out of 3 metrics we want to see what should be maximized/min to develop new criteria. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GR_oHwXxbNq", "reviewer": "Reviewer_nJ3h", "summary": "The authors propose and verify an assumption about the distribution of convolutional weight called CWDA, which reveals that the well-trained convolutional filters in each layer approximately follow a Gaussian-alike distribution. Based on this assumption, the authors analyze the similarity and the applicability problem among different pruning criteria. The point of this analysis is that norm-based metrics, particularly L1 and L2, behave quite similarly. Experimental results on various network structures, tasks, and datasets demonstrate the proposed analysis. The resulting conclusions are interesting. However, there are some issues in the paper. Detailed comments are as follows.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Contributions**:\n1.\tThe authors propose and verify an assumption called CWDA, which reveals that the well-trained convolutional filters approximately follow a Gaussian-alike distribution. \n\n2.\tBased on the proposed CWDA, the authors analyze the similarity and the applicability problem of different types of pruning criteria, which is an interesting perspective. These two blind spots are able to guide the researchers to design more reasonable criteria. \n\n**Questions and points needed to be improved**:\n1.\tThe title “Rethinking the pruning criteria…” is over-claim. In fact, the main point of the paper is to show that norm-based metrics behave quite similarly. The authors also analyze the similarity and applicability of other pruning criteria. However, the conclusion here is not clear. \n\n2.\tIn Figure 3 and Table 4, the authors only show the performance of Norm-based pruning criteria. It would be better for the authors to compare the performance between Activation-based [1-2], Importance-based [3-7], and BN-based criteria [8] on layer-wised pruning settings. These results would help the community to better understand different pruning criteria.\n\n3.\tThere are some repetitions in this paper. (1) References 11 and 26 are repeated. (2) References 12 and 34 are repeated. (3) Sections I and P are repeated in the supplemental material.\n\n4.\tSome papers in the references list have been published in a conference/journal. For example, “Wide residual networks” have been published in BMVC 2014. Please cite the conference/journal version instead of the arXiv one.\n\n5.\tThe authors do not reference the correct Lamma or Equation in Line 740, 762, 780, and 782 of the supplemental material.\n\n**References**\n\n[1]\tNetwork trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv 2016.\n\n[2]\tAn entropy-based pruning method for cnn compression. arXiv 2017.\n\n[3]\tPruning convolutional neural networks for resource efficient inference. ICLR 2017.\n\n[4]\tImportance Estimation for Neural Network Pruning. CVPR 2019.\n\n[5]\tSNIP: Single-shot Network Pruning based on Connection Sensitivity. ICLR 2019.\n\n[6]\tHRank: Filter Pruning using High-Rank Feature Map. CVPR 2020.\n\n[7]\tDiscrimination-aware Network Pruning for Deep Model Compression. TPAMI 2021.\n\n[8]\tLearning Efficient Convolutional Networks through Network Slimming. ICCV 2017.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please discuss the limitations of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "30"}, "extra_text": {}}, {"review_id": "7fh4sR0zFZT", "reviewer": "Reviewer_EQps", "summary": "This paper studies 1) the similarity between different pruning criteria for channel pruning, 2) the applicability of different pruning criteria, where applicability refers to the confidence in the importance ranking of filters.  The authors empirically show that for some common pruning strategies such as l1 and l2 pruning and other norm-based pruning, the importance ranking of the filters is almost identical. Moreover, they claim with theoretical and experimental support that the importance score of different filters is very close to each other under certain conditions.  The main contribution of this paper is to analyze these two phenomena for better pruning criteria in the future. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the broad topic of the paper -- understanding the pruning criteria beter. However, the current version of the paper leaves many questions unanswered and needs substantial improvement. I list my concerns below:\n\n- Writing: I think the writing can be improved significantly in terms of clarity, flow, and grammar. Here are some of the easy-to-fix typos I noticed:  \n    - line 62: unnecessary column at the end of the sentence.\n    - line 93: 'Taking a convolutional layers'. \n    - line 101: missing space after the comma.\n    - line 145: missing space after the comma.\n    - line 146: unclear sentence?\n    - line 151: consider about.\n    - line 162: n-dimensional.\n    - line 191: 'Since Proposition 1,'.\n    - line 194: 'closed to 0'.\n    - line 208: 'in more types of pruning criteria'.\n\n- Overall, I am not convinced with the importance of the problem at hand. It is not surprising that norm-based pruning criteria give almost the same ranking (Similarity). It is also not surprising that for some layers, the importance scores are very similar (Applicability). It would be interesting to compare the effect of pruning filters with similar importance scores on the performance. Perhaps they all perform well. In that case, the pruning criterion is reliable. If some of them do not perform well, then the pruning criterion does not produce a reliable ranking of filters' importance.  However, no such result is provided. Another question to ask is whether very similar importance scores relate to the incompressibility of that layer. Maybe the conclusion we should get from this observation is if importance scores are very similar, it is better not to prune any filters. However, none of these interesting points are discussed in the paper. So, I am having difficulty in understanding the authors' goal in analyzing 'similarity' and 'applicability' phenomena. Can the authors provide a stronger motivation for their study?\n\n- I am confused about the setup in CWDA (lines 84-98). In line 85, $F_{ij}$ are assumed to be iid. However, in line 96, it is said that parameters in the same channel of $F_{ij}$ are observed to be linearly correlated. According to this observation, iid assumption is unrealistic. Did the authors make this iid assumption to simplify the problem?\n\n- I also have concerns about the use of expectation and variance operators. As far as I understand, $\\mathbb{E}[X]$ refers to the average of the importance scores in one layer. Do the authors think of the importance scores as different realizations of the same random variable X? If so, 1) why is $X=(x_1,...,x_k)$ defined as a deterministic sequence? 2) the authors should make it clear that they are using an empirical approximation to $\\mathbb{E}[X]$. I feel like the authors only wanted to find the average of $X$ and misused expectation operator. Can the authors clarify their notation here?\n\n- There is a recent paper on layer-wise sparsity [1], where a new pruning criterion and strategy are proposed to avoid layer collapse.  The authors may want to have a discussion on that in the paper.\n\n- Minor: I would avoid abbreviations like 'can't' in an academic paper.\n\n[1] Lee, Jaeho, et al. \"Layer-adaptive Sparsity for the Magnitude-based Pruning.\" International Conference on Learning Representations. 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some limitations in Section 6. I suggest the authors discuss also their iid assumption and explain why they had to make that assumption although it is unrealistic. The authors do mention how their findings would impact the research community. They do not address the potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a study of pruning criteria by evaluating different methods and defining metrics that help understand the different properties of the various criteria out there. The topic of DNN pruning is quite rich today and I personally find it refreshing (as do some of the reviewers) to see a paper that tries to sort out the different methods. The paper is far from a complete solution to the problem, but it provides a good start. Given the importance and magnitude of the problem, I believe the paper is worthy of being published at NeurIPS and will likely motivate follow up papers that will improve the way we currently measure pruning criteria.
Decision: Accept (Poster)