Rethinking the Pruning Criteria for Convolutional Neural Network

1 Introduction . Pruning [ 1 , 2 , 3 , 4 ] a trained neural network is commonly seen in network compression . In particular , for CNNs , channel pruning refers to the pruning of the filters in the convolutional layers . There are several critical factors for channel pruning . Procedures . One-shot method [ 5 ] : Train a network from scratch ; Use a certain criterion to calculate filters ’ Importance Score , and prune the filters which have small Importance Score ; After additional training , the pruned network can recover its accuracy to some extent . Iterative method [ 1 , 6 , 7 ] : Unlike One-shot methods , they prune and fine-tune a network alternately . Criteria . The filters ’ Importance Score can be definded by a given criterion . From different ideas , many types of pruning criteria have been proposed , such as Norm-based [ 5 ] , Activation-based [ 8 , 9 ] , Importance-based [ 10 , 11 ] , BN-based [ 12 ] and so on . Strategy . Layer-wise pruning : In each layer , we can sort and prune the filters , which have small Importance Score measured by a given criterion . Global pruning : Different from layer-wise pruning , global pruning [ 12 , 13 ] sort the filters from all the layers through their Importance Score and prune them . In this work , we conduct our investigation on a variety of pruning criteria . As one of the simplest and most effective channel pruning criteria , ` 1 pruning [ 5 ] is widely used in practice . The core idea of this criterion is to sort the ` 1 norm of filters in one layer and then prune the filters with a small ` 1 norm . Similarly , there is ` 2 pruning which instead leverages the ` 2 norm [ 7 , 6 ] . ` 1 and ` 2 can be seen as the criteria which use absolute Importance Score of filters . Through the study of the distribution of norm , [ 4 ] demonstrates that these criteria should satisfy two conditions : ( 1 ) the variance of the norm of the filters can not be too small ; ( 2 ) the minimum norm of the filters should be small enough . Since ∗Corresponding author : pluo.lhi @ gmail.com ; ‡ co-first author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these two conditions do not always hold , a new criterion considering the relative Importance Score of the filters is proposed [ 4 ] . Since this criterion uses the Fermat point ( i.e. , geometric median [ 14 ] ) , we call this method Fermat . Due to the high calculation cost of Fermat point , [ 4 ] further relaxed the Fermat and then introduced another criterion denotes as GM . To illustrate each of the pruning criteria , let Fij ∈ RNi×k×k represent the jth filter of the ith convolutional layer , where Ni is the number of input channels for ith layer and k denotes the kernel size of the convolutional filter . In ith layer , there are Ni+1 filters . For each criteria , details are shown in Table 2 , where F denotes the Fermat point of Fij in Euclidean space . These four pruning criteria are called Norm-based pruning in this paper as they utilize norm in their design . Previous works [ 15 , 3 , 16 , 17 , 18 ] , including the criteria mentioned above , the main concerns commonly consist of ( a ) How much the model was compressed ; ( b ) How much performance was restored ; ( c ) The inference efficiency of the pruned network and ( d ) The cost of finding the pruned network . However , few works discussed the following two blind spots about the pruning criteria : ( 1 ) Similarity : What are the actual differences among these pruning criteria ? Taking the VGG16 and ResNet18 on ImageNet as an example , we show the ranks of filters ’ Importance Score under different criteria in Table 1 . It is obvious that they have almost the same sequence , leading to similar pruned structures . In this situation , the criteria used absolute Importance Score of filters ( ` 1 , ` 2 ) and the criteria used relative Importance Score of filters ( Fermat , GM ) may not be significantly different . ( 2 ) Applicability : What is the applicability of these pruning criteria to prune the CNNs ? There is a toy example w.r.t . ` 2 criterion . If the ` 2 norm of the filters in one layer are 0.9 , 0.8 , 0.4 and 0.01 , according to smaller-norm-less-informative assumption [ 19 ] , it ’ s apparent that we should prune the last filter . However , if the norm are close , such as 0.91 , 0.92 , 0.93 , 0.92 , it is hard to determine which filter should be pruned even though the first one is the smallest . In Fig . 1 , we demonstrate some real examples , i.e. , the visualization of Applicability problem under different networks and criteria . In this paper , we provide comprehensive observations and in-depth analysis of these two blind spots . Before that , in Section 2 , we propose an assumption about the parameters distribution of CNNs , called Convolution Weight Distribution Assumption ( CWDA ) , and use it as a theoretical tool to analyze the two blind spots . We explore the Similarity and Applicability problem of pruning criteria in the following order : ( 1 ) Norm-based criteria ( layer-wise pruning ) in Section 3 ; ( 2 ) Other types of criteria ( layer-wise pruning ) in Section 4 ; ( 3 ) and different types of criteria ( global pruning ) in Section 5 . Last but not least , we provide further discussion on : ( i ) the conditions for CWDA to be satisfied , ( ii ) how our findings help the community in Section 6 . In order to focus on the pruning criteria , all the pruning experiments are based on the relatively simple pruning procedure , i.e. , one-shot method . The main contributions of this work are two-fold : ( 1 ) We analyze the Applicability problem and the Similarity of different types of pruning criteria . These two blind spots can guide and motivate researchers to design more reasonable criteria . We also break some stereotypes , such as that the results of ` 1 and ` 2 pruning are not always similar . ( 2 ) We propose and verify an assumption called CWDA , which reveals that the well-trained convolutional filters approximately follow a Gaussian-alike distribution . Using CWDA , we succeeded in explaining the multiple observations about these two blind spots theoretically . 2 Weight Distribution Assumption . In this section , we propose and verify an assumption about the parameters distribution of the convolutional filters . ( Convolution Weight Distribution Assumption ) Let Fij ∈ RNi×k×k be the jth well-trained filter of the ith convolutional layer . In general2 , in ith layer , Fij ( j = 1 , 2 , ... , Ni+1 ) are i.i.d and follow such a distribution : Fij ∼ N ( 0 , Σidiag + ·Σiblock ) , ( 1 ) where Σiblock = diag ( K1 , K2 , ... , KNi ) is a block diagonal matrix and the diagonal elements of Σiblock are 0. is a small constant . The values of the off-block-diagonal elements are 0 and Kl ∈ Rk 2×k2 , l = 1 , 2 , ... , Ni . Σidiag = diag ( a1 , a2 , ... , aNi×k×k ) is a diagonal matrix and the elements of Σidiag are close enough . This assumption is based on the observation shown in the Fig . 2 . To estimate Σidiag + ·Σiblock , we use the correlation matrix FFT where F ∈ R ( Ni×k×k ) ×Ni+1 denotes all the parameters in ith layer . Taking a convolutional layer of ResNet18 trained on ImageNet as an example , we find that FFT is a block diagonal matrix . Specifically , each block is a k2 × k2 matrix and the off-diagonal elements are close to 0 . We visualize the jth filter Fij ∈ RNi×k×k in ith layer in Fig . 2 ( c ) , and this phenomenon reveals that the parameters in the same channel of Fij tend to be linearly correlated , and the parameters of any two different channels ( yellow and green channel in Fig . 2 ( c ) ) in Fij only have a low linear correlation . 2.1 Statistical test for CWDA . In fact , CWDA is not easy to be verified , e.g. , for ResNet164 trained on Cifar100 , the number of filters in the first stage is only 16 , which is too small to be used to estimate the statistics in CWDA accurately . Thus , We consider verifying four necessary conditions of CWDA : ( 1 ) Gaussian . Whether the weights of Fij approximately follows a Gaussian-alike distribution ; ( 2 ) Variance . Whether the variance of the diagonal elements of Σdiag are small enough ; ( 3 ) Mean . Whether the mean of weights of Fij is close to 0 . ( 4 ) The magnitude of . Whether is small enough . The results of the tests are shown in Appendix P , where we consider a variety of factors for the statistical tests , including different network structure , optimizer , regularization , initialization , dataset , 2In Section 6 , we make further discussion and analysis on the conditions for CWDA to be satisfied . training strategy , and other tasks in computer vision ( e.g. , semantic segmentation , detection and so on ) . The test results show that CWDA has a great generality for CNNs . 3 About the Norm-based criteria . We start from the criteria in Table 2 , which are widely cited and compared [ 21 , 22 , 23 , 24 , 25 ] . 3.1 Similarity . In this section , we further verify the observation that the Norm-based pruning criteria in Table 2 are highly similar from two perspectives . Empirically , we conducted large amount of experiments on image classification to investigate the similarities . Theoretically , we rigorously prove the similarities of the criteria in Table 2 in layer-wise pruning under CWDA . Empirical Analysis . ( 1 ) In Fig . 3 , we show the test accuracy of the ResNet56 after pruning and fine-tuning under different pruning ratios and datasets . The test accuracy curves of different pruning criteria at different stages are very close under different pruning ratios . This phenomenon implies that those pruned networks using different Norm-based criteria are very similar , and there are strong similarities among these pruning criteria . The experiments about other commonly used configs of pruning ratio can be found in Appendix L. ( 2 ) In Fig . 4 , we show the Spearman ’ s rank correlation coefficient3 ( Sp ) between different pruning criteria . The Sp in most convolutional layers are more than 0.9 , which means the network structures are almost the same after pruning . Note that the Sp in transition layer are relatively small , and the transition layer refers to the layer where the dimensions of the filter change , like the layer between stage 1 and stage 2 of a ResNet . The reason for this phenomenon may be that the layers in these areas are sensitive . It is interesting but will not greatly impact the structural similarity of the whole pruned network . The similar observations are shown in Fig . 2 in [ 16 ] , Fig . 6 and Fig . 10 in [ 5 ] . 3Sp is a nonparametric measurement of ranking correlation , and it assesses how well the relationship between two variables can be described using a monotonic function , i.e. , filters ranking sequence in the same layer under two criteria in this paper . Theoretical Analysis . Besides the experimental verification , the similarities via using layer-wise pruning among the criteria in Table 2 can also be proved theoretically in this section . Let C1 and C2 be two pruning criteria to calculate the Importance Score for all convolutional filters in one layer . If they can produce the similar ranks of Importance Score , we define that C1 and C2 are approximately monotonic to each other and use C1 ∼= C2 to represent this relationship . In Section 3.1 , we use the Sp to describe this relationship but it ’ s hard to be analyzed theoretically . Therefore , we focus on a stronger condition . Let X = ( x1 , x2 , ... , xk ) and Y = ( y1 , y2 , ... , yk ) be two given sequences4 . we first normalize their magnitude , i.e. , let X̂ = X/E ( X ) and Ŷ = Y/E ( Y ) . This operation does not change the ranking sequence of the elements of X and Y , because E ( X ) and E ( Y ) are constants , i.e. , X̂ ∼= Ŷ ⇔ X ∼= Y . After that , if both Var ( X̂/Ŷ ) and Var ( Ŷ/X̂ ) are small enough , then the Sp between X and Y is close to 1 , where X̂/Ŷ = ( x̂1/ŷ1 , .. , x̂k/ŷk ) . The reason is that in these situations , the ratio X̂/Ŷ and Ŷ/X̂ will be close to two constants a , b . For any 1 ≤ i ≤ k , x̂i ≈ a · ŷi and ŷi ≈ b · x̂i . So , ab ≈ 1 and a , b 6= 0 . Therefore , there exists an approximately monotonic mapping from ŷi to x̂i ( linear function ) , which makes the Sp between X and Y close to 1 . With this basic fact , we propose the Theorem 1 , which implies that many Norm-based pruning criteria produces almost the same ranks of Importance Score . Theorem 1 . Let n−dimension random variable X meet CWDA , and the pair of criteria ( C1 , C2 ) is one of ( ` 1 , ` 2 ) , ( ` 2 , Fermat ) or ( Fermat , GM ) , we have max { VarX ( Ĉ2 ( X ) Ĉ1 ( X ) ) , VarX ( Ĉ1 ( X ) Ĉ2 ( X ) ) } . B ( n ) , ( 2 ) where Ĉ1 ( X ) denotes C1 ( X ) /E ( C1 ( X ) ) and Ĉ2 ( X ) denotes C2 ( X ) /E ( C2 ( X ) ) . B ( n ) denotes the upper bound of left-hand side and when n is large enough , B ( n ) → 0 . Proof . ( See Appendix C ) . In specific , for ith convolutional layer of a CNN , since Fij ∈ Rn , j = 1 , 2 , ... Ni+1 , meet CWDA and the dimension n is generally large , we can obtain ` 1 ∼= ` 2 , ` 2 ∼= Fermat and Fermat ∼= GM according to Theorem 1 . Therefore , we have ` 1 ∼= ` 2 ∼= Fermat ∼= GM , which verifies the strong similarities among the criteria shown in Table 2 .

Summary: The goal of the paper is to bring into attention that many norm-based pruning criteria used for structured pruning are very similar, in that their ranking of the redundant filters is highly correlated. The key ingredient is the CWDA assumption that filters in a particular convolutional layer are iid and approximately follow a Gaussian distribution, which is shown based on extensive statistical hypothesis testing. Based on this assumption, they prove that these pruning criteria are roughly the same.