Rethinking the Pruning Criteria for Convolutional Neural Network

1 Introduction . Pruning [ 1 , 2 , 3 , 4 ] a trained neural network is commonly seen in network compression . In particular , for CNNs , channel pruning refers to the pruning of the filters in the convolutional layers . There are several critical factors for channel pruning . Procedures . One-shot method [ 5 ] : Train a network from scratch ; Use a certain criterion to calculate filters ’ Importance Score , and prune the filters which have small Importance Score ; After additional training , the pruned network can recover its accuracy to some extent . Iterative method [ 1 , 6 , 7 ] : Unlike One-shot methods , they prune and fine-tune a network alternately . Criteria . The filters ’ Importance Score can be definded by a given criterion . From different ideas , many types of pruning criteria have been proposed , such as Norm-based [ 5 ] , Activation-based [ 8 , 9 ] , Importance-based [ 10 , 11 ] , BN-based [ 12 ] and so on . Strategy . Layer-wise pruning : In each layer , we can sort and prune the filters , which have small Importance Score measured by a given criterion . Global pruning : Different from layer-wise pruning , global pruning [ 12 , 13 ] sort the filters from all the layers through their Importance Score and prune them . In this work , we conduct our investigation on a variety of pruning criteria . As one of the simplest and most effective channel pruning criteria , ` 1 pruning [ 5 ] is widely used in practice . The core idea of this criterion is to sort the ` 1 norm of filters in one layer and then prune the filters with a small ` 1 norm . Similarly , there is ` 2 pruning which instead leverages the ` 2 norm [ 7 , 6 ] . ` 1 and ` 2 can be seen as the criteria which use absolute Importance Score of filters . Through the study of the distribution of norm , [ 4 ] demonstrates that these criteria should satisfy two conditions : ( 1 ) the variance of the norm of the filters can not be too small ; ( 2 ) the minimum norm of the filters should be small enough . Since ∗Corresponding author : pluo.lhi @ gmail.com ; ‡ co-first author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these two conditions do not always hold , a new criterion considering the relative Importance Score of the filters is proposed [ 4 ] . Since this criterion uses the Fermat point ( i.e. , geometric median [ 14 ] ) , we call this method Fermat . Due to the high calculation cost of Fermat point , [ 4 ] further relaxed the Fermat and then introduced another criterion denotes as GM . To illustrate each of the pruning criteria , let Fij ∈ RNi×k×k represent the jth filter of the ith convolutional layer , where Ni is the number of input channels for ith layer and k denotes the kernel size of the convolutional filter . In ith layer , there are Ni+1 filters . For each criteria , details are shown in Table 2 , where F denotes the Fermat point of Fij in Euclidean space . These four pruning criteria are called Norm-based pruning in this paper as they utilize norm in their design . Previous works [ 15 , 3 , 16 , 17 , 18 ] , including the criteria mentioned above , the main concerns commonly consist of ( a ) How much the model was compressed ; ( b ) How much performance was restored ; ( c ) The inference efficiency of the pruned network and ( d ) The cost of finding the pruned network . However , few works discussed the following two blind spots about the pruning criteria : ( 1 ) Similarity : What are the actual differences among these pruning criteria ? Taking the VGG16 and ResNet18 on ImageNet as an example , we show the ranks of filters ’ Importance Score under different criteria in Table 1 . It is obvious that they have almost the same sequence , leading to similar pruned structures . In this situation , the criteria used absolute Importance Score of filters ( ` 1 , ` 2 ) and the criteria used relative Importance Score of filters ( Fermat , GM ) may not be significantly different . ( 2 ) Applicability : What is the applicability of these pruning criteria to prune the CNNs ? There is a toy example w.r.t . ` 2 criterion . If the ` 2 norm of the filters in one layer are 0.9 , 0.8 , 0.4 and 0.01 , according to smaller-norm-less-informative assumption [ 19 ] , it ’ s apparent that we should prune the last filter . However , if the norm are close , such as 0.91 , 0.92 , 0.93 , 0.92 , it is hard to determine which filter should be pruned even though the first one is the smallest . In Fig . 1 , we demonstrate some real examples , i.e. , the visualization of Applicability problem under different networks and criteria . In this paper , we provide comprehensive observations and in-depth analysis of these two blind spots . Before that , in Section 2 , we propose an assumption about the parameters distribution of CNNs , called Convolution Weight Distribution Assumption ( CWDA ) , and use it as a theoretical tool to analyze the two blind spots . We explore the Similarity and Applicability problem of pruning criteria in the following order : ( 1 ) Norm-based criteria ( layer-wise pruning ) in Section 3 ; ( 2 ) Other types of criteria ( layer-wise pruning ) in Section 4 ; ( 3 ) and different types of criteria ( global pruning ) in Section 5 . Last but not least , we provide further discussion on : ( i ) the conditions for CWDA to be satisfied , ( ii ) how our findings help the community in Section 6 . In order to focus on the pruning criteria , all the pruning experiments are based on the relatively simple pruning procedure , i.e. , one-shot method . The main contributions of this work are two-fold : ( 1 ) We analyze the Applicability problem and the Similarity of different types of pruning criteria . These two blind spots can guide and motivate researchers to design more reasonable criteria . We also break some stereotypes , such as that the results of ` 1 and ` 2 pruning are not always similar . ( 2 ) We propose and verify an assumption called CWDA , which reveals that the well-trained convolutional filters approximately follow a Gaussian-alike distribution . Using CWDA , we succeeded in explaining the multiple observations about these two blind spots theoretically . 2 Weight Distribution Assumption . In this section , we propose and verify an assumption about the parameters distribution of the convolutional filters . ( Convolution Weight Distribution Assumption ) Let Fij ∈ RNi×k×k be the jth well-trained filter of the ith convolutional layer . In general2 , in ith layer , Fij ( j = 1 , 2 , ... , Ni+1 ) are i.i.d and follow such a distribution : Fij ∼ N ( 0 , Σidiag + ·Σiblock ) , ( 1 ) where Σiblock = diag ( K1 , K2 , ... , KNi ) is a block diagonal matrix and the diagonal elements of Σiblock are 0. is a small constant . The values of the off-block-diagonal elements are 0 and Kl ∈ Rk 2×k2 , l = 1 , 2 , ... , Ni . Σidiag = diag ( a1 , a2 , ... , aNi×k×k ) is a diagonal matrix and the elements of Σidiag are close enough . This assumption is based on the observation shown in the Fig . 2 . To estimate Σidiag + ·Σiblock , we use the correlation matrix FFT where F ∈ R ( Ni×k×k ) ×Ni+1 denotes all the parameters in ith layer . Taking a convolutional layer of ResNet18 trained on ImageNet as an example , we find that FFT is a block diagonal matrix . Specifically , each block is a k2 × k2 matrix and the off-diagonal elements are close to 0 . We visualize the jth filter Fij ∈ RNi×k×k in ith layer in Fig . 2 ( c ) , and this phenomenon reveals that the parameters in the same channel of Fij tend to be linearly correlated , and the parameters of any two different channels ( yellow and green channel in Fig . 2 ( c ) ) in Fij only have a low linear correlation . 2.1 Statistical test for CWDA . In fact , CWDA is not easy to be verified , e.g. , for ResNet164 trained on Cifar100 , the number of filters in the first stage is only 16 , which is too small to be used to estimate the statistics in CWDA accurately . Thus , We consider verifying four necessary conditions of CWDA : ( 1 ) Gaussian . Whether the weights of Fij approximately follows a Gaussian-alike distribution ; ( 2 ) Variance . Whether the variance of the diagonal elements of Σdiag are small enough ; ( 3 ) Mean . Whether the mean of weights of Fij is close to 0 . ( 4 ) The magnitude of . Whether is small enough . The results of the tests are shown in Appendix P , where we consider a variety of factors for the statistical tests , including different network structure , optimizer , regularization , initialization , dataset , 2In Section 6 , we make further discussion and analysis on the conditions for CWDA to be satisfied . training strategy , and other tasks in computer vision ( e.g. , semantic segmentation , detection and so on ) . The test results show that CWDA has a great generality for CNNs . 3 About the Norm-based criteria . We start from the criteria in Table 2 , which are widely cited and compared [ 21 , 22 , 23 , 24 , 25 ] . 3.1 Similarity . In this section , we further verify the observation that the Norm-based pruning criteria in Table 2 are highly similar from two perspectives . Empirically , we conducted large amount of experiments on image classification to investigate the similarities . Theoretically , we rigorously prove the similarities of the criteria in Table 2 in layer-wise pruning under CWDA . Empirical Analysis . ( 1 ) In Fig . 3 , we show the test accuracy of the ResNet56 after pruning and fine-tuning under different pruning ratios and datasets . The test accuracy curves of different pruning criteria at different stages are very close under different pruning ratios . This phenomenon implies that those pruned networks using different Norm-based criteria are very similar , and there are strong similarities among these pruning criteria . The experiments about other commonly used configs of pruning ratio can be found in Appendix L. ( 2 ) In Fig . 4 , we show the Spearman ’ s rank correlation coefficient3 ( Sp ) between different pruning criteria . The Sp in most convolutional layers are more than 0.9 , which means the network structures are almost the same after pruning . Note that the Sp in transition layer are relatively small , and the transition layer refers to the layer where the dimensions of the filter change , like the layer between stage 1 and stage 2 of a ResNet . The reason for this phenomenon may be that the layers in these areas are sensitive . It is interesting but will not greatly impact the structural similarity of the whole pruned network . The similar observations are shown in Fig . 2 in [ 16 ] , Fig . 6 and Fig . 10 in [ 5 ] . 3Sp is a nonparametric measurement of ranking correlation , and it assesses how well the relationship between two variables can be described using a monotonic function , i.e. , filters ranking sequence in the same layer under two criteria in this paper . Theoretical Analysis . Besides the experimental verification , the similarities via using layer-wise pruning among the criteria in Table 2 can also be proved theoretically in this section . Let C1 and C2 be two pruning criteria to calculate the Importance Score for all convolutional filters in one layer . If they can produce the similar ranks of Importance Score , we define that C1 and C2 are approximately monotonic to each other and use C1 ∼= C2 to represent this relationship . In Section 3.1 , we use the Sp to describe this relationship but it ’ s hard to be analyzed theoretically . Therefore , we focus on a stronger condition . Let X = ( x1 , x2 , ... , xk ) and Y = ( y1 , y2 , ... , yk ) be two given sequences4 . we first normalize their magnitude , i.e. , let X̂ = X/E ( X ) and Ŷ = Y/E ( Y ) . This operation does not change the ranking sequence of the elements of X and Y , because E ( X ) and E ( Y ) are constants , i.e. , X̂ ∼= Ŷ ⇔ X ∼= Y . After that , if both Var ( X̂/Ŷ ) and Var ( Ŷ/X̂ ) are small enough , then the Sp between X and Y is close to 1 , where X̂/Ŷ = ( x̂1/ŷ1 , .. , x̂k/ŷk ) . The reason is that in these situations , the ratio X̂/Ŷ and Ŷ/X̂ will be close to two constants a , b . For any 1 ≤ i ≤ k , x̂i ≈ a · ŷi and ŷi ≈ b · x̂i . So , ab ≈ 1 and a , b 6= 0 . Therefore , there exists an approximately monotonic mapping from ŷi to x̂i ( linear function ) , which makes the Sp between X and Y close to 1 . With this basic fact , we propose the Theorem 1 , which implies that many Norm-based pruning criteria produces almost the same ranks of Importance Score . Theorem 1 . Let n−dimension random variable X meet CWDA , and the pair of criteria ( C1 , C2 ) is one of ( ` 1 , ` 2 ) , ( ` 2 , Fermat ) or ( Fermat , GM ) , we have max { VarX ( Ĉ2 ( X ) Ĉ1 ( X ) ) , VarX ( Ĉ1 ( X ) Ĉ2 ( X ) ) } . B ( n ) , ( 2 ) where Ĉ1 ( X ) denotes C1 ( X ) /E ( C1 ( X ) ) and Ĉ2 ( X ) denotes C2 ( X ) /E ( C2 ( X ) ) . B ( n ) denotes the upper bound of left-hand side and when n is large enough , B ( n ) → 0 . Proof . ( See Appendix C ) . In specific , for ith convolutional layer of a CNN , since Fij ∈ Rn , j = 1 , 2 , ... Ni+1 , meet CWDA and the dimension n is generally large , we can obtain ` 1 ∼= ` 2 , ` 2 ∼= Fermat and Fermat ∼= GM according to Theorem 1 . Therefore , we have ` 1 ∼= ` 2 ∼= Fermat ∼= GM , which verifies the strong similarities among the criteria shown in Table 2 .

Summary: This paper analyzes the current limitations of existing magnitude-based pruning methods. First, the paper focuses on the similarities between three methods and then focuses on the redundancy in large networks. The paper also analyzes the weight distribution for a well-trained network and propose CDWA as a way to prove this distribution.  My main concern with this paper is the contribution to the field.