Providing clear pruning threshold: A novel CNN pruning method via <i>L</i> <sub>0</sub> regularisation

IET Image ProcessingVolume 15, Issue 2 p. 405-418 ORIGINAL RESEARCH PAPEROpen Access Providing clear pruning threshold: A novel CNN pruning method via L0 regularisation Guo Li, Corresponding Author gxliguo@gmail.com North China Electric Power University, Beijing, 102206 China Correspondence Guo Li, North China Electric Power University, Beijing 102206, China. Email: gxliguo@gmail.comSearch for more papers by this authorGang Xu, North China Electric Power University, Beijing, 102206 ChinaSearch for more papers by this author Guo Li, Corresponding Author gxliguo@gmail.com North China Electric Power University, Beijing, 102206 China Correspondence Guo Li, North China Electric Power University, Beijing 102206, China. Email: gxliguo@gmail.comSearch for more papers by this authorGang Xu, North China Electric Power University, Beijing, 102206 ChinaSearch for more papers by this author First published: 14 December 2020 https://doi.org/10.1049/ipr2.12030AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Network pruning is a significant way to improve the practicability of convolution neural networks (CNNs) by removing the redundant structure of the network model. However, in most of the existing network pruning methods l1 or l2 regularisation is applied to parameter matrices and the manual selection of pruning threshold is difficult and labor-intensive. A novel CNNs network pruning method via l0 regularisation is proposed, which adopts l0 regularisation to expand the saliency gap between neurons. A half-quadratic splitting (HQS) based iterative algorithm is put forward to calculate the approximation solution of l0 regularisation, which makes the joint optimisation problem of regularisation term and training loss function can be solved by various gradient-based algorithms. Meanwhile, a hyperparameters selection method is designed to make most of the hyperparameters in the algorithm can be determined by examining the pre-trained model. The results of experiments on MNIST, Fashion-MNIST and CIFAR100 show that the proposed method can provide a much clearer pruning threshold by widening the saliency gap, and achieve a similar or even better compression performance, compared with the state-of-the-art studies. Nomenclature i layer index j filter index s number of filters g number of layers r regularisation redius α , β , λ , c i regularisation factor (n) current iteration number (N) number of iterations T auxiliary matrix W model parameter matrix W i layer parameter matrix w i j , t i j parameter set of the j-th filter in layer i Φ ( W i ) , Φ ( T i ) saliency vector in layer i ϕ i j W , ϕ i j T the j-th saliency value in layer i L ( W ) training loss 1 INTRODUCTION Convolution neural networks (CNNs) are powerful tools in various computer vision tasks [1-4]. However, CNNs have complicated network architectures and are storage massive and computation-intensive, which results in high deployment cost and limits application scenarios. As an effective method for reducing the deployment cost in both storage and computation, network compression has aroused increasing attention in the community. Numerous ways have been proposed to compress the network model, such as weight quantisation [5, 6], low-rank decomposition [7, 8], knowledge distillation [9] and network pruning. Weight quantisation compresses the model size and inference time by weight sharing and reducing the number of bits, which does no change in network structure. Therefore, it can not remove the structure redundancy and is often used with other methods. Low-rank decomposition decomposes a large convolution kernel into a smaller kernels or a cascade of small kernels to reduce computation load. But this type of method for decomposing the convolution kernel cannot delete the branches in the parallel structure, for example, the shortcut paths in resnet. Meanwhile, matrix decomposition is often computationally expensive. Knowledge distillation uses a pre-trained model to design a smaller architecture and then train from scratch. Network pruning achieves network simplification by removing unimportant substructures(e.g. filters, channels, layers) of the network without unaccepted accuracy drop. Network pruning starts with a pre-trained model, which is easier to keep model performance than knowledge distillation. Many of the network pruning methods focus on saliency metric [10-20]. However, the saliency metric in the pre-trained model is often gaussian distribution [13-15, 20, 21]. It is difficult to select a set of layer-wise pruning thresholds directly without sparse regularisation. In the line of regularisation, l1 [22-24] and l2 [17, 25-28] norm are mainstream choices. He et al. [20] stated that the effectiveness of norm-based pruning criterion depended on two requirements: (1)the norm deviation of the filters should be large, in other words, the numerical gap of norm between the important and unimportant filters should be large. (2)the minimum norm of the filters should be small. However, applying l1 or l2 regularisation in a parameters matrix will let all the parameters close to zero, which could hardly increase the saliency gap and results in more difficulties in adjusting the layer-wise pruning ratios in subsequent trimming steps. It should be noted that only the unimportant neurons other than all the neurons need to be regularised to zero and the others should be free from regularisation to find better values. In general, pruning algorithms are iteration-based, which have two iteration strategies: global iteration and layer-by-layer iteration. Global iterative pruning [17, 24] can globally consider the sensitivity of each layer to get the optimal pruning ratio, but the conventional l1 and l2 norm in global regularisation hardly increase the saliency difference between the important and unimportant neurons, which makes it hard to trim too many neurons in a single iteration and the trimming ratio of each layer needs to be carefully adjusted. For large depth models that require multiple pruning, the manual selection of layer-wise pruning ratio at each iteration is a huge workload. In opposite, the layer-by-layer pruning method [14, 15] can reach the best pruning result in some layers but prone to over-cutting, especially in the first few layers. Once previous layers are over-pruned, the following needs more parameters to build a stronger classifier to compensate for the loss of feature information. Because the pruning sensitivity and saliency distribution in different layers are diverse, to obtain the globally optimal solution, layer-wise regularisation coefficients and thresholds need to be carefully selected, which is often labor-intensive. In order to solve the shortcomings of the previous methods in the saliency gap and the difficulty of parameters tuning, we propose to use l0 norm to regularise the model parameters. l0 regularisation constrains some parameters to 0 while having no effect on other parameters, which makes some parameters free from regularisation to widen the saliency gap. Xu et al. [29] used HQS algorithm to solve l0 regularisation problem in the field of image filtering. However, classical HQS is hard to use in the field of network pruning directly due to the obscureness of hyperparameters and the differences in objective function of regularisation in different application areas. In this paper, an HQS-based iteration algorithm is proposed to get the approximate solution of l0 regularisation problem in the field of network pruning. A parameter selection method is also presented, which transforms ambiguous hyperparameters in the classical HQS algorithm into parameters that can be determined by directly observing the parameter distribution of the pre-trained model. Meanwhile, a strategy of iterative regularisation and one-shot trimming is adopted to further reduce the workload of parameters tuning: First, l0 regularisation is applied to the pre-trained model to make the saliency distribution of neurons as sparse as possible and increase the number of zero-saliency neurons. Then, all the neurons are globally trimmed according to their saliency only once. Finally the pruned model is retrained to compensate for loss of accuracy. Our major contributions are as follows: i. An iterative HQS based algorithm is firstly proposed to find the approximate solution of l0 regularisation in the field of network pruning, which can widen the saliency gap between important and unimportant neurons and provide clear pruning threshold for each layer. ii. A general hyperparameters selection method is presented. Some obscure parameters in HQS can be transformed into the new parameters set that can be obtained according to the observation of the pre-trained model histogram. We study the influence of each parameter in the new parameter set on the regularisation effect and give a general range for each parameter. ii. The strategy of global regularisation and one-shot trimming does not require sensitivity analysis and the multiple manual selections of layer-wise pruning ratios. It makes the proposed method can be directly applied to various network models and datasets without many hyperparameter attempts. The rest of the paper is organised as follows. Related works are described in Section 2. In Section 3, the proposed work method is described, including l0 regularisation, the calculation of hyperparameters and pruning strategy. Experiments on MNIST, Fashion-MNIST, and CIFAR100 dataset are introduced in Section 4. The conclusion is drawn in Section 5. 2 RELATED WORKS 2.1 l1 or l2 regularisation based methods Wen et al. [23] proposed a structured sparsity learning (SSL) method to regularise the substructures in network architecture. In their work, the parameters belonging to the same structure were regularised by group lasso regularisation. Liu et al. [24] introduced a scaling factor for each output channel, which was jointly trained with the network weights and constrained by l1 regularisation. In the trimming step, all the channels with the factor smaller than a global threshold were removed. As we discussed earlier, these types of l1 or l2 regularisation based method are hard to offer a clear pruning threshold, which makes the selection of layer-wise pruning ratios labor-intensive and often needs iterative pruning. Luo et al. [15] proposed a filter-level pruning method called Thinet, where the filters were pruned layer-by-layer according to its saliency without sparse regularisation. He et al. [20] proposed a saliency metric called geometric median (GM) and achieved state-of-the-art results, where the saliency metric was the distance between the filter and the geometric centre of the filters set. For this type of method, the layer-wise pruning ratios are often difficult to be determined due to the gaussian distribution of saliency metric. To achieve the ideal compression ratio on large models, a lot of attempts are usually inevitable. 2.2 l0 regularisation based methods Louizos et al. [30] proposed a method based on parameter prior distribution to use l0 regularisation in network training, which can be regarded as a global pruning method with layer-wise parameter. This method can achieve network compression during network training, but its compression rate is not as good as the method based on the pre-trained model. It is already difficult to select hyperparameters from the pre-trained model, from scratch can only be more difficult. Lin et al. [31] proposed structured sparsity regularisation (SSR) that integrated the l 2 , 1 , l 1 , l 2 , 0 regularisation and achieved state-of-the-art performance in many models. In their work, l0 sparse regularisation is firstly solved by alternating direction method of multipliers (ADMM), but most of the hyperparameters in their method are model-related. Moreover, the method of SSR is a layer-by-layer threshold-based pruning method. Once the filter is removed, it can not be restored. Thus, the layer-wise threshold should be selected carefully and the sensitive analysis before pruning is necessary. Meanwhile, the threshold in each layer is not controlled by an individual hyperparameter but two( α i and β), which leads to the coupling of the pruning threshold and the coefficient of the regularisation term. Thus, to obtain a good result, it is necessary to select a set of suitable α i whose element number is equal to the number of layers. It should be pointed out both the selection of hyperparameters and the layer-by-layer pruning strategy are time-consuming. 3 THE PROPOSED APPROACH 3.1 l0 sparse regularisation The simplified flowchart of our regularisation algorithm is depicted in Figure 1. Conventionally, applying regularisation to a pre-trained model can be described as minimising the following objective function: L = min W L ( W ) + λ ∑ i = 1 g p ( W i ) . (1)where W denotes all parameters of the model. L ( W ) is the training loss function. λ is regularisation coefficient. p ( · ) is the regularisation function. i and g are the layer index and the number of layers, respectively. W i is the parameter matrix of layer, which is a 2-D parameter matrix. For a convolution layer, the shape of filter is [ c o u t , c i n , h , w ] , so the shape of W i is [ c o u t , c i n × h × w ] . FIGURE 1Open in figure viewerPowerPoint Simplified flowchart of our regularisation algorithm s is the number of filters in layer1 and z is the number of weights in every filter. Saliency Hist is the histogram of the saliency vector elements. First, calculate the saliency of each filter. Then calculate the auxiliary variable: Setting the filter parameters in the weight matrix smaller than the regularisation radius to 0 such as w12, while keeping the other unchanged. Finally, the distance between the weight matrix and the auxiliary matrix is jointly optimised with training loss. During the iteration process, the regularisation radius is automatically decreased The objective function of joint optimisation of saliency-based l0 regularisation and training loss is: L = min W L ( W ) + λ ∑ i = 1 g Φ ( W i ) 0 , (2) Φ W i = { ϕ i j W | j ∈ [ 1 , s i ] } , ϕ i j W = φ w i j , (3) W = W i | i ∈ 1 , g . (4)where j is the filter index. Φ ( W i ) is the saliency vector, s i is its element number and related to regular granularity. φ ( · ) is the saliency function. To simplify mathematical derivations, let filter saliency is only related to its parameters(e.g. l1 norm), so we have the bottom equation in (3). It is worth mentioning that the saliency of filter ϕ i j might be not only related to its parameters w i j , and its definition is still an open question. Following the HQS principle [29] to solve this non-derivable problem, the objective function (2) is rewritten as follows: L = min W , T × L ( W ) + 1 s ∑ i = 1 g α Φ W i − Φ T i 2 2 + β Φ T i 0 , (5) s = ∑ i = 1 g s i , (6) T = T i | i ∈ 1 , g . (7)where W and T are two variables that need to be solved. α and β are both regularisation coefficients with positive values. The former is related to the balance between training loss and the similarity of ( W i , T i ) . The latter denotes the regularisation intensity of l0 norm. T i is an auxiliary variable that is directly constrained by l0 norm and has the same shape of W i . So far, (5) can be decomposed into optimising two parameters: W and T. Firstly, we get Φ ( T i ) by using l0 norm to constrain Φ ( W i ) , which is non-derivable. And then, minimising ∥ Φ ( W i ) − Φ ( T i ) ∥ 2 2 to make every element in Φ ( W i ) as similar as possible to its counterpart in Φ ( T i ) , which can be transferred to minimise ∥ W i − T i ∥ F 2 or ∥ W i − T i ∥ 1 2 . Both regularisation term are convex and derivable(only if the value is zero, the latter is not), so it can be optimised by many back-propagation-based methods, such as gradient descent. However, the auxiliary variables T limits the change of all parameters, which might be harmful to the compensation of performance loss caused by regularisation by optimising other filters. We definite a variant of T called version.2 to verify this intuition at the end of Sec 3.1.1, which is shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Definition of auxiliary variable T for version 1 and version 2. The saliency of the 2nd filter is smaller than regularisation radius. In the auxiliary variable version 1, all the filters are included, but only those smaller than the radius are included in version 2 3.1.1 Subproblem1: Optimising T In objective function (5), T is only related to the second term. In (3), we have assumed that the filter's saliency is only related to its parameters, so the ϕ i j T with different i j is irrelevant to each other and can be solved individually. Combing (5) and (3), the objective function of subproblem1 is: L 2 = min T ∑ i = 1 g α Φ ( W i ) − Φ ( T i ) 2 2 + β Φ ( T i ) 0 = min ϕ i j T ∈ T ∑ i = 1 g ∑ j = 1 s i α ϕ i j W − ϕ i j T 2 + β h ϕ i j T , (8) h x = 0 , x = 0 1 , e l s e . (9)where L2 is the second term of (5). ϕ i j W and ϕ i j T are the j-th elements of Φ ( W i ) and Φ ( T i ) , respectively. The closed-form solution of (8) is: ϕ i j T = 0 , ϕ i j W ≤ β α ϕ i j W , e l s e , (10) r = β α . (11)The mathematical derivation is in Appendix A.1. In Appendix A.2, we also analysis the difference between approximate and exact solution, and how the proposed method reduces it. Thus, the corresponding t i j for each ϕ i j T can be obtained by: t i j = 0 , ϕ i j T = 0 w i j , e l s e , (12) Finally, combining (12) and (10): t i j = 0 , ϕ i j W ≤ r w i j , e l s e . (13) Weight-wise regularisation For weight-wise pruning, the ϕ i j is the saliency of the j-th weight in layer i, and w i j is the value of the j-th weight in W i . A common choice of ϕ i j is the magnitude of weight value, so ϕ i j = | w i j | Filter-wise regularisation For filter-wise and channel-wise pruning, ϕ i j is the saliency of the j-th filter in layer i. w i j denotes the parameters set that includes all the parameters in the corresponding filter, which is the j-th row vector of matrix W i . If the saliency metric is l2 norm, ϕ i j = ∥ w i j ∥ 2 . Batch normalisation regularisation Batch normalisation is widely used in many network architectures, where every output channel x is normalised and then mapped to y by function y = a x + b . The scale parameter a can be used to minimise the channel outputs. Inspired by previous work [24], when the convolution layer i is followed by a batch norm layer, let the filter saliency ϕ i j in (13) equal to the corresponding scale parameter | a i j | . Sparse normalisation In general, the saliency distribution range in each layer is vastly different, which means that applying a global r in (10) might cause different constrain intensity in each layer. For example, if a layer's distribution range is smaller than r in all iterations, the corresponding T is always zero according to (13). To precisely controlling the regularisation intensity in each layer as wish, normalising the saliency histogram of different layers to a similar distribution range is needed. To this end, (13) is replaced to (14). ( ϕ i j W ( n ) ) ′ = ϕ i j W ( n ) D Φ ( W i ( 0 ) ) , t i j ( n ) ′ = 0 , ( ϕ i j W ( n ) ) ′ ≤ r w i j ( n ) , e l s e . (14)where the variables with superscript (n) are iteration-varying, n is the current iteration number. W i ( 0 ) is the parameter matrix of the pre-trained model, D is the variance function. In actual situations, the saliency vector in every layer is normalised using (14) once and then multiply by a layer-wise coefficient to scale to the same range. Using the following method to let most of the saliency values in the pre-trained model be small than 1 might be more directly: ( ϕ i j W ( n ) ) ′ = ϕ i j W ( n ) 95 t h Φ ( W i ( 0 ) ) . (15)where 95 t h ( Φ ( W i ( 0 ) ) ) is the 95th percentile of Φ ( W i ( 0 ) ) . This method discards the average value of the saliency vector that might be related to the layer's saliency, but the layer's saliency can be obtained after the first sparse regularisation. After performing the same regularisation on each layer, the more sparse the layers, the less important. 3.1.2 Subproblem2: Optimising W Solving W is equivalent to solving the following equation, which is formed by removing irrelevant variables in (5). L = min W L ( W ) + 1 s ∑ i = 1 g c i α W i − T i F 2 . (16)where we add a switch c i to control the regularisation strength of each layer. Generally, c i = 1 . There are various algorithms to solve this kind of minimisation problem and Frobenius norm loss is convex, so it has no great influence on the original model convergence. In our work, we use gradient descent to optimise this function. According to (13) and Figure 2, when pushing the parameters in the range of regularisation radius to zero, the change of elements that out of the range is limited at the same time. On the one hand, it can protect the performance and saliency gap by preventing big value changing. On the other hand, this is not in line with the intuition that pruning would make the change of all neurons. This intuition can be tested by rewriting the (16) as follows(called version.2): L = min W L ( W ) + 1 s ∑ i = 1 g c i α W i ′ − 0 F 2 , W i ′ = w i j | j ∈ { j | t i j = 0 , t i j ∈ T i } . (17)where W i ′ is a subset of W i , which consists of the elements that the counterpart values in T i are zero. In other words, only the elements in the range of regularisation radius are regularised to zero. Once the output channel j can be removed in the pruning step, all filters related to channel j in the next layer can also be removed. To avoid the weak channel output being amplified by the filters in the next layer, we regularise the filters with the low l2 norm and the corresponding input filters in the next layer. 3.1.3 Parameters updating and terminating condition The coefficient α highly impacts the similarity between W and T, Once we get W from (16), α will be update by (18), and then resolve T , W in order and so on. α ( n + 1 ) = σ α ( n ) , σ > 1 . (18) The proposed sparse regularisation algorithm is summarised in Algorithm 1, where the superscript n ∈ [ 0 , N ] represents the number of iterations, the variable with superscript 0 is its initial value, and with N is its ending value. The algorithm that use (16) in step 5 is called version.1, and (17) is called version.2. ALGORITHM 1. l0 Sparse Regularisation Input: Dataset D, model parameters W(0) Hyperparameter: α ( 0 ) , α ( N ) , β , σ Output: W ( N ) 1: Let α = α ( 0 ) . 2: while α ( n ) < α ( N ) do 3: Compute saliency matrix Φ ( W ( n ) ) from (3) 4: Compute T ( n + 1 ) from (14) 5: Compute W ( n + 1 ) from (16)(ver.1) or (17)(ver.2) 6: update α ( n + 1 ) ← σ α ( n ) by (18) 7: end while 8: return model parameters W ( N ) 3.2 The calculation of Hyperparameters There are four hyperparameters in our method, which are α ( 0 ) , α ( N ) , β and σ. To simplify, lets: r ( n ) = β α ( N ) . (19)where r ( n ) is the regularisation radius in iteration n, and is decreases with the increasing of iteration n, because: r ( n + 1 ) r ( n ) = 2 β / ( σ α ( N ) ) 2 β / α ( N ) = 1 σ , σ > 1 . (20) According to Algorithm 1, auxiliary variable T ( n + 1 ) is calculated by (14). From the mathematical point of view, when r ( n ) → 0 , ∥ T ( N ) ∥ 0 = ∥ W ( N ) ∥ 0 . Meanwhile, because α ( N ) increases with n, the constraint ability of T ( N ) to W ( N ) is gradually enhanced, according to (16), which makes more small parameters in W ( N ) closer to 0. Let r(0) and r ( N ) denote initialise and stopping radius, respectively. These two processes coordinate with each other and should follow the following principles: In the beginning, a large r(0) can provide a wide parameter search range to find more potential, and a small α(0) can balance larger regularisation losses to protect model performance. At the end of the iteration, small r ( n ) can provide more precise norm constraints, and higher α ( N ) is helpful to increase the number of zero elements in W ( N ) . In order to protect the model performance, the ratio of regular loss to training error (the first and second term in (2), respectively) in the final iteration should be strictly limited. According to principles 3, let k be the ratio of regularisation loss to training loss, according to (16): 1 s α ( N ) W ( N ) − T ( N ) F 2 = k · L ( W ( N ) ) . (21)where k is the ratio of regularisation loss to training loss. Assuming that model performance and training error are almost equal before and after pruning, the W ( N ) in the right hand of (21) can be replaced by W(0). According to (16) and (21), to protect the model performance, the α ( N ) should be smaller, so the overestimation of ∥ W ( N ) − T ( N ) ∥ F 2 is necessary. The similarity of W and T is increasing with the iteration number, so ∥ W ( 0 ) − T ( 0 ) ∥ F 2 > ∥ W ( N ) − T ( N ) ∥ F 2 . Thus, ∥ W ( N ) − T ( N ) ∥ F 2 in the left hand of (21) is replaced by ∥ W ( 0 ) − T ( 0 ) ∥ F 2 . The (21) can be rewritten as follows: 1 s α ( N ) W ( 0 ) − T ( 0 ) F 2 = k · L ( W ( 0 ) ) , (22) α ( N ) = k s · L ( W ( 0 ) ) W ( 0 ) − T ( 0 ) F 2 . (23) T(0) can be solved by combining (13)(19): t i j ( 0 ) = 0 , ϕ i j W ( 0 ) ≤ r ( 0 ) w i j ( 0 ) , e l s e , r ( 0 ) = β α ( 0 ) . (24) According to principles 1, a large r(0) is beneficial to expand the search range and a typical value is to cover the whole range of parameter values. Generally, r(0) can be easily obtained by observing the histogram of parameters. In further, if the model parameters are normalised, it can even be independent of the model. Once r(0) is defined, α ( N ) and β can be obtained by (23) and (24), respectively. According to principles 2, r ( n ) is a small value, which can also be obtained from parameter histogram. Once r ( n ) is defined, α ( N ) can be calculated by (24). Assume the maximum number of iterations is N, we have: α ( N ) = σ N α ( 0 ) , N = log σ α ( N ) α ( 0 ) = ln α ( N ) − ln α ( 0 ) ln σ . (25)where the superscript N of σ represents the N-th power of σ, and (N) of α denotes the maximum number of iterations. ALGORITHM 2. l0 Parameters Selection Input: Dataset ( X , Y ) , model parameters W(0), regular radius ( r ( 0 ) , r ( n ) ) , the ratio of regularisation loss to training error k, iteration step σ Output: Parameters α ( 0 ) , α ( N ) , β , σ 1: Compute α ( N ) by α ( N ) = k s · L ( W ( 0 ) ) ∥ W ( 0 ) − T ( 0 ) ∥ F 2 2: Compute β by β = α ( N ) ( r ( n ) ) 2 3: Compute α(0) by α ( 0 ) = β ( r ( 0 ) ) 2 4: return α ( 0 ) , α ( N ) , β , σ In HQS theory, small σ is good for getting close to the real solution of l0 regularisation, the typical range is (1.1, 2]. The hyperparameter selection method is summarised in Algorithm 2. 3.3 Pruning Various saliency metrics and pruning methods can be introduced to our algorithm framework. In our experiment, the saliency metric is defined as the Frobenius norm of filters, l2 norm of neurons, and the scale coefficients in batch normalisation layer. As is shown in Figure 6, compared with the baseline model and conventional regularisation methods, proposed method possesses a higher saliency difference, which means that a simple pruning theory can also meet good results and can save us from the despaired loop of threshold selection an