Title: GLOBALLY SOFT FILTER PRUNING FOR EFFICIENT CONVOLUTIONAL NEURAL NETWORKS

Abstract: This paper propose a cumulative saliency based Globally Soft Filter Pruning (GSFP) scheme to prune redundant filters of Convolutional Neural Networks (CNNs). Specifically, the GSFP adopts a robust pruning method, which measures the global redundancy of the filter in the whole model by using the soft pruning strategy. In addition, in the model recovery process after pruning, we use the cumulative saliency strategy to improve the accuracy of pruning. GSFP has two advantages over previous works: (1) More accurate pruning guidance. For a pre-trained CNN model, the saliency of the filter varies with different input data. Therefore, accumulating the saliency of the filter over the entire data set can provide more accurate guidance for pruning. On the other hand, pruning from a global perspective is more accurate than local pruning. (2) More robust pruning strategy. We propose a reasonable normalization formula to prevent certain layers of filters in the network from being completely clipped due to excessive pruning rate. Experiment results show that GSFP is effective on many classic CNN architectures and different data sets. Within my knowledge, GSFP is the first algorithm to combine global and soft pruning strategies. Notably, on MNIST and CIFAR-10, it achieves a much higher compression ratio compared with prior work while maintaining the same test accuracy.

1. INTRODUCTION:

In recent years, convolutional neural networks (CNNs) have developed rapidly and achieved remarkable performance, which is widely applied in the field of computer vision, natural language processing, etc. However, CNNs with high computational and storage volumes, in fact, are difficult to deploy on embedded systems. As a result, model compression for CNNs have received hot attention.

Pruning-based approachs aim to remove the unnecessary connections of the neural network. parameter pruning which include non-structured pruning and structured pruning, reduces network complexity by directly reducing the scales of weight. Non-structured pruning (Han et al. (2015) ; Guo et al. (2016) ) randomly deletes redundant neurons reducing model scales greatly. However, it is necessary to store additional coordinate information, resulting in a weak advantage in reducing compuational effort. Structured pruning (Wen et al. (2016) ; Mao et al. (2017) ; Luo et al. (2017) ; Zhu & Gupta (2018) ; He et al. (2018) ) takes into account both the amount of computation and storage required. It performs regular model clipping based on filter, channel, and even a layer. Since the model pruned has good regularity, it can significantly reduce the scales of calculation and storage at the same time, which has recently received extensive attention. Essentially, the work of this paper is based on the idea of filter pruning.

In terms of Filter pruning, the current algorithm can be divided into "Soft" and "Hard" according to the training strategy, or "Local" and "Global" according to the pruning range. "Hard" is divided into two cases. One is that each iteration performs a pruning operation, so the pruning is too frequent to allow enough time recovering the model capacity. The second is to manually cut off the gradient update after pruning and then retraining (Luo et al. (2017) , Dong et al. (2017) , Li et al. (2017) ). "Soft" means that each epoch performs a pruning operation, and the clipped filter can be updated during the retraining process (He et al. (2018) ). "Local" means to set the pruning rate for each layer (He et al. (2017) , Mao et al. (2017) ), and "Global" means that only one overall pruning rate is set, and the pruning rate of each layer is obtained by dynamic adjustment (Lin et al. (2018) ). Essentially, the work of this paper is based on the pruning strategy of global and soft.

Most of the previous works on filter pruning, especially with the global pruning strategy, is poorly robust. That is, the distribution of pruning rate in each layer is extremely uneven. The filters of some layers are cut off a lot, while some layers are hardly cut. In extreme cases, when setting a too high pruning rate, the global pruning strategy will cause the filters of a certain layer removed completely, resulting in low accuracy. For example, analyzing the pruning results of VGG-16 in Li et al. (2017) did not trim the third to seventh convolutional layers, while the percentage of FLOPs in other convolutional layers was reduced by more than 50% on CIFAR-10, as shown in Table 3 . In Lin et al. (2018) the overall VGG-16 pruning rate is about 50% on ImageNet, but the CONV3 3 layer pruned FLOPs is as high as 98%. In extreme cases, the pruning results of VGG-16 on CIFAR-10 are shown in Table 1 , both the GHFP and the GSFP methods produce a low accuracy setting the pruning rate of 61%, which is the result of the removal of certain layers. Conversely, in Mao et al. (2017) , the imbalanced pruning results is prevented by setting the pruning rate of each layer. But this requires workers to have good prior knowledge and it takes a lot of time to perform repeated experiments.

In this work, we made efforts to improve the robustness of the global filter pruning algorithm. We proposed a cumulative saliency based Globally Soft Filter Pruning (GSFP) scheme, which globally prunes the unsalient filters and has better robustness. The contributions of this work are as follows:

• we experimentally compared the effects of four filter pruning strategies (global and local, hard and soft) on the results. And we found that combining global pruning with soft pruning strategy will have better results. • we proposed a reasonable normalized saliency formula to improve the robustness of global pruning method.

• By accumulating saliency score across all batches, we obtain more accurate pruning guidance, as we restore the model capacity during retraining process. The flowchar of proposed GSFP scheme is shown in Figure 1 . In particular, we first got the saliency score of each filter through the saliency formula and normalized the saliency score of each filter by dividing the average saliency score of each layer. Then, accumulate the scores through the rebust saliency formula during each training iteration. After obtaining the cumulative saliency score, we sort the scores across all layers, and generate masks based on the prune rate to multiply the corresponding filters to complete the pruning operation. Finally, we iteratively tune the sparse network and dynamically update the filter saliency in a top-down manner.

l l n n l l l l n l N N n n l l n n N G N W s G W n n i l l I c i s i L N L sort c c c

2. MOTIVATION:

In section 1, we explain that the current filter pruning algorithm can be divided into "Soft" and "Hard", or "Local" and "Global". In order to spy on the impact of different pruning strategies on the results, we compared the combined effects of the above four pruning strategies. As shown in Table 1 , We test in extreme sparsity levels (Pruned FLOPS 61% on VGG-16 in CIFAR-10). "L" refers to "Local", "G" refers to "Global", "H" refers to "Hard", "S" refers to "Soft", "F" refers to "Filter" and "P" refers to "Pruning". Therefore, "LHFP" means Locally Hard Filter Pruning method, and so on.

The GHFP 1 method and the GHFP 2 method have the difference in the saliency score formula, and GHFP 2 uses a normalized score formula to achieve better robustness. This also applies to GSFP 1 and GSFP 2 . GSFP 3 uses cumulative normalized saliency formula to achieve more accurate pruning guidance compared to GSFP 2 . Table 1 : Pruning results on VGG-16 in CIFAR-10 and comparisons to other approaches. Here, "Local" refers to setting the pruning rate for each layer, and "Global" refers to setting the global pruning rate from the entire model. "Hard" refers to pruning once per iteration or pruned without gradient update, "Soft" refers to pruning once per epoch and the pruned filters can be updated when training the model after pruning. "Normalized" formula reference formula 5(The GHFP 2 Normalized formula has slightly difference) and "Cumulative Normalized" formula reference formula 6. The comparison results are shown in Table 1 and Figure 2(a) . We have found:

• soft pruning strategy is better than hard, because the soft method has enough time to recover model capacity after pruning. This is clearly shown in Figure 2 (a). Correspondingly, soft method also takes longer training time than hard.

• global pruning has worse robustness. As shown in Table 1 , the accuracy drops rapidly under extreme sparsity on GHFP 1 and GSFP 1 compared with LHFP and LSFP. Because the filters of entire layer is comletely cut off. One of the reasons is that the size of the filter is not considered in the saliency formula. Though certain normalization methods, the robustness of global pruning can be improved. • when the algorithm has better robustness, the global pruning can reach higher accuracy than the local. This can be obtained by comparing GHFP 2 and LHFP, GSFP 2 and LSFP.

In addition, we also found that cumulative saliency has more accurate pruning guidance. By accumulating saliency score across batches, GSFP 3 has achieved the best performance.

3. METHOD:



3.1. PRELIMINARIES:

We will formally introduce the symbol and annotations in this section. Consider a set of training examples

D = {X = {X 1 , X 2 , • • • , X N } , Y = {Y 1 , Y 2 , • • • , Y N }}

, where X i and Y i represent an input and a target output, respectively. We denote filters of the entire network as

W = W 1 1 , W 2 1 , • • • , W N L L , where W n l ∈ R C l ×K×K , 1 ≤ l ≤ L, 1 ≤ n ≤ N l .

C l denotes the number of channels for the l-th convolution layer; L denotes the number of layers; N l denotes the number of filters for the l-th convolution layer. During the training, the gradient value of the parameter is defined as

G = G 1 1 , G 2 1 , • • • , G N L L

. The shapes of input tensor U and output tensor V are N l × H l × W l and C l+1 × H l+1 × W l+1 , respectively. The saliency score corresponding to each filter as

S = s 1 1 , s 2 1 , • • • , s N L L

. By sorting the calculated saliency scores, we can get the mask of each filter according to the global pruning rate P. Noted that mask

M = M 1 1 , M 2 1 , • • • , M N L L

has two forms corresponding to filter mask and channel mask in Figure 1 . The parameters after pruning are defined as

W * = W 1 * 1 , W 2 * 1 , • • • , W N L * L .

In the pruning process, we hope to replace the original network parameters with more sparse and regular parameters to achieve a trade-off between reducing the amount of calculation and not losing accuracy as

L (D|W * ) = L (D|W • M) ≈ L (D|W)

, where L (•) is a loss function for the pruned network. The pruning problem translates into the following optimization problem:

min L (D|W * ) s.t. W * = W • M M 0 ≤ P L l=1 N l (1)

where M 0 is the number of filter pruning form(all M internal elements are zero) in M. In response to these problems, the key is to find a suitable mask. Therefore, we need to develop a set of rules to evaluate the importance of each filter.

3.2. GLOBALLY SOFT FILTER PRUNING (GSFP):

Most of previous works (Li et al. (2017) ; Luo et al. (2017) ; Lin et al. (2018) ) pruned the filter every training iteration. This pruning strategy leads to a high rate of mis-pruning, and the model after pruning recovery time is short so that it will reduce the capacity. In addition, the constructed saliency formula either does not take into account global redundancy or ignores normalization of each layer. This work proposed a more reasonable saliency formula to make the pruning strategy more robust and improved the accuracy of pruning by accumulating saliency scores. The details of GSFP is illustratively explained in Figure 1 , which can be divided into the following three steps.

Rebust Saliency Formula. Some works such as Li et al. (2017) defined the saliency formula as the sum of the absolute values of kernel weights. He et al. (2018) used the l 2 -norm to evaluate the importance of each filter. Molchanov et al. (2017) and Lin et al. (2018) proposed that not only the weight but also the gradient value affecting the loss function, which is derived by Taylor expansion.

|∆L (W * )| = L (D|W * ) -L (D|W) ≈ ∂L (D|W * ) ∂W * • W * (2)

However when the global pruning strategy is adopted, the pruning result will be uneven if the weights between the different layers are very different. Obviously, the extremely uneven pruning result is an important cause of poor pruning robustness. We all know that filter sizes are often different in different convolution layers. However, the size of the filter is not considered in the formula which is one of the reasons for the uneven pruning results. So we have modified the l 2 -norm formula as follows:

EQUATION

In order to highlight the saliency of the filter for different input data, we use the gradient value in back propagation as the multiplier factor. The specific definition is as follows:

G n l 1 = 1 C l • K • K • Ci c=1 K k1=1 K k2 |g n l (c, k 1 , k 2 )| (4)

Different convolutional layers have different parameter distributions, and they also accompany vanishing or exploding gradient problem during training. Taking into account these two phenomena, we normalize the gradient factor and the weighting factor separately, and finally construct a complete saliency formula as follows:

s n l = N l G n l 1 N l n=1 G n l 1 • N l W n l 2 N l n=1 W n l 2 (5)

Cumulative Saliency. In order to improve the accuracy of pruning, we accumulate the saliency scores of the filters in each batch during the training process. During the training stage, the saliency score is calculated while restoring the capacity of the model. The saliency score continues to accumulate in each iteration until the entire data set is traversed and then prune based on the saliency score, as shown in Figure 1 . We define the cumulative significance score as

C = c 1 1 , c 2 1 , • • • , c N L L .

The process of saliency accumulation is described as:

EQUATION

where I represents the number of iterations within a training epoch. This takes into account the saliency of the filter throughout the data set and provides more accurate guidance for subsequent pruning processing.

Filter Pruning. In the filter pruning step, we use a global pruning strategy to fully consider the redundancy of each layer. We only need to set a global pruning rate P. The global saliency scores of all filters Indx is constructed, which are sorted by a descending order, i.e., Indx = sort( c

1 1 , c 2 1 , • • • , c N L L

). Then set the top-P L l=1 N l Indx corresponding filters to zero. Note that if we set the value of selected N l P l filters to zero, the channel corresponding to the next layer of filters will be set zero simultaneously, i.e., W n l+1 ∈ R N l ×(1-P l )×K×K , 1 ≤ n ≤ N l+1 .

4. EXPERIMENTS:

To evaluate the effectiveness of the GSFP scheme, CNN model pruning were performed on three databases MNIST, CIFAR-10, and ImageNet. We implemented our filter pruning method in the PyTorch framework. To get the baseline accuracies for each network, we train each model from scratch on MNIST and CIFAR-10. For the ImageNet dataset, pre-trained model from torchvision were used as the baseline model. To further analyze the significance of the saliency score for pruning, we visualized the saliency scores for the four pruned models with different pruning rates in Figure 3 . In the figure, the 17th and 29th filters are critical filter. We can see that as the pruning rate increases, the saliency of the critical filter becomes more and more prominent during the process of model retraining. When we set the pruning rate to 70%, 80%, 85% and 90%, respectively, the corresponding convolution kernel of CONV1 layer remained is 7, 6 , 5 , 3 in Table 2 . In Figure 3 , we can clearly gauge the importance of each filter across the network. When the pruning rate is set to 90%, CONV1 can only retain 3 filters after global measurement, but as can be seen from the figure 3 , there are 5 filters that are obviously saliency, we need drop two important filters, so the classification accuracy drop about 0.32%. Therefore, visual saliency scores can help us find the most appropriate pruning rate. 

4.2. RESULT ON CIFAR-10:

For CIFAR-10 dataset, we have tested our GSFP on single-branch networks(VGG-16), multi-branch networks and compact networks(MobileNet and MobileNet-V2). We use several difference pruning rates to select the best trade-off between model computation and classification accuracy. Table 4 summaries the pruning results and Figure 2 (b) shows the training curve for different networks.

For example, in the case where the same classification without loss of accuracy for single-branch networks VGG-16 (Simonyan & Zisserman (2014) ), (Li et al. (2017) ) pruning method reduced FLOPs by 34.2%, while GSFP reduced FLOPs by 61.46%. The proposed scheme have achieved 2× better results. Note that VGG-16 on CIFAR-10 consists of 13 convolutional layers adding Batch Normalization and just 1 fully connected layer, in which the fully connected layers do not occupy large portions of parameters due to the small input size and less hidden units. For single-branch networks we adopted a strategy of combining filter pruning with channel pruning. Because the previous layer of filter pruning will affect the next layer of convolution filter channels. Detailed pruning results are shown in Table 3 . Compared with the pruning results listed in Li et al. (2017) , the pruning results of the GSFP are more balanced in each layer, which is why the GSFP can pruning more, similar to the buckets effect.

For multi-branch networks such as ResNet-32 and ResNet-110 models (He et al. (2016) ) on CIFAR-10, we need to consider the shortcut connection structure in the process of pruning. Since there is a cross-layer connection for each residual block of ResNet, we only perform filter pruning for the first layer convolution and the shortcut convolution layer of each block, and other layer filter pruning combined with channel pruning. The proposed method reduced FLOPs by 50% on ResNet-32 without losing the accuracy of the pre-trained model. With the same recognition accuracy, the proposed sheme achieved 7% more pruning rates compared to He et al. (2018) . For the ResNet-110, our scheme has a advantage of 12% on pruning rates compared with the best result of the state-of-the-art.

Moreover, we have also pruned two compact networks MobileNet (Howard et al. ( 2017)) and Mo-bileNetV2 (Sandler et al. (2018) ). Compact network usually uses group convolution unit to reduce the amount of calculations and parameters, so we only conducted filter pruning for these networks.

After pruning, it is found that MobileNet and MobileNetV2 have a lot of redundancy on the CIFAR-10 dataset, the amount of calculation pruning can reach 72.57% and 69.68%, respectively. We also test our pruning scheme on the large scale ImageNet classification task. ResNet is currently the most widely used network structure, so in the experiment we chose ResNet-18 and ResNet-32 for pruning. During the pruning process, since the ResNet cross-layer contains a lot of information, we do not pruned the shortcut convolution layer. Table 5 shows the pruning results of ResNet on Imagenet. For ResNet-18, GSFP reduced the FLOPs by 47.06% , with a Top-1 accuracy drop 2.95%.

For ResNet-34, under almost the same loss of precision, the GSFP was reduced by 3.75% compared to the He et al. (2018) . In addition, when the pruning was further increased, the calculation amount was reduced by 9.6%, while the Top1 precision loss was only decreased by 0.5%. Compared to Dong et al. (2017) , Li et al. (2017) and He et al. (2018) , our proposed GSFP has a larger pruning space. By analyzing the pruning results of convolution layers at different time points in Figure 4 , we can see that the strategy of globally soft pruning can automatically adjust the pruning result and improve the robustness of the pruning process. 

5. CONCLUSIONS:

In this work, we propose a scheme GSFP on filter pruning for convolutional neural networks, which combines global pruning strategy and soft pruning strategy. First, we designed the experiment and compared the previous work with four different pruning strategies (global and local, hard and soft).

In addition, it is proved by experiments that the global pruning strategy will reduce the pruning robustness and can be improved by a reasonable normalization formula. Finally, the pruning accuracy can be effectively improved by accumulating the saliency of the filter across all batches in the model recovery training process after pruning.

:



Figures and Tables:

Figure 1: An illustration of GSFP. Each circle with a color represents the saliency score of a filter, and the rectangle represents the generated filter mask. The schematic diagram is divided into four parts: the top represents the training process and the bottom represents the pruning process. The middle blue dashed box mainly illustrates the use of normalized saliency formula to eliminate saliency scores uneven between different layers. The red dashed box shows the process of saliency accumulation and generation filter masks by global ordering.

(a) Different method pruning curves on VGG-16.(b) Different networks pruning curves.

Figure 2: Different method and networks pruning training curves on CIFAR-10.

Figure 3: The saliency scores visualization for LeNet-5 on 10 epoch. The top figure shows saliency scores for CONV1, bottom figure shows the saliency scores for CONV2. The pruning rate are 70%, 80%, 85% and 90%, respectively.

Figure 4: Different layers of pruning distribution curves on ImageNet

The same data argumentation strategies were used with PyTorch official examples(Paszke et al. (2017)). For retraining of filter pruning, we use a constant learning rate 0.01 and retrained 100 epochs on MNIST. In CIFAR-10, we set the initial learning rate to 0.01, multiply by 0.1 per 50 epoch, and retrained 150 epochs. Finally, we retrained 100 epochs on ImageNet. The first 70 epochs learning rates are set to 0.001, and the last 30 epochs settings are 0.0001.

Table content: <table><tr><td>4.1 LENET ON MNIST MNIST is a well-known database of handwritten digits which contains about 60, 000 images for training and 10, 000 for testing. We perform experiments on a version of LeNet proposed in (LeCun Filters Channel FLOPs Error Baseline 20 -50 1 -20 100% -100% 0.75% Wen et al. (2016) 5 -19 1 -4 25% -7.6% 0.80% Wen et al. (2016) 3 -12 1 -3 15% -3.6% 1.00% GSFP (70%) 7 -14 1 -7 35% -9.8% 0.72% GSFP (80%) 6 -8 1 -6 30% -4.8% 0.77% GSFP (85%) 5 -6 1 -5 25% -3.0% 0.83% et al. Method GSFP (90%) 3 -4 1 -3 15% -1.2% 1.07%</td></tr></table>

Results after pruning unimportant filters and channels in LeNet-5

Table content: <table><tr><td>70%</td></tr><tr><td>80%</td></tr><tr><td>85%</td></tr><tr><td>90%</td></tr><tr><td>70%</td></tr><tr><td>80%</td></tr><tr><td>85%</td></tr><tr><td>90%</td></tr></table>

pruning model detail comparison on CIFAR-10.

Table content: <table/>

Comparison of pruning result on CIFAR-10. The MFLOPs denotes million floating-point operations. The values in parentheses represent the global filter pruning ratio.

Table content: <table/>

Comparison of pruning results on ImageNet. The GFLOPs is the giga floating-point operations. The Acc Drop is the accuracy of the pruned model minus that of the baseline model, so a smaller number of Acc Drop is better.

Table content: <table/>

Peer review (iclr2019):
 This paper proposes new heuristics to prune and compactify neural networks. The heuristics try to consider 1) filter weight and gradient normalization by their size, 2) saliency normalization across layers, 3) saliency accumulation across batch. The author claims that these can address problems previous studies had and experimental results show that the proposed method achieve higher compression ration with less loss of accuracy. This paper discusses how to determine the importance of filters. As cited in the paper, there have been various attempts to tackle the same problem and the paper contributes to the series of efforts. The paper introduces a new way to compute such importance values based on their observations. The method is tested on a few dataset and a various models and compared with some previous studies. I like the simple but yet effective method, however, I think it is not good enough for ICLR. 1. What is effective is not very clear. The paper pointed out issues of previous studies and proposed the new method based on the observations. However, only the final method is compared with other work and it did not examine which part of the method was essential. The paper needs more detailed analyses on the proposed method. For example, the readers would want to know if the normalization in Eq. (2) is really important or not. The readers would be also interested in a visualization like Fig. 2 without saliency normalization. 2. The numbers of previous studies come only from their papers. It is very difficult to know if the proposed method is actually better than the previous methods if the numbers just come from their papers. We want to compare the ideas, but not numbers. The essential ideas of other papers need to be abstracted and tested in the paper by itself. It relates to the first item above. "Baseline" should be a baseline method but not models without pruning. Numbers from other papers are still useful to show that the numbers in the paper are good in an absolute manner. 3. Weak theoretical reasoning Eq. (1) in the paper is not actually used for optimization while some previous methods do. If the proposed method is better than other methods which directly optimizes the loss, should we think that the formulation itself is bad? The paper discusses imbalanced pruned pruning results. It needs to show that it is actually bad. * minor things ** Table 1: Should the first row of "Wen et al. (2016)" have "5-19" and "1-5" or "4-19" and "1-4" for "Filters" and "Channels", respectively? ** I*d recommend another proofreading.