Learning compact ConvNets through filter pruning based on the saliency of a feature map

IET Image ProcessingVolume 16, Issue 1 p. 123-133 ORIGINAL RESEARCH PAPEROpen Access Learning compact ConvNets through filter pruning based on the saliency of a feature map Zhoufeng Liu, Corresponding Author Zhoufeng Liu lzhoufeng@hotmail.com orcid.org/0000-0003-0006-9959 School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, China Correspondence Zhoufeng Liu, Room 308, Basic Laboratory Building, Zhongyuan University of Technology, 41 Zhongyuan Road, Zhengzhou City, Henan Province, China. Email: lzhoufeng@hotmail.comSearch for more papers by this authorXiaohui Liu, Xiaohui Liu School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this authorChunlei Li, Chunlei Li School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this authorShumin Ding, Shumin Ding School of Energy & Environment, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this authorLiang Liao, Liang Liao School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this author Zhoufeng Liu, Corresponding Author Zhoufeng Liu lzhoufeng@hotmail.com orcid.org/0000-0003-0006-9959 School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, China Correspondence Zhoufeng Liu, Room 308, Basic Laboratory Building, Zhongyuan University of Technology, 41 Zhongyuan Road, Zhengzhou City, Henan Province, China. Email: lzhoufeng@hotmail.comSearch for more papers by this authorXiaohui Liu, Xiaohui Liu School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this authorChunlei Li, Chunlei Li School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this authorShumin Ding, Shumin Ding School of Energy & Environment, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this authorLiang Liao, Liang Liao School of Electrical and Information Engineering, Zhongyuan University of Technology, Zhengzhou, ChinaSearch for more papers by this author First published: 12 September 2021 https://doi.org/10.1049/ipr2.12338AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract With the performance increase of convolutional neural network (CNN), the disadvantages of CNN's high storage and high power consumption are followed. Among the methods mentioned in various literature, filter pruning is a crucial method for constructing lightweight networks. However, the current filter pruning method is still challenged by complicated processes and training inefficiency. This paper proposes an effective filter pruning method, which uses the saliency of the feature map (SFM), i.e. information entropy, as a theoretical guide for whether the filter is essential. The pruning principle use here is that the filter with a weak saliency feature map in the early stage will not significantly improve the final accuracy. Thus, one can efficiently prune the non-salient feature map with a smaller information entropy and the corresponding filter. Besides, an over-parameterized convolution method is employed to improve the pruned model's accuracy without increasing parameter at inference time. Experimental results show that without introducing any additional constraints, the effectiveness of this method in FLOPs and parameters reduction with similar accuracy has advanced the state-of-the-art. For example, on CIFAR-10, the pruned VGG-16 achieves only a small loss of 0.39% in Top-1 accuracy with a factor of 83.3% parameters, and 66.7% FLOPs reductions. On ImageNet-100, the pruned ResNet-50 achieves only a small accuracy degradation of 0.76% in Top-1 accuracy with a factor of 61.19% parameters, and 62.98% FLOPs reductions. 1 INTRODUCTION Convolutional neural networks (CNN) have shown significant advantages in the field of computer vision, such as classification [1-6], detection [7-9], and segmentation [10, 11]. In the past ten years, to pursue better performance, the general trend in developing convolutional neural networks is to increase the network's depth gradually. For example, the early neural network LeNet [1] architecture has only five layers. Later, the VGG [3] and GoogleNet [5] architecture has about 20 layers. In particular, the recent ResNet [4] and DenseNet [12] have increased to astonishing hundreds or even thousands of layers. As the number of layers of convolutional neural networks continues to increase, many networks have over-parameterization severe problems, which lead to over-determined (or over-complete) systems. More importantly, the high storage and high consumption of deep convolutional neural networks make it difficult to deploy them to edge computing devices such as automated robots, drones, smartphones, etc. One can have many methods to solve the above problem. Although good progress has been made in new hardware or special hardware acceleration, the convolutional neural network model should be further reduced in model parameters and FLOPs to generate compact and efficient models with reasonable accuracy compromises. Convolutional neural network model compression mainly resorts to parameter quantization [13], pruning [14-16], low-rank decomposition [17] and knowledge distillation [18], etc. Among them, the pruning method has become a mainstream branch of network compression and acceleration. On the one hand, the pruning process in model compression has many similarities with the synaptic pruning during mammalian brain development. Some rarely used synapses are gradually pruned to form a relatively stable adult brain [19]. On the other hand, learning a complex function from a small initial architecture may result in low accuracy. Therefore, it is necessary to first learn a task with a larger architecture and many parameters and then dynamically evolve the baseline network into an ideal sub-network by removing redundant and less important connections. So far, network pruning mainly includes two categories: unstructured pruning and structured pruning. Unstructured pruning [20-22] prunes the neural network's connections, resulting in a sparse matrix. Although it can produce a considerable reduction in network parameters, it cannot gain significant acceleration due to the need for specially customized software or hardware platforms, especially for modern networks with dense connection layers, such as ResNet [4] and DenseNet [12]. To avoid the limitations of the above-mentioned unstructured pruning, we believe that filter pruning is a better pruning method with many advantages: (1) The pruned model has no difference in network structure and can be run on a general hardware platform or BLAS library. (2) Memory usage will be significantly reduced. This memory reduction originates from the model itself and the activation in the middle, which is rarely mentioned in the previous literature. (3) The pruned Network can be further compressed and accelerated by other compression methods, such as parameter quantization [13], low-rank decomposition [17]. (4) The pruned model can significantly speed up other vision tasks, such as object detection or semantic segmentation. However, the existing filter pruning scheme still has some problems. Specifically, we pursue a higher compression/speedup ratio, but it will be restricted by machine time and human labour, especially for the pruning method of modifying the loss function during training. For example, Liu et al. [23] and Zhao et al. [24] imposed sparse constraints on the batch normalization layer's scale factor and identified channels with lower scale factors as unimportant. Huang et al. [25] and Lin et al. [26] introduced a mask to learn sparse structure pruning in which the filter corresponding to the scaling factor of zero is removed. As the loss has changed, the required retraining steps are burdensome in machine time and human labour and usually require another round of hyper-parameter tuning. These problems can be attributed to the lack of practical/theoretical guidance on the importance and redundancy of filters. This paper proposes an effective filter pruning method, which uses the feature map's saliency, measured by the one-dimensional entropy, as a theoretical guide for whether the filter is essential. It does not impose additional auxiliary constraints or retrain the model, thus simplifying the model pruning's complexity. Moreover, we observe that the earlier the more significant the feature map's information entropy, the more significant the contribution of the corresponding filter to the accuracy. Thus we can prune the feature map with a smaller information entropy to compress the model. However, filter pruning usually decreases the model performance. In this paper, an over-parameterized convolution method is adopted to improve the pruned model's accuracy without increasing the inference stage parameters. Notably, an over-parameterized convolutional layer is composed of a depthwise convolutional layer and a traditional convolutional layer. Furthermore, it can be folded into a compact single layer representation after the training phase. In other words, the over-parameterized convolutional network introduces performance gains without incurring any computational complexity increase at the inference stage. The main contributions of our method are as follows. The proposed pruning method in this paper does not impose additional auxiliary constraints or retrain the model, thus simplifying the complexity of model pruning. This paper proposes an effective and efficient pruning method. We accurately estimate the importance of the CNN feature map based on information entropy, though only a small part of the input image is delivered to the CNN model. It shows that we can efficiently prune the feature map and corresponding filter. To improve the pruned network's performance, the pruned network employs over-parameterized convolution to bring performance gains without incurring any computational complexity increase for the inference stage. The rest of this article is organized as follows. Related work is discussed in Section 2. We will introduce the pruning method proposed in this paper in detail in Section 3. Section 4 introduces experiments to compare with other experiments. Finally, we conclude this article in Section 5. 2 RELATED WORK In the past ten years, deep neural networks have developed rapidly. However, many researchers have found that deep models have over-parameterization severe problems. Denil et al. [27] prove that only a small subset of the original parameters can effectively reconstruct the network. Therefore, model compression and acceleration have become a hot topic in the deep learning community. In the paper, the following introduces several mainstream model compression and acceleration methods. 2.1 Lightweight network design With the need for deploying neural networks on edge devices, there has been a rising interest in building compact models in the recent literature, e.g. [28-35]. MobileNet [28, 29], and Xception [30] replace standard convolution with depthwise separable convolution, resulting in a small-scale network model due to the inefficiency of the vanilla convolution. SqueezeNet [31] substitutes 3 × 3 convolutional filters with 1 × 1 filters, making the number of input channels to 3 × 3 filters reduction and downsampling later in the network. ShuffleNet [32, 33] follows the idea of reducing the dense connection of channels and proposes the use of group convolution to reduce the computational complexity and realizes the exchange of information between groups with the help of channel shuffle. IGCV [34, 35] shares the same idea with ShuffleNet, except that it uses two-stage interleaved group convolution for information exchange. Although these lightweight networks have achieved good detection performance, this manual design cost is high and becomes difficult as the network becomes complex. Besides, there are also some recent works [36-38] on automatically designing network architectures. The search space of these methods is extremely large, thus one needs to train hundreds of models to distinguish good from bad ones. 2.2 Low-rank decomposition The literature [27] shows that neural networks tend to over-parameterize. That is, the parameters of each layer can be accurately restored from a small subset. Inspired by this, low-rank decomposition has become a standard method of network compression. Generally speaking, it presents the weight matrix with the product of two smaller matrices to approximate the convolution operation. Unlike pruning, its goal is to reduce the network's computational cost, not change the original number of filters. To this end, Jaderberg et al. [39] decomposed the weight matrix's spatial dimension into the product of two filters of rank 1. For a CNN trained on a text character recognition data set, they achieved a 4.5x speedup and a 1% decrease in accuracy. Denton [17] uses singular value decomposition (SVD) and other techniques to approximate a large matrix with the product of two low-rank matrices, and the original model loses less than 1%. Also, Zhang et al. [40] considered subsequent nonlinear units when learning low-rank decomposition to provide significant speedup. Although low-rank decomposition is beneficial to the compression and acceleration of CNNs, it usually causes a considerable loss of accuracy in the case of high compression ratios. 2.3 Pruning Based on the assumption that there is a lot of parameter redundancy in deep neural networks, removing unimportant or unnecessary parameters is the most intuitive idea through pruning. In other words, the pruning method can make the parameters in the model have a high degree of sparsity. Early network compression work mainly focused on the compression of the fully connected layers [15, 20, 41]. LeCun et al. [41] and Hassibi et al. [20] proposed a measurement method that prunes the network parameters with low significance values by calculating the loss function's Hessian matrix relative to the parameters. Based on the redundancy study between neurons, Srinivas and Babu [42] pruned the model without retraining. Han et al. [15] propose to remove the connections below the set threshold to reduce the number of parameters in CNNs. However, the method mentioned above will only result in a memory footprint reduction and will not necessarily reduce the computation time, which is mainly dominated by the convolutional layer. Those methods usually lead to the unstructured sparseness of CNNs, which will affect the flexibility of applications across different hardware platforms or libraries. To reduce the convolutional layer's computational cost, one can adopt a standard solution to remove redundant filters or feature maps, and it can be divided into two categories: filter pruning based on property importance and adaptive importance approach. Filter pruning based on property importance aims to use the intrinsic properties of CNNs as the basis for judging whether the filter is essential. Reference [16] proposed a magnitude-based pruning method that prunes the corresponding feature map by measuring the filter's norm. BO Ayinde et al. [43] proposed a pruning method based on the relative cosine distances in the feature space. However, this magnitude-based method is inefficient for judging the importance of the filter. Hu et al. [44] proposed a new pruning criterion, that is, if the output of some neurons is mostly zero, these activations should be redundant. Therefore, they calculated the average percentage of zero (APoZ) of each filter to measure the filter's importance. Chen Z et al. [45] designed a criterion directly related to the last accuracy of a network to evaluate each channel's importance, and then filters with low importance scores are removed according to the pruning ratio. He et al. [46] trimmed the filter closest to the median from the perspective of which filters are most replaceable. All the criteria mentioned above are ad hoc, which gives them the advantage of low time complexity but has limitations on the acceleration and compression ratio. The adaptive importance approach is to associate pruning with network training loss and retrain the network to generate pruning strategies. The literature [23] first applies regularization to the loss function and then prunes the channels corresponding to the scale factor close to zero in the batch normalization layer. Huang et al. [25] introduced a mask to learn a sparse structure pruning in which the filter corresponding to the scaling factor of zero is removed. Although these methods based on adaptive importance usually produce higher compression and speedup ratios, the required retraining steps need extra machine time and human labour. Luo et al. [47] regard pruning as an optimization problem. Whether to remove unimportant channels and corresponding filters in the current layer is based on the next layer's reconstruction error. However, the small local reconstruction errors may result in larger global output errors through the deep network propagation. The literature [48] proposed a TE-based criterion, iteratively pruning a filter and then fine-tuning the pruned network. However, this is too costly for deep networks. Overall, filter pruning is still an open issue so far. 3 PROPOSED METHOD In this section, we first describe the notations and preliminaries. Next, we present the framework of the SFM filter pruning. Then, we detailed the pruning process of SFM. Finally, we discuss how to improve the pruned model's accuracy without increasing parameters at inference time. 3.1 Preliminaries and notations In our method, Ti, j, k is employed to represent the ( i , j , k ) elements of the three-dimensional tensor T i , j , k R I × J × K , and its number and elements will keep unchanged when it is reshaped, e.g. a 4 D tensor T ∈ R I × J × K × L can be reshaped to a 3 D tensor T ∈ R I × ( J × K ) × L , or T ∈ R I × Q × L , where Q = J × K . For simplicity, a tensor and its reshaped version(s) share the same symbol in this paper. Suppose a CNN model has L convolutional layers, and these layers are interlaced with pooling, ReLU, and batch normalization. The i-th convolutional layer can be expressed as a 3-D filter W i = { w 1 i , w 2 i , … , w n i i } ∈ R n i × ( k i × k i ) × n i − 1 , where w j i , n i , and k i represent the j-th filter, the number of filters in the i-th layer, and the convolution kernel's size. This layer's feature map output is denoted as O i = { o 1 i , o 2 i , … , o n i i } ∈ R n i × h i × w i , where the j-th feature map o j i is generated by w j i . h i and w i the height and width of the current layer's feature map, respectively. For filter pruning W i is divided into two groups: the reserved filters and the pruned filters, which are considered unimportant. It should be pointed out that n i 1 and n i 2 are the number of essential filters and unimportant filters, respectively. More importantly, we have such constraints: W i 1 ∩ W i 2 = ∅ , W i 1 ∪ W i 2 = W i and n i 1 + n i 2 = n i . 3.2 The SFM framework The filter level pruning method realizes the feature map channel sparse in deep CNN to minimize accuracy degradation. It is imperative to preserve the feature discriminant capability and remove those filters with a minimum contribution to the feature discriminant capability. Differing from the previous filter pruning, our SFM selects unimportant filters to be pruned more efficiently. Figure 1 shows the overall pipeline of SFM. Its procedure consists of four basic operations: (1) calculate the saliency quantity of each feature map; (2) prune unimportant feature maps and corresponding filters; (3) improve the model performance with a cheap operation, and (4) fine-tune the whole network. More precisely, given a pre-trained CNN model, we can efficiently prune the non-salient feature map based on the fact filters with a weak saliency feature map in the early stage will not significantly improve the final accuracy. However, if we adopt a pruning strategy, the pruned model's accuracy will be decreased. Hence, to improve the network's performance, we use a cheap operation, called 'the over-parameterized convolution,' to replace the traditional convolution in the pruned network without increasing the parameters. Finally, the operation of fine-tuning is adopted to restore the model performance further. FIGURE 1Open in figure viewerPowerPoint The overall SFM pipelineNote: Given a pre-trained CNN model, it is pruned by using our proposed filter selection method, followed by cheap operation, i.e. the over-parameterized convolution, to improve the pruned model's accuracy. The four operations mentioned above are applied in an alternating manner. Furthermore, it is worth noting that the pruned model's performance is mainly dominated by pruning and cheap operation. Hence, in the following, we describe pruning's details based on feature maps' saliency and cheap operation. 3.3 Part 1: Pruning We first focus on the pruning part, whose critical is to evaluate each neuron's importance and remove those unimportant ones. 3.3.1 Overview of SFM pruning SFM prunes the least important filters from a pre-trained model to make the computation and memory reduction. The pruning process is illustrated in Figure 2, which is based on the feature map's saliency. Starting from a pre-trained model, we prune it with a predefined compression rate. We summarize our pruning framework as follows: Filter selection. The proposed method is based on the premise that the feature map's smaller saliency is less critical. Unlike existing methods restricted by training inefficiency, it requires only a small part of the input image to estimate the CNN feature map's importance accurately. Hence we can safely prune the corresponding filter with high efficiency. Pruning. The unimportant feature maps and their corresponding filters are pruned. Furthermore, the filters' channels in the next layer are also removed, resulting in a much smaller model. Iteration. Go to Step 1 to prune the next layer. FIGURE 2Open in figure viewerPowerPoint Illustration of the pruning partNote: These feature map and the associated filters (highlighted in orange in the first line) have little contribution to the overall performance, thus can be discarded, leading to a compact model. 3.3.2 Filter selection To determine which channel can be safely removed, we first focus on the dotted line in Figure 2 to introduce our data-driven feature map selection method, which determines the feature maps (and their associated filters) that are to be pruned away. Specifically, filter pruning can be formulated as an optimization problem: min I ( w j i ) ∑ i = 1 L ∑ j = 1 n i I w j i M w j i s . t . ∑ j n i I w j i = n i 2 (1) where I ( w j i ) is an indicator function which is equal to 1 if w j i is grouped to W i 2 , or 0 if w j i is grouped to W i 1 . The index M ( • ) measures the importance of an arbitrary filter in each convolutional layer. Thus minimizing Equation (1) is equivalent to removing the n i 2 least essential filters. The methods mentioned in the existing literature directly customize M ( • ) on the filters. However, the specially designed M ( • ) on filters ignore the distribution of input images and output labels, which is, we contend, why the method based on property importance shows sub-optimal performance. In contrast, the feature map can reflect both the filter properties and how the input images are converted to predict the final labels. More importantly, the literature [49] shows that even a single feature map in the same layer plays a different role in the network. Therefore, in this paper, we propose to define the evaluation function M ( • ) on the feature map. To this end, Equation (1) is reformulated as: min I w j i ∑ i = 1 L ∑ j = 1 n i I w j i E D M ̂ o j i D , : , : s . t . ∑ j = 1 n i I w j i = n i 2 (2)where D is the input image and M ̂ ( • ) is employed to estimate the information of feature maps o j i ( D , : , : ) generated by the filter w j i . In this paper, the expectation of M ̂ ( • ) is positively related to M ( • ) in Equation (1). If the feature map contains more information than a normal map, its corresponding filter is also more critical. Therefore, the problem falls in designing an appropriate function, reflecting the feature map's information richness. Prior work [44] utilizes feature maps' sparsity to remove the filters, i.e. feature maps with a large portion of zeros are pruned. However, this method is intractable since complicated processes and training inefficiency heavily limit it. Instead, we prune the feature map by exploiting their saliency, i.e. information entropy, an effective and efficient method to measure the information richness of the feature map due to the strong correlation between the early saliency of the feature map and the final settled accuracy. Our information measurement can be defined as: M ̂ o j i D , : , : = Entropy o j i D , : , : (3)where Entropy ( • ) is the entropy of a feature map for input image D. The proposed entropy solution is summarized in Algorithm 1. Either intuitively or theoretically, a feature map with greater entropy contains more information. Thus, the entropy can serve as a reliable measurement for information richness. ALGORITHM 1. Calculate the entropy of the i-th layer feature map Input: O i ( D , : , : ) , c = [ ] Output: c 1: for o j i ( D , : , : ) in O i ( D , : , : ) do: 2: get: one-dimensional list of a single feature map L 3: Scan for: discard duplicate values in L and calculate each value's probability to form a new list M. 4: for n in M do: 5: H ( o j i ( D , : , : ) ) = − ∑ n p n log 2 p n 6: copy: H ( o j i ( D , : , : ) ) into c 7: end for 8: end for The core of the pruning proposed in this paper is the calculation of feature map entropy. However, the entropy of the feature map may be sensitive to the different input images. In other words, the feature map's entropy o j i ( 1 , : , : ) is not necessarily equal to the entropy of the feature map o j i ( 4 , : , : ) . Fortunately, inspired by the literature [50], we can employ a small part of the input image to accurately estimate the importance of the CNN feature map by entropy expectation with high efficiency. Then, the entropy expectation of the feature map can be reformulated as: E D M ̂ o j i D , : , : ≈ ∑ b = 1 s Entropy o j i b , : , : (4) where b and s are the input images and the number of input images, respectively, and s is set to 500. Combining Equation (2) and Equation (5), we can derive the final objective function of filter selection as: min I w j i ∑ i = 1 L ∑ j = 1 n i I w j i ∑ b = 1 s Entropy o j i b , : , : s . t . ∑ j = 1 n i I w j i = n i 2 (5) It can be seen from Equation (5) that n i 2 unimportant feature maps and corresponding filters are pruned through the entropy of the feature map. 3.4 Part 2: Cheap operations Pruning may decrease the performance of the CNN model. Hence, it is necessary to improve the pruned model's accuracy without increasing parameters at inference time. The cheap operation stage can be conducted via any boosting the network's expressiveness technique. It is generally believed that the increase of network depth will improve network expression ability. Inspired by the literature [51], we employ an over-parameterized convolutional layer, i.e. a conventional layer with additional depthwise convolution. It can accelerate the training of various CNNs and consistently boost the performance of the pruned models. An over-parameterized convolutional layer's most notable benefit is that it can be folded into a compact single layer representation after the training phase. In other words, the network parameters via cheap operation are precisely equivalent to the original pruned model in the inference stage. To clearly describe the over-parameterized convolution process, one can introduce the conventional convolution and depthwise convolution, defined as follows. 3.4.1 Conventional convolution The input feature map is processed by a convolutional layer in a sliding window fashion, applying a set of convolution kernels to a corresponding sized patch at each window position. For convenience, assume that a patch of the i-th layer is the 2 D tensor P i ∈ R ( k i × k i ) × n i − 1 , where k i is the ker