CHIP: CHannel Independence-based Pruning for Compact Neural Networks

1 Introduction . Convolutional neural networks ( CNNs ) have obtained widespread adoptions in numerous important AI applications [ 17 , 48 , 47 , 12 , 11 , 44 , 35 ] . However , CNNs are inherently computation intensive and storage intensive , thereby posing severe challenges for their efficient deployment on resourceconstrained embedded platforms . To address these challenges , model compression is widely used to accelerate and compress CNN models on edge devices . To date , various types of compression strategies , such as network pruning [ 15 , 16 , 37 , 60 , 28 , 14 , 1 , 49 , 10 , 63 , 36 , 18 , 13 , 3 , 2 , 25 , 53 , 50 , 9 , 39 , 33 ] , quantization [ 15 , 55 , 43 , 8 ] , low-rank approximation [ 56 , 40 , 58 , 57 ] , knowledge distillation [ 22 , 41 ] and structured matrix-based construction [ 45 , 29 , 6 ] , have been proposed and explored . Among them , network pruning is the most popular and extensively studied model compression technique in both academia and industry . Based on their differences in pruning granularity , pruning approaches can be roughly categorized to weight pruning [ 16 , 15 ] and filter pruning [ 54 , 27 , 21 , 38 , 34 ] . Weight pruning focuses on the proper selection of the to-be-pruned weights within the filters . Although enabling a high compression ratio , this strategy meanwhile causes unstructured sparsity patterns , which are not well supported by the general-purpose hardware in practice . On the other hand , filter pruning emphasizes the removal of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the entire selected filters . The resulting structured sparsity patterns can be then properly leveraged by the off-the-shelf CPUs/GPUs to achieve acceleration in the real-world scenario . Existing Filter Pruning Methods . Motivated by the potential practical speedup offered by filter pruning , to date numerous research efforts have been conducted to study how to determine the important filters – the key component of efficient filter pruning . A well-known strategy is to utilize the norms of different filters to evaluate their importance . Such a `` smaller-norm-less-important '' hypothesis is adopted in several pioneering filter pruning works [ 27 , 19 ] . Later , considering the limitations of norm-based criterion in real scenarios , [ 20 ] proposes to utilize geometric medianbased criterion . More recently , first determining those important feature maps and then preserving the corresponding filters , instead of directly selecting the filters , become a popular strategy for filter pruning . As indicated in [ 31 ] , the features , by their natures , reflect and capture rich and important information and characteristics of both input data and filters , and hence measuring the importance of features can provide a better guideline to determine the important filters . Built on this pruning philosophy , several feature-guided filter pruning approaches [ 31 , 51 ] have been proposed and developed , and the evaluation results show their superior performance over the state-of-the-art filter-guided counterparts with respect to both task performance ( e.g. , accuracy ) and compression performance ( e.g. , model size and floating-point operations ( FLOPs ) reductions ) . Determining Importance : Intra-channel & Inter-channel Perspectives . These recent advancements on filter pruning indeed show the huge benefits of leveraging feature information to determine the importance of filters . To date , some feature-guided approaches measure the importance from the intra-channel perspective . In other words , no matter which importance metric is used , the importance of one feature map ( and its corresponding filter ) , is measured only upon the information of this feature map in its own channel . On the other aspect , the inter-channel perspective , which essentially determines the filter importance via using cross-channel information [ 20 , 42 , 46 , 52 , 26 ] , is still being further explored . To be specific , [ 20 ] and [ 42 ] adopt cross-channel geometric median and Hessian , respectively , to measure the channel importance . However , such measurement is based on filter instead of feature map information , and hence the rich and important feature characteristics are not properly identified and extracted . [ 52 , 26 ] also explore inter-channel-based filter pruning via introducing budget constraints across channels . However , such exploration and utilization of the inter-channel information are implicit and indirect , thereby limiting the practical pruning performance . Benefits of Inter-channel Perspective . In principle , the feature information across multiple channels , if being leveraged properly , can potentially provide richer knowledge for filter pruning than the intra-channel information . Specifically , this is because : 1 ) the importance of one filter , if being solely determined by its corresponding feature map , may be sensitive to input data ; while the cross-channel feature information can bring more stable and reliable measurement ; and 2 ) consider the essential mission of pruning is to remove the unnecessary redundancy , the inter-channel strategy can inherently better identify and capture the potential unnecessary correlations among different feature maps ( and the corresponding filters ) , and thereby unlocking the new opportunity of achieving better task and compression performance . Technical Preview and Contributions . Motivated by these promising potential benefits , in this paper we propose to explore and leverage the cross-channel feature information for efficient filter pruning . To be specific , we propose Channel Independence , a cross-channel correlation-based metric to measure the importance of filters . Channel independence can be intuitively understood as the measurement of `` replaceability '' : when the feature map of one filter is measured as exhibiting lower independence , it means this feature map tends to be more linearly dependent on other feature maps of other channels . In such a scenario , the contained information of this low-independence feature map is believed to have already been implicitly encoded in other feature maps – in other words , it does not contain useful information or knowledge . Therefore the corresponding filter , which outputs this low-independence feature map , is viewed as unimportant and can be safely removed without affecting the model capacity . Overall , the contributions of this paper are summarized as : • We propose channel independence , a metric that measures the correlation of multiple feature maps , to determine the importance of filters . Built from an inter-channel perspective , channel independence can identify and capture the filter importance in a more global and precise way , thereby providing a better guideline for filter pruning . • We systematically investigate and analyze the suitable quantification metric , the complexity of the measuring scheme and the sensitiveness & reliability of channel independence , and then we develop a low-cost fine-grained high-robustness channel independence calculation scheme for efficient filter pruning . • We empirically apply the channel independence-based importance determination in different filter pruning tasks . The evaluation results show that our proposed approach brings very high pruning performance with preserving high accuracy . Notably , on CIFAR-10 dataset our solution can bring 0.90 % and 0.94 % accuracy increase over baseline ResNet-56 and ResNet-110 models , respectively , and meanwhile the model size and FLOPs are reduced by 42.8 % and 47.4 % ( for ResNet-56 ) and 48.3 % and 52.1 % ( for ResNet-110 ) , respectively . On ImageNet dataset , our approach can achieve 40.8 % and 44.8 % storage and computation reductions , respectively , with 0.15 % accuracy increase over the baseline ResNet-50 model . 2 Preliminaries . Filter Pruning . For a CNN model with L layers , its l-th convolutional layer W l = { F l1 , F l 2 , · · · , F l cl } contains cl filters F li ∈ Rc l−1×kl×kl , where cl , cl−1 and kl denote the number of output channels , the number of input channels and the kernel size , respectively . In general , network pruning can be formulated as the following optimization problem : min { Wl } Ll=1 L ( Y , f ( X , W l ) ) , s.t . ∥W l∥0 ≤ κl , ( 1 ) where L ( · , · ) is the loss function , Y is the ground-truth labels , X is the input data , and f ( · , · ) is the output function of CNN model { W l } Ll=1 . Besides , ∥ · ∥0 is the ℓ0-norm that measures the number of non-zero filters in the set , and κl is the number of filters to be preserved in the l-th layer . Feature-guided Filter Pruning . Consider the feature maps , in principle , contain rich and important information of both filters and input data , approaches using feature information have become popular and achieved the state-of-the-art performance for filter pruning . To be specific , unlike the filter-guided methods that directly minimize the loss function involved with filters ( as Eq . 1 ) , the objective of feature-guided filter pruning is to minimize the following loss function : min { Al } Ll=1 L ( Y , Al ) , s.t . ∥Al∥0 ≤ κl , ( 2 ) where Al = { Al1 , Al2 , · · · , Alcl } ∈ R cl×h×w is a set of feature maps output from the l-th layer , and Ali ∈ Rh×w is the feature map corresponds to the i-th channel . In general , after the κl important feature maps are identified and selected , their corresponding κl filters are preserved after pruning . 3 The Proposed Method . 3.1 Motivation . As formulated in Eq . 2 , feature-guided filter pruning leverages the generated feature maps in each layer to identify the important filters . To achieve that , various types of feature information , such as the high ranks [ 31 ] and the scaling factors [ 51 ] , have been proposed and utilized to select the proper feature maps and the corresponding filters . A common point for these state-of-the-art approaches is that all of them focus on measuring the importance via using the information contained in each feature map . On the other hand , the correlation among different feature maps , as another type of rich information provided by the neural networks , is little exploited in the existing filter pruning works . Why Inter-channel Perspective ? We argue that the feature information across multiple channels is of significant importance and richness , and it can be leveraged towards efficient filter pruning . Such an inter-channel perspective is motivated by two promising benefits . First , filter pruning is essentially a data-driven strategy . When the importance of one filter solely depends on the information represented by its own generated feature map , the measurement of the importance may be unstable and sensitive to the slight change of input data . On the other hand , determining the importance built upon information contained in the multiple feature maps , if performed properly , can reduce the potential disturbance incurred by the change of input data , and thereby making the importance ranking more reliable and stable . Second , the inter-channel strategy , by its nature , can better model and capture the cross-channel correlation . In the context of model compression , these identified correlations can be interpreted as a type of architecture-level redundancy , which is exactly what filter pruning aims to remove . Therefore , inter-channel strategy can enable more aggressive pruning while still preserving high accuracy .

Summary: This paper presents a metric to evaluate pre-trained dense networks to identify the feature maps that need to be dropped in an optimal manner, such that the performance of the pruned network is maximized. Authors present a simple, yet effective approach to identify the feature maps that need to be removed from the network such that it does not compromise the performance. Authors demonstrate the improvement on ImageNet dataset, which clearly demonstrates the efficacy of the proposed metric.