CHIP: CHannel Independence-based Pruning for Compact Neural Networks

1 Introduction . Convolutional neural networks ( CNNs ) have obtained widespread adoptions in numerous important AI applications [ 17 , 48 , 47 , 12 , 11 , 44 , 35 ] . However , CNNs are inherently computation intensive and storage intensive , thereby posing severe challenges for their efficient deployment on resourceconstrained embedded platforms . To address these challenges , model compression is widely used to accelerate and compress CNN models on edge devices . To date , various types of compression strategies , such as network pruning [ 15 , 16 , 37 , 60 , 28 , 14 , 1 , 49 , 10 , 63 , 36 , 18 , 13 , 3 , 2 , 25 , 53 , 50 , 9 , 39 , 33 ] , quantization [ 15 , 55 , 43 , 8 ] , low-rank approximation [ 56 , 40 , 58 , 57 ] , knowledge distillation [ 22 , 41 ] and structured matrix-based construction [ 45 , 29 , 6 ] , have been proposed and explored . Among them , network pruning is the most popular and extensively studied model compression technique in both academia and industry . Based on their differences in pruning granularity , pruning approaches can be roughly categorized to weight pruning [ 16 , 15 ] and filter pruning [ 54 , 27 , 21 , 38 , 34 ] . Weight pruning focuses on the proper selection of the to-be-pruned weights within the filters . Although enabling a high compression ratio , this strategy meanwhile causes unstructured sparsity patterns , which are not well supported by the general-purpose hardware in practice . On the other hand , filter pruning emphasizes the removal of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the entire selected filters . The resulting structured sparsity patterns can be then properly leveraged by the off-the-shelf CPUs/GPUs to achieve acceleration in the real-world scenario . Existing Filter Pruning Methods . Motivated by the potential practical speedup offered by filter pruning , to date numerous research efforts have been conducted to study how to determine the important filters – the key component of efficient filter pruning . A well-known strategy is to utilize the norms of different filters to evaluate their importance . Such a `` smaller-norm-less-important '' hypothesis is adopted in several pioneering filter pruning works [ 27 , 19 ] . Later , considering the limitations of norm-based criterion in real scenarios , [ 20 ] proposes to utilize geometric medianbased criterion . More recently , first determining those important feature maps and then preserving the corresponding filters , instead of directly selecting the filters , become a popular strategy for filter pruning . As indicated in [ 31 ] , the features , by their natures , reflect and capture rich and important information and characteristics of both input data and filters , and hence measuring the importance of features can provide a better guideline to determine the important filters . Built on this pruning philosophy , several feature-guided filter pruning approaches [ 31 , 51 ] have been proposed and developed , and the evaluation results show their superior performance over the state-of-the-art filter-guided counterparts with respect to both task performance ( e.g. , accuracy ) and compression performance ( e.g. , model size and floating-point operations ( FLOPs ) reductions ) . Determining Importance : Intra-channel & Inter-channel Perspectives . These recent advancements on filter pruning indeed show the huge benefits of leveraging feature information to determine the importance of filters . To date , some feature-guided approaches measure the importance from the intra-channel perspective . In other words , no matter which importance metric is used , the importance of one feature map ( and its corresponding filter ) , is measured only upon the information of this feature map in its own channel . On the other aspect , the inter-channel perspective , which essentially determines the filter importance via using cross-channel information [ 20 , 42 , 46 , 52 , 26 ] , is still being further explored . To be specific , [ 20 ] and [ 42 ] adopt cross-channel geometric median and Hessian , respectively , to measure the channel importance . However , such measurement is based on filter instead of feature map information , and hence the rich and important feature characteristics are not properly identified and extracted . [ 52 , 26 ] also explore inter-channel-based filter pruning via introducing budget constraints across channels . However , such exploration and utilization of the inter-channel information are implicit and indirect , thereby limiting the practical pruning performance . Benefits of Inter-channel Perspective . In principle , the feature information across multiple channels , if being leveraged properly , can potentially provide richer knowledge for filter pruning than the intra-channel information . Specifically , this is because : 1 ) the importance of one filter , if being solely determined by its corresponding feature map , may be sensitive to input data ; while the cross-channel feature information can bring more stable and reliable measurement ; and 2 ) consider the essential mission of pruning is to remove the unnecessary redundancy , the inter-channel strategy can inherently better identify and capture the potential unnecessary correlations among different feature maps ( and the corresponding filters ) , and thereby unlocking the new opportunity of achieving better task and compression performance . Technical Preview and Contributions . Motivated by these promising potential benefits , in this paper we propose to explore and leverage the cross-channel feature information for efficient filter pruning . To be specific , we propose Channel Independence , a cross-channel correlation-based metric to measure the importance of filters . Channel independence can be intuitively understood as the measurement of `` replaceability '' : when the feature map of one filter is measured as exhibiting lower independence , it means this feature map tends to be more linearly dependent on other feature maps of other channels . In such a scenario , the contained information of this low-independence feature map is believed to have already been implicitly encoded in other feature maps – in other words , it does not contain useful information or knowledge . Therefore the corresponding filter , which outputs this low-independence feature map , is viewed as unimportant and can be safely removed without affecting the model capacity . Overall , the contributions of this paper are summarized as : • We propose channel independence , a metric that measures the correlation of multiple feature maps , to determine the importance of filters . Built from an inter-channel perspective , channel independence can identify and capture the filter importance in a more global and precise way , thereby providing a better guideline for filter pruning . • We systematically investigate and analyze the suitable quantification metric , the complexity of the measuring scheme and the sensitiveness & reliability of channel independence , and then we develop a low-cost fine-grained high-robustness channel independence calculation scheme for efficient filter pruning . • We empirically apply the channel independence-based importance determination in different filter pruning tasks . The evaluation results show that our proposed approach brings very high pruning performance with preserving high accuracy . Notably , on CIFAR-10 dataset our solution can bring 0.90 % and 0.94 % accuracy increase over baseline ResNet-56 and ResNet-110 models , respectively , and meanwhile the model size and FLOPs are reduced by 42.8 % and 47.4 % ( for ResNet-56 ) and 48.3 % and 52.1 % ( for ResNet-110 ) , respectively . On ImageNet dataset , our approach can achieve 40.8 % and 44.8 % storage and computation reductions , respectively , with 0.15 % accuracy increase over the baseline ResNet-50 model . 2 Preliminaries . Filter Pruning . For a CNN model with L layers , its l-th convolutional layer W l = { F l1 , F l 2 , · · · , F l cl } contains cl filters F li ∈ Rc l−1×kl×kl , where cl , cl−1 and kl denote the number of output channels , the number of input channels and the kernel size , respectively . In general , network pruning can be formulated as the following optimization problem : min { Wl } Ll=1 L ( Y , f ( X , W l ) ) , s.t . ∥W l∥0 ≤ κl , ( 1 ) where L ( · , · ) is the loss function , Y is the ground-truth labels , X is the input data , and f ( · , · ) is the output function of CNN model { W l } Ll=1 . Besides , ∥ · ∥0 is the ℓ0-norm that measures the number of non-zero filters in the set , and κl is the number of filters to be preserved in the l-th layer . Feature-guided Filter Pruning . Consider the feature maps , in principle , contain rich and important information of both filters and input data , approaches using feature information have become popular and achieved the state-of-the-art performance for filter pruning . To be specific , unlike the filter-guided methods that directly minimize the loss function involved with filters ( as Eq . 1 ) , the objective of feature-guided filter pruning is to minimize the following loss function : min { Al } Ll=1 L ( Y , Al ) , s.t . ∥Al∥0 ≤ κl , ( 2 ) where Al = { Al1 , Al2 , · · · , Alcl } ∈ R cl×h×w is a set of feature maps output from the l-th layer , and Ali ∈ Rh×w is the feature map corresponds to the i-th channel . In general , after the κl important feature maps are identified and selected , their corresponding κl filters are preserved after pruning . 3 The Proposed Method . 3.1 Motivation . As formulated in Eq . 2 , feature-guided filter pruning leverages the generated feature maps in each layer to identify the important filters . To achieve that , various types of feature information , such as the high ranks [ 31 ] and the scaling factors [ 51 ] , have been proposed and utilized to select the proper feature maps and the corresponding filters . A common point for these state-of-the-art approaches is that all of them focus on measuring the importance via using the information contained in each feature map . On the other hand , the correlation among different feature maps , as another type of rich information provided by the neural networks , is little exploited in the existing filter pruning works . Why Inter-channel Perspective ? We argue that the feature information across multiple channels is of significant importance and richness , and it can be leveraged towards efficient filter pruning . Such an inter-channel perspective is motivated by two promising benefits . First , filter pruning is essentially a data-driven strategy . When the importance of one filter solely depends on the information represented by its own generated feature map , the measurement of the importance may be unstable and sensitive to the slight change of input data . On the other hand , determining the importance built upon information contained in the multiple feature maps , if performed properly , can reduce the potential disturbance incurred by the change of input data , and thereby making the importance ranking more reliable and stable . Second , the inter-channel strategy , by its nature , can better model and capture the cross-channel correlation . In the context of model compression , these identified correlations can be interpreted as a type of architecture-level redundancy , which is exactly what filter pruning aims to remove . Therefore , inter-channel strategy can enable more aggressive pruning while still preserving high accuracy .

Summary: To determine the importance of filters in DNNs, this paper proposes a new concept, channel independence, which measures the correlation of multiple feature maps. Also, this paper systematically investigates and analyzes the quantification metric, measuring scheme, and sensitiveness & reliability of channel independence in the context of filter pruning. As a result, the proposed scheme has achieved outstanding performance compared to existing methods. Although this paper is well-written and presents a solution for network pruning that has steadily gained a lot of attention, there are some minor comments as a result of my review of this paper. (Please refer to my main review)