CHIP: CHannel Independence-based Pruning for Compact Neural Networks
Year: 2022

Abstract:
Filter pruning has been widely used for neural network compression because of its enabled practical acceleration. To date, most of the existing filter pruning works explore the importance of filters via using intra-channel information. In this paper, starting from an inter-channel perspective, we propose to perform efficient filter pruning using channel independence, a metric that measures the correlations among different feature maps. The less independent feature map is interpreted as containing less useful information/knowledge, and hence its corresponding filter can be pruned without affecting model capacity. We systematically investigate the quantification metric, measuring scheme and sensitiveness/reliability of channel independence in the context of filter pruning. Our evaluation results for different models on various datasets show the superior performance of our approach. Notably, on CIFAR-10 dataset our solution can bring 0.90% and 0.94% accuracy increase over baseline ResNet-56 and ResNet-110 models, respectively, and meanwhile the model size and FLOPs are reduced by 42.8% and 47.4% (for ResNet-56) and 48.3% and 52.1% (for ResNet-110), respectively. On ImageNet dataset, our approach can achieve 40.8% and 44.8% storage and computation reductions, respectively, with 0.15% accuracy increase over the baseline ResNet-50 model. The code is available at https://github.com/Eclipsess/CHIP_NeurIPS2021.

heading: 1 Introduction
text: Convolutional neural networks (CNNs) have obtained widespread adoptions in numerous important AI applications [17, 48, 47, 12, 11, 44, 35]. However, CNNs are inherently computation intensive and storage intensive, thereby posing severe challenges for their efficient deployment on resourceconstrained embedded platforms. To address these challenges, model compression is widely used to accelerate and compress CNN models on edge devices. To date, various types of compression strategies, such as network pruning [15, 16, 37, 60, 28, 14, 1, 49, 10, 63, 36, 18, 13, 3, 2, 25, 53, 50, 9, 39, 33], quantization [15, 55, 43, 8], low-rank approximation [56, 40, 58, 57], knowledge distillation [22, 41] and structured matrix-based construction [45, 29, 6], have been proposed and explored. Among them, network pruning is the most popular and extensively studied model compression technique in both academia and industry.
Based on their differences in pruning granularity, pruning approaches can be roughly categorized to weight pruning [16, 15] and filter pruning [54, 27, 21, 38, 34]. Weight pruning focuses on the proper selection of the to-be-pruned weights within the filters. Although enabling a high compression ratio, this strategy meanwhile causes unstructured sparsity patterns, which are not well supported by the general-purpose hardware in practice. On the other hand, filter pruning emphasizes the removal of
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the entire selected filters. The resulting structured sparsity patterns can be then properly leveraged by the off-the-shelf CPUs/GPUs to achieve acceleration in the real-world scenario.
Existing Filter Pruning Methods. Motivated by the potential practical speedup offered by filter pruning, to date numerous research efforts have been conducted to study how to determine the important filters – the key component of efficient filter pruning. A well-known strategy is to utilize the norms of different filters to evaluate their importance. Such a "smaller-norm-less-important" hypothesis is adopted in several pioneering filter pruning works [27, 19]. Later, considering the limitations of norm-based criterion in real scenarios, [20] proposes to utilize geometric medianbased criterion. More recently, first determining those important feature maps and then preserving the corresponding filters, instead of directly selecting the filters, become a popular strategy for filter pruning. As indicated in [31], the features, by their natures, reflect and capture rich and important information and characteristics of both input data and filters, and hence measuring the importance of features can provide a better guideline to determine the important filters. Built on this pruning philosophy, several feature-guided filter pruning approaches [31, 51] have been proposed and developed, and the evaluation results show their superior performance over the state-of-the-art filter-guided counterparts with respect to both task performance (e.g., accuracy) and compression performance (e.g., model size and floating-point operations (FLOPs) reductions).
Determining Importance: Intra-channel & Inter-channel Perspectives. These recent advancements on filter pruning indeed show the huge benefits of leveraging feature information to determine the importance of filters. To date, some feature-guided approaches measure the importance from the intra-channel perspective. In other words, no matter which importance metric is used, the importance of one feature map (and its corresponding filter), is measured only upon the information of this feature map in its own channel. On the other aspect, the inter-channel perspective, which essentially determines the filter importance via using cross-channel information [20, 42, 46, 52, 26], is still being further explored. To be specific, [20] and [42] adopt cross-channel geometric median and Hessian, respectively, to measure the channel importance. However, such measurement is based on filter instead of feature map information, and hence the rich and important feature characteristics are not properly identified and extracted. [52, 26] also explore inter-channel-based filter pruning via introducing budget constraints across channels. However, such exploration and utilization of the inter-channel information are implicit and indirect, thereby limiting the practical pruning performance.
Benefits of Inter-channel Perspective. In principle, the feature information across multiple channels, if being leveraged properly, can potentially provide richer knowledge for filter pruning than the intra-channel information. Specifically, this is because: 1) the importance of one filter, if being solely determined by its corresponding feature map, may be sensitive to input data; while the cross-channel feature information can bring more stable and reliable measurement; and 2) consider the essential mission of pruning is to remove the unnecessary redundancy, the inter-channel strategy can inherently better identify and capture the potential unnecessary correlations among different feature maps (and the corresponding filters), and thereby unlocking the new opportunity of achieving better task and compression performance.
Technical Preview and Contributions. Motivated by these promising potential benefits, in this paper we propose to explore and leverage the cross-channel feature information for efficient filter pruning. To be specific, we propose Channel Independence, a cross-channel correlation-based metric to measure the importance of filters. Channel independence can be intuitively understood as the measurement of "replaceability": when the feature map of one filter is measured as exhibiting lower independence, it means this feature map tends to be more linearly dependent on other feature maps of other channels. In such a scenario, the contained information of this low-independence feature map is believed to have already been implicitly encoded in other feature maps – in other words, it does not contain useful information or knowledge. Therefore the corresponding filter, which outputs this low-independence feature map, is viewed as unimportant and can be safely removed without affecting the model capacity. Overall, the contributions of this paper are summarized as:
• We propose channel independence, a metric that measures the correlation of multiple feature maps, to determine the importance of filters. Built from an inter-channel perspective, channel independence can identify and capture the filter importance in a more global and precise way, thereby providing a better guideline for filter pruning.
• We systematically investigate and analyze the suitable quantification metric, the complexity of the measuring scheme and the sensitiveness & reliability of channel independence, and then we develop a low-cost fine-grained high-robustness channel independence calculation scheme for efficient filter pruning.
• We empirically apply the channel independence-based importance determination in different filter pruning tasks. The evaluation results show that our proposed approach brings very high pruning performance with preserving high accuracy. Notably, on CIFAR-10 dataset our solution can bring 0.90% and 0.94% accuracy increase over baseline ResNet-56 and ResNet-110 models, respectively, and meanwhile the model size and FLOPs are reduced by 42.8% and 47.4% (for ResNet-56) and 48.3% and 52.1% (for ResNet-110), respectively. On ImageNet dataset, our approach can achieve 40.8% and 44.8% storage and computation reductions, respectively, with 0.15% accuracy increase over the baseline ResNet-50 model.

heading: 2 Preliminaries
text: Filter Pruning. For a CNN model with L layers, its l-th convolutional layer W l = {F l1,F l 2, · · · ,F l cl} contains cl filters F li ∈ Rc
l−1×kl×kl , where cl, cl−1 and kl denote the number of output channels, the number of input channels and the kernel size, respectively. In general, network pruning can be formulated as the following optimization problem:
min {Wl}Ll=1
L(Y , f(X ,W l)), s.t. ∥W l∥0 ≤ κl, (1)
where L(·, ·) is the loss function, Y is the ground-truth labels, X is the input data, and f(·, ·) is the output function of CNN model {W l}Ll=1. Besides, ∥ · ∥0 is the ℓ0-norm that measures the number of non-zero filters in the set, and κl is the number of filters to be preserved in the l-th layer.
Feature-guided Filter Pruning. Consider the feature maps, in principle, contain rich and important information of both filters and input data, approaches using feature information have become popular and achieved the state-of-the-art performance for filter pruning. To be specific, unlike the filter-guided methods that directly minimize the loss function involved with filters (as Eq. 1), the objective of feature-guided filter pruning is to minimize the following loss function:
min {Al}Ll=1
L(Y ,Al), s.t. ∥Al∥0 ≤ κl, (2)
where Al = {Al1,Al2, · · · ,Alcl} ∈ R cl×h×w is a set of feature maps output from the l-th layer, and Ali ∈ Rh×w is the feature map corresponds to the i-th channel. In general, after the κl important feature maps are identified and selected, their corresponding κl filters are preserved after pruning.

heading: 3 The Proposed Method

heading: 3.1 Motivation
text: As formulated in Eq. 2, feature-guided filter pruning leverages the generated feature maps in each layer to identify the important filters. To achieve that, various types of feature information, such as the high ranks [31] and the scaling factors [51], have been proposed and utilized to select the proper feature maps and the corresponding filters. A common point for these state-of-the-art approaches is that all of them focus on measuring the importance via using the information contained in each feature map. On the other hand, the correlation among different feature maps, as another type of rich information provided by the neural networks, is little exploited in the existing filter pruning works.
Why Inter-channel Perspective? We argue that the feature information across multiple channels is of significant importance and richness, and it can be leveraged towards efficient filter pruning. Such an inter-channel perspective is motivated by two promising benefits. First, filter pruning is essentially a data-driven strategy. When the importance of one filter solely depends on the information represented by its own generated feature map, the measurement of the importance may be unstable and sensitive to the slight change of input data. On the other hand, determining the importance built upon information contained in the multiple feature maps, if performed properly, can reduce the potential disturbance incurred by the change of input data, and thereby making the importance ranking more reliable and stable. Second, the inter-channel strategy, by its nature, can better model and capture the cross-channel correlation. In the context of model compression, these identified correlations can be interpreted as a type of architecture-level redundancy, which is exactly what filter pruning aims to remove. Therefore, inter-channel strategy can enable more aggressive pruning while still preserving high accuracy.

heading: 3.2 Channel Independence: A New Lens for Filter Importance
text: Key Idea. Motivated by these promising benefits, we propose to explore the filter importance from the inter-channel perspective. Our key idea is to use channel independence to represent the importance of each feature map (and its corresponding filter). To be specific, when one feature map of one channel is highly linearly dependent on other feature maps of other channels, it implies that its contained information has already been largely encoded in other feature maps. Consequently, even we remove the corresponding filter, the represented information and knowledge of its generated low-independence feature map can still be largely preserved and approximately reconstructed by other feature maps of other filters after the fine-tuning procedure. In other words, the filters that generate low-independence feature maps tend to exhibit more "replaceability", which can be interpreted as lower importance. Therefore, removing those filters with low channel-independence feature maps will be safe while still preserving high model capacity.
How to Measure Channel Independence? Next we discuss how to properly measure the independence of one feature map from others. To that end, four important questions need to be answered.
Question #1: Which mathematical metric should be adopted to quantify the independence of one feature map from other feature maps?
Analysis. Considering the entire set of feature maps generated from one layer is a 3-D tensor, we propose to extract the linear dependence information of each feature map within the framework of linear algebra. To be specific, given output feature map set of the l-th layer Al, we first matricize Al to Al = [al1 T ,al2 T , · · · ,alcl T ]T ∈ Rcl×hw, where a row vector ali ∈ Rhw is the vectorized Ali. In such a scenario, the linear independence of each vectorized feature map a l i, as a row of the matricized entire set of feature maps Al, can be measured via the existing matrix analysis tool.
The most straightforward solution is to use rank to determine the independence of ali, since rank mathematically represents the maximum number of linearly independent rows/columns of the matrix. For instance, we can remove one row from the matrix, and calculate the rank change of the matrix, and then identify the impact and the importance of the deleted row – the less rank change, the less independence (and the importance) of the removed row.
Our Proposal. However, in the context of filter pruning, we believe, the change of nuclear norm of the entire set of feature maps, is a better metric to quantify the independence of each feature map. This is because, as the ℓ1-norm of singular values of the matrix, the nuclear norm can reveal richer "soft" information on the impact of the deleted row on the matrix; while the rank, as the ℓ0-norm of the singular values, is too "hard" to reflect such change. For instance, as shown in Fig. 2, when we select ali, as one row of A
l, to be removed, the rank change of Al is almost the same regardless of our selection of ali; while the corresponding changes of nuclear norm vary significantly when different ali are deleted. Therefore, the change of nuclear norm can be viewed as a more precise metric to measure the linear independence of one feature map in a more fine-grained way. In general, the channel independence of one feature map is defined and calculated as below:
Definition 1 (Channel independence of single feature map) For the i-th layer with output feature maps Al = {Al1,Al2, · · · ,Alcl} ∈ R
cl×h×w , the Channel Independence (CI) of one feature map Ali ∈ Rh×w in the i-th channel is defined and calculated as:
CI(Ali) ≜ ∥Al∥∗ − ∥M li ⊙Al∥∗, (3)
where Al ∈ Rcl×hw is the matricized Al, ∥ · ∥∗ is the nuclear norm, ⊙ is the Hadamard product, and M li ∈ Rc l×hw is the row mask matrix whose i-th row entries are zeros and other entries are ones.
Question #2: What is the proper scheme to quantify the independence of multiple feature maps?
Analysis. Eq. 3 describes the measurement of channel independence for a single feature map. However, in practice filter pruning typically aims to remove multiple filters, which means the independence of the combination of multiple feature maps needs to be calculated. In the case of pruning m filters, such scenario corresponds to checking the changes of nuclear norm of the original cl-row Al after removing m rows (ali). A straightforward solution is to just calculate C cl
m changes of nuclear norms for all the possible m-row removal choices, and then select the one which corresponds to the smallest change. However, this strategy is very computationally expensive, and sometimes even intractable when cl is large. For instance, in order to identify the smallest nuclear norm change for pruning 50% filters of a 256-output channel ResNet-50 layer, such brutal-force measurement requires more than 5× 1075 times of nuclear norm calculation. Our Proposal. To address this computational challenge, we propose to leverage the independence of individual feature map to approximate the independence of their combination. To be specific, in order to determine m least independent rows in the Al, we first iteratively remove one row (ali) from Al and calculate the corresponding nuclear norm change between the remaining (cl − 1)-row matrix
and the original cl-row Al. Then, among the cl calculated changes, we identify the m smallest ones and the corresponding removed ali. Those selected m vectorized feature maps a l i are interpreted as the less independent from other feature maps, and hence their corresponding filters F li are less important ones that should be pruned. In general, this individual independence-based measurement can closely approximate the combined independence of multiple feature maps (see Definition 2). Such approximation requires much less computational complexity (reduction from O(C(N,κ)) to O(N)); while still achieving superior filter pruning performance (see Section 4 for evaluation results). Definition 2 (Approximated channel independence of combined multiple feature maps) For the i-th layer with output feature maps Al ∈ Rcl×h×w , the channel independence of combined m feature maps {Albi} m i=1, where A l bi ∈ Rh×w is in the bi-th channel, is defined and approximated as:
CI({Albi} m i=1) ≜ ∥Al∥∗ − ∥M lb1,··· ,bm ⊙A l∥∗ ≈ m∑ i=1 CI(Albi), (4)
where M lb1,··· ,bm is the multi-row mask matrix, in which the b1, · · · , bm-th row entries are zeros and all the other entries are ones.
Question #3: How is the sensitiveness of channel independence related to the distribution of input data?
Our Observation. Consider our proposed channel independence-based filter pruning is a data-driven approach, its reliability with different distributions of input data should be carefully ensured and examined. To that end, we perform empirical evaluations on channel independence with respect to multiple input images. We observe that the average channel independence of each feature map is very stable at the batch level. In other words, we can simply input small batches of image samples, and calculate the average channel independence, and then such averaged channel independence with a small number of input data can be used to estimate the channel independence with all the input data. As illustrated in Fig. 4, for the same feature map, the average channel independence in different batches remains very similar, thereby indicating that our channel independence-based approach is robust against different input data.
Question #4: Is this one-shot importance determination scheme good enough? Do we need to further learn and adjust the pruning mask from the data?
Our Observation. As described above, our proposed scheme calculates the channel independence to identify the filter importance. Considering our approach is built on one-shot calculation, a natural extension is to further adjust the importance ranking via additional learning. To be specific, if we interpret the filter pruning is a channel-wise masking operation over the entire weight tensor, the
Algorithm 1 CHannel Independence-based Pruning (CHIP) procedure for the l-th layer
Input: Pre-trained weight tensor W l, N sets of feature maps Al = {Al1,Al2, · · · ,Alcl} ∈ R cl×h×w
from N input samples, and the desired number of filters to be preserved κl. Output: Pruned weight tensor W lprune.
1: for each input sample do 2: Flatten feature maps: Al := reshape(Al, [cl, hw]); 3: for i = 1 to cl do 4: CI calculation: Calculate CI(Ali) via Equation 3; 5: end for 6: end for 7: Averaging: Average CI(Ali) under all N input samples; 8: Sorting: Sort {CI(Ali)}c l
i=1 in ascending order; 9: Pruning: Prune cl − κl filters in W l corresponding to the cl − κl smallest CI(Ali);
10: Fine-tuning: Obtain final W lprune via fine-tuning W l with removing the pruned filter channels.
selection of channel mask can be learned from data, and such learning process can use the pruning mask determined by our approach as the initialization. Though in principle this learning-based strategy is expected to enable additional performance improvement, our empirical evaluations show that the consecutive learning procedure does not easily bring further accuracy increase (with the target compression ratio) or compression ratio increase (with the target accuracy) – more experimental details are reported in Supplementary Material. We hypothesize the reason for such phenomenon is that, our proposed nuclear norm change-based channel independence, though only requires one-time calculation, already identifies and captures the importance of feature maps (and its corresponding filters) with high quality, and hence further learning-based adjustment of pruning mask does not easily provide additional improvement.
The Overall Algorithm. After addressing the above four problems, we can then integrate our proposals and observations to develop the entire filter pruning procedure from the inter-channel perspective. Algorithm 1 describes and summarizes the overall scheme for our proposed CHannel Independence-based filter Pruning (CHIP) algorithm.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Baselines Models and Datasets. To demonstrate the effectiveness and generality of our proposed channel independence-based approach, we evaluate its pruning performance for various baseline models on different image classification datasets. To be specific, we conduct experiments for three CNN models (ResNet-56, ResNet-110 and VGG-16) on CIFAR-10 dataset [24]. Also, we further evaluate our approach and compare its performance with other state-of-the-art pruning methods for ResNet-50 model on large-scale ImageNet dataset [5].
Pruning and Fine-tuning Configurations. We conduct our empirical evaluations on Nvidia Tesla V100 GPUs with PyTorch 1.7 framework. To determine the importance of each filter, we randomly sample 5 batches (640 input images) to calculate the average channel independence of each feature map in all the experiments. After performing the channel independence-based filter pruning, we then perform fine-tuning on the pruned models with Stochastic Gradient Descent (SGD) as the optimizer. To be specific, we perform the fine-tuning for 300 epochs on CIFAR-10 datasets with the batch size, momentum, weight decay and initial learning rate as 128, 0.9, 0.05 and 0.01, respectively. On the ImageNet dataset, fine-tuning is performed for 180 epochs with the batch size, momentum, weight decay and initial learning rate as 256, 0.99, 0.0001 and 0.1, respectively.

heading: 4.2 Evaluation and Comparison on CIFAR-10 Dataset
text: Table 1 shows the evaluation results of the pruned ResNet-56, ResNet-110 and VGG-16 models on CIFAR-10 dataset. To be consistent with prior works, we evaluate the performance for two scenarios: targeting high accuracy and targeting high model size and FLOPs reductions.
ResNet-56. For ResNet-56 model, our channel independence-based approach can bring 0.90% accuracy increase over the baseline model with 42.8% and 47.4% model size and FLOPs reductions, respectively. When we adopt aggressive compression with 71.8% and 72.3% model size and FLOPs reductions, we can still achieve high performance – our solution enables 1.33% higher accuracy than HRank [31] with the similar model size and computational costs.
ResNet-110. For ResNet-110 model, our approach can bring 0.94% accuracy increase over the baseline model with 48.3% and 52.1% model size and FLOPs reductions, respectively. When we perform aggressive pruning with 68.3% and 71.6% model size and FLOPs reductions, our pruned model can still achieve 0.13% higher accuracy over the baseline model.
VGG-16. For VGG-16 model, our approach can bring 81.6% and 58.1% model size and FLOPs reductions, respectively, with only 0.1% accuracy drop. Moreover, with 83.3% and 66.6% storage and computational cost reductions, our pruned model can achieve 1.38% higher accuracy than the model using other pruning approaches under a similar compression ratio. For even higher FLOPs reduction (78.6%), our method can bring nearly 2% accuracy increase over the prior works.

heading: 4.3 Evaluation and Comparison on ImageNet Dataset
text: Table 2 summarizes the pruning performance of our approach for ResNet-50 on ImageNet dataset. It is seen that when targeting a moderate compression ratio, our approach can achieve 40.8% and 44.8% storage and computation reductions, respectively, with 0.15% accuracy increase over the baseline model. When we further increase the compression ratio, our approach still achieves superior performance than state-of-the-art works. For instance, compared with SCOP [51], our approach shows higher accuracy (0.12%) in moderate compression region and the same accuracy in high compress region; while meanwhile enjoying a much smaller model size and fewer FLOPs.

heading: 5 Conclusion
text: In this paper, we propose to use channel independence, an inter-channel perspective-motivated metric, to evaluate the importance of filters for network pruning. By systematically exploring the quantification metric, measuring scheme, and sensitiveness and reliability of channel independence, we develop CHIP, a CHannel Independence-based filter pruning for neural network compression. Extensive evaluation results on different datasets show our proposed approach brings significant storage and computational cost reductions while still preserving high model accuracy.
Broader Impact
As technology advances, cell phones, laptops, wearable gadgets and intelligent connected vehicles with specific chips are required to handle more complicated tasks by deploying neural networks. However, more powerful networks will cost more memory size and running time. Network pruning is the main strategy to reduce the memory size and accelerate the run-time during the inference stage. Benefiting from pruning techniques and specific designs for hardware [62, 4], IoT (Internet of Things) devices are able to execute complex projects based on small and efficient models.
Acknowledgements and Funding Disclosure
Bo Yuan would like to thank the support from National Science Foundation (NSF) award CCF1937403. Saman Zonouz would like to thank the support from NSF CPS and SATC programs.

Review summaries:
The authors propose to explore and leverage the cross-channel feature information for filter pruning. Specifically, the authors propose a cross-channel correlation-based metric to measure the importance of filters. In this case, the low-independence feature maps are considered redundant. As a result, the corresponding filters that output the low-independence feature maps are viewed as uninformative and can be safely removed. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of the proposed method. However, the experiments are not sufficient. Please see my detailed comments below.

This paper proposes a channel pruning method named CHIP to improve the existing channel pruning frameworks. The proposed method measures the importance of channels through channel independence. Extensive experiments show the effectiveness of trunk pruning.

To determine the importance of filters in DNNs, this paper proposes a new concept, channel independence, which measures the correlation of multiple feature maps. Also, this paper systematically investigates and analyzes the quantification metric, measuring scheme, and sensitiveness & reliability of channel independence in the context of filter pruning. As a result, the proposed scheme has achieved outstanding performance compared to existing methods. Although this paper is well-written and presents a solution for network pruning that has steadily gained a lot of attention, there are some minor comments as a result of my review of this paper. (Please refer to my main review)

This paper presents a metric to evaluate pre-trained dense networks to identify the feature maps that need to be dropped in an optimal manner, such that the performance of the pruned network is maximized. Authors present a simple, yet effective approach to identify the feature maps that need to be removed from the network such that it does not compromise the performance. Authors demonstrate the improvement on ImageNet dataset, which clearly demonstrates the efficacy of the proposed metric.