CHIP: CHannel Independence-based Pruning for Compact Neural Networks
Authors: Yang Sui
Miao Yin
Yi Xie
Huy Phan
Saman Aliari Zonouz
Bo Yuan
Venue: NeurIPS 2021 Poster (2021)
Keywords: model compression
pruning
filter pruning
channel pruning

Abstract: Filter pruning has been widely used for neural network compression because of its enabled practical acceleration. To date, most of the existing filter pruning works explore the importance of filters via using intra-channel information. In this paper, starting from an inter-channel perspective, we propose to perform efficient filter pruning using Channel Independence, a metric that measures the correlations among different feature maps. The less independent feature map is interpreted as containing less useful information$/$knowledge, and hence its corresponding filter can be pruned without affecting model capacity. We systematically investigate the quantification metric, measuring scheme and sensitiveness$/$reliability of channel independence in the context of filter pruning. Our evaluation results for different models on various datasets show the superior performance of our approach. Notably, on CIFAR-10 dataset our solution can bring $0.75\%$ and $0.94\%$ accuracy increase over baseline ResNet-56 and ResNet-110 models, respectively, and meanwhile the model size and FLOPs are reduced by  $42.8\%$ and  $47.4\%$ (for ResNet-56) and $48.3\%$ and $52.1\%$ (for ResNet-110), respectively. On ImageNet dataset, our approach can achieve $40.8\%$ and $44.8\%$ storage and computation reductions, respectively, with $0.15\%$ accuracy increase over the baseline ResNet-50 model. The code is available at https://github.com/Eclipsess/CHIP_NeurIPS2021.

CHIP: CHannel Independence-based Pruning for Compact Neural Networks
Yang Sui Miao Yin
Yi Xie
Huy Phan
Saman Zonouz
Bo Yuan
Department of Electrical and Computer Engineering
Rutgers University
Piscataway, NJ 08854, USA
{yang.su sui, min, xie, phy., saman saman. zotoooz)ruttgerss....
bo yuan@ssesoe. rutgers. edu
Abstract
Filter pruning has been widely used for neural network compression because of its enabled practical acceleration. To date, most of the existing filter pruning works explore the importance of filters via using intra-channel information. In this paper, starting from an inter-channel perspective, we propose to perform efficient filter pruning using channel independence, metric that measures the correlations among different feature maps. The less independent feature map is interpreted as containing less useful information knowledge, and hence its corresponding filter can be pruned without affecting model capacity. We systematically investigate the quantification metric, measuring scheme and sensitiveness /reliability of channel independence in the context of filter pruning. Our evaluation results for different models on various datasets show the superior performance our approach. Notably, on CIFAR-10 10 dataset our solution can bring 0.90% and 0.94% accuracy increase over baseline ResNet-56 and ResNet-1 10 models, respectively, and meanwhile the model size and FL.PSs are reduced by 42.8% and 47.4% (for ResNet-56) and 48.3% and 52.1% (for ResNet- 110), respectively. achieve 40.8% and 44.8% storage and computation reductions, respectively, with 0. 15% accuracy increase over the baseline ResNet-50 model. The code is available at https//ggitbbb.cmm//cepes////CC Neeu NeurlPS2221
ImageNet dataset, our approach can
1 Introduction
Convolutional neural networks (CNNs) have obtained widespread adoptions in numerous important Al applications 17 47 12
35 However, CNNs are inherently computation intensive and storage intensive, thereby posing severe challenges for their efficient deployment on resource- constrained embedded platforms. To address these challenges, model compression is widely used to accelerate and compress CNN models edge devices. To date, various types of compression strategies, such as network pruning [15
37 60, 28 [4]
49 10
63
36
18
[3]
2 25 50 g
39 33
quantization [3 55 43 8
low-rank approximation [56
40
58 57, knowledge distillation
29 6), have been proposed and explored. Among them, network pruning is the most popular and extensively studied model compression technique in both academia and industry.
22
and structured matrix based construction
45
Based on their differences in pruning granularity. pruning approaches can be roughly categorized to weight pruning [6. 15] and filter pruning [54 27 21
Weight pruning focuses on the proper selection of the to-be-pruned pruned weights within the filters. Although enabling high compression ratio, this strategy meanwhile causes unstructured sparsity patterns, which are not well supported by the general-purpose hardware in practice. On the other hand, filter pruning emphasizes the removal of
34
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Importance
Calculation
Importance
Calculation
Importance
Calculation
Input Image
Importance Calculation
d
Prune
Least
Important
Filters
Feature Maps
Importance
(a) Feature information- informaaion-based filter pruning from an intra-channel perspective.
b
Importance
Calculation
Input Image
Prune
Least
Important
Filters
Feature Maps
Importance Score
Pruned Filters
Feature information- based filter pruning from an inter-channel perspective.
Figure 1: Intra-channel vs Inter-channel perspectives for filter pruning.
the entire selected filters. The resulting structured sparsity patterns can be then properly leveraged by the off-the-shell CPUs/GPUs to achieve acceleration in the real-world scenario.
Existing Filter Pruning Methods. Motivated by the potential practical speedup offered by filter pruning, to date numerous research efforts have been conducted to study how to determine the important filters the key component of efficient filter pruning. A well-known strategy is to utilize the norms of different filters to evaluate their importance. Such
wwwallrr-noom-eee----mrrre hypothesis is adopted in several pioneering filter pruning works [27] [9] Later, considering the limitations of norm-based criterion in real scenarios, [20] proposes to utilize geometric median- based criterion. More recently, first determining those important feature maps and then preserving the corresponding filters, instead of directly selecting the filters, become a popular strategy for filter pruning. As indicated (31), the features, by their natures, reflect and capture rich and important information and characteristics of both input data and filters, and hence measuring the importance of features can provide better guideline to determine the important filters. Built this pruning philosophy, several cature-guided filter pruning approaches [3 have been proposed and developed, and the evaluation results show their superior performance over the ttate-of-the-arr filte-guided counterparts with respect to both task performance (e.g.. accuracy) and compression performance (e.g., model size and soating-poin operations (FLOPs) reductions).
Determining Importance: Intra-channel & Inter-channel Perspectives. These recent advance- ments filter pruning indeed show the huge benefits of leveraging feature information determine the importance of filters. To date, some feature- guided approaches measure the importance from the intra-channel perspective. other words, no matter which importance metric used, the importance of one feature map (and its corresponding filter). is measured only upon the information of this feature map its own channel. On the other aspect, the inter-channel perspective, which essentially determines the filter importance via using cross-channel information 20
52 26), is still being further explored. To be specific, (20) and [4] adopt cross-chand coosnel geometric median and Hessian, re- spectively, to measure the channel importance. However, such measurement is based on filter instead of feature map information, and hence the rich and important feature characteristics are not properly identified and extracted. budget constraints across channels. However, such exploration and utilization of the inter-channel information are implicit and indirect, thereby limiting the practical pruning performance.
42 46
522 26 also explore inter-channel-based filter pruning via introducing

---

Benefits of Inter-channel Perspective. In principle, the feature information across multiple channels, if being leveraged properly, can potentially provide richer knowledge for filter pruning than the intra-channel information. Specifically, this is because: 1) the importance of one filter, if being solely determined by its corresponding feature map, may be sensitive to input data; while the cross-channel feature information can bring more stable and reliable measurement; and 2) consider the essential mission of pruning is to remove the unnecessary redundancy, the inter-channel strategy can inherently better identify and capture the potential unnecessary correlations among different feature maps (and the corresponding filters), and thereby unlocking the new opportunity of achieving better task and compression performance.
Technical Preview and Contributions. Motivated by these promising potential benefits, in this paper we propose to explore and leverage the cross-channel feature information for efficient filter pruning. To specific, we propose Channel Independence, a cross-channel correlation-base metric to measure the importance of filters. Channel independence can be intuitively understood as the measurement of "replaceebility": when the feature map of one filter is measured as exhibiting lower independence, it means this feature map tends to be more linearly dependent on other feature maps of other channels. In such a scenario, the contained information of this low-independence feature map is believed to have already been implicitly encoded in other feature maps in other words, it does not contain useful information or knowledge. Therefore the corresponding filter, which outputs this low- low-independence feature map, is viewed as unimportant and can be safely removed without affecting the model capacity. Overall, the contributions of this paper are summarized as:
We propose channel independence, a metric that measures the correlation of multiple feature maps, to determine the importance of filters. Built from an inter-channel perspective, channel independence can identify and capture the filter importance in more global and precise way, thereby providing a better guideline for filter pruning.
We systematically investigate and analyze the suitable quantification metric, the complexity of the measuring scheme and the sensitiveness & reliability of channel independence, and then we develop low-cost ahe-ggraied high-robustness channel independence calculation scheme for efficient filter pruning.
We empirically apply the channel independence- based importance determination in different filter pruning tasks. The evaluation results show that our proposed approach brings very high pruning performance with preserving high accuracy. Notably, on CIFAR- 10 dataset solution can bring 0.90% and 0.94% accuracy increase ResNet- 10 models, respectively, and meanwhile the model size and FLOPS are reduced by 42.8% and 47.4% (for ResNet-56) and 48.3% and 52.1% (for ResNet-1 10), respectively. On ImageNet dataset, our approach can achieve 40.8% and 44.8% storage and computation reductions, respectively, with 0.15% 15% accuracy increase over the baseline ResNet-50 model.
baseline ResNet-56 and
2 Preliminaries
Filter Pruning. For a CNN model with L layers, its 1-th convolutional layer W (F, F.r} contains et filters Fi E xxk- xx' where e' c' and k denote the number of output channels, the number of input channels and the kernel size, respectively. In general, network pruning can be formulated as the following optimization problem:
s.t. || S n
min (W D
(1)
where C(..-) )) is the loss function, y is the ground-truth labels, X' is the input data, and f(...) the output function of CNN model [W')E, Besides, || llo is the fn-mrmm that measures the number of non-zero filters in the set, and v² is the number of filters to be preserved in the I-th layer.
Feature-guided Filter Pruning. Consider the feature maps, in principle, contain rich and important information both filters and input data, approaches using feature information have become popular and achieved the state-of-the-art performance for filter pruning. To be specific, unlike the filter- methods that directly minimize the loss function involved with filters (as Eq. D. the objective of feature- ceature-guided filter pruning is to minimize the following loss function:
3

---

min C(), A'), (4
|| S k
where A' = (A/, A4,
A/..... A E Re xhxx is set of feature maps output from the I-th layer, and A E /xx is the feature map corresponds to the i-th channel. In general, after the x' important feature maps are identified and selected, their corresponding x² filters are preserved after pruning.
3
The Proposed Method
3.1 Motivation
As formulated in Eq. 2 feature-guided -uuied filter pruning leverages the generated feature maps in each layer to identify the important filters. To achieve that, various types of feature information, such as the high ranks [31] and the scaling factors [5]], have been proposed and utilized to select the proper feature maps and the corresponding filters. A common point for these state-of-the-art approaches is that all of them focus on measuring the importance via using the information contained in each feature map. On the other hand, the correlation among different feature maps, another type of rich information provided by the neural networks, is little exploited in the existing filter pruning works.
Why Inter-channel Perspective? We argue that the feature information across multiple channels is of significant importance and richness, and it can be leveraged towards efficient filter pruning. Such inter-channel perspective is motivated by two promising benefits. First, filter pruning is essentially a data-driven strategy. When the importance of one filter solely depends on the information represented by its own generated feature map, the measurement of the importance may be unstable and sensitive to the slight change of input data. On the other hand, determining the importance built upon information contained in the multiple feature maps, if performed properly, can reduce the potential disturbance incurred by the change of input data, and thereby making the importance ranking more reliable and stable. Second, the inter-channel strategy, by its nature, can better model and capture the cross-channel correlation. In the context of model compression, these identified correlations can be interpreted as a type of architecture-leve redundancy, which is exactly what filter pruning aims to remove. Therefore, inter-channel strategy can enable more aggressive pruning while still preserving high accuracy,
3.2 Channel Independence: A New Lens for Filter Importance
Key Idea. Motivated by these promising benefits, we propose to explore the filter importance from the inter-channel perspective. Our key idea is to use channel independence to represent the importance of each feature map (and its corresponding filter). To be specific, when one feature map of one channel highly linearly dependent on other feature maps of other channels, implies that its contained information has already been largely encoded in other feature maps. Consequently. even we remove the corresponding filter, the represented information and knowledge of its generated low-independence feature map can still be largely preserved and approximately reconstructed by other feature maps of other filters after the fine-tuning procedure. In other words, the filters that generate low-independence feature maps tend to exhibit more "replaceability" which can be interpreted as lower importance. Therefore, removing those filters with low channel- indeeeddeneeeee feature maps will be safe while still preserving high model capacity.
How to Measure Channel Independence? Next we discuss how properly measure the indepen- dence of one feature map from others. To that end, four important questions need to be answered.
Question #1: Which mathematical metric should be adopted quantify the independence of one feature map from other feature maps?
Analysis. Considering the entire set of feature maps generated from one layer is 3-D tensor, we propose to extract the linear dependence information of each feature map within the framework of linear algebra. To be specific, given output feature map set of the 1-th layer A', we first matricize A' to A [a|" : al , ..... a 2 E Re xhw where row vector af E /R is the vectorized A/. In such scenario, the linear independence of each vectorized feature map al, row of the matricized entire set of feature maps A', can be measured via the existing matrix analysis tool.

---

Noclear Norm
A of Rofk
2
0
10 20
30 40 50 60
10
20
30
40
50
60
Index of feature map
Index of feature map
(a) The change of rank.
(b) The change of nuclear norm.
Figure 2: The change of (a) rank and (b) nuclear norm of the entire set of feature maps (A') when one feature map (a') is removed. The x-axis represents the index of the feature map that is removed. The y-axis represents the corresponding rank/nuclear norm change of the entire set of feature maps. The feature maps are output from one layer of the ResNet-50 model with input ImageNet image. It is seen that change of nuclear norm can better reveal the impact of the deleted feature map on the entire set of feature maps.
The most straightforward solution is to use rank to determine the independence of a'. since rank mathematically represents the maximum number of linearly independent rows/columns of the matrix. For instance, we can remove one row from the matrix, and calculate the rank change of the matrix, and then identify the impact and the importance of the deleted row the less rank change, the less independence (and the importance) of the removed row.
Our Proposal. However, in the context of filter pruning, we believe, the change of nuclear norm of the entire set of feature maps, is a better metric to quantify the independence of each feature map. This is because, as the -n-ormm of singular values of the matrix, the nuclear "soft" information on the impact of the deleted row on the matrix; while the rank, as the 4o-ormm of the singular values, is too "hard" to reflect such change. For instance, as shown Fig. 2 when we select a'. as one row of A', to be removed, the rank change of A' is almost the same regardless of our selection of a;: while the corresponding changes of nuclear norm vary significantly when different at are deleted. Therefore, the change of nuclear norm can be viewed as a more precise metric to measure the linear independence of one feature map in the channel independence of one feature map is defined and calculated as below:
can reveal richer
fine-grained way. In general,
Definition (Channel independence of single feature map) For the i-th layer with output feature maps A' {A,, Ab,,... Ald) IRee xhxw the Channel Independence (CI) of one feature map A kax x in the i-th channel is defined and calculated as:
CI(AA) & ||A'||, - | o A'll.
where A IRd xhw is the matricized A', 11 -1- I|= is the nuclear norm, O is the Hadamard product, and M E Ree xhw is the row mask matrix whose i-th row entries are zeros and other entries are ones.
Question #2: What is the proper scheme to quantify the independence of multiple feature mups?
Analysis. Eq. describes the measurement of channel independence for a single feature map. However, in practice filter pruning typically aims to remove multiple filters, which means the independence of the combination of multiple feature maps needs be calculated. In the case of pruning filters, such scenario corresponds to checking the changes of nuclear norm of the original c-rrow A' after removing m rows (at). A straightforward solution is to just calculate C changes of nuclear norms for all the possible m--ow removal choices, and then select the one which corresponds to the smallest change. However, this strategy is very computationally expensive, and sometimes even intractable when el large. For instance, in order to identify the smallest nuclear norm change for pruning 50% filters of 256-output channel ResNet-50 layer, such brutal-force measurement requires more than 5 x 75 times of nuclear norm calculation.
Our Proposal. To address this computational challenge, we propose leverage the independence of individual feature map to approximate the independence of their combination. To be specific, in order to determine m least independent rows in the A', we first iteratively remove row (a)) from A' and calculate the corresponding nuclear norm change between the remaining (c' 1)-row matrix
5

---

Calculate the Upper Bound
Nuclear Norm
Upper Bound
Input
Filters
Feature Maps
Reshaped Feature Maps
Calculation (+ CI)
Pruned Filters
Figure 3: Example of filter pruning process using the change nuclear norm-based channel indepen-
dence (CT) criterion.
and the original -oow A'. Then, among the e' calculated changes, we identify the m smallest ones and the corresponding removed a/. Those selected m vectorized feature maps at are interpreted the less independent from other feature maps, and hence their corresponding filters F are less important ones that should be pruned. In general, this individual independence- closely approximate the combined independence of multiple feature maps (see Definition 2). Such approximation requires much less computational complexity (reduction from OCC(N, ()) ) to O(N)): while still achieving superior filter pruning performance (see Section 4 for evaluation results).
beseedeeceeeeeedddde based measurement
Definition 2 (Approximated channel independence of combined multiple feature maps) For the i-th layer with output feature maps A' E TRe xhxw the channel independence of combined m
y feature maps {A,, i== where A!, Rhbx is in the by-th channel, is defined and approximated
CI(LA 1m A |A IM, d... O AAll R Σ CI(AL)).
(4)
where M,.... ..m the multi-row mask matrix, in which the by, bm row entries are zeros and all the other entries are ones.
Question #3: How is the sensitiveness of channel independence related to the distribution of input data?
Our Observation. Consider our proposed channel independence- based filter pruning is data-driven approach, its reliability with different distributions of input data should be carefully ensured and examined. To that end, we perform empirical evaluations on channel independence with respect to multiple input images. We observe that the average channel independence of each feature map is very stable at the batch level. In other words, we can simply input small batches of image samples, and calculate the average channel independence, and then such averaged channel independence with a small number of input data can be used to estimate the channel independence with all the input data. As illustrated in Fig. 4 for the same feature map, the average channel independence different batches remains very similar, thereby indicating that our channel independence-basee approach is robust against different input data.
Question #4: this one-shot importance determination scheme good enough? Do we need to further Tearn and adjust the pruning mask from the data?
Our Observation. As described above, our proposed scheme calculates the channel independence to identify the filter importance. Considering our approach is built on one-shot calculation, natural extension is to further adjust the importance ranking via additional learning. specific, if we interpret the filter pruning is channel-wise masking operation over the entire weight tensor, the

---

Index batch
50
Index of batch
Index of batch
Incex of betch
-00
40
15
15
15
5 53
10
20
30
10
201
30
Index of fenture map
Index of fenture map
Index of feature map
Index of feature map (d) ReaNct-56_41
(a) ReaNet-56_11
(b) ResNot-56_21
(c) ResNot-56_31
Iniex of batch
60
Indox of batch
88 80
Index of batch
60-
Index of botch
80
40-
40-
15
-0
15
15
10
Index fenture map
Index feature map
Index of feature map
Index of Teature map
(e) ReaNet- 10_11
(1) ResNet-110.22
ResNer-1
ResNet- I 110_41
Index holch
80
Index of hatch
8 8
Index of batch
88
Index of batch
15-
10
La t
L 8
50
1000
50
100
250
500
750
1000
200
400
fenture map
feature map
Index feature map
Index of feature map
(i) ResNet-50_11
(i ResNet-50_21
(0) ResNot-50331
(1) ResNet-50_44
Figure 4: The channel independence of feature maps for one layer in ResNet-50. Here the channel independence is averaged for one batch of input images. The x-axis is the index of the feature map. The y-axis the index of batches of input images. Here the batch size 128. Different colors denote the different values of channel independence. It is seen that the average channel independence very stable regardless of different input data batches.
Algorithm 1 CHannel sandepenndece--baseeee Pruning (CHIP) procedure for the /-th layer
Input: Pre-trained weight tensor W N sets of feature maps A' {Ai. Al, Aly R xhxm from N input samples, and the desired number of filters to be preserved x!.
Output: Pruned weight tensor W Wprune.
1: for each input sample do
2: Flatten feature maps: A' := reshape(A (A' [c', hw]);;
3: for i= 1 to et do
4:
CI calculation: Calculate CT(A() via Equation
5: end for
6: end for
7: Averaging: Average CI(A)) under all N input samples;
8: Sorting: Sort (CT(A!)) in ascending order;
9: Pruning: Prune c¹ rel filters in W corresponding to the d' - n²' smallest CI(A));
10: Fine-tuning: Obtain final W prane via fine-tuning W with removing the pruned filter channels.
selection of channel mask can be learned from data, and such learning process can use the pruning mask determined by our approach as the initialization. Though in principle this learning- based strategy expected to enable additional performance improvement, our empirical evaluations show that the consecutive learning procedure does not easily bring further accuracy increase (with the target compression ratio) or compression ratio increase (with the target accuracy) - more experimental details are reported in Supplementary Material. We hypothesize the reason for such phenomenon is that, our proposed nuclear norm change-based channel independence, though only requires one-time calculation, already identifies and captures the importance of feature maps (and its corresponding filters) with high quality, and hence further learning- based adjustment of pruning mask does not easily provide additional improvement.
The Overall Algorithm. After addressing the above four problems, we can then integrate our proposals and observations to develop the entire filter pruning procedure from the inter-channel perspective. Algorithm I describes and summarizes the overall scheme for our proposed CHannel Independence-basee filter Pruning (CHIP) algorithm.
7

---

Experiments
4.1 Experimental Settings
Baselines Models and Datasets. demonstrate the effectiveness and generality of our proposed channel iilependenne-basee approach, evaluate its pruning performance for various baseline models on different image classification datasets. To be specific, we conduct experiments for three CNN models (ResNet-56, ResNet-1100 10 and VGG-16) CIFAR-10 dataset [24). Also, further evaluate our approach and compare its performance with other state-of-the- pruning methods for ResNet-50 model large-scale ImageNet dataset [5]
Pruning and Fine-tuning Configurations. We conduct our empirical evaluations Nvidia Tesla V100 GPUs with PyTorch 1.7 framework. To determine the importance of each filter, we randomly sample batches (640 input images) to calculate the average channel independence each feature map in all the experiments. After performing the channel independence-based filter pruning, we then perform fine-tuning on the pruned models with Stochastic Gradient Descent (SGD) as the optimizer. To be specific, perform the fine-tuning for 300 epochs CIFAR- 10 datasets with the batch size, momentum, weight decay and initial learning rate as 128, 0.9, 0.05 and 0.01, respectively. the ImageNet dataset, fine-tuning is performed for 180 epochs with the batch size, momentum, weight decay and initial learning rate as 256, 0.99, 0.0001 and 0.1, respectively.
Table 1: Experimental results CIFAR-10 dataset.
Top-1 Accuracy (%)
Method
Params, (1%)
FLOPs (1%)
Baseline
Pruned
A
ResNet-56
- (2016) 27
93.04
93.06
+0.02
0.73M(13.7)
90.90M(27.6)
NISP (2018)
59
93.04
93.01
-0.03
0.49M(42.4)
81.00M(35.5)
GAL (2019)
33
93.26
93.38
+0.12
0.75M(11.8
78.30M(37.6
HRank (2020)
[3]
93.26
93.52
+0.26
0.71M(16.8)
88.72M(29)
CHIP (Ours)
93.26
94.16
+0.90
0.48M(42.8)
65.94M(47.4)
GÄL (2019) [32
93.26
91.38
-1.68
0.29M(65.9)
49.99M(60.2)
LASSO (2017) 211
92.80
91.80
-1.00
N/A
62.00M(50.66
HRank (2020) 31
93.26
90.72
-2.54
0.27M(68. 1)
32.52M(74.1)
CHIP (Ours)
93.26
92.05
-1.21
0.24M(71.8)
34.79M(72.3)
ResNet- 110
1 -orm (2016)
27
93.53
93.30
-0.23
1.16M(32.44
155. (5..00M(8.7))
HRank (2020)
31
93.50
94.23
+0.73
1.04M(39.4 4)
148. 118.70M(4)
CHIP (Ours)
93.50
94.44
+0.94
0.89M(48.3)
121.09M(52.1)
GAL (2019)
[32
93.50
92.74
-0.76
0.95M(44.8)
130. (30.20M(48.5)
HRank (2020)
31
93.50
92.65
-0.85
0.53M(68.77 7)
79.30M(68.66
CHIP (Ours)
93.50
93.63
+0.13
0.54M(68.3)
71.69M(71.6)
VGG-16
SSS (2018) 23
93.96
93.02
-0.94
3.93M(73.8)
133.13M(41.5
GAL (2019)
B2
93.96
93.77
-0.19
3.36M(77.6)
189 49M(39.6)
HRank (2020)
B1
93.96
93.43
-0.53
2.51M(82.9)
145. (55.61((53.5)
CHIP (Ours)
93.96
93.86
-0.10
2.76M(81.6)
131.17M(58.1)
GAL (2019) [32
93.96
93,42
-0.34
2.67M(82.2)
17L38M
HRank (2020)
B1
93.96
92.34
-1.62
2.64M(82.1)
108.61
CHIP (Ours)
93.96
93.72
-0.24
2.50M(83.3)
104.78M(66.6)
HHRank (2020)
31
93.96
91.23
-2.33
1.78M92.0)
73.70M(76.3)
CHIP (Ours)
93.96
93.18
-0.78
1.90M(87.3)
66.95M(78.6)
4.2 Evaluation and Comparison on CIFAR-10 Dataset
Table shows the evaluation results of the pruned ResNer-56, ResNet- 10 and VGG- 16 models on CIFAR-I0 dataset. To be consistent with prior works, we evaluate the performance for two scenarios: targeting high accuracy and targeting high model size and FLOPs reductions.

---

ResNet-56. For ResNet-56 model, our channel independence-basee approach can bring 0.90% accuracy increase over the baseline model with 42.8% and 47.4% model size and FLOPs reductions, respectively. When we adopt aggressive compression with 71.8% and 72.3% model size and FLOPS reductions, we can still achieve high performance our solution enables 1.33% higher accuracy than HRank [11 with the similar model size and computational costs.
ResNet-110. For ResNet-111 10 model, our approach can bring 0.94% accuracy increase over the baseline model with 48.3% and 52.1 1%1% model size and FLOPs reductions, respectively. When we perform aggressive pruning with 68.3% and 71.6% model size and FLOPs reductions, our pruned model can still achieve 0.13% higher accuracy over the baseline model.
VGG-16. For VGG- 16 model, approach can bring 81.6% and 58.1% model size and FLOPs reductions, respectively, with only 0. accuracy drop. Moreover, with 83.3% and 66.6% storage and computational cost reductions, our pruned model model using other pruning approaches under a similar compression ratio. For even higher FLOPSs reduction (78.6%), our method can bring nearly 2% accuracy increase over the prior works.
achieve 1. 1.38% higher accuracy than the
4.3 Evaluation and Comparison on ImageNet Dataset
Table 2 summarizes the pruning performance of approach for ResNet-50 ImageNet dataset. seen that when targeting a moderate compression ratio, our approach can achieve 40.8% and 44.8% storage and computation reductions, respectively, with 0. 0.15% accuracy increase baseline model. When we further increase the compression ratio, our approach still achieves superior performance than state-of-the-art works. For instance, compared with SCOP (5I), our approach shows higher accuracy (0.12% 12%) in moderate compression region and the same accuracy in high compress region; while meanwhile enjoying much smaller model size and fewer FLOPS.
the
Table Experimental results ImageNet dataset.
Top-1 Accuracy (%)
Top-5 Accuracy (%)
Method
Params. FLOPs
Baseline Pruned
A
Baseline Pruned
A
((%)
((%)
ResNet-50
ThiNet (2017)
37
72.88
72.04
-0.84
91.14
90.67
-0.47
33.7
36.8
SFP [19
76.15
74.61
-1.54
92.87
92.06
-0.81
N/A
41.8
Autopruner (2020)
(36
76.15
74.76
-1.39
92.87
92.15
-0.72
N/A
48.7
FPGM (2019) 20
76.15
75.59
-0.56
92.87
92.63
-0.24
37.5
42.2
Taylor (2019)
38
76.18
74.50
-1.68
N/A
N/A
N/A
44.5
44.9
C-SGD (2019)
75.33
74.93
-0.40
92.56
92.27
-0.29
N/A
46.2
GAL (2019)
32
76.15
71.95
-4.20
92.87
90.94
-1.93
16.9
43
RRBP (2019) 61
76.10
73.00
-3.10
92.90
91.00
-1900
N/A
54.5
PFP (2020)
B0
76.13
75.91
-0.22
92.87
92.81
-0.06
18.1
10.8
HRank (2020)
31
76.15
74.98
-1.17
92.87
92.33
-0.54
36.6
43.7
SCOP (2020)
51
76.15
75.95
-0.20
92.87
92.79
-0.08
42.8
45.3
CHIP (Ours)
76.15
76.30
+0.15
92.87
93.02
+0.15
40.8
44.8
CHIP (Ours)
76.15
76.15
0.00
92.87
92.91
+0.04
44.2
48.7
PFP (2020) 30
76.13
75.21
-0.92
92.87
92.43
-0.44
30.1
SCOP (2020)
51
76.15
75.26
-0.89
92.87
92.53
-0.34
51.8
54.6
CHIP (Ours)
76.15
75.26
-0.89
92.87
92.53
-0.34
56.7
62.8
HRank (2020)
31
76.15
71.98
-4.17
92.87
91.01
1.86
46.0
62.1
HRank (2020)
BI
76.15
69.10
-7.05
92.87
89.58
-3.29
67.5
76.0
CHIP (Ours)
76.15
73.30
-2.85
92.87
91.48
-1.39
68.6
76.7

---

5 Conclusion
In this paper, we propose to use channel independence, an inter-channel perspective-motte mottivaeed metric, to evaluate the importance of filters for network pruning. By systematically exploring the quantification metric, measuring scheme, and sensitiveness and reliability of channel independence, develop CHIP, CHannel independence-based filter pruning for neural network compression. Extensive evaluation results on different datasets show our proposed approach brings significant storage and computational cost reductions while still preserving high model accuracy.
Broader Impact
As technology advances, cell phones, laptops, wearable gadgets and intelligent connected vehicles with specific chips are required to handle more complicated tasks by deploying neural networks. However, more powerful networks will cost more memory size and running time, Network pruning is the main strategy to reduce the memory size and accelerate the run-time during the inference stage. Benefiting from pruning techniques and specific designs for hardware [62 4), loT (Internet of Things) devices are able to execute complex projects based on small and efficient models.
Acknowledgements and Funding Disclosure
Bo Yuan would like thank the support from National Science Foundation (NSF) award CCF- 1937403. Saman Zonouz would like to thank the support from NSF CPS and SATC programs.
References
[1] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state neural network pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems, volume 2, pages 129 146, 2020.
[2] Chunhua Deng. Siyu Liao, Yi Xie, Keshab K Parhi, Xuchai Qian, and Bo Yuan. Permdnn: Efficient com- pressed dan architecture with permuted diagonal matrices. In 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 189 202. IEEE, 2018.
[3] Chunhua Deng. Siyu Liao, and Bo Yuan. Permenn: Energy-efficien convolutional neural network hardware architecture with permuted diagonal structure. IEEE Transactions Computers, 70(2):163-173, 2020.
[4] Chunhua Deng, Yang Sui, Siyu Liao, Xuchai Qian, and Bo Yuan. Gospa: An energy-effiiien high- performance globally optimized sparse convolutional neural network accelerator. In 202/ ACM/EEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 1110-1123, IEEE, 2021.
Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-254 Ieee, 2009.
[6] Caiwen Ding. Siyu Liao, Yanzhi Wang. Zhe Li, Ning Liu, Youwei Zhuo Chao Wang. Xuchai Qian, Yu Bai, Geng Yuan, et al. Cirenn: accelerating and compressing deep neural networks using circulln weight matrices. In Proceedings of the 50th Annual IEEF/ACM International Symposiun on Microarchitecture, pages 395-408, 2017.
[7] Xiaohan Ding, Guiguang Ding. Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4943-4953, 2019.
[8] Steven Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra Modha. Learned step size quantization. In International Conference on Learning Representations, 2019.
[9] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
[10] Shangqian Gao, Feihu Huang. Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks. In Proceddings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1899-1908, 2020.
[11] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440-1443 2015.
10

---

[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 580-587, 2014.
[13] Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmep: Differentiahle markow channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1539-1547, 2020.
[14] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, arXiv preprint arXiv: [40,:6490044, 2016.
[15] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:
2015.
[16] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135-1143, 2015.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[18] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2009-2018. 2020.
[19] Yang He, Guoliang Kang. Xuanyi Dong. Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the Twenty-Seven Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 2234-2240, 2018.
[20] Yang He. Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 4340-4349, 2019.
[21] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference Computer Vision, pages 1389-1397, 2017.
[22] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini
2015.
[23] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 304-320, 2018
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.023400, 2018.
[26] Carl Lemaire, Andrew Achkar, and Pierre-Mare Jodoin. Structured pruning of neural networks with budget- regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9108-9116, 2019.
[27] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: /10.6/00000..... 2016.
[28] Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8018-8027, 2020.
[29] Siyu Liao and Bo Yuan. Cireconv: A structured convolution with low complexity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4287-4294, 2019.
[30] Lucas Liehenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provahle filter pruning for efficient neural networks. In International Conference on Learning Representations, 2019.
[31] Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank fcature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1529-1538, 2020.
[32] Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang. and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2790-2799. 2019.
[33] Tao Lin, Scbastian U Stich, Luis Barba, Daniil Dmitriex, and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint arXiv:2006.072. 2020.
[34] Jing Liu, Bohan Zhuang, Zhuangwei Zhuang. Yong Guo, Junzhou Huang. Jinhui Zhu, and Mingkui Tan. Discrimination aware network pruning for deep model compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
11

---

[35] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 3431-3440, 2015.
[36] Jian-Hao Luo and Jianxin Wu, Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107:107461, 2020.
[37] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 5058-5066 2017.
[38] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Juri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1724-11272, 2019.
[39] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646-4655. PMLR, 2019.
[40] Yu Pan, Jing Xu. Maolin Wang, Jinmian Ye, Fei Wang. Kun Bai, and Zenglin Xu. Compressing recurrent neural networks with tensor ring for action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4683-4694 2019.
[41] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3967-3976, 2019.
[42] Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. International Conference on Machine Learning, pages 51 51-55122, 2019.
[43] Mohammad Rastegari, Vicente Ordonez. Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifi- cation using binary convolutional neural networks. In European Conference on Computer Vision, pages 525-542. Springer. 2016.
[44] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01443 2015.
[45] Vikas Sindhwani, Tara N Sainath, and Sanjiv Kumar. Structured transforms for small- mmll--oootprit deep learning. arXiv preprini arXiv:1510.0172: 2015.
[46] Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito,, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error. arXiv preprint arXiv:/808.00 08558, 2018.
[47] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[48] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, PMLR, 2019.
[49] Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33, 2020.
[50] Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5018-50100000 5028, 2021.
[51] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing XU, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. In Advances in Neural Information Processing Systems, pages 10936-10947, 2020.
[52] Rishabh Tiwari, Udbhav Bamba, Arna Chavan, and Deepak Gupta. Chipnet: Budget- pruning with heaviside continuous approximations. In International Conference on Learning Representations, 2020.
[53] Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
[54] Wei Wen, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, pages 2074-2082, 2016.
[55] Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin. and Hongkai Xiong. Deep neural network compression with single and multiple level quantization. arXiv preprint arXiv: 1803.0328, 2018.
[56] Yinchong Yang, Denis Krompass, and Volker Tresp. Tensor-train recurrent neural networks for video classification. In International Conference on Machine Learning, pages 3891-3900, 2017.
[57] Miao Yin. Siyu Liao, Xiao Xino-Yang Liu, Xiaodong Wang, and Bo Yuan. Compressing recurrent neural networks using hierarchical tucker tensor decomposition. arXiv preprint arXiv:2005.04300 2020.
12

---

[58] Miao Yin. Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor decomposition-bases based dnn model compression with optimization framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10674-10683, 2021.
[59] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching Yung Lin, and Larry Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194 9203, 2018.
[60] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using altemattng direction method of multipliers. Proceedings of the European Conference on Computer Vision (ECCV), pages 184-199, 2018.
[61] Yuefu Zhou, Ya Zhang, Yanfeng Wang, and Qi Tian. Accelerate enn via recursive bayesian pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3306-3315, 2019.
[62] Chaoyang Zhu, Kejie Huang, Shuyuan Yang. Ziqi Zhu, Hejia Zhang, and Haihin Shen. An efficient hardware accelerator for structured sparse convolutional neural networks on fpgas. IEEE Transactions on Very Large Scale Inlegration (VLSI) Systems, (8(9):1953-1965 2020.
[63] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang. and Jinhui Zhu. Discrimination aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pages 875 886, 2018.
13


Reviews:
[{"review_id": "yurLg_qrcU1", "reviewer": "Reviewer_TZ9P", "summary": "The authors propose to explore and leverage the cross-channel feature information for filter pruning. Specifically, the authors propose a cross-channel correlation-based metric to measure the importance of filters. In this case, the low-independence feature maps are considered redundant. As a result, the corresponding filters that output the low-independence feature maps are viewed as uninformative and can be safely removed. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of the proposed method. However, the experiments are not sufficient. Please see my detailed comments below.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Contributions**: \n1.\tThe authors propose a new metric to measure the correlation of multiple feature maps. Relying on the proposed metric, the proposed method is able to identify important filters from a global perspective.\n\n2.\tThe authors develop a low-cost fine-grained channel independence metric for efficient filter pruning.\n\n3.\tExperiments on CIFAR-10 and ImageNet show that the proposed method is able to achieve high pruning ratios while still achieving promising performance.\n\n**Questions and points needed to be improved**: \n1.\tCCP [1] also considers the inter-channel relationship during channel pruning. More discussions on the difference between the proposed method and CCP are required. Moreover, it would be better for the authors to provide more discussion on several methods that use gradient-based metrics [2-4] to measure the importance of filters.\n\n2.\tIn the experiments on ImageNet, it takes too many epochs (180) to finetune the pruned models comparing with other methods (20 in SCOP [5]). It is unclear whether the good performance is resulting from longtime training.\n\n3.\tTo demonstrate the effectiveness of the proposed pruning metric, it would be better for the authors to show the accuracy-pruning rate trade-off curve of different methods, which will strengthen the paper.\n\n4.\tSome important details are missing. How to set the pruning rate of each layer? Do the authors set the same pruning rate for each layer?\n\n5.\tIn Algorithm 1, the authors compute the channel independence (CI) criterion among $N$ samples. However, the effect of different $N$ is unclear. It would be better for the authors to conduct more experiments to investigate the effect of different $N$.\n\n6.\tThe authors conduct experiments on heavyweight networks (e.g., ResNet-56, ResNet-110, VGG-16, ResNet-50). However, the performance of the proposed method on lightweight networks is still unknown. It would be better for the authors to conduct more experiments on lightweight models (e.g., MobileNetV1 [6] and MobileNetV2 [7]).\n\n**Reference**: \n\n[1] Collaborative Channel Pruning for Deep Networks. ICML 2019.\n\n[2] Importance Estimation for Neural Network Pruning. CVPR 2019.\n\n[3] SNIP: Single-shot Network Pruning based on Connection Sensitivity. ICLR 2019.\n\n[4] Discrimination-aware Network Pruning for Deep Model Compression. TPAMI 2021.\n\n[5] SCOP: Scientific Control for Reliable Neural Network Pruning. NeurIPS 2020.\n\n[6] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv 2017.\n\n[7] MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please discuss the limitations of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "rfEmRnY5pEn", "reviewer": "Reviewer_Z9Gi", "summary": "To determine the importance of filters in DNNs, this paper proposes a new concept, channel independence, which measures the correlation of multiple feature maps. Also, this paper systematically investigates and analyzes the quantification metric, measuring scheme, and sensitiveness & reliability of channel independence in the context of filter pruning. As a result, the proposed scheme has achieved outstanding performance compared to existing methods. Although this paper is well-written and presents a solution for network pruning that has steadily gained a lot of attention, there are some minor comments as a result of my review of this paper. (Please refer to my main review)", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Scalability/Compatibility of the proposed method (Request for additional experiments)\n\n1-1) Recently, activation functions with negative values such as Swish and Mish are widely used. In general, pruning is vulnerable to activation functions including negative values due to the influence of shift parameters, so please show whether the proposed method has robust properties to these activation functions.\n\n1-2) As you know, fine-tuning requires a significant training cost, so many filter pruning techniques that achieve excellent performance without fine-tuning have been proposed. It is necessary to show whether good performance can be maintained even if fine-tuning is removed from the proposed method.\n\n1-3) It would be good to present the results of various networks for ImageNet dataset such as ResNet-34/101 or MobileNet.\n\n1-4) It is necessary to show whether the proposed method is effective not only for classification but also for more practical detection and segmentation networks.\n\n1-5) Recently, there have been many attempts to perform dynamic pruning rather than static pruning within filter pruning, and therefore it would be good to include the comparison result and analysis with dynamic pruning schemes.\n\n2. Request for additional explanation\n\n2-1) Authors need to present an example in which the difference in the importance of filters occurs in the intra-channel and the inter-channel at the beginning of Chapter 1 or Chapter 3. And if possible, it would be good to present a numerical value indicating the proportion of such filters among all filters.\n\n2-2) It seems that the complexity of including the inter-channel is relatively higher than considering only the intra-channel. Therefore, it is necessary to analyze the complexity.\n\n2-3) Recently, many kernel pruning studies have been attempted, and therefore it would be good to explain the advantages of filter pruning considering inter-channel compared to kernel pruning targeting the kernel in the filter.\n\n2-4) It seems that the baseline accuracy is slightly lower than the values originally presented in ResNet and other papers. Authors should explain how to obtain baseline accuracy.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See my main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "pzA66XbG2d2", "reviewer": "Reviewer_JzDy", "summary": "This paper presents a metric to evaluate pre-trained dense networks to identify the feature maps that need to be dropped in an optimal manner, such that the performance of the pruned network is maximized. Authors present a simple, yet effective approach to identify the feature maps that need to be removed from the network such that it does not compromise the performance. Authors demonstrate the improvement on ImageNet dataset, which clearly demonstrates the efficacy of the proposed metric.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper appropriately addresses the scientific question of how to choose the correct set of feature maps for a pre-trained dense network and prune it. The idea of the paper is clear and the paper is well written. However, I have some comments/suggestions which need to be addressed during the rebuttal.\n\n1. Authors state that the existing methods do not consider the inter-feature relations when choosing the feature maps to be eliminated during pruning. While I am not aware if any such method exists or not, it is important to note that most of the pruning method methods such that add additional objective (pruning-related) do actually consider this aspect in a implicit sense. For example, the “L1-regularization” approach increases sparsity at a layer level or at the entire network level. During this process, it takes an aggregate over a set of feature maps, thus tying them together. Similarly, there exist budget-aware pruning methods (“BAR” and “CHIPNET”) that introduce budget constraints in the pruning process, and this budget ties the different feature maps together during the selection process. My point here is that in such schemes, the mask corresponding to any feature map when goes towards a value of 0, it helps the other maps to go towards 1 in some sens – thus they are not really independent of each other. Authors seem to have completely missed this whole set of literature in their discussion. I would recommend the authors to accordingly soften their claim and add a discussion related to the above methods in the paper. \n\n2. Authors claim that their one-shot static pruning approach prunes the networks in an optimal sense, and state that they have experimented with dynamic learning methods where further to-be-pruned feature maps are identified as a part of the learning process. However, in the supplementary material authors demonstrate this only in 3-4 lines with a figure that contains insufficient details. My personnel experience with static and dynamic pruning has has revealed that no matter how well the static pruning  is done, dynamic learning contrbutes significantly to the pruning results. It is certainly possible that the method proposed in the paper is better and its static nature is good enough, however, this needs to be rigorously evaluated before such a claim can be made. In this regard, I would like the authors to present thorough details for the extensive set of experiments that they have conducted to reach this conclusion.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have not provided limitations of their work. I would recommend that the authors discuss limitations if any. Furthers, authors are recommended to add a line or 2 on the societal impact of their work - for example are there any assumptions made in designing the method that could lead to bias in any form that could adversely impact the society.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "T_W_95g10a", "reviewer": "Reviewer_nHQh", "summary": "This paper proposes a channel pruning method named CHIP to improve the existing channel pruning frameworks. The proposed method measures the importance of channels through channel independence. Extensive experiments show the effectiveness of trunk pruning.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros. \n(1)\tThe paper is well motivated and well-written. The contributions are clearly presented.\n\n(2)\tModel compression is an important topic in scaling deep models on mobile devices. It is essential to design an effective channel pruning method.\n\n(3)\tExtensive experiments are conducted to show the effectiveness of CHIP.\n\nCons.\n(1)\tThe authors utilize the nuclear norm to measure the independence among channels. It is not clear for me to see the relationship between the nuclear norm and correlations between channels. Experiments should be done to show this.\n\n(2)\tThe hyper-parameter $\\mathcal{k}^l$ is crucial to determine the model size of the pruned model. But it is not specified in the paper. Note that the performance of the pruned model is strongly related to the layer-wise configuration of $\\mathcal{k}^l$.\n\n(3)\tSince the motivation is to design the pruning method by exploring the correlation between channels. It would be great to visualize the correlation before and after pruning.\n\n(4)\tIt would be better to see how CHIP performs on MobileNet. EfficientNet, etc.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not address the limitations and potential negative social impact of their work. The authors can discuss how the proposed pruning method affects the development of edge AI.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new pruning method for model compression that utilizes channel independence (CI). CI is given by the change of nuclear norm of the entire set of feature maps between before and after removing a channel. The method is compared with other methods extensively, and it is reported that the proposed method shows a better performance than other existing ones.  

This paper is overall well-written and it is easy to follow. The proposed method can be easily implemented. It is nice that such a simple method gives good performance improvement.  
On the other hand, there are still some concerns as follows:   
1. The authors state that the existing methods do not consider inter-channel correlation. However, it is a too strong claim and ignores several related works. For example, CCP [R1] and Spectral-Pruning [R2] consider correlation between channels, and other methods implicitly takes inter-channel correlation into account. The authors should add discussions about relation to those existing methods, and tone down the claim.    
2. The proposed method utilizes a nuclear norm type criterion to measure channel independence instead of directly measuring channel correlations. The current form of the paper does not give sufficient justification about why the proposed one is appropriate. This point should be discussed properly, for example, by showing the strong correlation between channel independence (correlation score) and nuclear norm in numerical experiments.    
3.  Some details of experimental settings such as the number of preserved filters $k_l$ are missing. These information should be included. I also recommend the authors to include accuracy-pruning rate trade-off curve of different pruning metrics.  

[R1] Collaborative Channel Pruning for Deep Networks. ICML 2019.   
[R2]  Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error. IJCAI 2020.

The concerns I listed above should be addressed. On the other hand, the proposed method shows fairly nice performance which is informative for the community. In summary, I recommend this paper for acceptance, while I think the authors should fix the issues listed above.  
Decision: Accept (Poster)