Automated Channel Pruning with Learned Importance
Authors: Łukasz Treszczotko
Pawel Kubik
Venue: ICLR 2022 Submitted (2022)
Keywords: channel pruning
knowledge distillation

Abstract: Neural network pruning allows for significant reduction of model size and latency. However, most of the current network pruning methods do not consider channel interdependencies and a lot of manual adjustments are required before they can be applied to new network architectures. Moreover, these algorithms are often based on hand-picked, sometimes complicated heuristics and can require thousands of GPU computation hours.  In this paper, we introduce a simple neural network pruning and fine-tuning framework that requires no manual heuristics, is highly efficient to train (2-6 times speed up compared to NAS-based competitors) and produces comparable performance. The framework contains 1) an automatic channel detection algorithm that groups the interdependent blocks of channels; 2) a non-iterative pruning algorithm that learns channel importance directly from feature maps while masking the coupled computational blocks using Gumbel-Softmax sampling and 3) a hierarchical knowledge distillation approach to fine-tune the pruned neural networks. We validate our pipeline on ImageNet classification, human segmentation and image denoising, creating lightweight and low latency models, easy to deploy on mobile devices. Using our pruning algorithm and hierarchical knowledge distillation for fine-tuning we are able to prune EfficientNet B0, EfficientNetV2 B0 and MobileNetV2  to 75% of their original FLOPs with no loss of accuracy on ImageNet. We release a set pruned backbones as Keras models - all of them proved beneficial when deployed in other projects.

21 Efforts directed towards deployment of neural networks on low- pwwfefrrroaccce devices such as mobile phones Under review as a conference paper at ICLR 2022
ACPLI: AUTOMATED CHANNEL PRUNING WITH LEARNED IMPORTANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network pruning allows for significant reduction of model size and latency. How- ever, most of the current network pruning methods do not consider channel interdepen- dencies and a lot of manual adjustments are required before they can be applied to new network architectures. Moreover, these algorithms often based on hand-picked, some- times complicated heuristics and can require thousands of GPU computation hours. this paper, we introduce simple neural network pruning and fine-tuning tuning framework that requires manual heuristics, is highly efficient to train (2-6 times speed up compared NAS-based competitors) and produces comparable performance. The framework contains 1) an automatic channel detection algorithm that groups the interdependent blocks of channels; 2) a non- non-iterative pruning algorithm that learns channel importance directly from feature maps while masking the coupled computational blocks using Gumbel-Softman sampling and 3) a hicrarchical knowledge distillation approach to fine-tune the pruned neural networks. We validate pipeline ImageNet classification, human segmentation image denoising, creating lightweight and low latency models, easy to deploy on mobile devices. Using our pruning algorithm and hierarchical knowledge distillation for fine- tuning are able prune EfficientNet B0, EfficientNetV2 BO and MobileNet 75% of their original FLOPs with no loss accuracy on ImageNet. We release a set pruned backbones as Keras models all of them proved beneficial when deployed in other projects.
2
3
5
8
9
11
20
INTRODUCTION
22 or TVs, created a demand for smaller and faster models. This has led to advances in neural network compression techniques, which allow to minimize existing large-scale architectures and adjust them to fit specific hardware requirements. Some techniques have been especially successful in this area. Neural network quantization approaches (Nagel et al., 2021) not only decreased the size of the models, but also enabled us to utilize specialized computing accelerators like DSPs. Unfortunately, other techniques, such network pruning (Liu et al., 2020), are not equally effective in low-resource environments.
25
27
28 Early attempts naive weight pruning introduced sparse computations, which render them inefficient in practical scenarios (Han et al., 2015; Guo et al., 2016). Channel pruning (Li et al., 2016; Liu et al., 2017; 202 Herrmann al., 2020; Liu et al., 2019b) delivers significant improvements in terms of both memory consumption and execution speed, devices.
31
the preferred approach if we want to deploy our models on mobile
However, the majority of existing approaches to channel pruning share several drawbacks:
34
1. Little effort has been made to address channel interdependencies that occur in the majority of the architectures, with Liu et al. (2021a) a) being a notable exception. Many popular network architectures contain residual connections inspired by ResNet (He et al., 2015). Feature maps added in residual connections must hold the same shapes, which is likely to be violated when channels are removed independently. We refer to channels involved in this kind of dependency coupled. Automating the process adding pruning logic to the network consideration channel interdependencies extremely important in practical considerations.
36
37
39
40
2.
Most methods require an expensive and time-consuming fine-tuning process after channels are removed. Some authors use an iterative approach, where channels are removed a number steps, and fine-tuning is performed between these steps. Either way, the fine-tuning process often requires a significant number of GPU hours complete.
42
43
3. Channels in any given convolution platforms, e.g. SNPE (Qualcomm). are optimized for specific numbers of input and output channels and pruning channels independently can give little to no speed-up.
being considered independently. However, target
46
48 In order to overcome these issues we introduce an end-to-end channel pruning pipeline which can be deployed on wide array of neural networks in an automated way. Our main insights are that: (1) Channel

---

Under review as conference paper at ICLR 2022
Top 1 accuracy on ImageNet
18
23
Do pronee
bil puuurnedd bi, Leacher
Jeather
700
FLOPs M
Figure ImageNet accuracy pruned EfficientNet BO and Considering FLOPs/accuracy trade-off the some pruned models are better than FBNetV2, which to our knowledge has SOTA results in its FLOPs range.
importance can be learned from the feature maps using simple additional networks and no hand-crafted channel importance metric is needed. (2) Neural network computational graphs should be partitioned in 52 way which enables removing channels jointly if they are coupled, c.g., they belong to convolutions whose outputs are later added (the case in point being skip connections). (3) Hierarchical knowledge distillation (a variant of classical knowledge distillation in which multiple teacher networks used consecutively) is the preferred way fine-tuning networks after channels are removed since significantly speeds up training, results in better accuracy and can be used with little or no data augmentation.
55
Our contributions can be summarized as follows:
58
New pruning algorithm. introduce new pruning algorithm which channel importance learned directly from the feature maps. The process of choosing channels is one-shot and requires just couple of GPU hours.
61
Automated method for grouping operations that should be pruned jointly, based on channel interdependencies. We introduce relatively simple way of inserting the pruning logic into networks which allows discard the error-prone process of manual inspection. This makes our solution easy to scale and be deployed for segmentation, detection or image denoising.
63
64
65
Hierarchical knowledge distillation We employ novel approach to fine-tuning pruned networks. The approach is related to the one presented in Mirzadeh et al. (2020). The major insight is to gradually increase the complexity of the teacher network. It leads to much quicker training, and yet, achieves much better final accuracy, while not requiring advanced and time-consuming data augmentation procedures.
86
69
Up to 25% reduction if FLOPs with loss accuracy on ImageNet. We validate our channel pruning pipeline on:
71
EfficientNet (Tan & Le, 2019), MobileNetV2 (Sandler et al., 2018) and EfficientNetV2 (Tan
73
Le, 2021) models for classification on ImageNet;
74
a human segmentation model based on EfficientNet with EfficientDet like Tan et al. (2019) segmentation head;
PMRID (Wang et al., 2020) network for RawRGB image denoising.
77
Figure we can see that for FLOPs between 200M and 300M our prumed models outperform V2 which used, among other things, superkernels to modify existing EfficientNet architecture and produced models that outperform EfficientNet itself. This shows that pruning and fine- tuning pipeline (which is much simpler than NAS algortithms used in Wan ct al. (2020)) can generate better results. Moreover, EfficientNet BO pruned to 75% of its original FLOPs has the accuracy the original model. Interestingly EfficientNet B1 pruned to match EfficientNet BO outperforms BO by approximately I % in top-1 accuracy on ImageNet.
78
80
83
The whole framework of pruning and fine-tuning we introduce in this paper requires little computational resources. The pruning algorithm usually only takes couple hours complete single GPU. Using hierarchical knowledge distillation further speeds up the tne-ttning process.
96
87
2
RELATED WORK
88 Channel selection. Many channel pruning methods employ greedy approach where channel removal
interleaved with expensive fine-tuning of the network (Luo et al., 2018; Liu et al., 2015; He et al., 2017).

---

Under review as a conference paper at ICLR 2022
90
Similar, but a more affordable approach, is to periodically prune channels throughout a single training procedure (Liu et al., a02 Guo et al., 2020; Chen et al., 2020). Ye et al. (2020) and Hou et al. (2021) point out flaws in the idea of greedy channel removal and propose to selectively restore channels in the pruned network. Liu et al. (2019b) trains an auxiliary neural network to quickly evaluate pruned networks and select the best one using an evolutionary algorithm. Other methods jointly train neural network and learn importance scores for its channels using channel gating mechanism. In (Chen et al., 2020), this is achieved by randomly enable and disable channels during each iteration of the training. Gradient descent was used to update the importance scores in Hermann al. (2020); Lin et al. (2020); Ye et al. (2020) and is based on the idea for optimizing hyperparameters in neural architecture search in Liu al. (2019a) and Xie et al. (2018). These gradient-based baaed methods rely on Gumbel-Softmax reparametrization trick (Jang et al., 2016 6) to enable paopppropggg through the gates distribution. Hermann et al. (2020) proposes a variant of such a method where the logits of the channel gates are trainable parameters, as well as variant where the logits are produced by an auxiliary neural network that accepts a feature map. Selecting channels based network input introduces an overhead that is unacceptable on lemiiee devices. Our solution contains a similar idea, but we ensured that the auxiliary networks can be safely removed after the training.
92
93
95
96
90
99
100
101
102
103
104
105
Channel coupling. The channel coupling pattern occurs in many modern architectures inspired by ResNet (He et al., 2015), such as MobileNet (Sandler et al., 2018), EfficientNet (Tan & Le, 2019; 2021) or FBNet (Wan et al., 2020). Many studies seem ignore this issue (Hermaan et al., 2020; Lin et al., 2020; Ye et al., 2020); other resolve this issue by manually grouping interdependent layers or providing model-specific
106
107
108
109
heuristics (Shao et al., 2021; Hou et al., 2021: Guo et al., 2020; Liu et al., 2021b). Independently to our efforts, an automated solution for grouping channels has been proposed in Liu et al. (2021a). We propose a similar algorithm (see section 4), and additionally offer an extension for handling concatenations.
110
111
112
Measuring speed-up. Many pruning methods are parametrised by a fraction of channels to prune, either globally or per-layer (Lin et al., 2020; Ye et al., 2020; Herrmann et al., 2020). Overall network FLOPs! better corresponds the usual business requirements. In Chen al. (2020) and Liu al. (2021a), the maximal FLOPs parameter is included in their stopping criteria and importance scores of channels are adjusted according to their computation cost. Similarly to Guo et al. (2020), we construct a loss function that introduce a penalty for exceeding the provided FL.PSs budget and use a part differentiable importance optimization.
113
114
115
116
117
118
119
Knowledge distillation. It has been noted that Knowledge distillation can perform poorly when there is a large discrepancy in complexity between student and teacher networks (Cho & Hariharan, 2019). Cho Hariharan (2019) evaluate a step-wise approach, in which the intermediate teacher networks are trained by distilling knowledge from the original large teacher and then find it ineffective. Mirzadeh et al. (2020) propose using teacher assistant to bridge the complexity gap. Hou et al. (2021) apply knowledge distillation fine-tune pruned network, but do not address aforementioned issues. We propose an inverted version of the step-wise approach from Cho & Hariharan (2019), and train train our pruned network with increasingly larger teachers. Such chains can be naturally formed for model families like EfficientNet (Tan & Le, 2019) and EfficientNNeV2 (Tan & Le, 2021). We also observe that in case of generic knowledge distillation, the final results can be improved by (even slightly) disturbing the student model with channel pruning before starting the distillation.
120
121
122
123
124
125
126
127
128
129
3 PRUNING METHOD
130
131
The basic idea behind our channel pruning algorithm is to set up scheme in which the importance of channels is being learned from the feature maps generated by convolutions in neural networks. We assign each channel score corresponding to its importance that updated at each training step and used to approximate behavior of the prumed network by appropriate masking (Liu et al., 2017; Herrmann et al., apply probabilistic approach where channels in feature maps are masked with samples from random variables with values in (0, 1). This is a continuous relaxation approach to solving a discrete problem. The distributions of these random variables depend on the values of corresponding logits (which can be though of proxies for channel scores and have values in R). These logits are learned during the pruning stage. More precisely, given a feature map of size (B, H. W. C) (B is batch size, H/ and W are spatial dimension and C' is the number of channels) and a logits variable, for each channel separately parametrized by the corresponding logit in logits. We mask the feature map by multiplying it by the sampled values.
132
133
134
135
2020). Similarly to Herraan et al. (2020)
136
137
139
139
140
141
sample- using Gumbel-Softmas (Jang et al., 2016) the random variable
142
143
144
We do not consider each feature map individually - instead, we extend our understanding of channels from 145 a single feature map series of operations occurring within network. The intuition is that element- wise operations, like activation functions, propagate channels forward throughout the network, while convolutional layers consume their input channels and create new ones, Pruming sequential models is trivial but in more complicated cases, like models with residual connections, there exist additional couplings between channels, introduced by operations that accept multiple inputs, e.g. element- wise sum, multiplication (Fig. 2). Because
148
147
148
148
a number of floating-pe point operations

---

Under review as conference paper at ICLR 2022
150
coupled channels must be pruned jointly to ensure valid shapes, we use single random variable to mask each set of coupled channels (see Section 4 for details about automatic detection of coupled channels).
152
Although logits be treated standalone trainable variables, choose learn them from the feature maps in a feedback-loop mechanism. This is because the latter approach is faster to train, results logits which (once converted to probabilities) have lower entropy and produces better results. Once we decide on the feature maps from which will learn the optimal logits values, place simple neural networks called logit predictor modules
153
154
158
take these feature maps as inputs. These modules are build of 3x3 depthwise convolution followed by Ixl convolution and global mean pooling along spatial dimensions. The output output vector of each such module is later used to update the value of the corresponding logits variable (using exponential moving average) as in Figure
157
158
159
160
The masking operations should always be placed just before the convolution operations that absorb the channels (see Figure 2). The placement of logit predictors is more involved and in cases more complicated than the relatively simple one presented in Figure 2, we choose to follow simple heuristic to place them after convolutions with largest kernel sizes.
182
163
184
During the pruning phase we augment the task-specific loss with an auxiliary lutency-based loss, It is based the expected number of FLOPs in the pruned network, which is computed by using all the logits have attached network. We train network weights and logit predictor modules jointly so that the network can adjust to channels being phased out.
168
167
Logit
channel
Predictor
logits
moving average update
Conv
Activ.
Sum
Conv
Conv
Activ.
Gumbel-
Softmax
masking
Figure 2: An subset of a network with logit predictor and masking. The colors indicate the correspondence between channels. The logit predictor takes a feature map produced by the sum operation and use to predict an update for the channel logits.
168
3.1 PRUNING LARGER BLOCKS OF CHANNELS
189
We allow for blocks of channels (instead of just individual channels) to be treated jointly, so that blocks of predefined size will be chosen discarded together. This is especially important for platforms where convolutions are optimized with a specific block size channels mind, e.g., for SNPE (Qualcomm) this number is 32 and pruning individual channels often makes little sense.
171
172
173
LAYER GROUPING ALGORITHM
174
Although channel coupling has been observed the literature, relevant groups operations seem to be usually established via network-specific heuristics or manual annotation. A notable exception is Liu et al. (2021a) where the problem is described at length and algorithm for finding the groups is derived. The algorithm is then tested on architectures based on ResNet. However, unlike our solution, it not support concatenation operations. For clarity. we focus on convolutional neural networks, but the proposed strategy be extended to other kinds of architectures.
175
178
177
178
180
SOLUTION
To overcome the issues delineated in Section 3 and make channel pruning available for most off- shelf architectures have developed an algorithm that capable of automatically detecting channel interdependencies between feature maps generated by operations in the network.
182
183
184
To keep track of all the places where channels have to he considered in a synchronised way, we introduce the concept of an orbit. An orbit can be thought as subset of operations that are interdependent from the point of considered jointly when removing channels. Naively removing channels without taking into account these interdependencies may result in an invalid network. For example, if we remove output channel from one of the convolutions the left in Figure 2,
185
196
view of channel pruning. Operations the same orbit need
187

---

Under review as a conference paper at ICLR 2022
C1
C4
C1
C4
C2
concat
C5
C2
concat
C5
C3
C6
C3
C6
extended orbit
(b) final orbits (red and blue)
Figure 3: Breaking up an extended orbit. An extended orbit is broken up into two final orbits. Nodes C'1 and C2 must have their channels pruned jointly. Node C3 can be pruned separately.
189
the number of channels will no longer match for the Sum operation. A typical network has multiple orbits. It is easiest to understand this concept by seeing how orbits are build, which we delineate in Algorithm 1 below.
190
191
192
First, we fix some notation to make matters more intuitive. All operations a typical convolutional neural network can be described as being of the following types:
193
194
1.
sources are the operation where new channels are being created, namely regular convolution layers (not depthwise!) and dense layers;
195
196
sinks are the operation where channels are being absorbed, namely regular convolution layers (not depthwise!) and dense layers;
197
3. continuators all the operations with single input tensor that simply pass the channels forward, c.g., batch normalization, mean pooling, resize, activations;
189
200
4. joiners are operations with multiple input tensors of the same shape which join these tensors without altering the shape, namely element-wise addition and multiplication;
201
Typically, continuator operations
not problematic since they do not alter the channels structure and have a single predecessor and a single output. is the joiner operations that introduce interdependencies between channels. For brevity, from now on we will only speak of convolutions as sources and sinks, but everything applies just as well to dense layers.
203
204
208
Note that some sources can be sinks at the same time and vice versa. We refer operations that are either sinks or sources as source-sinks. To identify all the subgraphs the network where channels have to be considered jointly we run an exhaustive-search type algorithm which has two distinct phases:
207
208
In the fist phase search for extended orbits, where the coupled operations brought together. In Algorithm we describe how extended orbits are created. The input neural network directed acyclic graph (DAG). The algorithm amounts to removing all inbound edges from convolution nodes and finding all weakly connected components in the resulting graph. The extended orbits are then these weakly connected components once we restore the inbound edges convolution nodes.
210
211
213
214
The second phase similar to first one. take the extended orbit and then mark concatenation nodes (which play special role, since they group channels from separate sources) inside sinks and repeat the process. Most notably,
extended orbits found in phase one we do the following:
215
discard extended orbits in which there are concatenation nodes followed by joiner nodes, as it makes the whole process much more difficult to implement. We do not prune channels within such orbits. In Figure we give an example of extended orbit and how is broken up into final orbits.
217
218
219
Algorithm Scarching for extended orbits
Input: network DAG with layers represented as nodes
1: := {p= P path starting and ending with a convolution with no convolutions inside the path }
2: for each path p in P remove the last node
3: for every distinct node nı paths in P, create empty color set for the node Cn.
4: X ; {z x the initial node a path P
5: for 2' in X do
6: pick unused color c
7: add color e color sets of all the nodes on all the paths starting in
8: end for
9:
while there exist nodes with multiple colors do
pick node with multiple colors (c1, .2k ) at random
11: if any node the DAG has a color in (C2...., Cx) switch the color to ei
12: end while

---

Under review as a conference paper at ICLR 2022
220
5
PRUNING, FINE-TUNING AND HIERARCHICAL KNOWLEDGE DISTILLATION
221
5.1
PRUNING STAGE
222
The pruning workflow is the same for all types of tasks. We first find all final orbits in the network and attach logit predictors. Final orbits determine both: which parts of the network are being pruned and which of them are pruned jointly. The FLOPs per pixel can be automatically computed (and are differentiable with respect to the channel logits as in (Fig. 2). We can compute FLOPs for the original network and then set some FLOPs target. In practice we compute PFPP (FLOPs per pixel the input tensor divided by 1000), to have
223
224
225
226
a value that is independent of the input size. The latency loss is then given by ReLU "target_KFPP kFPP - 1). We add this loss to the quality loss related to the task, e.g., cross entropy in classification. To avoid an overly aggressive reduction of PPP,, we anneal the loss using exponential decay so that the beginning of training the annealing multiplier is O. and approaches 1. as the training progresses.
227
220
229
230
Once the pruning phase is over we retain or discard output channels in convolutions based on channel interdependence discovered by applying Algorithm and the values of logits variables learned by logit
232
233
predictors.
234
5.2
HUNING AND HIERARCHICAL KNOWLEDGE DISTILLATION
235
We propose to fine tunetune pruned models with a method we call hierarchical knowledge distillation. This approach relies on increasing the complexity of the teacher network in discrete steps. Given a fine-tuning budget of K GPU hours, and N teacher networks we train the network for K/N GPU hours with each of these teacher networks, starting with the smallest one. Our loss is Los + 3Lad where Les the standard cross entropy loss and Led is the distillation loss. Using higher weight term for the Lid is crucial to prevent overfitting and produce better results.
236
238
239
Hierarchical knowledge distillation consistently performs much better than just using the original model the teacher. The comparisons can be seen in Section 6.2. Given an array of models with increasing FLOPs requirements, like EfficientNet Tan & Le (2019) and EfficientNetV2 Tan & Le (2021). ), it is possible to cheaply train new models for missing FLOPs values. This may produce better results in terms of FLOPs/accuracy trade-off and require less computational resources.
242
243
244
245
246
It is perplexing that trying to use hierarchical knowledge distillation on an unpruned network does not work anywhere near as well. Our intuition is that pruning provides some kind of initial perturbation to network weights and architecture which prove beneficial from the point of view of gradient descent optimization. Are there any other types of model perturbations which boost the effectiveness of this type of knowledge distillation? These are the questions we could try to address as our future research. It would be also interesting see how this approach performs when applied recent state-of-the-arr methods based on neural architecture search Wang et al. (2021).
247
248
248
250
251
252
253
6
EXPERIMENTS
254
All the experiments we perform adhere to the same schedule: (1) We first run the pruning algorithm with additional latency losses (usually 1-10 epochs, depending on the task). (2) We then fine-tune the pruned model (without resetting its weights). The experiments for classification on ImageNet presented in
255
256
257
Section 6.2. Experiments for image denoising and human segmentation are presented in Sections A.2. and
258
A 2.2 respectively.
259
6.1
HYPERPARAMETERS FOR THE PRUNING PHASE
260
For the pruning phase, during which channels to be removed are being chosen, the setup is roughly the same for each task. The logits predictor is always two layer network with 3 x 3 depthwise convolution followed by 1 convolution and global mean pooling. We set the batch size to 16 and the training updating the
262
263
channel gates distributions as described in section 3. The initial value of channel logits is set to 3.0 so that
264
initially there little to no masking. There is an additional loss that penalizes the entropy of all the logits so
285
that at the end of the pruning phase the channel enabling probabilities (which we get by applying softmax logits) are far away from 0.5. The temperature for Gumbel-Softmal is constant - 0.5.
268
287
6.2
CLASSIFICATION ON IMAGENET
268
We prune EfficientNet BO, EfficientNet B1 (Tan & Le, 2019). MobileNetV2 (Sandler al., 2018), and EfficientNerV2 (Tan & Le, 2021). We choose these since they are already highly optimized for mobile devices and relatively small. FfficientNerV2 is a recent state-of-the-art architecture optimized for mobile GPUs and
269
270
271
DSPs. All the models are taken from their official Keras implementations' except for EfficientNetV2. Larger

---

Under review as a conference paper at ICLR 2022
Table I: Top
ImageNet accuracy and FLOPs for for EfficientNet B0 and BI pruned
(a) B0 pruned
Standard training
B1 teacher
Model
BO teacher
FLOPs (G)
(after using BO first)
original
77.30
0.393
m&
73.49
73.72
0.156
m2
74.88
75.00
0.197
m10 m22
76.01
76.56
0.248
76.54
77.30
0,299
NPRR Hou et al. (2021)
77.00
0.346
(b) B1 pruned
Standard training
B2 teacher (after using B first)
Model original ml
BI teacher
FLOPPs (G)
79.10
0.700
77.02
76.99 77.74
0.299
77.26
0.348
m16
77.66
78.35
0,400
272
networks like the VGG19 or the ResNet family has been predominant in channel pruming literature, but are rarely suitable for resource-limine limited devices, where the need for optimization is biggest. The phase where channels chosen usually lasts little more than single epoch on ImageNet. We split the ImageNet train data into two parts, leaving about
273
275
the data for early-stopping.
276
Following Section 5.2 we use multiple teacher networks. The details are as follows:
EffliceentNet BO: fine-tune the models for 40 epochs with BO teacher and then further fine-tune with a B1 for another 40 epochs;
278
279
EfficientNet B1: fine-tune the models for 25 epochs with B1 as teacher and then we further fine-tune with B2 for another 25 epochs.
280
281
Mo1ileNetV2: fine-tune the models for 40 epochs with MobileNetV2 as teacher and then we further fine-tune with EfficientNet BO for another 40 epochs.
283
EfficientNetV2 BO: fine-tune the models for 16 epochs with BOV2 as, then fine-tune the models for 16 epochs with B1V2 as teacher and finally fine-tune the models for epochs with B2V2 as teacher.
284
295
The interesting thing noticed is that using knowledge distillation without pruning does help all. For example we tried fine-tuning MobileNetV2 with EfficientNet BO teacher right away and top Imagenet accuracy fell from 71.52% to 71.12%. We conjecture that some kind of initial perturbation is needed for knowledge distillation work. In case this perturbation is channel pruning.
287
298
299
290
Batch size is to 192 for BO and MobileNetV2 fine-tuning. For B1 and EfficientNet BO batch size 128. The input image resolution is (224, 224). We use only random crop and flip as augmentations. For training we use one NVidia RTX3090 GPU. For the pruning phase we set the batch size to 16 and, quite importantly, we freeze all batch normalization layers. We use Adam optimizer for all the training runs. During lask-ne phase the learning rate is set to 0.0001. For fine-tume tuming we use exponential decay with learning rate initially set to 0.0001 and the decay rate set to 0.001.
291
292
293
294
296
2.
COMPARISONS AND DISCUSSION
297
Few authors have attempted to prune EfficientNet (Tan Lc, 2019). We can compare our results with Hou al. (2021), where only one model is presented, which was also fine-tuned with knowledge distillation. We provide much wider FLOPss spectrum for BO and prune BI as well. It interesting the FLOPs level of BO outperforms BO by wide margin. The results are Table 1.
298
298
that BI pruned
300
301
Comparisons for MobileNetV2 are quite difficult due the inconsistencies between different versions of the model taken by different authors their baseline. For instance in Hou et al. (2021) the authors first take an over prened backbone which they proceed to prune. In Liu et al. (2019b) the largest version of MobileNetV2 is taken (585M FL.OPs) and then pruned. Some of the authors run the fine-tuning for much longer than we do. Notably, in Ye et al. (2020) the fine-tuning is run 4 GPUs with batch size 512 and for 250 epochs which is considerably more expensive than our approach. Detailed results are in Table 2 and Figure 5a. Again using hierarchical knowledge distillation we are able to fine-tune the model pruned to 75% % of original FLOPs so
303
304
306
307
that it has 0.7% higher accuracy than the original.

---

Under review as conference paper at ICLR 2022
Top accuracy on ImageNer
Ciooestee praned 31 tarcher
ImageNe
T0p accuracy
FLOPs
FLOPs M
(a) EfficientNet BO pruned
(b) EfficientNet B1 pruned
Figure 4: ImageNet accuracy pruned EfficientNet BO and B1. Considering FLOPs/aceuracy trade-off the some pruned models are better than FBNetV2.
Table 2: Top
ImageNet accuracy and FLOPs for MobileNetV2 pruned
teacher (after using MobileNetV2 first)
Standard training
MobileNetV2 teacher
Model
FLOPs (G)
original
71.52
0.301
m5
67.58
67.99
0.135
GSPE Ye et al. (2020)
68.8
0,138
m6
67.08
68.76
0.140
META Liu
(2019b)
68.2
0.140
GFP Liu et al. (2021a)
69.16
0,150
GSPE Ye et al. (2020)
69.7
0.152
m7
69.79
70.05
0.170
GSPE Ye et al. (2020)
70.4
0,170
m8
69.47
71.28
0.199
GSPE et al. (2020)
71,2
0,201
GSPE et al. (2020)
71.6
0,220
m9
70.92
72.22
0.228
When it comes EfficientNetV2,, able to outperform the original model's results ImageNet with the help hierarchical EKD, inasmuch as pruned version of BO (70% FLOPs of the original model) has higher top accuracy than the original. See Table 3 and Figure 5b.
310
CONCLUSION
312
313
Using an automated solution to process coupled channels in neural network architectures and simple scheme learn channel importance,
able prune models with varying architectures for different underlying tasks. For fine-tuning pruned classification networks we use hierarchical knowledge distillation
315
318
which produces much better results than just using the original model as a teacher. The whole pruning pipeline requires much less computational resources than some of the state-of-the-ee NAS based solutions for finding efficient FLOPs/ accuracy trade-off's like Wang al. (2021).
318
100 5000
Pead/Wor Fee www...u PUUNCC
(a) ImageNet accuracy of pruned MohileNetV2, For finetun- ing we use knowledge-distille with both original model and EfficientNet B0 as teachers.
PLS/
(b) RfficientNetV2 B0 pruned
Figure 5: Pruning results for MobileNerV2 and EfficientNet22

---

Under review as a conference paper at ICLR 2022
Table 3: Top ImageNet accuracy and FLOPs for EffiicientNeV22 BO pruned
Model
Standard training
B0 teacher
hierarchical teachers
FLOPs (G)
original m22
78.67
0,722
77.59
78.93
0.506
m17
77.25
78.37
0.431
m15
76.70
77.64
0,379
m12
75.59
76.36
0.299
319
REFERENCES
320
Zhiqiang Chen, Ting-Bing Xu, Changde Du, Cheng-Lin Liu, and Huiguang He. Dynamical channel pruning by conditional accuracy change for deep neural networks. IEEE trunsactions neural networks and learning systems, 02(2):799-813 2020.
322
323
J. Cho and B. Hariharan. On the efficacy of knowledge distillation, pp- 4793-4801, nov 2019. doi: 10.
324
1 1109/CCV.2019 9.099. URL https: wwwp::eewwwwete.eeeeeeiioccom wwtp::/sssttttttttoteeeoooe wtt. 1109//..11......
325
2019 9, 00489.
326
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self-supervised structure-sensitive learning and new benchmark for human parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
327
328
328
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1539-1547, 2020.
330
331
332
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dons. arXiv preprint arXiv:1608.c 04493, 2016.
333
334
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.02626 2015.
335
336
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
337
338
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
339
340
Charles Herrmann, Richard Strong Bowen, and Ramin Zabih. Channel selection using gumbel softmax. In European Conference on Computer Vision, pp. 241-257. Springer. 2020.
341
342
Yuenan Hou, Zheng Ma, Chunxiao Liu, Zhe Wang, and Chen Change Loy. Network pruning via resource reallocation. CoRR, 10s/2103.01 2021. URL wttp:///wwwwwwwwaaaae/eeooe 22103333 01847.
343
344 Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 16 144, 2016.
346
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 08710. 2016.
347
348
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. CoRR, abs/2002.10179, 2020. URL https: //axxiv. org/ abs 20020000222 10179.
348
350
351
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp-
352
353
806-814, 2015.
354
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2019a.
355
Jay Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey. arXiv preprint arXiv:2005.0445 04475, 2020.
356
357 L Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7021-7032. PMLR, 18-24 Jul
358
359
350
2021a. URL https: /pwpceeeiiggs..../////ppp./...sp mlr, press/ v139 1111/11/llllii 1iu2lab html.

---

Under review as a conference paper at ICLR 2022
362
Xiangcheng Liu, Jian Cao, Hongyi Yao, Wenyu Sun, and Yuan 7hang. Adapruner: Adaptive channel pruning and effective weights inheritance. arXiv preprint arXtv:2109.06397 2021b.
384
Zechun Liu, Haoyuan Mu, X. Zhang, Zichao Guo, X. Yang, K. Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3295-3304, 2019b.
385
366
367
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-274- 2017.
368
369
370
Jian- Wian-Hac Hao Luo, Hao Zhang, Hong- Yu Zhou, Chen- Wei Xie, Jianxin Wu, and Weiyao Lin. Thinet: pruning cnn filters for a thinner net. IEEE transactions pultern analysis and machine intelligence, 41(10);
371
372
2525-2538, 2018.
373
Seyed Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. Proceedings of the AAAI Conference on Artificial Intelligence, 34:5191-5198, 04 2020. doi: 10. 1...669/aaai.v34444.555. v34i04.5965
374
375
376
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021.
377
378
Qualcomm. Snpe: Snapdragon neural processing engine. attps://develooer..c
com/
379
httees/ddfauut///eee aallaauttffllssee htt///lllls///eeeeee docs/ snpe/.
380
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiee Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CoRR, abs/1801.044 .04381, 2018. URL http: //arxvaaavvv.v//aa///a/ wwwg/bs/////eee.eeeee/..ee//eeee 1801. 04381.
382
383
Wenqi Shao, Hang Yu, Zhaoyang Zhang, Hang Xu, Zhenguo Li, and Ping Luo. Bwep: Probabilistic learning-to-prune channels for convnets via batch whitening. arXiv preprint 1eXiv:2105.00 06423, 2021.
384
385
Mennatullah Siam, Heba Mahgoub, Mohamed Zahran, Senthil Yogamani, Martin Jagersand, and Ahmad EI- Sallab. Modnet: Motion and appearance based moving object detection network for autonomous driving. In 2018 st International Conference on Intelligent Transportation Systems (ITSC); pp. 2859-2864, 2018. doi: 10. 1.111//TTC...01..85560
386
387
389
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CORR, abs//900.1199990 2019. URL http://arxiy org/ abs 190//995...1 10..9...11.6... 11946.
390
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training. CoRR, abs/2104.00298,
392
2021. URL https: / arxiv, .rw.//orgraaaa.. attaa/aa/221000 /2004. 98.
393
Mingxing Tan, Ruoming Pang, and Quoe V. Le. Efficientdet: Scalable and efficient object detection. CoRR, abs/1911 0bs//9911.0900 109070, 2019. URL / arxiv v. wttg/ass/////eee 09070.
394
395
Alvin Wan, Xiaoliang Dai. Peizhao Zhang, Zijian Hc, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu. Tao Xu, Kan Chen, Peter Vajda, and Joseph E. Gonzalez. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
396
397
399
Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra, Alphanet: Improved training of supernet with alpha-divergence. arXiv preprint arXiv:2102.0799 0795444 2021.
400
401
Yuzhi Wang, Haibin Huang, Qin Xu, Jiaming Liu, Yiqun Liu, and Jue Wang. Practical deep raw image denoising mobile devices, 2020.
402
403
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv: 1872.0992, 2018.
404
405
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. CoRR, abs/2003.01794, 2020. URL https: 017944
408
407
/
10

---

Under review as conference paper at ICLR 2022
408
A
APPENDIX
A.I LAYER WIDTHS VISUALIZATION
410
how layer width looks like after pruning. The pattern that emerge are quite 411 telling. EfficientNets are build of a series of meta-blocks. .egg 2,3, ...... 7 in EfficientNet BO, where each meta-block consists of a number of MBCONV blocks the same spatial resolution. It appears that in each 413 such meta-block the most important block usually the first one, and block importance decays proportionally to the depth of the block inside the meta-bloct See Figure 6 in the Appendix.
is quite interesting
414
205
(a) EfficientNet BO pruned
(b) EfficientNet BI pruned
Figure 6: Visualisation of the layer width after channels removed. There noticeable patter in which the first block in a series of residual blocks at the same spatial resolution is the most important one and the algorithm is reluctant to remove the channels. Later blocks seem to be less informative, proportionally to their depth.
415
A.2
FURTHER RESULTS
416
A.2.1
RAWRGB IMAGE DENOISING
417
We prune a recent state-of-the-art network for RawRGB image denoising on mobile devices introduced in Wang et al. (2020). We train the models on SIDD Medium dataset https: /wwww eecs. yorku. ha/-kamell hameel/sidee sidd/ dat php. We first extract 256x256 patches for training and validation and then test the networks on SIDD validation dataset https: / www. eecs. yorku. ca/ ~kame s dd benchmare rk php. The batch size is set to 16, learning rate is 0.0001 and we use Adam optimizer. The loss is mean absolute error. We train the original model for 150 epochs, prune it and then train the original model for another 150 epochs. The pruned models are fine-tuned for 150 epochs as well. For comparison we also train from scratch smaller (linearly scaled down) versions of the original model. The results can be seen Table 4 and Figure? 7.
418
420
421
423
424
426
A.2.2
HUMAN SEGMENTATION
427
For semantic segmentation we use a private dataset for training human segmentation models for real time prediction in video bokeh task. This is dictated by the need to have superior edge quality which is missing in publicly available data for segmentation. The dataset consists of 120k real image/mask pair and 50k synthetic ones. Apart from loU also compute edge IoU, which pays attention only the edges of the masks and can be thought of as a proxy for edge quality. The baseline architecture consists of an EfficientNet
428
430
431
Table 4: Pruning results for image denoising and human segmentation.
(a) PSNR and kFPP for pruned PM-
RID model
(b) Pruning results for pruned human segmentation model.
Model
PSNR
kFPP
Model
loU
Edge loU
kFPP (G)
baseline
51.84
29,9
baseline
0.9414
0.4039
40.8
m25
51.84
25.4
m30
0.9440
0.3977
30.3
m22
51.79
22.1
m25
0,9423
0.3848
25.4
m2
51.80
19.6
m:22
0.9431
0.3816
22
m5
51.67
15.1
19
0.9420
0.3574
19.3
m12
51.41
12.3
m15
0.9372
0.3354
14.6
mll
0.9295
3213
11.3
m&8
0.9253
0.2939
7.5
m5
0.9050
0.2265
4.5
11

---

Under review as conference paper at ICLR 2022
$2.0
PNRD
Interly stated
$1.9
PMRID pruncd
Valisation PSNR
53.4
51.3
10.2
KFP>
Figure 7: Validation results for pruned RawRGB denoising models.
669
0:42
0.40
070
(a) IoU
(b) Edge IoU
Figure 8: Validation results for pruned human segmentation.
432 BO (Tan & Le, 2019) backbone, EfficientDet (Tan et al., 2019) (modified slightly to allow for easier channel pruning) fusion block and detail branch (Siam et al., 2018) to preserve edge quality. The backbone network pretrained on ImageNet. We train the original model for 70 epochs, prune and then fine-tune models for 50 epochs. The validation results are presented in Table 4. The validation dataset is split of a modified version of LIP dataset (Gong al., 2017), where objects belonging people (such handbags,
434
pruned
435
437
etc.) are also considered part of these people. This done, so that we can train models for video bokeh effect. The results are in Table 4b and are visualized in Figures 8a and 8b.
438
Notice that the smallest pruned model is compressed to around 10% of the size of the original one. Even in these extreme compression scenario approach produces model with IoU higher than 90%. ToU starts dropping only after we have removed more than 60% the original FLOPs. This is an observation which, our experience, is true for many more architectures for segmentation, the one being presented here is just one example. Edge loU starts falling much more quickly, perhaps beacause employ no sedeificccffcc loss.
441
442
443
12


Reviews:
[{"review_id": "wisNMfjLjd8", "reviewer": "Reviewer_b957", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Pros\n- This paper aims at solving the coupled channels in channel pruning, which is important in practice and needs more investigations.\n- The evaluated tasks (image denoising and human segmentation) seem to be quite interesting and seldomly discussed in related pruning works.\n- It is good to observe and conclude that KD only works for pruned networks but not for unpruned (original) networks. However, more investigations are welcomed to make confident claims and hypotheses.\n\n#### Cons\n- There are only a few baseline pruning methods compared. Meanwhile, results on ResNet should be compared. Otherwise, it is difficult to judge the effectiveness of the proposed method.\n- The tables show merely the same results as those in the figures (e.g., Table 1 & Figure 4, Tables 2/3 & Figure 5.\n- No ablation studies. It seems that the most performance gain is from the hierarchical KD, which is not exclusive to other pruning methods. Applying hierarchical KD seems not to have enough methodological novelty or contributions.\n- The writing clarity should be moderately addressed. For example, the descriptions of the two phases in Section 4.1 are quite informal and vague, and the relation between Algorithm 1 and the described steps in the main text is also unclear. Also, it is worthy to explain why \"concatenation\" is unique to others and important in your pruning method (if so).\n\n#### Details\n- It is not clear that how the proposed method, compared with baselines, uniquely solves the issue of pruning \"specific numbers of input and output channels\".\n- The claim that the proposed method is \"easy to scale and be deployed for segmentation, detection or image denoising\" needs more explanations.\n- It is not fully correct to claim \"between 200M and 300M our pruned models outperform FBNetV2\" from Figure 1 since FBNetV2 obtains an accuracy of ~76 with fewer FLOPs.\n- For the four operation types introduced by the authors, which one does the \"concatenation\" belong to?\n\n- Minor issues \n  - In Abstract: \"We release a set pruned ... \" -> \"a set of\"\n  - In Section 2: \"Selecting channels based network ...\" -> \"based on\"\n  - In Section 2: \"and train train our pruned network ...\" -> \"and train our pruned network\"\n  - In Section 3: \"The output output vector of ...\" -> \"The output vector of\"\n  - Figure 2: \"An subset of ...\" -> \"A subset\"\n  - In Section 3.1: \"... a specific block size o channels ...\" -> \"of\"\n  - In Section 4.1: \"In the fist phase\" -> \"first\"", "summary_of_the_paper": "This paper proposes a new channel pruning method, which firstly finds the coupled channels in the network, then prunes channels with learned logit predictors, and finally uses hierarchical knowledge distillation (KD) to fine-tune the pruned network. Empirical evaluations have been conducted with EfficientNets and MobileNets on ImageNet, an image denoising dataset, and a private human segmentation dataset.", "summary_of_the_review": "This paper solves an important practical problem in channel pruning with interesting solutions and results. However, in terms of novelty, clarity, and evaluations, this paper needs to be polished and completed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3d6y_UYyNZ", "reviewer": "Reviewer_Www2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The proposed pruning scheme solves the dimension mismatching problem, when pruning some specific model architecture (such as ResNet).\n2) The hierarchical KD can promote the performance, which avoids the problem: “too large teacher does not always make better student”.\n\nWeakness:\n1)\tAlthough each part of the proposed method is effective, the overall algorithm is still cumbersome. It has multiple stages. In contrast, many of existing pruning methods do not need fine-tuning.\n2)\tTechnical details and formulations are limited. It seems that the main novelty reflected in the scheme or procedure novelty. \n3)\tThe experimental results are not convincing. The compared methods are few. Although few authors have attempted to prune EfficientNet, other networks can be compressed in experiments such as ResNet. In addition, the performance gains compared with SOTAs are also marginal, which are also within 1%.\n4)\tThe paper is poorly written. There are many typos and some are listed as follows:\n       --In caption of Figure 2, “An subset of a network” should be “A subset of a network”.\n       --In Line157 of Page4, “The output output vector” should be “The output vector”.\n       --In Line283 of Page7, “B0V2 as,” should be “B0V2 as teacher,”.\n       --In Line301 of Page7, “due the inconsistencies” should be “due to the inconsistencies”.\n", "summary_of_the_paper": "This paper proposes a neural network pruning and fine-tuning framework for model compression. It can automatically prune the channels by learning the channel importance. The contributions are: 1) A new pruning scheme is proposed by learning the channel importance; 2) The pruning logic is introduced in the pruning scheme. Thus, some grouping operations are pruned jointly. The pruning problems of residual connections in ResNet can be solved; 3) Hierarchical knowledge distillation is added in the fine-tuning phase to speed up training. Experimental results show effectiveness of the proposed method.", "summary_of_the_review": "This paper proposes a new neural network pruning and fine-tuning framework for model compression. Each part of the proposed method is effective and can solve some challenges. However, the novelty and the technical details are limited, and the experimental results are not sufficient (see weakness). Therefore, my rating is marginally below the acceptance threshold. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3c_lCEPPD-r", "reviewer": "Reviewer_Gsnx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The research about channel pruning is insufficient, with only six related works, and the author should summarize the differences with them. \n2) Hierarchical distillation of knowledge seems to be a traditional approach, which should not be a contribution to this work. In the experiments, the authors should verify the effectiveness of the proposed method by comparing it with various channel pruning and knowledge distillation methods. \n3) The author should analyze the differences between the two parts, eg. New pruning algorithm and Hierarchical knowledge distillation, with other methods in the ablation study.", "summary_of_the_paper": "This paper proposes a hierarchical knowledge distillation method for neural network pruning. Experiments demonstrate that the whole pruning pipeline requires much less computational resources than some of the state-of-the-art NAS based solutions for finding efficient FLOPs / accuracy trade-offs.", "summary_of_the_review": "See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A method for pruning neural networks is proposed.  Reviewers raised several concerns, including poor technical presentation and insufficient experimental validation with respect to both baseline methods and ablation studies.  All reviewer ratings lean toward reject and the authors did not provide a response.
Decision: Reject