ACPLI: AUTOMATED CHANNEL PRUNING
Year: 2021

Abstract:
Neural network pruning allows for significant reduction of model size and latency. How1 ever, most of the current network pruning methods do not consider channel interdepen2 dencies and a lot of manual adjustments are required before they can be applied to new 3 network architectures. Moreover, these algorithms are often based on hand-picked, some4 times complicated heuristics and can require thousands of GPU computation hours. In 5 this paper, we introduce a simple neural network pruning and fine-tuning framework that 6 requires no manual heuristics, is highly efficient to train (2-6 times speed up compared to 7 NAS-based competitors) and produces comparable performance. The framework contains 8 1) an automatic channel detection algorithm that groups the interdependent blocks of 9 channels; 2) a non-iterative pruning algorithm that learns channel importance directly from 10 feature maps while masking the coupled computational blocks using Gumbel-Softmax 11 sampling and 3) a hierarchical knowledge distillation approach to fine-tune the pruned 12 neural networks. We validate our pipeline on ImageNet classification, human segmentation 13 and image denoising, creating lightweight and low latency models, easy to deploy on 14 mobile devices. Using our pruning algorithm and hierarchical knowledge distillation for 15 fine-tuning we are able to prune EfficientNet B0, EfficientNetV2 B0 and MobileNetV2 16 to 75% of their original FLOPs with no loss of accuracy on ImageNet. We release a set 17 pruned backbones as Keras models all of them proved beneficial when deployed in other 18 projects. 19

Neural network pruning allows for significant reduction of model size and latency. How-1 ever, most of the current network pruning methods do not consider channel interdepen-2 dencies and a lot of manual adjustments are required before they can be applied to new3 network architectures. Moreover, these algorithms are often based on hand-picked, some-4 times complicated heuristics and can require thousands of GPU computation hours. In5 this paper, we introduce a simple neural network pruning and fine-tuning framework that6 requires no manual heuristics, is highly efficient to train (2-6 times speed up compared to7 NAS-based competitors) and produces comparable performance. The framework contains8 1) an automatic channel detection algorithm that groups the interdependent blocks of9 channels; 2) a non-iterative pruning algorithm that learns channel importance directly from10 feature maps while masking the coupled computational blocks using Gumbel-Softmax11 sampling and 3) a hierarchical knowledge distillation approach to fine-tune the pruned12 neural networks. We validate our pipeline on ImageNet classification, human segmentation13 and image denoising, creating lightweight and low latency models, easy to deploy on14 mobile devices. Using our pruning algorithm and hierarchical knowledge distillation for15 fine-tuning we are able to prune EfficientNet B0, EfficientNetV2 B0 and MobileNetV216 to 75% of their original FLOPs with no loss of accuracy on ImageNet. We release a set17 pruned backbones as Keras models - all of them proved beneficial when deployed in other18 projects.19
1 INTRODUCTION20
Efforts directed towards deployment of neural networks on low-performance devices such as mobile phones21 or TVs, created a demand for smaller and faster models. This has led to advances in neural network22 compression techniques, which allow us to minimize existing large-scale architectures and adjust them to23 fit specific hardware requirements. Some techniques have been especially successful in this area. Neural24 network quantization approaches (Nagel et al., 2021) not only decreased the size of the models, but also25 enabled us to utilize specialized computing accelerators like DSPs. Unfortunately, other techniques, such as26 network pruning (Liu et al., 2020), are not equally effective in low-resource environments.27
Early attempts of naive weight pruning introduced sparse computations, which render them inefficient in28 practical scenarios (Han et al., 2015; Guo et al., 2016). Channel pruning (Li et al., 2016; Liu et al., 2017;29 2021a; Herrmann et al., 2020; Liu et al., 2019b) delivers significant improvements in terms of both memory30 consumption and execution speed, and is the preferred approach if we want to deploy our models on mobile31 devices.32
However, the majority of existing approaches to channel pruning share several drawbacks:33
1. Little effort has been made to address channel interdependencies that occur in the majority of the34 architectures, with Liu et al. (2021a) being a notable exception. Many popular network architectures35 contain residual connections inspired by ResNet (He et al., 2015). Feature maps added in residual36 connections must hold the same shapes, which is likely to be violated when channels are removed37 independently. We refer to channels involved in this kind of dependency as coupled. Automating38 the process of adding pruning logic to the network in consideration of channel interdependencies is39 extremely important in practical considerations.40
2. Most methods require an expensive and time-consuming fine-tuning process after channels are41 removed. Some authors use an iterative approach, where channels are removed in a number of steps,42 and fine-tuning is performed between these steps. Either way, the fine-tuning process often requires43 a significant number of GPU hours to complete.44
3. Channels in any given convolution are being considered independently. However, some target45 platforms, e.g. SNPE (Qualcomm), are optimized for specific numbers of input and output channels46 and pruning channels independently can give little to no speed-up.47
In order to overcome these issues we introduce an end-to-end channel pruning pipeline which can be48 deployed on a wide array of neural networks in an automated way. Our main insights are that: (1) Channel49
Similar, but a more affordable approach, is to periodically prune channels throughout a single training90 procedure (Liu et al., 2021a; Guo et al., 2020; Chen et al., 2020). Ye et al. (2020) and Hou et al. (2021)91 point out flaws in the idea of greedy channel removal and propose to selectively restore channels in the92 pruned network. Liu et al. (2019b) trains an auxiliary neural network to quickly evaluate pruned networks93 and select the best one using an evolutionary algorithm. Other methods jointly train a neural network and94 learn importance scores for its channels using channel gating mechanism. In (Chen et al., 2020), this is95 achieved by randomly enable and disable channels during each iteration of the training. Gradient descent96 was used to update the importance scores in Herrmann et al. (2020); Lin et al. (2020); Ye et al. (2020) and is97 based on the idea for optimizing hyperparameters in neural architecture search in Liu et al. (2019a) and Xie98 et al. (2018). These gradient-based methods rely on Gumbel-Softmax reparametrization trick (Jang et al.,99 2016) to enable back-propagating through the gates distribution. Herrmann et al. (2020) proposes a variant100 of such a method where the logits of the channel gates are trainable parameters, as well as a variant where101 the logits are produced by an auxiliary neural network that accepts a feature map. Selecting channels based102 network input introduces an overhead that is unacceptable on resource-limited devices. Our solution contains103 a similar idea, but we ensured that the auxiliary networks can be safely removed after the training.104
Channel coupling. The channel coupling pattern occurs in many modern architectures inspired by ResNet105 (He et al., 2015), such as MobileNet (Sandler et al., 2018), EfficientNet (Tan & Le, 2019; 2021) or FBNet106 (Wan et al., 2020). Many studies seem to ignore this issue (Herrmann et al., 2020; Lin et al., 2020; Ye et al.,107 2020); other resolve this issue by manually grouping interdependent layers or providing model-specific108 heuristics (Shao et al., 2021; Hou et al., 2021; Guo et al., 2020; Liu et al., 2021b). Independently to our109 efforts, an automated solution for grouping channels has been proposed in Liu et al. (2021a). We propose a110 similar algorithm (see section 4), and additionally offer an extension for handling concatenations.111
Measuring speed-up. Many pruning methods are parametrised by a fraction of channels to prune, either112 globally or per-layer (Lin et al., 2020; Ye et al., 2020; Herrmann et al., 2020). Overall network FLOPs1113 better corresponds to the usual business requirements. In Chen et al. (2020) and Liu et al. (2021a), the114 maximal FLOPs parameter is included in their stopping criteria and importance scores of channels are115 adjusted according to their computation cost. Similarly to Guo et al. (2020), we construct a loss function that116 introduce a penalty for exceeding the provided FLOPs budget and use it as a part differentiable importance117 optimization.118
Knowledge distillation. It has been noted that Knowledge distillation can perform poorly when there is119 a large discrepancy in complexity between student and teacher networks (Cho & Hariharan, 2019). Cho120 & Hariharan (2019) evaluate a step-wise approach, in which the intermediate teacher networks are trained121 by distilling knowledge from the original large teacher and then find it ineffective. Mirzadeh et al. (2020)122 propose using a teacher assistant to bridge the complexity gap. Hou et al. (2021) apply knowledge distillation123 to fine-tune pruned network, but do not address aforementioned issues. We propose an inverted version of124 the step-wise approach from Cho & Hariharan (2019), and train train our pruned network with increasingly125 larger teachers. Such chains can be naturally formed for model families like EfficientNet (Tan & Le, 2019)126 and EfficientNetV2 (Tan & Le, 2021). We also observe that in case of generic knowledge distillation, the127 final results can be improved by (even slightly) disturbing the student model with channel pruning before128 starting the distillation.129
3 PRUNING METHOD130
The basic idea behind our channel pruning algorithm is to set up a scheme in which the importance of131 channels is being learned from the feature maps generated by convolutions in neural networks. We assign132 each channel a score corresponding to its importance that is updated at each training step and used to133 approximate behavior of the pruned network by appropriate masking (Liu et al., 2017; Herrmann et al.,134 2020). Similarly to Herrmann et al. (2020) we apply a probabilistic approach where channels in feature135 maps are masked with samples from random variables with values in (0, 1). This is a continuous relaxation136 approach to solving a discrete problem. The distributions of these random variables depend on the values of137 corresponding logits (which can be though of as proxies for channel scores and have values in R). These138 logits are learned during the pruning stage. More precisely, given a feature map of size (B,H,W,C) (B139 is batch size, H and W are spatial dimension and C is the number of channels) and a logits variable, for140 each channel separately we sample — using Gumbel-Softmax (Jang et al., 2016) — the random variable141 parametrized by the corresponding logit in logits. We mask the feature map by multiplying it by the142 sampled values.143
We do not consider each feature map individually — instead, we extend our understanding of channels from144 a single feature map to a series of operations occurring within a network. The intuition is that element-wise145 operations, like activation functions, propagate channels forward throughout the network, while convolutional146 layers consume their input channels and create new ones. Pruning sequential models is trivial but in more147 complicated cases, like models with residual connections, there exist additional couplings between channels,148 introduced by operations that accept multiple inputs, e.g. element-wise sum, multiplication (Fig. 2). Because149
1a number of floating-point operations
coupled channels must be pruned jointly to ensure valid shapes, we use a single random variable to mask150 each set of coupled channels (see Section 4 for details about automatic detection of coupled channels).151
Although logits can be treated as standalone trainable variables, we choose to learn them from the feature152 maps in a feedback-loop mechanism. This is because the latter approach is faster to train, results in logits153 which (once converted to probabilities) have lower entropy and produces better results. Once we decide on154 the feature maps from which we will learn the optimal logits values, we place simple neural networks called155 logit predictor modules that take these feature maps as inputs. These modules are build of 3x3 depthwise156 convolution followed by 1x1 convolution and global mean pooling along spatial dimensions. The output157 output vector of each such module is later used to update the value of the corresponding logits variable158 (using exponential moving average) as in Figure 2.159
The masking operations should always be placed just before the convolution operations that absorb the160 channels (see Figure 2). The placement of logit predictors is more involved and in cases more complicated161 than the relatively simple one presented in Figure 2, we choose to follow a simple heuristic to place them162 after convolutions with largest kernel sizes.163
During the pruning phase we augment the task-specific loss with an auxiliary latency-based loss. It is based164 on the expected number of FLOPs in the pruned network, which is computed by using all the logits we have165 attached to the network. We train network weights and logit predictor modules jointly so that the network166 can adjust to channels being phased out.167
3.1 PRUNING LARGER BLOCKS OF CHANNELS168
We allow for blocks of channels (instead of just individual channels) to be treated jointly, so that blocks169 of a predefined size will be chosen or discarded together. This is especially important for platforms where170 convolutions are optimized with a specific block size o channels in mind, e.g., for SNPE (Qualcomm) this171 number is 32 and pruning individual channels often makes little sense.172
4 LAYER GROUPING ALGORITHM173
Although channel coupling has been observed in the literature, relevant groups of operations seem to be174 usually established via network-specific heuristics or manual annotation. A notable exception is Liu et al.175 (2021a) where the problem is described at length and an algorithm for finding the groups is derived. The176 algorithm is then tested on architectures based on ResNet. However, unlike our solution, it does not support177 concatenation operations. For clarity, we focus on convolutional neural networks, but the proposed strategy178 can be extended to other kinds of architectures.179
4.1 SOLUTION180
To overcome the issues delineated in Section 3 and make channel pruning available for most off-the-181 shelf architectures we have developed an algorithm that is capable of automatically detecting channel182 interdependencies between feature maps generated by operations in the network.183
To keep track of all the places where channels have to be considered in a synchronised way, we introduce the184 concept of an orbit. An orbit can be thought as subset of operations that are interdependent from the point of185 view of channel pruning. Operations in the same orbit need to be considered jointly when removing channels.186 Naively removing channels without taking into account these interdependencies may result in an invalid187 network. For example, if we remove an output channel from one of the convolutions on the left in Figure 2,188
the number of channels will no longer match for the Sum operation. A typical network has multiple orbits.189 It is easiest to understand this concept by seeing how orbits are build, which we delineate in Algorithm 1190 below.191
First, we fix some notation to make matters more intuitive. All the operations in a typical convolutional192 neural network can be described as being of the following types:193
1. sources are the operation where new channels are being created, namely regular convolution layers194 (not depthwise!) and dense layers;195
2. sinks are the operation where channels are being absorbed, namely regular convolution layers (not196 depthwise!) and dense layers;197
3. continuators are all the operations with a single input tensor that simply pass on the channels198 forward, e.g., batch normalization, mean pooling, resize, activations;199
4. joiners are operations with multiple input tensors of the same shape which join these tensors200 without altering the shape, namely element-wise addition and multiplication;201
Typically, continuator operations are not problematic since they do not alter the channels structure and have202 a single predecessor and a single output. It is the joiner operations that introduce interdependencies between203 channels. For brevity, from now on we will only speak of convolutions as sources and sinks, but everything204 applies just as well to dense layers.205
Note that some sources can be sinks at the same time and vice versa. We refer to operations that are either206 sinks or sources as source-sinks. To identify all the subgraphs in the network where channels have to be207 considered jointly we run an exhaustive-search type algorithm which has two distinct phases:208
In the fist phase we search for extended orbits, where the coupled operations are brought together. In209 Algorithm 1 we describe how extended orbits are created. The input is a neural network directed acyclic210 graph (DAG). The algorithm amounts to removing all inbound edges from convolution nodes and finding all211 weakly connected components in the resulting graph. The extended orbits are then these weakly connected212 components once we restore the inbound edges in convolution nodes.213
The second phase is similar to the first one. For all extended orbits found in phase one we do the following:214 take the extended orbit and then mark concatenation nodes (which play a special role, since they group215 channels from separate sources) inside as sinks and repeat the process. Most notably, we discard extended216 orbits in which there are concatenation nodes followed by joiner nodes, as it makes the whole process much217 more difficult to implement. We do not prune channels within such orbits. In Figure 3 we give an example of218 an extended orbit and how is broken up into final orbits.219
Algorithm 1 Searching for extended orbits Input: network DAG with layers represented as nodes
1: P := {p : p is a path starting and ending with a convolution with no convolutions inside the path } 2: for each path p in P remove the last node 3: for every distinct node ni on paths in P , create an empty color set for the node Cni = {} 4: X := {x : x is the initial node of a path in P } 5: for x in X do 6: pick an unused color c 7: add color c to color sets of all the nodes on all the paths in P starting in x 8: end for 9: while there exist nodes with multiple colors do
10: pick a node with multiple colors {c1, c2, . . . , ck} at random 11: if any node in the DAG has a color in {c2, . . . , ck} switch the color to c1 12: end while
5 PRUNING, FINE-TUNING AND HIERARCHICAL KNOWLEDGE DISTILLATION220
5.1 PRUNING STAGE221
The pruning workflow is the same for all types of tasks. We first find all final orbits in the network and attach222 logit predictors. Final orbits determine both: which parts of the network are being pruned and which of them223 are pruned jointly. The FLOPs per pixel can be automatically computed (and are differentiable with respect224 to the channel logits as in (Fig. 2). We can compute FLOPs for the original network and then set some225 FLOPs target. In practice we compute kFPP (FLOPs per pixel of the input tensor divided by 1000), to have226 a value that is independent of the input size. The latency loss is then given by ReLU(kFPP/target_kFPP−1).227 We add this loss to the quality loss related to the task, e.g., cross entropy in classification. To avoid an228 overly aggressive reduction of kFPP , we anneal the loss using exponential decay so that at the beginning of229 training the annealing multiplier is 0. and approaches 1. as the training progresses.230
Once the pruning phase is over we retain or discard output channels in convolutions based on channel231 interdependence discovered by applying Algorithm 1 and the values of logits variables learned by logit232 predictors.233
5.2 FINE-TUNING AND HIERARCHICAL KNOWLEDGE DISTILLATION234
We propose to fine-tune pruned models with a method we call hierarchical knowledge distillation. This235 approach relies on increasing the complexity of the teacher network in discrete steps. Given a fine-tuning236 budget of K GPU hours, and N teacher networks we train the network for K/N GPU hours with each of237 these teacher networks, starting with the smallest one. Our loss is Lce + 5Lkd where Lce is the standard238 cross entropy loss and Lkd is the distillation loss. Using higher weight term for the Lkd is crucial to prevent239 overfitting and produce better results.240
Hierarchical knowledge distillation consistently performs much better than just using the original model as241 the teacher. The comparisons can be seen in Section 6.2. Given an array of models with increasing FLOPs242 requirements, like EfficientNet Tan & Le (2019) and EfficientNetV2 Tan & Le (2021), it is possible to cheaply243 train new models for missing FLOPs values. This may produce better results in terms of FLOPs/accuracy244 trade-off and require less computational resources.245
It is perplexing that trying to use hierarchical knowledge distillation on an unpruned network does not work246 anywhere near as well. Our intuition is that pruning provides some kind of initial perturbation to network247 weights and architecture which prove beneficial from the point of view of gradient descent optimization.248 Are there any other types of model perturbations which boost the effectiveness of this type of knowledge249 distillation? These are the questions we could try to address as our future research. It would be also250 interesting to see how this approach performs when applied to recent state-of-the-art methods based on251 neural architecture search Wang et al. (2021).252
6 EXPERIMENTS253
All the experiments we perform adhere to the same schedule: (1) We first run the pruning algorithm with254 additional latency losses (usually 1-10 epochs, depending on the task). (2) We then fine-tune the pruned255 model (without resetting its weights). The experiments for classification on ImageNet are presented in256 Section 6.2. Experiments for image denoising and human segmentation are presented in Sections A.2.1 and257 A.2.2, respectively.258
6.1 HYPERPARAMETERS FOR THE PRUNING PHASE259
For the pruning phase, during which channels to be removed are being chosen, the setup is roughly the same260 for each task. The logits predictor is always a two layer network with 3× 3 depthwise convolution followed261 by 1× 1 convolution and global mean pooling. We set the batch size to 16 and run the training updating the262 channel gates distributions as described in section 3. The initial value of channel logits is set to 3.0 so that263 initially there little to no masking. There is an additional loss that penalizes the entropy of all the logits so264 that at the end of the pruning phase the channel enabling probabilities (which we get by applying softmax to265 logits) are far away from 0.5. The temperature for Gumbel-Softmax is constant - 0.5.266
6.2 CLASSIFICATION ON IMAGENET267
We prune EfficientNet B0, EfficientNet B1 (Tan & Le, 2019), MobileNetV2 (Sandler et al., 2018), and268 EfficientNetV2 (Tan & Le, 2021). We choose these since they are already highly optimized for mobile devices269 and relatively small. EfficientNetV2 is a recent state-of-the-art architecture optimized for mobile GPUs and270 DSPs. All the models are taken from their official Keras implementations2 except for EfficientNetV2. Larger271
2https://www.tensorflow.org/api_docs/python/tf/keras/applications
networks like the VGG19 or the ResNet family has been predominant in channel pruning literature, but are272 rarely suitable for resource-limited devices, where the need for optimization is biggest. The phase where273 channels are chosen usually lasts a little more than a single epoch on ImageNet. We split the ImageNet train274 data into two parts, leaving about 5% of the data for early-stopping.275
Following Section 5.2 we use multiple teacher networks. The details are as follows:276
• EfficientNet B0: fine-tune the models for 40 epochs with B0 as teacher and then we further277 fine-tune with a B1 for another 40 epochs;278
• EfficientNet B1: fine-tune the models for 25 epochs with B1 as teacher and then we further279 fine-tune with a B2 for another 25 epochs.280
• MobileNetV2: fine-tune the models for 40 epochs with MobileNetV2 as teacher and then we281 further fine-tune with a EfficientNet B0 for another 40 epochs.282
• EfficientNetV2 B0: fine-tune the models for 16 epochs with B0V2 as, then fine-tune the models283 for 16 epochs with B1V2 as teacher and finally fine-tune the models for 16 epochs with B2V2 as284 teacher.285
The interesting thing we noticed is that using knowledge distillation without pruning does not help at all.286 For example we tried fine-tuning MobileNetV2 with EfficientNet B0 teacher right away and top 1 Imagenet287 accuracy fell from 71.52% to 71.12%. We conjecture that some kind of initial perturbation is needed for288 knowledge distillation to work. In our case this perturbation is channel pruning.289
Batch size is set to 192 for B0 and MobileNetV2 fine-tuning. For B1 and EfficientNetV2 B0 batch size is290 128. The input image resolution is (224, 224). We use only random crop and flip as augmentations. For291 training we use one NVidia RTX3090 GPU. For the pruning phase we set the batch size to 16 and, quite292 importantly, we freeze all batch normalization layers. We use Adam optimizer for all the training runs.293 During mask-learning phase the learning rate is set to 0.0001. For fine-tuning we use exponential decay with294 learning rate initially set to 0.0001 and the decay rate set to 0.001.295
6.2.1 COMPARISONS AND DISCUSSION296
Few authors have attempted to prune EfficientNet (Tan & Le, 2019). We can compare our results with Hou297 et al. (2021), where only one model is presented, which was also fine-tuned with knowledge distillation. We298 provide a much wider FLOPs spectrum for B0 and prune B1 as well. It is interesting to see that B1 pruned to299 the FLOPs level of B0 outperforms B0 by a wide margin. The results are in Table 1.300
Comparisons for MobileNetV2 are quite difficult due the inconsistencies between different versions of the301 model taken by different authors as their baseline. For instance in Hou et al. (2021) the authors first take an302 over-pruned backbone which they proceed to prune. In Liu et al. (2019b) the largest version of MobileNetV2303 is taken (585M FLOPs) and then pruned. Some of the authors run the fine-tuning for much longer than we do.304 Notably, in Ye et al. (2020) the fine-tuning is run on 4 GPUs with batch size 512 and for 250 epochs which is305 considerably more expensive than our approach. Detailed results are in Table 2 and Figure 5a. Again using306 hierarchical knowledge distillation we are able to fine-tune the model pruned to 75% of original FLOPs so307 that it has 0.7% higher accuracy than the original.308
When it comes to EfficientNetV2, we are able to outperform the original model’s results on ImageNet with309 the help of hierarchical EKD, inasmuch as the pruned version of B0 (70% of the FLOPs of the original310 model) has higher top 1 accuracy than the original. See Table 3 and Figure 5b.311
7 CONCLUSION312
Using an automated solution to process coupled channels in neural network architectures and a simple313 scheme to learn channel importance, we are able to prune models with varying architectures for different314 underlying tasks. For fine-tuning pruned classification networks we use hierarchical knowledge distillation315 which produces much better results than just using the original model as a teacher. The whole pruning316 pipeline requires much less computational resources than some of the state-of-the-art NAS based solutions317 for finding efficient FLOPs / accuracy trade-offs like Wang et al. (2021).318
REFERENCES319 Zhiqiang Chen, Ting-Bing Xu, Changde Du, Cheng-Lin Liu, and Huiguang He. Dynamical channel pruning320 by conditional accuracy change for deep neural networks. IEEE transactions on neural networks and321 learning systems, 32(2):799–813, 2020.322
J. Cho and B. Hariharan. On the efficacy of knowledge distillation. pp. 4793–4801, nov 2019. doi: 10.323 1109/ICCV.2019.00489. URL https://doi.ieeecomputersociety.org/10.1109/ICCV.324 2019.00489.325
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self-supervised326 structure-sensitive learning and a new benchmark for human parsing. In Proceedings of the IEEE327 Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.328
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmcp: Differentiable markov channel pruning329 for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern330 Recognition, pp. 1539–1547, 2020.331
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. arXiv preprint332 arXiv:1608.04493, 2016.333
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient334 neural networks. arXiv preprint arXiv:1506.02626, 2015.335
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition,336 2015.337
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In338 Proceedings of the IEEE international conference on computer vision, pp. 1389–1397, 2017.339
Charles Herrmann, Richard Strong Bowen, and Ramin Zabih. Channel selection using gumbel softmax. In340 European Conference on Computer Vision, pp. 241–257. Springer, 2020.341
Yuenan Hou, Zheng Ma, Chunxiao Liu, Zhe Wang, and Chen Change Loy. Network pruning via resource342 reallocation. CoRR, abs/2103.01847, 2021. URL https://arxiv.org/abs/2103.01847.343
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint344 arXiv:1611.01144, 2016.345
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient346 convnets. arXiv preprint arXiv:1608.08710, 2016.347
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao.348 Hrank: Filter pruning using high-rank feature map. CoRR, abs/2002.10179, 2020. URL https:349 //arxiv.org/abs/2002.10179.350
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional351 neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.352 806–814, 2015.353
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2019a.354
Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. Pruning algorithms to accelerate convolutional355 neural networks for edge applications: A survey. arXiv preprint arXiv:2005.04275, 2020.356
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen,357 Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression.358 In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine359 Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7021–7032. PMLR, 18–24 Jul360 2021a. URL https://proceedings.mlr.press/v139/liu21ab.html.361
Xiangcheng Liu, Jian Cao, Hongyi Yao, Wenyu Sun, and Yuan Zhang. Adapruner: Adaptive channel pruning362 and effective weights inheritance. arXiv preprint arXiv:2109.06397, 2021b.363
Zechun Liu, Haoyuan Mu, X. Zhang, Zichao Guo, X. Yang, K. Cheng, and Jian Sun. Metapruning: Meta364 learning for automatic neural network channel pruning. 2019 IEEE/CVF International Conference on365 Computer Vision (ICCV), pp. 3295–3304, 2019b.366
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning367 efficient convolutional networks through network slimming. In Proceedings of the IEEE international368 conference on computer vision, pp. 2736–2744, 2017.369
Jian-Hao Luo, Hao Zhang, Hong-Yu Zhou, Chen-Wei Xie, Jianxin Wu, and Weiyao Lin. Thinet: pruning370 cnn filters for a thinner net. IEEE transactions on pattern analysis and machine intelligence, 41(10):371 2525–2538, 2018.372
Seyed Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.373 Improved knowledge distillation via teacher assistant. Proceedings of the AAAI Conference on Artificial374 Intelligence, 34:5191–5198, 04 2020. doi: 10.1609/aaai.v34i04.5963.375
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen376 Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021.377
Qualcomm. Snpe: Snapdragon neural processing engine. https://developer.qualcomm.com/378 sites/default/files/docs/snpe/.379
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Inverted380 residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CoRR,381 abs/1801.04381, 2018. URL http://arxiv.org/abs/1801.04381.382
Wenqi Shao, Hang Yu, Zhaoyang Zhang, Hang Xu, Zhenguo Li, and Ping Luo. Bwcp: Probabilistic383 learning-to-prune channels for convnets via batch whitening. arXiv preprint arXiv:2105.06423, 2021.384
Mennatullah Siam, Heba Mahgoub, Mohamed Zahran, Senthil Yogamani, Martin Jagersand, and Ahmad El-385 Sallab. Modnet: Motion and appearance based moving object detection network for autonomous driving.386 In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2859–2864, 2018.387 doi: 10.1109/ITSC.2018.8569744.388
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks.389 CoRR, abs/1905.11946, 2019. URL http://arxiv.org/abs/1905.11946.390
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training. CoRR, abs/2104.00298,391 2021. URL https://arxiv.org/abs/2104.00298.392
Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object detection. CoRR,393 abs/1911.09070, 2019. URL http://arxiv.org/abs/1911.09070.394
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew395 Yu, Tao Xu, Kan Chen, Peter Vajda, and Joseph E. Gonzalez. Fbnetv2: Differentiable neural architecture396 search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer397 Vision and Pattern Recognition (CVPR), June 2020.398
Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved training of399 supernet with alpha-divergence. arXiv preprint arXiv:2102.07954, 2021.400
Yuzhi Wang, Haibin Huang, Qin Xu, Jiaming Liu, Yiqun Liu, and Jue Wang. Practical deep raw image401 denoising on mobile devices, 2020.402
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv403 preprint arXiv:1812.09926, 2018.404
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks405 provably exist: Pruning via greedy forward selection. CoRR, abs/2003.01794, 2020. URL https:406 //arxiv.org/abs/2003.01794.407
A APPENDIX408
A.1 LAYER WIDTHS VISUALIZATION409
It is quite interesting to see how layer width looks like after pruning. The pattern that emerge are quite410 telling. EfficientNets are build of a series of meta-blocks, .e.g, 2, 3, . . . , 7 in EfficientNet B0, where each411 meta-block consists of a number of MBCONV blocks at the same spatial resolution. It appears that in each412 such meta-block the most important block is usually the first one, and block importance decays proportionally413 to the depth of the block inside the meta-block. See Figure 6 in the Appendix.414
A.2 FURTHER RESULTS415
A.2.1 RAWRGB IMAGE DENOISING416
We prune a recent state-of-the-art network for RawRGB image denoising on mobile devices introduced417 in Wang et al. (2020). We train the models on SIDD Medium dataset https://www.eecs.yorku.418 ca/~kamel/sidd/dataset.php. We first extract 256x256 patches for training and validation and419 then test the networks on SIDD validation dataset https://www.eecs.yorku.ca/~kamel/sidd/420 benchmark.php. The batch size is set to 16, learning rate is 0.0001 and we use Adam optimizer. The421 loss is mean absolute error. We train the original model for 150 epochs, prune it and then train the original422 model for another 150 epochs. The pruned models are fine-tuned for 150 epochs as well. For comparison we423 also train from scratch smaller (linearly scaled down) versions of the original model. The results can be seen424 in Table 4 and Figure 7.425
A.2.2 HUMAN SEGMENTATION426
For semantic segmentation we use a private dataset for training human segmentation models for real time427 prediction in video bokeh task. This is dictated by the need to have superior edge quality which is missing428 in publicly available data for segmentation. The dataset consists of 120k real image/mask pair and 50k429 synthetic ones. Apart from IoU we also compute edge IoU, which pays attention only to the edges of the430 masks and can be thought of as a proxy for edge quality. The baseline architecture consists of an EfficientNet431
B0 (Tan & Le, 2019) backbone, EfficientDet (Tan et al., 2019) (modified slightly to allow for easier channel432 pruning) fusion block and a detail branch (Siam et al., 2018) to preserve edge quality. The backbone network433 is pretrained on ImageNet. We train the original model for 70 epochs, prune and then fine-tune the pruned434 models for 50 epochs. The validation results are presented in Table 4. The validation dataset is a split of a435 modified version of LIP dataset (Gong et al., 2017), where objects belonging to people (such as handbags,436 etc.) are also considered part of these people. This is done, so that we can train models for video bokeh437 effect. The results are in Table 4b and are visualized in Figures 8a and 8b.438
Notice that the smallest pruned model is compressed to around 10% of the size of the original one. Even in439 these extreme compression scenario our approach produces a model with IoU higher than 90%. IoU starts440 dropping only after we have removed more than 60% of the original FLOPs. This is an observation which, in441 our experience, is true for many more architectures for segmentation, the one being presented here is just442 one example. Edge IoU starts falling much more quickly, perhaps beacause we employ no edge-specific loss.443

Review summaries:
This paper proposes a neural network pruning and fine-tuning framework for model compression. It can automatically prune the channels by learning the channel importance. The contributions are: 1) A new pruning scheme is proposed by learning the channel importance; 2) The pruning logic is introduced in the pruning scheme. Thus, some grouping operations are pruned jointly. The pruning problems of residual connections in ResNet can be solved; 3) Hierarchical knowledge distillation is added in the fine-tuning phase to speed up training. Experimental results show effectiveness of the proposed method.