Dynamic Channel Pruning: Feature Boosting and Suppression

1 Introduction . State-of-the-art vision and image-based tasks such as image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , object detection ( Ren et al. , 2017 ; Huang et al. , 2017 ) and segmentation ( Long et al. , 2015 ) are all built upon deep convolutional neural networks ( CNNs ) . While CNN architectures have evolved to become more efficient , the general trend has been to use larger models with greater memory utilization , bandwidth and compute requirements to achieve higher accuracy . The formidable amount of computational resources used by CNNs present a great challenge in the deployment of CNNs in both cost-sensitive cloud services and low-powered edge computing applications . One common approach to reduce the memory , bandwidth and compute costs is to prune over-parameterized CNNs . If performed in a coarse-grain manner this approach is known as channel pruning ( Ye et al. , 2018 ; He et al. , 2017 ; Liu et al. , 2017 ; Wen et al. , 2016 ) . Channel pruning evaluates channel saliency measures and removes all input and output connections from unimportant channels— generating a smaller dense model . A saliency-based pruning method , however , has threefold disadvantages . Firstly , by removing channels , the capabilities of CNNs are permanently lost , and the resulting CNN may never regain its accuracy for difficult inputs for which the removed channels were responsible . Secondly , despite the fact that channel pruning may drastically shrink model size , without careful design , computational resources can not be effectively reduced in a CNN without a detrimental impact on its accuracy . Finally , the saliency of a neuron is not static , which can be illustrated by the feature visualization in Figure 1 . Here , a CNN is shown a set of input images , certain channel neurons in a convolutional output may get highly excited , whereas another set of ∗Equal contribution , corresponding authors . images elicit little response from the same channels . This is in line with our understanding of CNNs that neurons in a convolutional layer specialize in recognizing distinct features , and the relative importance of a neuron depends heavily on the inputs . The above shortcomings prompt the question : why should we prune by static importance , if the importance is highly input-dependent ? Surely , a more promising alternative is to prune dynamically depending on the current input . A dynamic channel pruning strategy allows the network to learn to prioritize certain convolutional channels and ignore irrelevant ones . Instead of simply reducing model size at the cost of accuracy with pruning , we can accelerate convolution by selectively computing only a subset of channels predicted to be important at run-time , while considering the sparse input from the preceding convolution layer . In effect , the amount of cached activations and the number of read , write and arithmetic operations used by a well-designed dynamic model can be almost identical to an equivalently sparse statically pruned one . In addition to saving computational resources , a dynamic model preserves all neurons of the full model , which minimizes the impact on task accuracy . In this paper , we propose feature boosting and suppression ( FBS ) to dynamically amplify and suppress output channels computed by the convolutional layer . Intuitively , we can imagine that the flow of information of each output channel can be amplified or restricted under the control of a “ valve ” . This allows salient information to flow freely while we stop all information from unimportant channels and skip their computation . Unlike pruning statically , the valves use features from the previous layer to predict the saliency of output channels . With conventional stochastic gradient descent ( SGD ) methods , the predictor can learn to adapt itself by observing the input and output features of the convolution operation . FBS introduces tiny auxiliary connections to existing convolutional layers . The minimal overhead added to the existing model is thus negligible when compared to the potential speed up provided by the dynamic sparsity . Existing dynamic computation strategies in CNNs ( Lin et al. , 2017 ; Odena et al. , 2017 ; Bolukbasi et al. , 2017 ) produce on/off pruning decisions or execution path selections . Training them thus often resorts to reinforcement learning , which in practice is often computationally expensive . Even though FBS similarly use non-differentiable functions , contrary to these methods , the unified losses are still wellminimized with conventional SGD . We apply FBS to a custom CIFAR-10 ( Krizhevsky et al. , 2014 ) classifier and popular CNN models such as VGG-16 ( Simonyan & Zisserman , 2015 ) and ResNet-18 ( He et al. , 2016 ) trained on the ImageNet dataset ( Deng et al. , 2009 ) . Empirical results show that under the same speed-ups , FBS can produce models with validation accuracies surpassing all other channel pruning and dynamic conditional execution methods examined in the paper . 2 Related Work . 2.1 Structured Sparsity . Since LeCun et al . ( 1990 ) introduced optimal brain damage , the idea of creating more compact and efficient CNNs by removing connections or neurons has received significant attention . Early literature on pruning deep CNNs zero out individual weight parameters ( Hassibi et al. , 1994 ; Guo et al. , 2016 ) . This results in highly irregular sparse connections , which were notoriously difficult for GPUs to exploit . This has prompted custom accelerator solutions that exploit sparse weights ( Parashar et al. , 2017 ; Han et al. , 2016 ) . Although supporting both sparse and dense convolutions efficiently normally involves some compromises in terms of efficiency or performance . Alternatively , recent work has thus increasingly focused on introducing structured sparsity ( Wen et al. , 2016 ; Ye et al. , 2018 ; Alvarez & Salzmann , 2016 ; Zhou et al. , 2016 ) , which can be exploited by GPUs and allows custom accelerators to focus solely on efficient dense operations . Wen et al . ( 2016 ) added group Lasso on channel weights to the model ’ s training loss function . This has the effect of reducing the magnitude of channel weights to diminish during training , and remove connections from zeroed-out channels . To facilitate this process , Alvarez & Salzmann ( 2016 ) additionally used proximal gradient descent , while Li et al . ( 2017 ) and He et al . ( 2018a ) proposed to prune channels by thresholds , i.e . they set unimportant channels to zero , and fine-tune the resulting CNN . The objective to induce sparsity in groups of weights may present difficulties for gradient-based methods , given the large number of weights that need to be optimized . A common approach to overcome this is to solve ( He et al. , 2017 ) or learn ( Liu et al. , 2017 ; Ye et al. , 2018 ) channel saliencies to drive the sparsification of CNNs . He et al . ( 2017 ) solved an optimization problem which limits the number of active convolutional channels while minimizing the reconstruction error on the convolutional output . Liu et al . ( 2017 ) used Lasso regularization on channel saliencies to induce sparsity and prune channels with a global threshold . Ye et al . ( 2018 ) learned to sparsify CNNs with an iterative shrinkage/thresholding algorithm applied to the scaling factors in batch normalization . There are methods ( Luo et al. , 2017 ; Zhuang et al. , 2018 ) that use greedy algorithms for channel selection . Huang et al . ( 2018 ) and He et al . ( 2018b ) adopted reinforcement learning to train agents to produce channel pruning decisions . PerforatedCNNs , proposed by Figurnov et al . ( 2016 ) , use predefined masks that are model-agnostic to skip the output pixels in convolutional layers . 2.2 Dynamic Execution . In a pruned model produced by structured sparsity methods , the capabilities of the pruned neurons and connections are permanently lost . Therefore , many propose to use dynamic networks as an alternative to structured sparsity . During inference , a dynamic network can use the input data to choose parts of the network to evaluate . Convolutional layers are usually spatially sparse , i.e . their activation outputs may contain only small patches of salient regions . A number of recent publications exploit this for acceleration . Dong et al . ( 2017 ) introduced low-cost collaborative layers which induce spatial sparsity in cheap convolutions , so that the main expensive ones can use the same sparsity information . Figurnov et al . ( 2017 ) proposed spatially adaptive computation time for residual networks ( He et al. , 2016 ) , which learns the number of residual blocks required to compute a certain spatial location . Almahairi et al . ( 2016 ) presented dynamic capacity networks , which use the gradient of a coarse output ’ s entropy to select salient locations in the input image for refinement . Ren et al . ( 2018 ) assumed the availability of a priori spatial sparsity in the input image , and accelerated the convolutional layer by computing non-sparse regions . There are dynamic networks that make binary decisions or multiple choices for the inference paths taken . BlockDrop , proposed by Wu et al . ( 2018 ) , trains a policy network to skip blocks in residual networks . Liu & Deng ( 2018 ) proposed conditional branches in deep neural networks ( DNNs ) , and used Q-learning to train the branching policies . Odena et al . ( 2017 ) designed a DNN with layers containing multiple modules , and decided which module to use with a recurrent neural network ( RNN ) . Lin et al . ( 2017 ) learned an RNN to adaptively prune channels in convolutional layers . The on/off decisions commonly used in these networks can not be represented by differentiable functions , hence the gradients are not well-defined . Consequently , the dynamic networks above train their policy functions by reinforcement learning . There exist , however , methods that workaround such limitations . Shazeer et al . ( 2017 ) introduced sparsely-gated mixture-of-experts and used a noisy ranking on the backpropagate-able gating networks to select the expensive experts to evaluate . Bolukbasi et al . ( 2017 ) trained differentiable policy functions to implement early exits in a DNN . Hua et al . ( 2018 ) learned binary policies that decide whether partial or all input channels are used for convolution , but approximate the gradients of the non-differentiable policy functions with continuous ones . 3 Feature Boosting and Suppression . We start with a high-level illustration ( Figure 2 ) of how FBS accelerates a convolutional layer with batch normalization ( BN ) . The auxiliary components ( in red ) predict the importance of each output channel based on the input features , and amplify the output features accordingly . Moreover , certain output channels are predicted to be entirely suppressed ( or zero-valued as represented by ) , such output sparsity information can advise the convolution operation to skip the computation of these channels , as indicated by the dashed arrow . It is notable that the expensive convolution can be doubly accelerated by skipping the inactive channels from both the input features and the predicted output channel saliencies . The rest of this section provides detailed explanation of the components in Figure 2 . 3.1 Preliminaries . For simplicity , we consider a deep sequential batch-normalized ( Ioffe & Szegedy , 2015 ) CNN with L convolutional layers , i.e . xL = F ( x0 ) = fL ( · · · f2 ( f1 ( x0 ) ) · · · ) , where the lth layer fl : RCl−1×Hl−1×Wl−1 → RCl×Hl×Wl computes the features xl ∈ RCl×Hl×Wl , which comprise of Cl channels of features with height Hl and width Wl . The l th layer is thus defined as : fl ( xl−1 ) = ( γl · norm ( convl ( xl−1 , θl ) ) + βl ) + . ( 1 ) Here , additions ( + ) and multiplications ( · ) are element-wise , ( z ) + = max ( z , 0 ) denotes the ReLU activation , γl , βl ∈ RCl are trainable parameters , norm ( z ) normalizes each channel of features z across the population of z , with µz , σ 2 z ∈ RCl respectively containing the population mean and variance of each channel , and a small prevents division by zero : norm ( z ) = z− µz√ σ2z + . ( 2 ) Additionally , convl ( xl−1 , θl ) computes the convolution of input features xl−1 using the weight tensor θl ∈ RC l×Cl−1×k2 , where k is the kernel size . Specifically , FBS concerns the optimization of convl ( xl−1 , θl ) functions , as a CNN spends the majority of its inference time in them , using k2Cl−1ClHlWl multiply-accumulate operations ( MACs ) for the lth layer .

Summary: This paper proposed a feature boosting and suppression method for dynamic channel pruning. To be specific, the proposed method firstly predicts the importance of each channel and then use an affine function to amplify/suppress the importance of different channels. However, the idea of dynamic channel pruning is not novel. Moreover, the comparisons in the experiments are quite limited.