Dynamic Channel Pruning: Feature Boosting and Suppression

1 Introduction . State-of-the-art vision and image-based tasks such as image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , object detection ( Ren et al. , 2017 ; Huang et al. , 2017 ) and segmentation ( Long et al. , 2015 ) are all built upon deep convolutional neural networks ( CNNs ) . While CNN architectures have evolved to become more efficient , the general trend has been to use larger models with greater memory utilization , bandwidth and compute requirements to achieve higher accuracy . The formidable amount of computational resources used by CNNs present a great challenge in the deployment of CNNs in both cost-sensitive cloud services and low-powered edge computing applications . One common approach to reduce the memory , bandwidth and compute costs is to prune over-parameterized CNNs . If performed in a coarse-grain manner this approach is known as channel pruning ( Ye et al. , 2018 ; He et al. , 2017 ; Liu et al. , 2017 ; Wen et al. , 2016 ) . Channel pruning evaluates channel saliency measures and removes all input and output connections from unimportant channels— generating a smaller dense model . A saliency-based pruning method , however , has threefold disadvantages . Firstly , by removing channels , the capabilities of CNNs are permanently lost , and the resulting CNN may never regain its accuracy for difficult inputs for which the removed channels were responsible . Secondly , despite the fact that channel pruning may drastically shrink model size , without careful design , computational resources can not be effectively reduced in a CNN without a detrimental impact on its accuracy . Finally , the saliency of a neuron is not static , which can be illustrated by the feature visualization in Figure 1 . Here , a CNN is shown a set of input images , certain channel neurons in a convolutional output may get highly excited , whereas another set of ∗Equal contribution , corresponding authors . images elicit little response from the same channels . This is in line with our understanding of CNNs that neurons in a convolutional layer specialize in recognizing distinct features , and the relative importance of a neuron depends heavily on the inputs . The above shortcomings prompt the question : why should we prune by static importance , if the importance is highly input-dependent ? Surely , a more promising alternative is to prune dynamically depending on the current input . A dynamic channel pruning strategy allows the network to learn to prioritize certain convolutional channels and ignore irrelevant ones . Instead of simply reducing model size at the cost of accuracy with pruning , we can accelerate convolution by selectively computing only a subset of channels predicted to be important at run-time , while considering the sparse input from the preceding convolution layer . In effect , the amount of cached activations and the number of read , write and arithmetic operations used by a well-designed dynamic model can be almost identical to an equivalently sparse statically pruned one . In addition to saving computational resources , a dynamic model preserves all neurons of the full model , which minimizes the impact on task accuracy . In this paper , we propose feature boosting and suppression ( FBS ) to dynamically amplify and suppress output channels computed by the convolutional layer . Intuitively , we can imagine that the flow of information of each output channel can be amplified or restricted under the control of a “ valve ” . This allows salient information to flow freely while we stop all information from unimportant channels and skip their computation . Unlike pruning statically , the valves use features from the previous layer to predict the saliency of output channels . With conventional stochastic gradient descent ( SGD ) methods , the predictor can learn to adapt itself by observing the input and output features of the convolution operation . FBS introduces tiny auxiliary connections to existing convolutional layers . The minimal overhead added to the existing model is thus negligible when compared to the potential speed up provided by the dynamic sparsity . Existing dynamic computation strategies in CNNs ( Lin et al. , 2017 ; Odena et al. , 2017 ; Bolukbasi et al. , 2017 ) produce on/off pruning decisions or execution path selections . Training them thus often resorts to reinforcement learning , which in practice is often computationally expensive . Even though FBS similarly use non-differentiable functions , contrary to these methods , the unified losses are still wellminimized with conventional SGD . We apply FBS to a custom CIFAR-10 ( Krizhevsky et al. , 2014 ) classifier and popular CNN models such as VGG-16 ( Simonyan & Zisserman , 2015 ) and ResNet-18 ( He et al. , 2016 ) trained on the ImageNet dataset ( Deng et al. , 2009 ) . Empirical results show that under the same speed-ups , FBS can produce models with validation accuracies surpassing all other channel pruning and dynamic conditional execution methods examined in the paper . 2 Related Work . 2.1 Structured Sparsity . Since LeCun et al . ( 1990 ) introduced optimal brain damage , the idea of creating more compact and efficient CNNs by removing connections or neurons has received significant attention . Early literature on pruning deep CNNs zero out individual weight parameters ( Hassibi et al. , 1994 ; Guo et al. , 2016 ) . This results in highly irregular sparse connections , which were notoriously difficult for GPUs to exploit . This has prompted custom accelerator solutions that exploit sparse weights ( Parashar et al. , 2017 ; Han et al. , 2016 ) . Although supporting both sparse and dense convolutions efficiently normally involves some compromises in terms of efficiency or performance . Alternatively , recent work has thus increasingly focused on introducing structured sparsity ( Wen et al. , 2016 ; Ye et al. , 2018 ; Alvarez & Salzmann , 2016 ; Zhou et al. , 2016 ) , which can be exploited by GPUs and allows custom accelerators to focus solely on efficient dense operations . Wen et al . ( 2016 ) added group Lasso on channel weights to the model ’ s training loss function . This has the effect of reducing the magnitude of channel weights to diminish during training , and remove connections from zeroed-out channels . To facilitate this process , Alvarez & Salzmann ( 2016 ) additionally used proximal gradient descent , while Li et al . ( 2017 ) and He et al . ( 2018a ) proposed to prune channels by thresholds , i.e . they set unimportant channels to zero , and fine-tune the resulting CNN . The objective to induce sparsity in groups of weights may present difficulties for gradient-based methods , given the large number of weights that need to be optimized . A common approach to overcome this is to solve ( He et al. , 2017 ) or learn ( Liu et al. , 2017 ; Ye et al. , 2018 ) channel saliencies to drive the sparsification of CNNs . He et al . ( 2017 ) solved an optimization problem which limits the number of active convolutional channels while minimizing the reconstruction error on the convolutional output . Liu et al . ( 2017 ) used Lasso regularization on channel saliencies to induce sparsity and prune channels with a global threshold . Ye et al . ( 2018 ) learned to sparsify CNNs with an iterative shrinkage/thresholding algorithm applied to the scaling factors in batch normalization . There are methods ( Luo et al. , 2017 ; Zhuang et al. , 2018 ) that use greedy algorithms for channel selection . Huang et al . ( 2018 ) and He et al . ( 2018b ) adopted reinforcement learning to train agents to produce channel pruning decisions . PerforatedCNNs , proposed by Figurnov et al . ( 2016 ) , use predefined masks that are model-agnostic to skip the output pixels in convolutional layers . 2.2 Dynamic Execution . In a pruned model produced by structured sparsity methods , the capabilities of the pruned neurons and connections are permanently lost . Therefore , many propose to use dynamic networks as an alternative to structured sparsity . During inference , a dynamic network can use the input data to choose parts of the network to evaluate . Convolutional layers are usually spatially sparse , i.e . their activation outputs may contain only small patches of salient regions . A number of recent publications exploit this for acceleration . Dong et al . ( 2017 ) introduced low-cost collaborative layers which induce spatial sparsity in cheap convolutions , so that the main expensive ones can use the same sparsity information . Figurnov et al . ( 2017 ) proposed spatially adaptive computation time for residual networks ( He et al. , 2016 ) , which learns the number of residual blocks required to compute a certain spatial location . Almahairi et al . ( 2016 ) presented dynamic capacity networks , which use the gradient of a coarse output ’ s entropy to select salient locations in the input image for refinement . Ren et al . ( 2018 ) assumed the availability of a priori spatial sparsity in the input image , and accelerated the convolutional layer by computing non-sparse regions . There are dynamic networks that make binary decisions or multiple choices for the inference paths taken . BlockDrop , proposed by Wu et al . ( 2018 ) , trains a policy network to skip blocks in residual networks . Liu & Deng ( 2018 ) proposed conditional branches in deep neural networks ( DNNs ) , and used Q-learning to train the branching policies . Odena et al . ( 2017 ) designed a DNN with layers containing multiple modules , and decided which module to use with a recurrent neural network ( RNN ) . Lin et al . ( 2017 ) learned an RNN to adaptively prune channels in convolutional layers . The on/off decisions commonly used in these networks can not be represented by differentiable functions , hence the gradients are not well-defined . Consequently , the dynamic networks above train their policy functions by reinforcement learning . There exist , however , methods that workaround such limitations . Shazeer et al . ( 2017 ) introduced sparsely-gated mixture-of-experts and used a noisy ranking on the backpropagate-able gating networks to select the expensive experts to evaluate . Bolukbasi et al . ( 2017 ) trained differentiable policy functions to implement early exits in a DNN . Hua et al . ( 2018 ) learned binary policies that decide whether partial or all input channels are used for convolution , but approximate the gradients of the non-differentiable policy functions with continuous ones . 3 Feature Boosting and Suppression . We start with a high-level illustration ( Figure 2 ) of how FBS accelerates a convolutional layer with batch normalization ( BN ) . The auxiliary components ( in red ) predict the importance of each output channel based on the input features , and amplify the output features accordingly . Moreover , certain output channels are predicted to be entirely suppressed ( or zero-valued as represented by ) , such output sparsity information can advise the convolution operation to skip the computation of these channels , as indicated by the dashed arrow . It is notable that the expensive convolution can be doubly accelerated by skipping the inactive channels from both the input features and the predicted output channel saliencies . The rest of this section provides detailed explanation of the components in Figure 2 . 3.1 Preliminaries . For simplicity , we consider a deep sequential batch-normalized ( Ioffe & Szegedy , 2015 ) CNN with L convolutional layers , i.e . xL = F ( x0 ) = fL ( · · · f2 ( f1 ( x0 ) ) · · · ) , where the lth layer fl : RCl−1×Hl−1×Wl−1 → RCl×Hl×Wl computes the features xl ∈ RCl×Hl×Wl , which comprise of Cl channels of features with height Hl and width Wl . The l th layer is thus defined as : fl ( xl−1 ) = ( γl · norm ( convl ( xl−1 , θl ) ) + βl ) + . ( 1 ) Here , additions ( + ) and multiplications ( · ) are element-wise , ( z ) + = max ( z , 0 ) denotes the ReLU activation , γl , βl ∈ RCl are trainable parameters , norm ( z ) normalizes each channel of features z across the population of z , with µz , σ 2 z ∈ RCl respectively containing the population mean and variance of each channel , and a small prevents division by zero : norm ( z ) = z− µz√ σ2z + . ( 2 ) Additionally , convl ( xl−1 , θl ) computes the convolution of input features xl−1 using the weight tensor θl ∈ RC l×Cl−1×k2 , where k is the kernel size . Specifically , FBS concerns the optimization of convl ( xl−1 , θl ) functions , as a CNN spends the majority of its inference time in them , using k2Cl−1ClHlWl multiply-accumulate operations ( MACs ) for the lth layer .

Summary: This manuscript presents a nice method that can dynamically prune some channels in a CNN network to speed up the training. The main strength of the proposed method is to determine which channels to be suppressed based upon each data sample without incurring too much computational burden or too much memory consumption.  The good thing is that the proposed pruning strategy does not result in a big performance decrease. Overall, this is a nicely written paper and may be empirically useful for training a very large CNN. Nevertheless, the authors did not present a real-world application in which it is important to speed up by 2 or 3 times at a small cost, so it is hard to judge the real impact of the proposed method.