Succinct Network Channel and Spatial Pruning via Discrete Variable QCQP

1 INTRODUCTION . Deep neural networks are the bedrock of artificial intelligence tasks such as object detection , speech recognition , and natural language processing ( Redmon & Farhadi , 2018 ; Chorowski et al. , 2015 ; Devlin et al. , 2019 ) . While modern networks have hundreds of millions to billions of parameters to train , it has been recently shown that these parameters are highly redundant and can be pruned without significant loss in accuracy ( Han et al. , 2015 ; Guo et al. , 2016 ) . This discovery has led practitioners to desire training and running the models on resource-constrained mobile devices , provoking a large body of research on network pruning . Unstructured pruning , however , does not directly lead to any practical acceleration or memory footprint reduction due to poor data locality ( Wen et al. , 2016 ) , and this motivated research on structured pruning to achieve practical usage under limited resource budgets . To this end , a line of research on channel pruning considers completely pruning the convolution filters along the input and output channel dimensions , where the resulting pruned model becomes a smaller dense network suited for practical acceleration and memory footprint reduction ( Li et al. , 2017 ; Luo et al. , 2017 ; He et al. , 2019 ; Wen et al. , 2016 ; He et al. , 2018a ) . However , existing channel pruning methods perform the pruning operations with a greedy approach and does not consider the inherent quadratic coupling between channels in the neighboring layers . Although these methods are easy to model and optimize , they can not safely remove inactive weights during the pruning procedure , suffer from discrepancies with the true objective , and prohibit the strict satisfaction of the required resource constraints during the pruning process . The ability to specify hard target resource constraints into the pruning optimization process is important since this allows the user to run the pruning and optional finetuning process only once . When the pruning process ignores the target specifications , the users may need to apply multiple rounds of pruning and finetuning until the specifications are eventually met , resulting in an extra computation overhead ( Han et al. , 2015 ; He et al. , 2018a ; Liu et al. , 2017 ) . In this paper , we formulate a principled optimization problem that prunes the network layer channels while respecting the quadratic coupling and exactly satisfying the user-specified FLOPs and j . Note that we use W ( l ) · , j to denote the tensor W ( l ) · , j , · , · , following the indexing rules of NumPy ( Van Der Walt et al. , 2011 ) . memory constraints . This new formulation leads to an interesting discrete variable QCQP ( Quadratic Constrained Quadratic Program ) optimization problem , which directly maximizes the importance of neurons in the pruned network under the specified resource constraints . Also , we increase the pruning granularity beyond channels and jointly prune individual 2D convolution filters spatially for greater efficiency . Furthermore , we generalize our formulation to cover nonsequential convolution operations , such as skip connections , and propose a principled optimization framework for handling various architectural implementations of skip connections in ResNet ( He et al. , 2016 ) . Our experiments on CIFAR-10 and ImageNet datasets show the state of the art results compared to other channel pruning methods that start from pretrained networks . 2 MOTIVATION . In this section , we first discuss the motivation of our method concretely . Suppose the weights in a sequential CNN form a sequence of 4-D tensors , W ( l ) ∈ RCl−1×Cl×Kl×Kl ∀l ∈ [ L ] where Cl−1 , Cl , and Kl represent the number of input channels , the number of output channels , and the filter size of l-th convolution weight tensor , respectively . We denote the feature map after l-th convolution as X ( l ) ∈ RCl×Hl×Wl . Concretely , X ( l ) j = σ ( X ( l−1 ) W ( l ) · , j ) = σ ( ∑Cl−1 i=1 X ( l−1 ) i ∗W ( l ) i , j ) , where σ is the activation function , ∗ denotes 2-D convolution operation , and denotes the sum of channel-wise 2-D convolutions . Now consider pruning these weights in channel-wise direction . We show that naive channel-wise pruning methods prevent exact specification of the target resource constraints due to unpruned inactive weights and deviate away from the true objective by ignoring quadratic coupling between channels in the neighboring layers . 2.1 INACTIVE WEIGHTS . According to Han et al . ( 2015 ) , network pruning produces dead neurons with zero input or output connections . These dead neurons cause inactive weights1 , which do not affect the final output activations of the pruned network . These inactive weights may not be excluded automatically through the standard pruning procedure and require additional post-processing which relies on ad-hoc heuristics . For example , Figure 1 shows a standard channel pruning procedure that deletes weights across the output channel direction but fails to prune the inactive weights . Concretely , deletion of weights on j-th output channel of l-th convolution layer leads to W ( l ) · , j = 0Cl−1 , Kl , Kl . Then , X ( l ) j becomes a dead neuron since X ( l ) j = σ ( X ( l−1 ) W ( l ) · , j ) = σ ( ∑Cl−1 i=1 X ( l−1 ) i ∗W ( l ) i , j ) = 0Cl , Hl , Wl . 1Rigorous mathematical definition of inactive weights is provided in Supplementary material D. The convolution operation on the dead neuron results in a trivially zero output , as below : X ( l+1 ) p = σ ( Cl∑ i=1 X ( l ) i ∗W ( l+1 ) i , p ) = σ ( Cl∑ i=1 1i 6=jX ( l ) i ∗W ( l+1 ) i , p +X ( l ) j︸︷︷︸ dead ∗W ( l+1 ) j , p︸ ︷︷ ︸ inactive︸ ︷︷ ︸ =0Hl+1 , Wl+1 ) . ( 1 ) Equation ( 1 ) shows that the dead neuron X ( l ) j causes weights W ( l+1 ) j , p , ∀p ∈ [ Cl+1 ] to be inactive . Such inactive weights do not account for the actual resource usage , even when they remain in the pruned network , which prevents the exact modeling of the user-specified hard resource constraints ( FLOPs and network size ) . Furthermore , inactive weights unpruned during the pruning procedure are a bigger problem for nonsequential convolutional networks due to their skip connections . To address this problem , we introduce a quadratic optimization-based algorithm that provably eliminates all the inactive weights during the pruning procedure . 2.2 QUADRATIC COUPLING . Existing channel pruning methods remove channels according to their importance . However , measuring a channel ’ s contribution to the network should also take into account the channels in the neighboring layers , as illustrated in Figure 2 . In the example , we define the importance of a channel as the absolute sum of weights in the channel , as in Li et al . ( 2017 ) , and assume the objective is to maximize the absolute sum of weights in the whole pruned network , excluding the inactive weights . We compare two different channel pruning methods : ( a ) a standard channel pruning method that greedily prunes each channel independently , and ( b ) our pruning method that considers the effect of the channels in neighboring layers when pruning . As a result of running each pruning algorithms , ( a ) will prune the second output channel of the first convolution and the third output channel of the second convolution , and ( b ) will prune the first output channel of the first convolution , the third output channel of the second convolution , and the first input channel of the second convolution . The objective values for each pruned networks are ( a ) 18 and ( b ) 21 , respectively . This shows that the coupling effect of the channels in neighboring layers directly affects the objective values , and finally results in a performance gap between ( a ) and ( b ) . We call this coupling relationship as the quadratic coupling between the neighboring layers and formulate the contributions to the objective by quadratic terms of neighboring channel activations . To address this quadratic coupling , we propose a channel pruning method based on the QCQP framework with importance evaluation respecting both the input and the output channels . 3 METHOD . In this section , we first propose our discrete QCQP formulation of channel pruning for the sequential convolutional neural networks ( CNNs ) . Then , we present an extended version of our formulation for joint channel and shape pruning of 2D convolution filters . The generalization to the nonsequential convolution ( skip addition and skip concatenation ) is introduced in Supplementary material A . 3.1 FORMULATION OF CHANNEL PRUNING FOR SEQUENTIAL CNNS . To capture the importance of weights in W ( l ) , we define the importance tensor as I ( l ) ∈ RCl−1×Cl×Kl×Kl+ . Following the protocol of Han et al . ( 2015 ) ; Guo et al . ( 2016 ) , we set I ( l ) = γl ∣∣W ( l ) ∣∣ where γl is the ` 2 normalizing factor in l-th layer or ‖vec ( W ( l ) ) ‖−1 . Then , we define the binary pruning mask as A ( l ) ∈ { 0 , 1 } Cl−1×Cl×Kl×Kl . For channel pruning in sequential CNNs , we define channel activation r ( l ) ∈ { 0 , 1 } Cl to indicate which indices of channels remain in the l-th layer of the pruned network . Then , the weights inW ( l ) i , j are active if and only if r ( l−1 ) i r ( l ) j = 1 , which leads to A ( l ) i , j = r ( l−1 ) i r ( l ) j JKl . For example , in Figure 2b , r ( l−1 ) = [ 1 , 1 , 1 ] ᵀ , r ( l ) = [ 0 , 1 ] ᵀ , and r ( l+1 ) = [ 1 , 1 , 0 ] ᵀ , therefore , A ( l ) = 0 10 1 0 1 ⊗ JKl and A ( l+1 ) = [ 0 0 01 1 0 ] ⊗ JKl+1 . We wish to directly maximize the sum of the importance of active weights after the pruning procedure under given resource constraints : 1 ) FLOPs , 2 ) memory , and 3 ) network size . Concretely , our optimization problem is 2 maximize r ( 0 : L ) L∑ l=1 〈 I ( l ) , A ( l ) 〉 ( 2 ) subject to L∑ l=0 al ∥∥∥r ( l ) ∥∥∥ 1 + L∑ l=1 bl ∥∥∥A ( l ) ∥∥∥ 1 ≤M A ( l ) = r ( l−1 ) r ( l ) ᵀ ⊗ JKl ∀l ∈ [ L ] r ( l ) ∈ { 0 , 1 } Cl . In our formulation , the actual resource usage of the pruned network is exactly computed by specifying the number of channels in the pruned network ( = ‖r ( l ) ‖1 ) and the pruning mask sparsity ( = ‖A ( l ) ‖1 ) in each layer . Concretely , the left hand side of the inequality in the first constraint in Equation ( 2 ) indicates the actual resource usage . Table 1 shows al , bl terms used for computing usage of each resource . Note that this optimization problem is a discrete nonconvex QCQP of the channel activations [ r ( 0 ) , . . . , r ( L ) ] , where the objective , which is the same with the objective in Section 2.2 , respects the quadratic coupling of channel activations ( = r ( l ) ) . Please refer to Supplementary material E for the details on the standard QCQP form of Equation ( 2 ) .

Summary: In this manuscript, a new pruning method is proposed by considering the inherent quadratic constraint between consecutive layers. Without this constraint, inactive weights cannot be safely removed. Even with the same objective function, the optimized result is different, as shown in the motivation section. Based on this observation, the pruning task is models as a QCQP optimization problem. And a faster algorithm to solve this problem is proposed. Moreover, the pruning on filter size can also be modeled as the QCQP problem, making the pruning on both channel and filter size feasible.