SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning

1 INTRODUCTION . Deep neural networks have consistently grown in size over the last years with increasing performance . However , this increase in size leads to slower inference , higher computational requirements and higher cost . To reduce the size of the networks without affecting their performance , a large number of pruning algorithms have been proposed ( e.g. , LeCun et al. , 1990 ; Hassibi et al. , 1993 ; Reed , 1993 ; Han et al. , 2015 ; Blalock et al. , 2020 ) . Pruning can either be unstructured , i.e . removing individual weights , or structured , i.e . removing entire substructures like nodes or channels . Single-shot pruning methods , as investigated in this work , usually consist of three steps : 1 ) training , 2 ) pruning , 3 ) another training step often referred to as fine-tuning . Unstructured pruning can significantly reduce the number of parameters of a neural network with only little loss in the accuracy , but the resulting networks often show only a marginal improvement in training and inference time , unless specialized hardware is used ( He et al. , 2017 ) . In contrast , structured pruning can directly reduce inference time and even training time when applied at initialization ( Lee et al. , 2018 ) . To exploit these advantages , in this work , we focus on structured pruning . Global pruning removes structure by structure from all available structures of a network until a predefined percentage of pruned structures is reached . Recent examples for global structured pruning methods are NN Slimming ( Liu et al. , 2017 ) , C-OBD and EigenDamage ( Wang et al. , 2019a ) . Local pruning , on the other hand , first subdivides all global structures into subsets ( e.g . layers ) and removes a percentage of structures of each subset . Recent examples for local pruning methods are HRank ( Lin et al. , 2019 ) , CCP ( Peng et al. , 2019 ) , FPGM ( He et al. , 2019 ) and Variational Pruning ( Zhao et al. , 2019 ) . Most local pruning schemes use a predefined layer-wise pruning ratio , which fixes the percentage of structures removed per layer . While this prevents the layers from collapsing , it also reduces some of the degrees of freedom , since some layers may be less important than others . Other local pruning methods like AMC ( He et al. , 2018 ) learn the layer-wise pruning ratios in a first step . A key challenge for global saliency-based pruning is to find an objective which can be efficiently calculated to make the approach scalable to large-scale , modern neural networks . While second-order pruning methods are usually more accurate than first-order methods ( Molchanov et al. , 2019 ) , calculating the full second-order saliency objective is intractable for modern neural networks . Therefore , most saliency-based pruning methods such as OBD ( e.g. , LeCun et al. , 1990 ) or C-OBD ( Wang et al. , 2019a ) evaluate the effect of removing a single weight or structure on the loss of the neural network in isolation . However , this neglects possible correlations between different structures , which are captured by the off-diagonal second-order terms , and hence , can significantly harm the estimation of the sensitivities . Finding a global second-order method that both considers off-diagonal terms and scales to modern neural networks is still an unsolved research question . Our main goal in this work is to devise a simple and efficient second-order pruning method that considers all global correlations for structured sensitivity pruning . In addition , we want to highlight the benefits that such methods may have over other structured global and local pruning schemes . Our contributions are as follows : • We develop two novel saliency-based pruning methods for second-order structured pruning ( SOSP ) and analyze them theoretically . We show that both of our methods drastically improve on the complexity of a naive second-order approach , which is usually intractable for modern neural networks . Further , we show that our SOSP-H method , which is based on fast Hessian-vector products , has the same low complexity as first-order methods , while taking the full Hessian into account . • We compare the performance and the scaling of SOSP-H to that of SOSP-I , which is based on the well-known Gauss-Newton approximation . While both methods perform on par , SOSP-H shows better scaling . We then benchmark our SOSP methods against a variety of state-of-the-art pruning methods and show that they achieve comparable or better results at lower computational costs for pruning . • We exploit the structure of the pruning masks found by our SOSP methods to widen architectural bottlenecks , which further improves the performance of the pruned networks . We diagnose layers with disproportionally low pruning ratios as architectural bottlenecks . PyTorch code implementing our method will be published upon acceptance of this manuscript . Related work is discussed in the light of our results in the Discussion section ( Sec . 4 ) . 2 SOSP : SECOND-ORDER STRUCTURED PRUNING . A neural network ( NN ) maps an input x ∈ Rd to an output fθ ( x ) ∈ RD , where θ ∈ RP are its P parameters . NN training proceeds , after random initialization θ = θ0 of the weights , by minibatch stochastic gradient descent on the empirical loss L ( θ ) : = 1N ∑N n=1 ` ( fθ ( xn ) , yn ) , given the training dataset { ( x1 , y1 ) , . . . , ( xN , yN ) } . In the classification case , y ∈ { 1 , . . . , D } is a discrete ground-truth label and ` ( fθ ( x ) , y ) : = − log σ ( fθ ( x ) ) y the cross-entropy loss , with σ : RD → RD the softmax-function . For regression , y ∈ RD and ` ( fθ ( x ) , y ) = 12 ‖fθ ( x ) − y‖ 2 is the squared loss . Structured pruning aims to remove weights or rather entire structures from a NN fθ with parameters θ . A structure can be a filter ( channel ) in a convolutional layer , a neuron in a fully-connected layer , or an entire layer in a parallel architecture . We assume the NN in question has been segmented into S structures s = 1 , . . . , S , which can potentially be pruned . We define the notation θs ∈ RP as the vector whose only nonzero components are those weights from θ that belong to structure s.1 Then , a pruning mask is a set M = { s1 , . . . , sm } of structures , and applying a mask M to a NN fθ means to consider the NN with parameter vector θ\M : = θ − ∑ s∈M θs . We now develop our pruning methods that incorporate global correlations into their saliency assessment by efficiently including the second-order loss terms . Our method SOSP-I allows a direct interpretation in terms of individual loss sensitivities , while our main method SOSP-H remains very efficient for large-scale networks due to its Hessian-vector product approximation . 1We require that each weight is assigned to at most one structure . In practice , we associate with each structure those weights that go into the structure , rather than those that leave it . The basic idea behind both our pruning methods is to select the pruning mask M so as to ( approximately ) minimize the joint effect on the network loss λ ( M ) : = ∣∣L ( θ ) − L ( θ\M ) ∣∣ of removing all structures in M , subject to a constraint on the overall pruning ratio . To circumvent this exponentially large search space , we approximate the loss up to second order , so that λ2 ( M ) = ∣∣∣∣∣∣ ∑ s∈M θTs dL ( θ ) dθ − 1 2 ∑ s , s′∈M θTs d2L ( θ ) dθ dθT θs′ ∣∣∣∣∣∣ ( 1 ) collapses to single-structure contributions plus pairwise correlations . The first-order terms λ1 ( s ) : = θs ·dL ( θ ) /dθ in ( 1 ) are efficient to evaluate by computing the gradient dL ( θ ) /dθ ∈ RP once and then a ( sparse ) dot product for every s. In contrast , the network Hessian H ( θ ) : = d2L ( θ ) /dθ2 ∈ RP×P in ( 1 ) is prohibitively expensive to compute or store in full . We therefore propose two different schemes to efficiently overcome this obstacle . We name the full methods SOSP-I ( individual sensitivities ) and SOSP-H ( Hessian-vector product ) . 2.1 SOSP-I : SALIENCY FROM INDIVIDUAL SENSITIVITIES . SOSP-I approximates each individual term θTs H ( θ ) θs′ in ( 1 ) efficiently , as we will show in Eq . ( 5 ) . We will therefore consider an upper bound to Eq . ( 1 ) which measures all sensitivities individually : λI2 ( M ) = ∑ s∈M ∣∣∣∣θTs dL ( θ ) dθ ∣∣∣∣+ 12 ∑ s , s′∈M ∣∣θTs H ( θ ) θs′ ∣∣ . ( 2 ) The absolute values are to prevent cancellations among the individual sensitivities of the network loss to the removal of structures , i.e . the derivatives λ1 ( s ) , and individual correlations θTs H ( θ ) θs′ . While objectives other than λI2 are equally possible in the method , including λ2 and variants with the absolute values not pulled in all the way , we found λI2 to empirically perform best overall . Then , SOSP-I iteratively selects the structures to prune , based on the objective ( 2 ) : Starting from an empty pruning mask M = { } , we iteratively add to M the structure s /∈ M that minimizes the overall sensitivity λI2 ( M ∪ { s } ) . In practice , the algorithm pre-computes the matrix Q ∈ RS×S , Qs , s′ : = 1 2 ∣∣θTs H ( θ ) θs′ ∣∣+ ∣∣∣∣θTs dL ( θ ) dθ ∣∣∣∣ · δs=s′ , ( 3 ) and selects at each iteration a structure s /∈M to prune by argmin s/∈M λI2 ( M ∪ { s } ) − λI2 ( M ) = argmin s/∈M ( Qs , s + 2 ∑ s′∈M Qs , s′ ) , ( 4 ) terminating at the desired pruning ratio . To computeQ efficiently , we show in App . B that the Hessian terms can be approximated as θTs H ( θ ) θs′ ≈ 1 N ′ N ′∑ n=1 ( φ ( xn ) θs ) T Rn ( φ ( xn ) θs′ ) . ( 5 ) Herein , the NN gradient φ ( x ) : = ∇θfθ ( x ) ∈ RD×P forms the basis of the well-established GaussNewton approximation ( see App . B for more details ) used for the Hessian , and the matrices Rn ∈ RD×D are diagonal plus a rank-1 contribution ( App . B ) . For further efficiency gains , the sum runs over a random subsample of size N ′ < N . In practice , one pre-computes all ( sparse ) products φ ( xn ) θs ∈ RD starting from the efficiently computable gradient φ ( xn ) , before aggregating a batch onto the terms θTs H ( θ ) θs′ . Eq . ( 5 ) also has an interpretation as output correlations between certain network modifications , without derivatives ( App . C ) .

Summary: This paper presents two novel approximation techniques for approximating the importance of a filter (the estimated loss difference between pruned and pre-trained networks) using second order approximations. The main difficulties of second order approximation lie in the computational burden of computing the full Hessian matrix. To circumvent this issue, this paper presents two novel ways to do so. The first approach, dubbed SOSP-I, linearize the neural network and only consider the coupling of the loss function. The second approach, dubbed SOSP-H, approximate the loss difference using a Hessian-vector product. Both approaches have much better efficiency compared to using the Hessian and empirically perform a little better than existing approaches. Overall, the approximations seem novel to me and I find the appendix helpful in understanding the derivations. However, their empirical advantages seem rather limited. Additionally, I think some of the related papers are missed and they need to be discussed and compared against.