SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning

1 INTRODUCTION . Deep neural networks have consistently grown in size over the last years with increasing performance . However , this increase in size leads to slower inference , higher computational requirements and higher cost . To reduce the size of the networks without affecting their performance , a large number of pruning algorithms have been proposed ( e.g. , LeCun et al. , 1990 ; Hassibi et al. , 1993 ; Reed , 1993 ; Han et al. , 2015 ; Blalock et al. , 2020 ) . Pruning can either be unstructured , i.e . removing individual weights , or structured , i.e . removing entire substructures like nodes or channels . Single-shot pruning methods , as investigated in this work , usually consist of three steps : 1 ) training , 2 ) pruning , 3 ) another training step often referred to as fine-tuning . Unstructured pruning can significantly reduce the number of parameters of a neural network with only little loss in the accuracy , but the resulting networks often show only a marginal improvement in training and inference time , unless specialized hardware is used ( He et al. , 2017 ) . In contrast , structured pruning can directly reduce inference time and even training time when applied at initialization ( Lee et al. , 2018 ) . To exploit these advantages , in this work , we focus on structured pruning . Global pruning removes structure by structure from all available structures of a network until a predefined percentage of pruned structures is reached . Recent examples for global structured pruning methods are NN Slimming ( Liu et al. , 2017 ) , C-OBD and EigenDamage ( Wang et al. , 2019a ) . Local pruning , on the other hand , first subdivides all global structures into subsets ( e.g . layers ) and removes a percentage of structures of each subset . Recent examples for local pruning methods are HRank ( Lin et al. , 2019 ) , CCP ( Peng et al. , 2019 ) , FPGM ( He et al. , 2019 ) and Variational Pruning ( Zhao et al. , 2019 ) . Most local pruning schemes use a predefined layer-wise pruning ratio , which fixes the percentage of structures removed per layer . While this prevents the layers from collapsing , it also reduces some of the degrees of freedom , since some layers may be less important than others . Other local pruning methods like AMC ( He et al. , 2018 ) learn the layer-wise pruning ratios in a first step . A key challenge for global saliency-based pruning is to find an objective which can be efficiently calculated to make the approach scalable to large-scale , modern neural networks . While second-order pruning methods are usually more accurate than first-order methods ( Molchanov et al. , 2019 ) , calculating the full second-order saliency objective is intractable for modern neural networks . Therefore , most saliency-based pruning methods such as OBD ( e.g. , LeCun et al. , 1990 ) or C-OBD ( Wang et al. , 2019a ) evaluate the effect of removing a single weight or structure on the loss of the neural network in isolation . However , this neglects possible correlations between different structures , which are captured by the off-diagonal second-order terms , and hence , can significantly harm the estimation of the sensitivities . Finding a global second-order method that both considers off-diagonal terms and scales to modern neural networks is still an unsolved research question . Our main goal in this work is to devise a simple and efficient second-order pruning method that considers all global correlations for structured sensitivity pruning . In addition , we want to highlight the benefits that such methods may have over other structured global and local pruning schemes . Our contributions are as follows : • We develop two novel saliency-based pruning methods for second-order structured pruning ( SOSP ) and analyze them theoretically . We show that both of our methods drastically improve on the complexity of a naive second-order approach , which is usually intractable for modern neural networks . Further , we show that our SOSP-H method , which is based on fast Hessian-vector products , has the same low complexity as first-order methods , while taking the full Hessian into account . • We compare the performance and the scaling of SOSP-H to that of SOSP-I , which is based on the well-known Gauss-Newton approximation . While both methods perform on par , SOSP-H shows better scaling . We then benchmark our SOSP methods against a variety of state-of-the-art pruning methods and show that they achieve comparable or better results at lower computational costs for pruning . • We exploit the structure of the pruning masks found by our SOSP methods to widen architectural bottlenecks , which further improves the performance of the pruned networks . We diagnose layers with disproportionally low pruning ratios as architectural bottlenecks . PyTorch code implementing our method will be published upon acceptance of this manuscript . Related work is discussed in the light of our results in the Discussion section ( Sec . 4 ) . 2 SOSP : SECOND-ORDER STRUCTURED PRUNING . A neural network ( NN ) maps an input x ∈ Rd to an output fθ ( x ) ∈ RD , where θ ∈ RP are its P parameters . NN training proceeds , after random initialization θ = θ0 of the weights , by minibatch stochastic gradient descent on the empirical loss L ( θ ) : = 1N ∑N n=1 ` ( fθ ( xn ) , yn ) , given the training dataset { ( x1 , y1 ) , . . . , ( xN , yN ) } . In the classification case , y ∈ { 1 , . . . , D } is a discrete ground-truth label and ` ( fθ ( x ) , y ) : = − log σ ( fθ ( x ) ) y the cross-entropy loss , with σ : RD → RD the softmax-function . For regression , y ∈ RD and ` ( fθ ( x ) , y ) = 12 ‖fθ ( x ) − y‖ 2 is the squared loss . Structured pruning aims to remove weights or rather entire structures from a NN fθ with parameters θ . A structure can be a filter ( channel ) in a convolutional layer , a neuron in a fully-connected layer , or an entire layer in a parallel architecture . We assume the NN in question has been segmented into S structures s = 1 , . . . , S , which can potentially be pruned . We define the notation θs ∈ RP as the vector whose only nonzero components are those weights from θ that belong to structure s.1 Then , a pruning mask is a set M = { s1 , . . . , sm } of structures , and applying a mask M to a NN fθ means to consider the NN with parameter vector θ\M : = θ − ∑ s∈M θs . We now develop our pruning methods that incorporate global correlations into their saliency assessment by efficiently including the second-order loss terms . Our method SOSP-I allows a direct interpretation in terms of individual loss sensitivities , while our main method SOSP-H remains very efficient for large-scale networks due to its Hessian-vector product approximation . 1We require that each weight is assigned to at most one structure . In practice , we associate with each structure those weights that go into the structure , rather than those that leave it . The basic idea behind both our pruning methods is to select the pruning mask M so as to ( approximately ) minimize the joint effect on the network loss λ ( M ) : = ∣∣L ( θ ) − L ( θ\M ) ∣∣ of removing all structures in M , subject to a constraint on the overall pruning ratio . To circumvent this exponentially large search space , we approximate the loss up to second order , so that λ2 ( M ) = ∣∣∣∣∣∣ ∑ s∈M θTs dL ( θ ) dθ − 1 2 ∑ s , s′∈M θTs d2L ( θ ) dθ dθT θs′ ∣∣∣∣∣∣ ( 1 ) collapses to single-structure contributions plus pairwise correlations . The first-order terms λ1 ( s ) : = θs ·dL ( θ ) /dθ in ( 1 ) are efficient to evaluate by computing the gradient dL ( θ ) /dθ ∈ RP once and then a ( sparse ) dot product for every s. In contrast , the network Hessian H ( θ ) : = d2L ( θ ) /dθ2 ∈ RP×P in ( 1 ) is prohibitively expensive to compute or store in full . We therefore propose two different schemes to efficiently overcome this obstacle . We name the full methods SOSP-I ( individual sensitivities ) and SOSP-H ( Hessian-vector product ) . 2.1 SOSP-I : SALIENCY FROM INDIVIDUAL SENSITIVITIES . SOSP-I approximates each individual term θTs H ( θ ) θs′ in ( 1 ) efficiently , as we will show in Eq . ( 5 ) . We will therefore consider an upper bound to Eq . ( 1 ) which measures all sensitivities individually : λI2 ( M ) = ∑ s∈M ∣∣∣∣θTs dL ( θ ) dθ ∣∣∣∣+ 12 ∑ s , s′∈M ∣∣θTs H ( θ ) θs′ ∣∣ . ( 2 ) The absolute values are to prevent cancellations among the individual sensitivities of the network loss to the removal of structures , i.e . the derivatives λ1 ( s ) , and individual correlations θTs H ( θ ) θs′ . While objectives other than λI2 are equally possible in the method , including λ2 and variants with the absolute values not pulled in all the way , we found λI2 to empirically perform best overall . Then , SOSP-I iteratively selects the structures to prune , based on the objective ( 2 ) : Starting from an empty pruning mask M = { } , we iteratively add to M the structure s /∈ M that minimizes the overall sensitivity λI2 ( M ∪ { s } ) . In practice , the algorithm pre-computes the matrix Q ∈ RS×S , Qs , s′ : = 1 2 ∣∣θTs H ( θ ) θs′ ∣∣+ ∣∣∣∣θTs dL ( θ ) dθ ∣∣∣∣ · δs=s′ , ( 3 ) and selects at each iteration a structure s /∈M to prune by argmin s/∈M λI2 ( M ∪ { s } ) − λI2 ( M ) = argmin s/∈M ( Qs , s + 2 ∑ s′∈M Qs , s′ ) , ( 4 ) terminating at the desired pruning ratio . To computeQ efficiently , we show in App . B that the Hessian terms can be approximated as θTs H ( θ ) θs′ ≈ 1 N ′ N ′∑ n=1 ( φ ( xn ) θs ) T Rn ( φ ( xn ) θs′ ) . ( 5 ) Herein , the NN gradient φ ( x ) : = ∇θfθ ( x ) ∈ RD×P forms the basis of the well-established GaussNewton approximation ( see App . B for more details ) used for the Hessian , and the matrices Rn ∈ RD×D are diagonal plus a rank-1 contribution ( App . B ) . For further efficiency gains , the sum runs over a random subsample of size N ′ < N . In practice , one pre-computes all ( sparse ) products φ ( xn ) θs ∈ RD starting from the efficiently computable gradient φ ( xn ) , before aggregating a batch onto the terms θTs H ( θ ) θs′ . Eq . ( 5 ) also has an interpretation as output correlations between certain network modifications , without derivatives ( App . C ) .

Summary: This paper presents saliency-based second-order structured pruning methods, namely SOSP-I and SOSP-H. The proposed methods are designed to capture the correlations among all structures and layers, known as second-order structure (Hessian). In particular, SOSP-I employs Hessian approximation while SOSP-H employs exact Hessian. Overall, the idea of this paper is very clear, and I kinda like the discussion part of it. The expand-pruning is also interesting.