A Gradient Flow Framework For Analyzing Network Pruning

1 INTRODUCTION . The use of Deep Neural Networks ( DNNs ) in intelligent edge systems has been enabled by extensive research on model compression . “ Pruning ” techniques are commonly used to remove “ unimportant ” filters to either preserve or promote specific , desirable model properties . Most pruning methods were originally designed to compress trained models , with the goal of reducing inference costs only . For example , Li et al . ( 2017 ) ; He et al . ( 2018 ) proposed to remove filters with small ` 1/ ` 2 norm , thus ensuring minimal change in model output . Molchanov et al . ( 2017 ; 2019 ) ; Theis et al . ( 2018 ) proposed to preserve the loss of a model , generally using Taylor expansions around a filter ’ s parameters to estimate change in loss as a function of its removal . Recent works focus on pruning models at initialization ( Lee et al . ( 2019 ; 2020 ) ) or after minimal training ( You et al . ( 2020 ) ) , thus enabling reduction in both inference and training costs . To estimate the impact of removing a parameter , these methods use the same importance measures as designed for pruning trained models . Since such measures focus on preserving model outputs or loss , Wang et al . ( 2020 ) argue they are not well-motivated for pruning models early-on in training . However , in this paper , we demonstrate that if the relationship between importance measures used for pruning trained models and the evolution of model parameters is established , their use early-on in training can be better justified . In particular , we employ gradient flow1 to develop a general framework that relates state-of-theart importance measures used in network pruning through the norm of model parameters . This framework establishes the relationship between regularly used importance measures and the evolution of a model ’ s parameters , thus demonstrating why measures designed to prune trained models also 1gradient flow refers to gradient descent with infinitesimal learning rate ; see Equation 6 for a short primer . perform well early-on in training . More generally , our framework enables better understanding of what properties make a parameter dispensable according to a particular importance measure . Our findings follow . ( i ) Magnitude-based pruning measures remove parameters that contribute least to reduction in loss . This enables magnitude-based pruned models to achieve faster convergence than magnitude-agnostic measures . ( ii ) Loss-preservation based measures remove parameters with the least tendency to change , thus preserving first-order model evolution dynamics . This shows the use of loss-preservation is justified for pruning models early-on in training as well . ( iii ) Gradient-norm based pruning is linearly related to second-order model evolution dynamics . Increasing gradient norm via pruning for even slightly trained models can permanently damage earlier layers , producing poorly performing architectures . This behavior is a result of aggressively pruning filters that maximally increase model loss . We validate our claims on several VGG-13 , MobileNet-V1 , and ResNet-56 models trained on CIFAR-10 and CIFAR-100 . 2 RELATED WORK . Several pruning frameworks define importance measures to estimate the impact of removing a parameter . Most popular importance measures are based on parameter magnitude ( Li et al . ( 2017 ) ; He et al . ( 2018 ) ; Liu et al . ( 2017 ) ) or loss preservation ( Molchanov et al . ( 2019 ; 2017 ) ; Theis et al . ( 2018 ) ; Gao et al . ( 2019 ) ) . Recent works show that using these measures , models pruned at initialization ( Lee et al . ( 2019 ) ; Wang et al . ( 2020 ) ; Hayou et al . ( 2021 ) ) or after minimal training ( You et al . ( 2020 ) ) achieve final performance similar to the original networks . Since measures for pruning trained models are motivated by output or loss preservation , Wang et al . ( 2020 ) argue they may not be well suited for pruning models early-on in training . They thus propose GraSP , a measure that promotes preservation of parameters that increase the gradient norm . Despite its success , the foundations of network pruning are not well understood . Recent work has shown that good “ subnetworks ” that achieve similar performance to the original network exist within both trained ( Ye et al . ( 2020 ) ) and untrained models ( Frankle & Carbin ( 2019 ) ; Malach et al . ( 2020 ) ; Pensia et al . ( 2020 ) ) . These works thus prove networks can be pruned without loss in performance , but do not indicate how a network should be pruned , i.e , which importance measures are preferable . In fact , Liu et al . ( 2019 ) show reinitializing pruned models before retraining rarely affects their performance , indicating the consequential differences among importance measures are in the properties of architectures they produce . Since different importance measures perform differently ( see Appendix E ) , analyzing popular measures to determine which model properties they tend to preserve can reveal which measures lead to better-performing architectures . From an implementation standpoint , pruning approaches can be placed in two categories . The first , structured pruning ( Li et al . ( 2017 ) ; He et al . ( 2018 ) ; Liu et al . ( 2017 ) ; Molchanov et al . ( 2019 ; 2017 ) ; Gao et al . ( 2019 ) ) , removes entire filters , thus preserving structural regularity and directly improving execution efficiency on commodity hardware platforms . The second , unstructured pruning ( Han et al . ( 2016b ) ; LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ) , is more fine-grained , operating at the level of individual parameters instead of filters . Unstructured pruning has recently been used to reduce computational complexity as well , but requires specially designed hardware ( Han et al . ( 2016a ) ) or software ( Elsen et al . ( 2020 ) ) that are capable of accelerating sparse operations . By clarifying benefits and pitfalls of popular importance measures , our work aims to ensure practitioners are better able to make informed choices for reducing DNN training/inference expenditure via network pruning . Thus , while results in this paper are applicable in both structured and unstructured settings , our experimental evaluation primarily focuses on structured pruning early-on in training . Results on unstructured pruning are relegated to Appendix H . 3 PRELIMINARIES : CLASSES OF STANDARD IMPORTANCE MEASURES . In this section , we review the most successful classes of importance measures for network pruning . These measures will be our focus in subsequent sections . We use bold symbols to denote vectors and italicize scalar variables . Consider a model that is parameterized as Θ ( t ) at time t. We denote the gradient of the loss with respect to model parameters at time t as g ( Θ ( t ) ) , the Hessian as H ( Θ ( t ) ) , and the model loss as L ( Θ ( t ) ) . A general model parameter is denoted as θ ( t ) . The importance of a set of parameters Θp ( t ) is denoted as I ( Θp ( t ) ) . Magnitude-based measures : Both ` 1 norm ( Li et al . ( 2017 ) ) and ` 2 norm ( He et al . ( 2018 ) ) have been successfully used as magnitude-focused importance measures and generally perform equally well . Due to its differentiability , ` 2 norm can be analyzed using gradient flow and will be our focus in the following sections . I ( Θp ( t ) ) = ‖Θp ( t ) ‖22 = ∑ θi∈Θp ( θi ( t ) ) 2 . ( 1 ) Loss-preservation based measures : These measures generally use a first-order Taylor decomposition to determine the impact removing a set of parameters has on model loss . Most recent methods ( Molchanov et al . ( 2019 ; 2017 ) ; Ding et al . ( 2019 ) ; Theis et al . ( 2018 ) ) for pruning trained models are variants of this method , often using additional heuristics to improve their performance . L ( Θ ( t ) −Θp ( t ) ) − L ( Θ ( t ) ) ≈ −ΘTp ( t ) g ( Θ ( t ) ) . ( 2 ) The equation above implies that the loss of a pruned model is higher ( lower ) than the original model if parameters with a negative ( positive ) value for ΘTp ( t ) g ( Θ ( t ) ) are removed . Thus , for preserving model loss , the following importance score should be used . I ( Θp ( t ) ) = ∣∣ΘTp ( t ) g ( Θ ( t ) ) ∣∣ . ( 3 ) Increase in gradient-norm based measures : Wang et al . ( 2020 ) argue loss-preservation based methods are not well-motivated for pruning models early-on in training . They thus propose GraSP , an importance measure that prunes parameters whose removal increases the gradient norm and can enable fast convergence for a pruned model . ‖g ( Θ ( t ) −Θp ( t ) ) ‖22 − ‖g ( Θ ( t ) ) ‖ 2 2 ≈ −2Θ T p ( t ) H ( Θ ( t ) ) g ( Θ ( t ) ) . ( 4 ) The above equation implies that the gradient norm of a pruned model is higher than the original model if parameters with a negative value for ΘTp ( t ) H ( Θ ( t ) ) g ( Θ ( t ) ) are removed . This results in the following importance score . I ( Θp ( t ) ) = Θ T p ( t ) H ( Θ ( t ) ) g ( Θ ( t ) ) . ( 5 ) As mentioned before , these importance measures were introduced for pruning trained models ( except for GraSP ) , but are also used for pruning models early-on in training . In the following sections , we revisit the original goals for these measures , establish their relationship with evolution of model parameters over time , and provide clear justifications for their use early-on in training . 4 GRADIENT FLOW AND NETWORK PRUNING . Gradient flow , or gradient descent with infinitesimal learning rate , is a continuous-time version of gradient descent . The evolution over time of model parameters , gradient , and loss under gradient flow can be described as follows . ( Parameters over time ) ∂Θ ( t ) ∂t = −g ( Θ ( t ) ) ; ( Gradient over time ) ∂g ( Θ ( t ) ) ∂t = −H ( Θ ( t ) ) g ( Θ ( t ) ) ; ( Loss over time ) ∂L ( t ) ∂t = −‖g ( Θ ( t ) ) ‖22 . ( 6 ) Recall that standard importance measures based on loss-preservation ( Equation 3 ) or increase in gradient-norm ( Equation 5 ) are derived using a first-order Taylor series approximation , making them exactly valid under the continuous scenario of gradient flow . This indicates analyzing the evolution of model parameters via gradient flow can provide useful insights into the relationships between different importance measures . To this end , we use gradient flow to develop a general framework that relates different classes of importance measures through the norm of model parameters . As we develop this framework , we explain the reasons why importance measures defined for pruning trained models are also highly effective when used for pruning early-on in training . In Appendix B , we further extend this framework to models trained using stochastic gradient descent , showing that up to a first-order approximation , the observations developed here are valid under SGD training too .

Summary: This paper proposes a detailed analysis on pruning heuristics, and its applications to early pruning. It thoroughly analyzed magnitude-based pruning, loss-preservation based pruning, and gradient-norm based pruning. The paper demonstrated the results on CIFAR-10 and CIFAR-100 datasets. it's very timely research to guide the audience which heuristic is better. My major concern is the novelty over existing pruning heuristics, since the techniques have all been proposed before. The other concern is the evaluation and the scale of the dataset. Given the results in table 2 different by less than a percent, and Cifar training is very noisy, it's hard to tell the difference. Just like the Lottery Ticket hypothesis works on Cifar but does not work on ImageNet, different pruning heuristics needs to be verified on the large scale ImageNet dataset in order to be convincing.