Structured Pruning Meets Orthogonality

Several recent works empirically found finetuning learning rate is crucial to the final performance in structured neural network pruning . It is shown that the dynamical isometry broken by pruning answers for this phenomenon . How to develop a filter pruning method that maintains or recovers dynamical isometry and is scalable to modern deep networks remains elusive up to now . In this paper , we present orthogonality preserving pruning ( OPP ) , a regularization-based structured pruning method that maintains the dynamical isometry during pruning . Specifically , OPP regularizes the gram matrix of convolutional kernels to encourage kernel orthogonality among the important filters meanwhile driving the unimportant weights towards zero . We also propose to regularize batch-normalization parameters for better preserving dynamical isometry for the whole network . Empirically , OPP can compete with the ideal dynamical isometry recovery method on linear networks . On non-linear networks ( ResNet56/VGG19 , CIFAR datasets ) , it outperforms the available solutions by a large margin . Moreover , OPP can also work effectively with modern deep networks ( ResNets ) on ImageNet , delivering encouraging performance in comparison to many recent filter pruning methods . To our best knowledge , this is the first method that effectively maintains dynamical isometry during pruning for large-scale deep neural networks . 1 INTRODUCTION . Neural network pruning aims to remove the parameters without seriously compromising the performance . It normally consists of three steps ( Reed , 1993 ; Han et al. , 2015 ; 2016b ) : pretrain a dense model ; prune the unnecessary connections or neurons with some criteria ; finetune to regain performance . Pruning is usually categorized into two groups , unstructured pruning ( a.k.a . element-wise pruning ) and structured pruning ( a.k.a . filter pruning or channel pruning ) . The former chooses a scalar weight as the basic pruning element ; the latter chooses a 3d filter as the basic pruning element . In general , structured pruning is more favored for acceleration on commodity hardware because of its consequent regular sparsity ; unstructured pruning results in irregular sparsity , which can be considerable without performance degradation but hard to exploit for acceleration if not using customized hardware ( Han et al. , 2016a ; 2017 ) . Recent structured pruning works ( Renda et al. , 2020 ; Le & Hua , 2021 ; Wang et al. , 2021a ) showed an interesting phenomenon : in the finetuning step , using a larger learning rate ( LR ) helps to achieve a significantly better final performance ( e.g. , ResNet34 pruned at speedup 1.32× can be improved by over 1 % top-1 accuracy on ImageNet ( Deng et al. , 2009 ) using finetuning LR 1e-2 vs. 1e-3 ) . The reason behind is shown by ( Wang et al. , 2021a ) relevant to dynamical isometry ( Saxe et al. , 2014 ) , a nice property of neural networks that are easy to train without gradient vanishing or explosion ( Glorot & Bengio , 2010 ; Pascanu et al. , 2013 ) . They mainly made two observations for explanation . First , the weights removal operation immediately breaks the dynamical isometry of the pretrained network . Second , SGD training in finetuning can help recover it ; a larger LR help recover it faster and better , thus making the final performance stronger . Although ( Wang et al. , 2021a ) provided a sound explanation , a more practical issue is how to recover the broken dynamical isometry or maintain it during pruning . In this regard , ( Wang et al. , 2021a ) proposed to apply weight orthogonalization based on QR decomposition ( Trefethen & Bau III , 1997 ; Mezzadri , 2006 ) to the pruned model . However , the method was shown to only work for linear networks . On modern deep convolutional neural networks ( CNNs ) , the method is still far from satisfactory . In this paper , we present orthogonality preserving pruning ( OPP ) , a new filter pruning method that maintains dynamical isometry well during pruning . The main idea of OPP is to promote kernel orthogonality among the kept filters meanwhile pushing the weights to be pruned rather close to zero . By doing so , the subsequent weights removal operation will barely hurt the dynamical isometry of the network . Specifically , we propose to regularize the gram matrix of weights : all the entries representing the correlation between the pruned filters and the others are encourage to diminish to zero . This is the first technical contribution of our method . The second one lies in how to treat the cross-correlation entries of kept filters . Inspired by the proposed orthogonalization initialization in ( Saxe et al. , 2014 ) , we add a kernel orthogonality term to the kept filters , which promotes dynamical isometry during pruning . In addition , modern deep models are typically equipped with batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) . However , previous filter pruning papers rarely explicitly take BN into account ( except two ( Liu et al. , 2017 ; Ye et al. , 2018 ) ; the differences between our work and theirs will be discussed in Sec . 3.2 ) to mitigate the side effect when it is removed because its associated filter is removed . By the idea of preserving dynamical isometry , they should not be ignored since their removal breaks dynamical isometry as well . Therefore , we propose to regularize the learnable parameters of BN to minimize the influence of its absence . Empirically , the proposed pruning algorithm is easy to implement and delivers encouraging results compared to many existing filter pruning methods . Notably , this is the first method that effectively maintains dynamical isometry during pruning on modern large-scale deep networks . Contributions . We make three contributions in this paper : • We present the first filter pruning method ( orthogonality preserving pruning ) that can effectively maintain dynamical isometry when pruning modern deep networks , through a customized weight gram matrix as regularization target . • Apart from weight regularization , we also propose to regularize the batch normalization parameters to better maintain dynamical isometry . This has been overlooked by most previous pruning papers , while we show it is an indispensable part if we aim to maintain dynamical isometry for the whole network . • Practically , the proposed method is scalable to modern large-scale deep neural networks ( e.g. , ResNets ) and datasets ( e.g. , ImageNet ) . It achieves promising pruning performance in comparison to many state-of-the-art filter pruning methods . 2 RELATED WORK . Neural network pruning . In terms of pruning granularity , pruning methods mainly fall into structured pruning ( a.k.a . filter pruning or channel pruning ) ( Li et al. , 2017 ; Wen et al. , 2016 ; He et al. , 2017 ; 2018a ; Wang et al. , 2021c ) and unstructured pruning ( a.k.a . element-wise pruning ) ( Han et al. , 2015 ; 2016b ; LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Singh & Alistarh , 2020 ) . Structured pruning results in regular sparsity after pruning , easy to be translated to acceleration on commodity hardware . In contrast , unstructured pruning produces irregular sparsity , beneficial to compression while hard to leverage for practical acceleration ( Wen et al. , 2016 ; Wang et al. , 2019b ) unless with special hardware support ( Han et al. , 2016a ; 2017 ) . For more comprehensive coverage , we recommend surveys ( Sze et al. , 2017 ; Cheng et al. , 2018a ; b ; Deng et al. , 2020 ; Wang et al. , 2021b ) . We focus on filter pruning in this work for acceleration . Most pruning papers focus on finding a better pruning criterion to select unimportant parameters to remove . The solutions primarily follow two paradigms ( Reed , 1993 ) : regularization-based and importance-based . The former selects unimportant parameters by adding a sparsity-inducing penalty term , which is jointly optimized with the original loss objective function ( e.g. , ( Wen et al. , 2016 ; Lebedev & Lempitsky , 2016 ; Louizos et al. , 2018 ; Liu et al. , 2017 ; Ye et al. , 2018 ) ) . The latter selects unimportant parameters through certain derived mathematical formula ( e.g. , ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; 2016b ; Li et al. , 2017 ; Molchanov et al. , 2017 ; 2019 ) ) . Notably , there is no strict boundary between the two paradigms . Several works ( Ding et al. , 2018 ; Wang et al. , 2019b ; 2021c ) manage to get the best from both worlds – they select unimportant weights by an importance criterion and add a penalty term for sparsity as well . Our method in this paper also belongs to this group , while it achieves stronger performances . Dynamical isometry and orthogonality . Dynamical isometry was first introduced by ( Saxe et al. , 2014 ) , where it can be achieved ( for linear MLP models ) by the orthogonality of weight matrix at initialization . Recent works on this topic mainly focus on how to maintain dynamical isometry during training instead of only for initialization ( Xie et al. , 2017 ; Huang et al. , 2018 ; Bansal et al. , 2018 ; Huang et al. , 2020 ; Wang et al. , 2020 ) . These methods are developed independent of pruning , thus not directly relevant to the proposed method in this work . However , the insights from these works inspire us to our proposed approach ( see Sec . 3.2 ) and possibly more in the future . Pruning + dynamical isometry . One particular paper that inspires us to this work is ( Wang et al. , 2021a ) , where the authors leverage dynamical isometry ( Saxe et al. , 2014 ) to explain the performance boosting effect ( also noted by ( Renda et al. , 2020 ; Le & Hua , 2021 ) ) of using a larger finetuning LR in pruning : pruning hurts dynamical isometry ; finetuning can recover dynamical isometry and a larger LR helps recover it faster ( and possibly better ) , thus delivering superior final performance . The explanation further clears the mystery about the value of network pruning ( Liu et al. , 2019 ; Crowley et al. , 2018 ) . Previously , ( Liu et al. , 2019 ; Crowley et al. , 2018 ) found the small models can be trained from scratch with comparable accuracy to the counterparts pruned from a pretrained large model , thus they argue no value of inheriting weights in structured pruning . ( Wang et al. , 2021a ) pointed out that this argument is actually built upon sub-optimal finetuning LR setups . With the proper ones , filter pruning consistently outperforms training from scratch . 3 METHODOLOGY . 3.1 PRELIMINARIES : DYNAMICAL ISOMETRY AND ORTHOGONALITY . The definition of dynamical isometry is that the input-output Jacobian of a network has as many singular values ( JSVs ) as possible close to 1 ( Saxe et al. , 2014 ) . With it , the error signal can preserve its norm under propagation without serious amplification or attenuation , which in turn helps the convergence of ( very deep ) networks . For a single fully-connected layer W , a sufficient and necessary condition to realize dynamical isometry is orthogonality , i.e. , WTW = I , as shown below , y = Wx , ||y|| = √ yTy = √ xTWTWx = ||x|| , iff . WTW = I , ( 1 ) where I stands for identity matrix . Orthogonality of a weight matrix can be easily realized by matrix orthogonalization techniques such as QR decomposition ( Trefethen & Bau III , 1997 ; Mezzadri , 2006 ) . Exact ( namely all the Jacobian singular values are exactly 1 ) dynamical isometry can be achieved for linear networks since multiple linear layers essentially reduce to a single 2d weight matrix . In contrast , the convolutional and non-linear cases are much complicated . Previous work ( Wang et al. , 2021a ) has shown that merely considering convolution or ReLU ( Nair & Hinton , 2010 ) renders the weight orthogonalization method much less effective in terms of recovering dynamical isometry after pruning , let alone considering modern deep networks with BN ( Ioffe & Szegedy , 2015 ) and residuals ( He et al. , 2016 ) . The primary goal of our paper is to bridge this gap . Following the seminal work of ( Saxe et al. , 2014 ) , several papers propose to maintain orthogonality during training instead of sorely for the initialization . There are primarily two groups of orthogonalization methods for convolutional neural networks : kernel orthogonality ( Xie et al. , 2017 ; Huang et al. , 2018 ; 2020 ) and orthogonal convolution ( Wang et al. , 2020 ) : KKT = I ⇒ Lorth = KKT − I , / kernel orthogonality KKT = I ⇒ Lorth = KKT − I , / orthogonal convolution ( 2 ) where clearly the difference lies in the weight matrix K vs. K. ( 1 ) K denotes the original weight matrix in a convolutional layer . Weights of a conv layer make up a 4d tensor RN×C×H×W , where N stands for the number of output channels , C for the number of input channels , H and W for the height and width of conv kernel . Then , K is a reshaped version of the 4d tensor : K ∈ RN×CHW ( if N < CHW ; otherwise , K ∈ RCHW×N ) . ( 2 ) In contrast , K ∈ RNHfoWfo×CHfiWfi stands for the doubly block-Toeplitz ( DBT ) matrix representation of K ( Hfo stands for the output feature map height , Hfi for the input feature map height . Wfo and Wfi can be inferred the same way for width ) . It was showed by ( Wang et al. , 2020 ) that orthogonal convolution is more effective than kernel orthogonality ( Xie et al. , 2017 ) in that the latter is only a necessary but insufficient condition of the former . In this work , we will evaluate both methods to see how effective they are in maintaining/recovering the broken dynamical isometry .

Summary: Dynamic isometry is shown to be a useful property that enable effective gradient propagation through the forward/backward. However, pruning will largely damage such a structure. This paper studies how to maintain the “dynamic isometry” property during pruning. Specifically, after getting an initial assessment of filter importance, the algorithm will maintain the partially kernel orthogonality of the important filters. They also propose to regularize the BN parameters to future boost the performance.