Structured Pruning Meets Orthogonality
Authors: Huan Wang
Yun Fu
Venue: ICLR 2022 Submitted (2022)
Keywords: network pruning
structured pruning
dynamical isometry
model compression

Abstract: Several recent works empirically found finetuning learning rate is crucial to the final performance in structured neural network pruning. It is shown that the \emph{dynamical isometry} broken by pruning answers for this phenomenon. How to develop a filter pruning method that maintains or recovers dynamical isometry \emph{and} is scalable to modern deep networks remains elusive up to now. In this paper, we present \emph{orthogonality preserving pruning} (OPP), a regularization-based structured pruning method that maintains the dynamical isometry during pruning. Specifically, OPP regularizes the gram matrix of convolutional kernels to encourage kernel orthogonality among the important filters meanwhile driving the unimportant weights towards zero. We also propose to regularize batch-normalization parameters for better preserving dynamical isometry for the whole network. Empirically, OPP can compete with the \emph{ideal} dynamical isometry recovery method on linear networks. On non-linear networks (ResNet56/VGG19, CIFAR datasets), it outperforms the available solutions \emph{by a large margin}. Moreover, OPP can also work effectively with modern deep networks (ResNets) on ImageNet, delivering encouraging performance in comparison to many recent filter pruning methods. To our best knowledge, this is the \emph{first} method that effectively maintains dynamical isometry during pruning for \emph{large-scale} deep neural networks.

Under review as a conference paper at ICLR 2022
STRUCTURED PRUNING MEETS ORTHOGONALITY
Anonymous authors
Paper under double-blind review
ABSTRACT
Several recent works empirically found finetuning learning rate is crucial to the final performance in structured neural network pruning. It is shown that the dy- namical isometry broken by pruning answers for this phenomenon. How to de- velop filter pruning method that maintains recovers dynamical isometry and is scalable to modern deep networks remains elusive up to now. In this paper, we present orthogonality preserving pruning (OPP), regularization-hasel structured pruning method that maintains the dynamical isometry during pruning. Specifi- cally, OPP regularizes the gram matrix of convolutional kernels encourage ker- nel orthogonality among the important filters meanwhile driving the unimportant weights towards zero. We also propose to regularize batch- noccchoomalizationn parame- ters for better preserving dynamical isometry for the whole network. Empirically, OPP can compete with the ideal dynamical isometry recovery method on linear networks. On non-linear networks (ResNet56/VGGi19 CIFAR datasets), it out- performs the available solutions by a large margin. Morcover, OPP can also work effectively with modern deep networks (ResNets) on ImageNet, delivering couraging performance in comparison to many recent filter pruning methods. To best knowledge, this is the first method that effectively maintains dynamical isometry during pruning for large-scale deep neural networks.
INTRODUCTION
Neural network pruning aims to remove the parameters without seriously compromising the perfor- mance. It normally consists of three steps (Reed, 1993; Han et al., 2015; 2016b): pretrain a dense model; prune the unnecessary connections mance. Pruning is usually categorized into two groups, unstructured pruning (a.k.a. element-wiss wise pruning) and structured pruning (a.k.a. filter pruning or channel pruning). The former chooses a scalar weight as the basic pruning element; the latter chooses a 3d filter the basic pruning el- ement. In general, structured pruning is more favored for acceleration on commodity hardware because of its consequent regular sparsity; unstructured pruning results in irregular sparsity, which can be considerable without performance degradation but hard to exploit for acceleration if not using customized hardware (Han et al., 2016a; 2017).
neurons with some criteria; finetune regain perfor-
Recent structured pruning works (Renda et al., 2020; Le & Hua, 2021; Wang et al., 2021a) showed an interesting phenomenon: in the finetuning step, using larger learning rate (LR) helps achieve a significantly better final performance (e.g., ResNet34 pruned at speedup 1.32 x can be improved by over 1% top-1 accuracy on ImageNet (Deng et al., 2009) using finetuning LR 1e-2 vs. 3). The reason behind is shown by (Wang et al., a))1a) relevant to dynamical isometry (Saxe et al., 2014), nice property of neural networks that are easy train without gradient vanishing or ex- plosion (Glorot & Bengio, 2010; Pascanu et al., 2013). They mainly made two observations for explanation. First, the weights removal operation immediately breaks the dynamical isometry of the pretrained network. Second, SGD training in finetuning can help recover it; larger LR help recover it faster and better, thus making the final performance stronger. Although (Wang et al., 2021a) provided a sound explanation, a more practical issue is how recover the broken dynamical isometry or maintain it during pruning. In this regard, (Wang et al., 2021a) proposed to apply weight orthogonalization based on QR decomposition (Trefethen & Bau III, 1997; Mezzadri, 2006) to the pruned model. However, the method was shown to only work for linear networks. On modern deep convolutional neural networks (CNNs), the method is still far from satisfactory.

---

Under review as a conference paper at ICLR 2022
In this paper, we present orthogonality preserving pruning (OPP), new filter pruning method that maintains dynamical isometry well during pruning. The main idea of OPP is to promote kernel or- thogonality among the kept filters meanwhile pushing the weights to be pruned rather close to zero. By doing so, the subsequent weights removal operation will barely hurt the dynamical isometry of the network. Specifically, we propose regularize the gram matrix of weights: all the entries representing the correlation between the pruned filters and the others are encourage to diminish to zero. This is the first technical contribution of our method. The second one lies in how to treat the correloornlation entries of kept filters. Inspired by the proposed orthogonalization initializa- tion in (Saxe et al., 2014), we add a kernel orthogonality term to the kept filters, which promotes dynamical isometry during pruning.
In addition, modern deep models are typically equipped with batch normalization (BN) (loffe & Szegedy, 2015). However, previous filter pruning papers rarely explicitly take BN into account (except two (Liu et al., 2017; Ye al., 2018); the differences between our work and theirs will be discussed in Sec. 3.2) mitigate the side effect when it is removed because its associated fil- ter is removed. By the idea of preserving dynamical isometry, they should not be ignored since their removal breaks dynamical isometry well. Therefore, we propose to regularize the learnable parameters of BN to minimize the influence of its absence.
Empirically, the proposed pruning algorithm is easy to implement and delivers encouraging results compared to many existing filter pruning methods. Notably, this is the first method that effectively maintains dynamical isometry during pruning on modern large-scale deep networks.
Contributions. We make three contributions in this paper:
We present the first filter pruning method (orthogonality preserving pruning) that can ef- fectively maintain dynamical isometry when pruning modern deep networks, through a customized weight gram matrix as regularization target.
Apart from weight regularization, we also propose to regularize the batch normalization pa- rameters to better maintain dynamical isometry. This has been overlooked by most previous pruning papers, while we show it is an indispensable part if we aim to maintain dynamical isometry for the whole network.
Practically, the proposed method is scalable to modern large-scale deep neural networks (e.g., ResNets) and datasets (e.g., ImageNet). It achieves promising pruning performance in comparison to many state-of-the-ar filter pruning methods.
2 RELATED WORK
Neural network pruning. terms of pruning granularity, pruning methods mainly fall into struc- tured pruning (a.k.a. filter pruning or channel pruning) (Li et al., 2017: Wen et al., 2016; He et al., 2017; 2018a; Wang et al., 2021c) and unstructured pruning (a.k.a. element-wise pruning) (Han et al., 2015; 2016b; LeCun et al., 1990: Hassibi & Stork, 1993; Singh & Alistarh, 2020). Structured prun- ing results in regular sparsity after pruning, easy to be translated to acceleration on commodity hardware. In contrast, unstructured pruning produces irregular sparsity, beneficial to compression while hard to leverage for practical acceleration (Wen et al., 2016; Wang et al., 2019b) unless with special hardware support (Han et al., 2016a; 2017). For more comprehensive coverage, we recom- mend surveys (Sze et al., 2017; Cheng et al., 2018;;b; Deng et al., 2020; Wang et al., 2021b). We focus on filter pruning in this work for acceleration.
Most pruning papers focus on finding a better pruning criterion to select unimportant parameters to remove. The solutions primarily follow two paradigms (Reed, 1993): regularization- based and importance-based. The former selects unimportant parameters by adding sparsity-inducing penalty term, which is jointly optimized with the original loss objective function (e.g., (Wen et al., 2016; Lebedev & Lempitsky, 2016; Louizos et al., 2018; Liu et al., 2017; Ye et al., 2018)). The latter selects unimportant parameters through certain derived mathematical formula (e.g., (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; 2016b; Li et al., 2017: Molchanov et al., 2017; 2019)). Notably, there is no strict boundary between the two paradigms. Several works (Ding et al., 2018: Wang et al., 2019b; 2021c) manage to get the best from both worlds they select unimportant weights by an importance criterion and add penalty term for sparsity as well. Our method this paper also belongs this group, while it achieves stronger performances.
2

---

Under review as a conference paper at ICLR 2022
Dynamical isometry and orthogonality. Dynamical isometry was first introduced by (Saxe al., 2014), where it can be achieved (for linear MLP models) by the orthogonality of weight matrix at initialization. Recent works on this topic mainly focus on how to maintain dynamical isometry during training instead of only for initialization (Xie et al., 2017: Huang et al., 2018; Bansal et al., 2018; Huang et al., 2020; Wang et al., 2020). These methods are developed independent of pruning, thus not directly relevant to the proposed method in this work. However, the insights from these works inspire us our proposed approach (see Sec. 3.2) and possibly more in the future.
Pruning+ dynamical isometry. One particular paper that inspires us to this work is (Wang et al., 2021a), where the authors leverage dynamical isometry (Saxe et al., 2014) explain the perfor- mance boosting effect (also noted by (Renda et al., 2020; Le & Hua, 2021)) of using larger fine- tuning LR in pruning: pruning hurts dynamical isometry; finetuning can recover dynamical isometry and a larger LR helps recover it faster (and possibly better), thus delivering superior final perfor- mance. The explanation further clears the mystery about the value of network pruning (Liu et al., 2019; Crowley et al., 2018). Previously, (Liu et al., 2019; Crowley et al., 2018) found the small models can be trained from scratch with comparable accuracy to the counterparts pruned from a pretrained large model, thus they argue no value of inheriting weights in structured pruning. (Wang et al., 202 la) pointed out that this argument is actually built upon sub-optimal finetuning LR setups. With the proper ones, filter pruning consistently outperforms training from scratch.
3 METHODOLOGY
3.1 PRELIMINARIES: DYNAMICAL ISOMETRY AND ORTHOGONALITY
The definition of dynamical isometry is that the input-output Jacobian of a network has as many sin- gular values (JSVs) possible close to I (Saxe et al., 2014). With it, the error signal can preserve its norm under propagation without serious amplification or attenuation, which in turn helps the con- vergence of (very deep) networks. For single fully-connected layer W, a sufficient and necessary condition to realize dynamical isometry is orthogonality, Le., W W = I, as shown below,
y = Wx,
(1)
||y|| = yy'y= VXTWTWx = ||x||, if. W
where stands for identity matrix. Orthogonality of a weight matrix can be easily realized by matrix orthogonalization techniques such QR decomposition (Trefethen & Bau III, 1997; Mez- zadri, 2006). Exact (namely all the Jacobian singular values are exactly 1) dynamical isometry single 2d can be achieved for linear networks since multiple linear layers essentially reduce weight matrix. In contrast, the convolutional and non-linear cases are much complicated. Previous work (Wang et al., 2021a) has shown that merely considering convolution or ReLU (Nair & Hin- ton, 2010) renders the weight orthogonalization method much less effective in terms of recovering dynamical isometry after pruning. let alone considering modern deep networks with BN (Ioffe & Szegedy, 2015) and residuals (He et al., 2016). The primary goal of our paper is to bridge this gap.
Following the seminal work of (Saxe et al., 2014), several papers propose to maintain orthogonality during training instead of sorely for the initialization. There are primarily two groups of orthogo- nalization methods for convolutional neural networks: kernel orthogonality (Xie et al., 2017; Huang et al., 2018; 2020) and orthogonal convolution (Wang et al., 2020):
KKT = I + Corth K K - I, kernel orthogonality
(2)
TKTT = 1 Lorth = KKT - -, orthogonal convolution
where clearly the difference lies in the weight matrix K vs. K. (1) K denotes the original weight matrix in a convolutional layer. Weights of a conv layer make up 4d tensor Cxx R xW where N stands for the number of output channels, C for the number of input channels, H and W for the height and width of conv kernel. Then, K is a reshaped version of the 4d tensor; K e IR NXCHW (if N < CHW; otherwise, K E I CHW x.N). (2) In contrast, IC E the doubly Toeelitz (DBT) matrix representation of K (H stands for the output feature map height, Hy; for the input feature map height. Wjo and Wfr ft can be inferred the same way for width).
stands for
was showed by (Wang et al., 2020) that orthogonal convolution is more effective than kernel or- thogonality (Xie et al., 2017) in that the latter only a necessary but insufficient condition of the
3

---

Under review as a conference paper at ICLR 2022
Target gram matrix WWT
Target gram matrix WWT
2
2
3
3
1
N
N
(a) Kernel Orthogonality
(b) Kernel Orthogonality for Pruning
Figure 1: Regularization target comparison between the proposed scheme (b) and kernel orthog- onality (a). Green part stands for zero entries (this figure is best viewed in color). Index to N denotes the filter indices. In (b, c), filter and N are the unimportant filters to be removed. (a) Regularization target of pure kernel orthogonality (an identity matrix), no pruning considered, (b) Regularization target when considering both kernel orthogonality and filter pruning.
former. this work, we will evaluate both methods to see how effective they are maintain- ing/recovering the broken dynamical isometry.
3.2
ORTHOGONALITY PRESERVING PRUNING (OPP)
The proposed method made up of two parts with the same goal: maintaining as much dynamical isometry during pruning possible. First, we explain how to employ regularization to do so for convolutional (Conv) or fully-connected (FC) layers. Second, we propose to regularize batch nor- malization layers given their prevailing use as a standard component deep networks nowadays.
(1) Orthogonality meets filter pruning. From previous works (Lee et al., 2020; Wang et al., 202 we know recovering the broken dynamical isometry incurred by pruning imperative. Considering orthogonality regularization can encourage dynamical isometry, pretty straightforward solution is build upon the existing kernel orthogonality regularization schemes. Specifically, kernel orthogo- nality regularizes the weight gram matrix to be close to identity matrix (see Fig. (a)). In our case, we aim to remove some filters, so naturally we can regularize the weight gram matrix to be close to partial identity matrix, with the diagonal entries at the pruned filters zeroed (see Fig. 1(b)).
Specifically, for the 1-th layer, we sort the filters by their -nomms and select those with the least L1 norms unimportant filters (denoted set St). Then, the proposed regularization term is,
L1 Σ IwW.T @ -ill,, b, = 0f S₁, else 1,
where W denotes the weight matrix; Î represents the partial identity matrix, whose diagonal entries of pruned filters are instead of 1; and || || denotes the Frobenius norm.
(2) BN regularization. There are mainly two kinds of learnable layers in a deep neural network: Conv/FC and BN. The above regularization (Eq. (3)) maintains dynamical isometry for Conv/FC layers. By the idea of preserving dynamical isometry for the whole network, BN is not ignorable since removing filters will change the internal feature distributions. the learned BN statistics do not change accordingly, the error will accumulate. Dynamical isometry will be impaired still. Look the following BN formulation (loffe & Szegedy, 2015),
W * X
Vo=+c + +33,
(4)
where * stands for convolution; stands for the running mean, x² for the running variance; is small amount for numerical stability; the two learnable parameters are and 3. When a filter is zeroed out, the output feature map channel is not zeroed out. These non-zero values are fed into the subsequent layers and play a part in the network. The dynamical isometry of other parameters will depend these values. Once these values zeroed, dynamical isometry will be damaged,
4

---

Under review as a conference paper at ICLR 2022
Algorithm Orthogonality Preserving Pruning (OPP)
1: Input: Pretrained model @ layer-wise pruning ratio ri of (-th layer, for E (1, 2.. L.).
2: Input: Regularization ceiling T, penalty coefficient update interval Ku. penalty granularity A.
3: Init: Iteration i = 0. Ay = for all filter j. Set pruned filter indices Sj by L -nom sorting.
4: while Xj T, for j E St do
5: if i % K2 = then
6:
7: end A X A for j E St.
Update regularization co-efficient in Eq. (6)
8: Network forward, loss (Eq. (6)) backward, parameter update by stochastic gradient descent.
9: Update iteration: i= + I.
end while
11:
Remove filters in St to obtain smaller model O'.
Finetune O' to regain accuracy.
13: Output: Finetuned model O'.
causing (irrecoverable) performance degradation. To overcome this, we propose to regularize the y and B of pruned feature map channels zero. The penalty term can be formulated
C2=
15 83. Σ jE
(5)
This regularization term is simple to implement in any deep learning frameworks, also ready to work with other filter pruning methods. The merits of this BN regularization will be demonstrated in experiments (see Tab. 3). To sum, with the proposed regularization terms, the total error function is
> m Lole + 2 ( L2),
+
(6)
where Lete stands for the original classification loss. For the unimportant (or pruned) filters. the coefficient À grows gradually (by a predefined constant A per K₂ iterations, up to ceiling 7) during training to ensure the pruned parameters are rather close to zero (inspired by (Wang al., 2019b; 2021c)). For the important (or remaining) filters, À is set to a constant (10 - in our experiments). Our algorithm can be summarized Algorithm 1.
Discussion.
Prior works (Liu et al., 2017; Ye et al., 2018) also propose regularize BN for pruning. Our BN regularization method starkly different from theirs. (1) In terms of motivation or goal, (Liu et al., 2017; Ye al., 2018) regularize y to learn unimportant filters, namely, regularizing BN to indirectly decide which filters are unimportant. In contrast, in our method, unimportant filters are decided by their L1 norms. We adopt BN regularization for totally different consideration to mitigate the side effect of breaking dynamical isometry, which is not mentioned at all in their works. (2) In terms of specific technique, (Liu et al., 2017; Ye al., 2018) only regularize the scale factor (because it is enough to decide which filters are unimportant) while we regularize both the learnable parameters because only regularizing one still impairs dynamical isometry of the network. Besides, we employ different regularization strength for different parameters (by the group of important filters vs. unimportant filters), while (Liu et al., 2017; Ye et al., 2018) simply use the same penalty strength for all parameters this is another key difference because regularizing all parameters (including those that are meant to kept) will damage dynamical isometry, which exactly what we want to avoid. In short, in either general motivation or specific technical details, our proposed BN regularization is distinct from previous works (Liu et al., 2017; Ye et al., 2018).
EXPERIMENTAL RESULTS
Datasets and networks. We first conduct analyses with MLP-7-Linear network on MNIST (Le- Cun et al., 1998). Then compare our method to other plausible solutions with ResNet56 (He et al., 2016)/GGG19 (Simonyan & Zisserman, 2015) on CIFARI0/100 (Krizhevsky, 2009). Next we eval- uate our algorithm on the large-scale ImageNet dataset (Deng et al., 2009) with ResNet34/50 (He et al., 2016) and MobileNetV2 (Sandler et al., 2018). Finally, we conduct ablation study to show the efficacy of the proposed BN regularization. On ImageNet, we take the official Py Torch (Paszke et al.,
5

---

Under review as a conference paper at ICLR 2022
3.0
Moan JSV
Mean |SV
Mcan
2.5
1r=0.001
2.0
1r-0.00
Ir=0.001
1.5
20
40
80
20
40
60
80
20
40
80
Test accuracy (N6
Tost accuracy (80
Text sscuras (%)
92
92
MMM
40
1r-0,001
1r-0.001
Ir-0,001
1r-0.01
Ir-0.01
Ir-
20
40
80
20
60
80
20
40
80
Epoch
Epoch
Epoch
91.36 90.54/ 0.0040
92.79/ / 92.77 / 1.0000
92.82/ 92.77 3,4875 (c) OPP (ours)
(a) L1
(b) OrthP
Figure 2: Mean JSV and test accuracy during finetuning with different setups (network: MLP-7- Linear, dataset: MNIST). L1 refers to the -norm pruning (Li et al., 2017). OrthP the dynamical isometry recovery method proposed by (Wang et al., 2021a). Below each plot are, in order, the best accuracy of LR 1e-2, the best accuracy of LR 1e-3, 3, and the mean JSV right after pruning (i.e. without finetuning). LR 1e-2 and 1e-3 are short for two finetuming LR schedules following (Wang et al., 202 (021a)): a): 10:1e-2, 30: 1e-3,, 60: 1--4, #epochs: 90}, {0 :0=33, 45 1 #epochs: : 90). The accuracies are averaged by 5 random runs. For reference, the unpruned model has mean JSV 2.4987, test accuracy 92.77.
2019) pre-trained models' base models to maintain comparability with other methods. On other datasets, we train our own base models with comparable or better accuracies than those reported in the original papers. Our code and trained models will be made publicly available.
Training settings. Given limited space, we defer the detailed hyper-parameter settings (e.g., batch size, weight decay) to the Appendix. Important ones (e.g.. LR) will be mentioned in this main text.
Comparison methods. We compare with (Wang et al., 2021a), which is the only work proposing a method (called OrthP) recover broken dynamical isometry for pruning pretrained models, far as we know. Another method worth comparison is Approximated Isometry (AI) (Lee et al., 2020), which initially proposed for recovering broken dynamical isometry randomly initialized net- works, but still, it can be used in case of pruning a pretrained model. Thus, we also compare with it. Besides, since maintaining orthogonality is the key and there many existing orthogonality regularization papers (Xie et al., 2017: Wang et al., 2020; Huang et al., 2018; 2020; Wang et al., 2020), straightforward solution is to combine them with -n-momm pruning (Li al., 2017) to see if they can help maintain or recover the broken dynamical isometry. There are two possible combi- nation schemes: 1) apply orthogonal regularization methods before applying Ln-norm pruning. 2) apply orthogonal regularization methods after Lı-norm pruning, namely, in finetuning. Two rep- resentative orthogonality regularization methods are selected because of their proved effectivvness: kernel orthogonality (Xie et al., 2017) and convolutional orthogonality (Wang et al., 2020), so in to- tal there are 4 combinations: L1 + KernOrth (Xie et al., 2017). 1). L1 + OrthConv (Wang et al., 2020), KernOrth (Xie et al., 2017) + L1. OrthConv (Wang et al., 2020) + L1-
Liyer-wise pruning ratios. We adopt manually preset pruning ratios in our method following (Li et al., 2017; He et al., 2018a; Wang al., 2019b; 2021c) to maintain comparability with prior works. Moreover, presetting pruning ratios help us control irrelevant factors precisely, excluding their interference our analyses. Please refer the Appendix for summary of all pruning ratios fixed adopted in this work and the discussion about the reason of employing fixed pruning ratios.
Comparison metries. (1) We compare the final test accuracy after finetuning with the similar FLOPs budget- - this is currently the most prevailing metric to compare different filter pruning meth- ods in classification. Concretely, we compare two settings: a relatively large finetuning LR (1e-2) and a small one (1e-3). We introduce these settings because previous works (Renda et al., 2020; Le & Hua, 2021: Wang et al., 2021a) showed that finetuning LR has a great impact on the final performance, From this metric, we can see how sensitive different methods are to the finetuning LR.

---

Under review as a conference paper at ICLR 2022
Table 1: Test accuracy (%) comparison among different dynamical isometry maintenance recov- ery methods on CIFARI0/100. "Seratch" stands for training from scratch. Each setting is randomly for 3 times, mean (std) accuracies reported. "KernOrth" means Kernel Orthogonalization (Xie et al., 2017); "OrthConv" means Convolutional Orthogonalization (Wang et al., 2020). Two finetun- ing LR schedules are evaluated here: initial LR 1e-2 vs, 1e-3. "Ace. diff." refers to the accuracy gap of LR 1 over LR 1e-3.
ResNet56 + CIFAR1O: Baseline accuracy 93. 78%, Params: 0.85M, FLOPs: 0.25G
Pruning ratio r Sparsity/Seeedu
0.3
0.5
0.7 70.57%/3359x
0.9
0.95
31.14%//14x 49.82%//1999
00.39%/11.41x 95.19%
Initial
finetuning
LR 1e-2
Scratch
93.16 (0.16)
92.78 (0.23)
92.11 (0.12)
88.36 (0.20)
84.60 (0.14)
L1 (Li et al., 2017)
93.79 (0.06)
93.51 (0.07)
92.26 (0.17)
88.71 (0.15)
84.63 (0.28)
L1 OrthP (Wang et al., a)
93.69 (0.02)
91.96 (0.19)
86.01 (0.34)
82.62 (0.05)
L1 Al (Lee et al., 2020)
(0.12)
93.24 (0.09)
(0.14)
86.67 (0.08)
83.34 (0.18)
11 + KernOrth (Xie al., 2017)
93.49 (0.04)
93.30 (0.19)
91.71 (0.14)
84.78 (0.34)
80.87 (0.47)
J. + OrthConv (Wang al., 2020)
92.54 (0.09)
92.41 (0.07)
(0.16)
84.52 (0.27)
80.23 (1.19)
KerOOrth (Xie et al., 2017) + L1
93.49 (0.07)
92.82 (0.10)
90.54 (0.25)
85.47 (0.20)
79.48 (0.81)
OrthConv (Wang al., 2020) L
93.63 (0.17)
93.28 (0.20)
92.27 (0.13)
86,70 (0.07)
83.21 (0.61)
OPP (ours)
93,99 (0.18)
93.44 (0.05)
92.43 (0.20)
89.50 (0.08)
85,94 (0.09)
Tritial
Finetunt ng
TIR 1e-3
La (Li et al., 2017)
93.43 (0.06)
(0.10)
91.77 (0.11)
87.57 (0.09)
83.10 (0.12)
OPP (ours)
93.54 (0.08)
93.32 (0.11)
92.00 (0.08)
89.09 (0.10)
85,47 (0.22)
Acc. diff. (L1)
0.36
0.39
0.49
1.14
1.53
Acc. diff. (OPP)
0.45
0.12
0.43
0.41
0.47
+ Baseline accuracy 74.02%, Params: 20.08M, FLOPs: 0.80G
Pruning ratio Sparsity/Speedup
0.1
0.3
0.5
0.7
0.9
19.24%
51.01%/1.97x
74.87%/3.60x
90.98%//.84x
98.96%/44.22x
Initial
finetuning ng
LR 1e-2
Scratch
72.84 (0.25)
71,88 (0.14)
70.79 (0.08)
66.51 (0.11)
54.37 (0.40)
L, (Li et al., 2017)
74.01 (0.18)
73.01 (0.22)
71.49 (0.14)
66.05 (0.04)
51.36 (0.11)
La + OrthP (Wang
74.00 (0.04)
72.30 (0.49)
68.09 (0.24)
62.22 (0.15)
48.07 (0.31)
L, Al (Lee al. 2020)
74.05 (0.12)
73.08 (0.26)
71.12
64.91 (0.23)
49.79 (0.11)
L1 + KernOrth (Xic al., 2017)
(0.26)
72.53 (0.09)
71.23 (0.10)
(0.14)
50.75 (0.30)
L1 + OrthConv (Wang al., 2020)
(0.10)
72.25 (0.31)
(0.11)
64.51 (0.43)
48.31 (0.18)
KernOrth (Xie al., 2017) L1
73.73 (0.23)
72.41 (0.12)
70.31 (0.12)
(4.19
50.72 (0.87)
OrthConv (Wang al., 2020)
73.55 (0.18)
72.67 (0.09)
71.24 (0.23)
65.66 (0.10)
50.53 (0.46)
OPP (ours)
74.30 (0.20)
72.84 (0.07)
71.71 (0.09)
67.91 (0.18)
57.81 (0.28)
Initial
Finetuning
LR 1e-3
L1 (Li et al., 2017)
73.67 (0.05)
72.04 (0.12)
70.21 (0.02)
64.72 (0.17)
48.43 (0.44)
OPP (ours) Aec. diff. (ET)
73.83 (0.02)
72.29 (0.07)
71.16 (0.12)
67.47 (0.17)
56.73 (0.34)
0.34
0.97
1.28
1.33
2.93
Acc. diff. (OPP)
0.47
0.55
0.55
0.44
1.08
(2) We also compare the test accuracy before finetuning from this metric, will see how robust different methods are right after weight removal. This metric is also used by previous pruning works (e.g., g.. (Wang et al., 2019a)) and broadly utilized to derive advanced weight importance criteria based on Hessian (LeCun et al., 1990; Hassibi & Stork, 1993; Singh & Alistarh, 2020).
4.1 ANALYSIS WITH M ON MNIST
MLP-7-Lineat is seven-layer linear MLP. It is adopted in (Wang et al., 2021a) for analysis because linear MLP is the only network that can achieve exact dynamical isometry (all JSVs are exactly 1) so far. Their proposed dynamical isometry recovery method OrthP (Wang et al., 2021a) was shown can achieve exact isometry on linear MI.P networks. Since we claim our method OPP can maintain dynamical isometry too, conceivably,
method should play similar role to OrthP in pruning. To confirm this, we prune the MLP-7-Linear network with our method (exactly following the settings in (Wang et al., 2021a) for fair comparison).
Results are presented in Fig. 2, where we plot the mean JSVs and test accuracies. Mean JSV is a metric to measure dynamical isometry (Lee et al., 2020; Wang et al., 2021a). Fig. 2(b) is the one equipped with OrthP, which can exactly recover dynamical isometry (note its mean JSV right after pruning is 1.0000), it works as the oracle here. We have the following observations. (1) OrthP
7

---

Under review as a conference paper at ICLR 2022
Table 2: Speedup comparison on ImageNet. FLOPs: ResNet34: 3.66G, ResNet50: 4.09G.
Method
Network
Base top-1 (%) Pruned top- (%) Top-1 drop (%) Speedup
Li (pruned-B) (I.i al., 2017)
73.23
72.17
1.06
1.32x
Li (pruncd-B, reimpl.) (Wang 2021a)
73.31
73.67
-0.36
1.32x
Taylor-FO (Molchanov 2019)
ResNet34
73.31
72.83
0,48
1.29x
GReg-2 (Wang al., 202
73.31
73.61
-0.30
1.32x
OPP (ours)
73.31
73.71
-0.40
1.32x
ProvablcFP (Licbenwein ct al., 2020)
76.
75.21
0.92
1,43x
GReg-I (Wang et al,, 2021c)
ResNet50
76.13
76.27
-0.14
1,49
OPP (ours)
76.
76.40
-0.27
1.49
IncReg (Wang et al., 20196)
75.60
72.47
3.13
2,00x
SFP (Hle et al., 2018a)
76.15
74.61
1,54
1,72x
HRank (Lin
2020)
76.15
74.98
1.17
1.78x
Taylor-FO (Molchanov et 2019)
76.18
74.50
1.82x
Factorized
al., 2019)
76.15
74.55
2.33x
DCP (Zhuang a 2018)
ResNet50
76.01
74.95
1.06
2.25x
CCP-AC (Peng al., 2019)
76.15
75.32
0.83
2.18x
C-SGD-50 (Ding et al., 2019)
75.34
74.54
0.80
2.26x
GReg-2 (Wang et al. 2021c)
76.13
75.36
0.77
2.31
CC (Li et 2021)
76.15
75,59
0.56
2.12x
OPP (ours)
76.13
75.60
0.53
2.31
I.PPC (He et al, 2020)
76.15
74.46
1.69
2,55x
GReg-2 (Wang et al., 2021c)
76.13
74.93
1.20
2.56x
ResNet50
CC (Li 2021)
76.15
74.54
1.61
2.68x
OPP (ours)
76.13
75.05
2.56x
IncReg (Wang et al., 2019h)
75.60
71.07
4,53
3.00x
Taylor-FO (Molchanov et al., 2019)
76.18
71.69
3,05
ResNet50
GReg-2 (Wang et al,, 202
76.13
73.90
3.06X
OPP (ours)
76.13
74.41
1,52
3.06
MobileNetV2-0.755 (Sandler 2018)
71.88
69.80
2.08
1.36x
AMC (Hc 2018b)
71.88
70.80
L08
1.36x
MobileNetV2
CC (Li al., 2021)
71.88
70.91
0.97
1,40x
OPP (ours)
71.88
71.01
0.87
1.41
improves the best accuracy from 91.36/90.54 to 92.79/92.77 using OPP, we obtain 92.81/92. Namely, in terms accuracy, our method is as good as the oracle scheme, not better. (2) Note the mean JSV right after pruning the L1 Lormorm pruning destroys the mean JSV from 2.4987 to 0.0040, and OrthP brings it back to 1.0000. In comparison, OPP achieves 3.4875, also comparable to OrthP.
4.2 RESNET56+CIFARI / VGG19+CIFAR 100
Here we compare our method to other plausible solutions on the CIFAR10/100 datasets (Krizhevsky, 2009) with non-linear convolutional networks (ResNet56 and VGG19). From the results in Tab. 1, we have the following observations.
(1) OrthP (Wang al., 2021a) and AI (Lee al., 2020) do not work well L, +OrthP underperform the original L1 under all the five pruning ratios for both ResNet56 and VGG19. +1+AI also under- performs L1 on ResNet56 for all pruning ratios. On VGG19, +1+AI only works for small pruning ratio regime (such as r = 0.1 and r 0.3).
(2) For KernOrth vs. OrthConv, the results look mixed OrthConv is better when applied before the L1 Li-norm pruning. This is reasonable since OrthConv has been shown more effective than Kern in enforcing more dynamical isometry (Wang al., 2020), which turn can stand more of the damage from pruning.
(3) Of special note is that, none of the above methods actually outperform the -nom pruning or the simple scratch training in aggressive pruning cases (the last two columns). It means that neither enforcing more isometry before pruning nor compensating isometry after pruning can help dynamical isometry recovery. In stark contrast,
proposed OPP method outperforms L -nom pruning and scratch consistently under most pruning ratios, especially in large pruning ratio regime. Besides. note that the accuracy trend: in general, with larger pruning ratio, the advantage of OPP over L, or Scratch is more pronounced. This is because a larger pruning ratio means the dynamical isometry is damaged more, where our method can help more, thus harvest performance gains.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Ablation Study:
Test accuracy (without finetuning) comparison with or without the pro-
posed BN regularization.
ResNet56 + CIFARIO: Baseline accuracy 93.78%,
Params: 0.85M, FLOPs: 0.25G
Pruning ratio
0.3
0.5
0.7
0.9
0.95
OPP (w/ BN reg)
92.79 (0.03)
92.23 (0.08)
90.46 (0.21)
44,25 (2.46)
16.52 (0,43)
OPP (w/ BN reg)
92.94 (0.14)
92.48 (0.19)
90.48 (0.09)
70.53 (1.69)
23.05 (2.61)
Acc. diff.
+0.15
+0.25
+0.02
+26.28
+6.53
VGG19 + CIFARTO0: Baseline accuracy 74.02%,
Params: 20.08M, FLOPs: 0.80G
Pruning ratio
0.1
0.3
0.5
0.7
0.9
OPP (w/o BN reg) OPP (w/ BN reg)
73.01 (0.13)
71.26 (0. 19)
68.67 (0.10)
61.70 (0.46)
1.75 (0.38)
73.44 (0.07)
71.61 (0.12)
69.28 (0.25)
65.15 (0.20)
2.84 (1.13)
Acc. diff.
+0.43
+0.35
+0.51
+3.45
+1.09
(4) Tab. 1, we also present the results when the initial finetuning LR is 1e-3. In (Wang et al., 2021a), is is shown that if the broken dynamical isometry can be well maintained//rccoveeedd the final performance gap between LR 1e-2 and 1e-3 will be diminished. Since we claim our method is good maintaining dynamical isometry, the performance gap should become smaller. This is em- pirically verified in the table (note the Acc. diff. of OPP vs. that of L1). In general, the accuracy gap between LR 1 e-2 and LR 1e-3 of OPP is smaller than that of Lı-norm pruning. Two exceptions are r = 0.3 for ResNet56 and r = 0.1 for VGG9. We conceive this is because the speedup ratio small. The accuracy difference between LR 1e-2 and 1e-3 is small itself. Despite them, the general picture from the table is that the accuracy gap between LR 1e-3 and 1e-2 2 turns smaller with our method, which is good sign that dynamical isometry effectively maintained.
4.3 IMAGENET
We further evaluate OPP on ImageNet (Deng et al., 2009) in comparison to many existing filter pruning algorithms. Results are shown in Tab. 2. Our method is consistently better than the others across different speedup ratios. Moreover, with a larger speedup ratio, the advantage of our method more pronounced. For example, OPP outperforms Taylor-FO (Molchanov et al., 2019) by 1.15% in terms of the top-1 accuracy drop at the 2.31 xppeedup track; at 3.06 speedup, OPP leads Taylor- FO (Molchanov et al., 2019) by 2.87%. This shows OPP is more robust to more aggressive pruning. The reason is easy to see more aggressive pruning hurts dynamical isometry more (Lee et al., 2019; Wang et al., 2021a). Our method can find more use in these cases since can recover the broken dynamical isometry during pruning. This phenomenon has been observed consistently (see also the results in Tab. and ablation study in Tab. 3).
4.4 ABLATION STUDY
Finally, we conduct ablation study regarding regularizing the two learnable parameters in BN lay- ers. The results are presented in Tab. 3, where we compare the accuracy right after pruning (namely, without finetuning). As seen, when the BN regularization is switched off, the performance degrades. A clear trend is: BN regularization is more helpful under the larger sparsity. This reiterates the im- portance of regularizing BN for dynamical isometry recovery, especially in the aggressive pruning.
CONCLUSION
Dynamical isometry maintenance or recovery is crucial to neural network pruning. In this work, we present a dynamical isometry preserving pruning method (named orthogonality preserving pruning) based on regularization. Specifically, we propose modified weight gram matrix as regularization target which drives the unimportant weights towards zero meanwhile regularizing the remaining ones to archive orthogonality. Besides, we propose to regularize the BN parameters to mitigate its damage to dynamical isometry. The proposed method is shown much more effective than existing isometry recovery counterparts. The resulted pruning algorithm also performs better compared to many recent filter pruning approaches. To our best knowledge, this is the first dynamical isometry preserving filter pruning method that scales the large-scale ImageNet dataset.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regular- izations in training deep networks? In NeurIPS, 2018. 3
Jian Cheng, Pei-song Wang, Gang Li, Qing-hao Hu, and Han-qing Lu. Recent advances in effi- cient computation of deep convolutional neural networks. Frontiers of Information Technology & Electronic Engineering, 19(1))64-777, 2018a.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35 (1):126-136, 2018b. 2
Elliot J Crowley, Jack Turner, Amos Storkey, and Michael O'Boyle. A closer look at structured pruning for neural network compression. arXiv preprint arXiv: 1810,0462, 2018. 3
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In CVPR, 2009. 5,
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4); 485-532, 2020. 2
Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. In AAAI, 2018. 2
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In CVPR, 2019. 8
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
S. Han, X. Liu, and H. Mao. EIE: efficient inference engine compressed deep neural network.
ACM Sigarch Computer Architecture News, 44(3):243-254, 2016a. 1,
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015. 1,2
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016b. I,
Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, et al. ESE: Efficient speech recognition engine with sparse 1stm fpga. In ACM/SIGDA International Symposium on Proell-rogra Gate Arrays, 2017. 1, 2
B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NeurIPS, 1993. 2,
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 3.5
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In ISCAI, 2018a. 2, 6, 8
Yang He. Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. CVPR, 2020.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: Automl for model compression and acceleration mobile devices. In ECCV, 2018b. 8
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In AAAI, 2018. 3,
10

---

Under review as a conference paper at ICLR 2022
Lei Huang, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, and Ling Shao. Controllable orthog onalization in training dnns. In CVPR, 2020. 6
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 2, 3,
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 5,
Duong Le and Binh-Son Hua. Network pruning that matters: A case study on retraining variants. In ICLR, 2021. 1,3,6 6
V. Lebedev and Lempitsky. Fast convnets using group-wise brain damage. In CVPR, 2016.
Y. LeCun, J. Denker, and S. A. Solla. Optimal brain damage. In NeurIPS, 1990. 2, 7
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11))22782332, 1998.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019. 9
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. In ICLR, 2020. 4, 6, 7,
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017. 2, 6, 7, 8, 14
Tuanhui Li, Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convo- lutional neural networks via factorized convolutional filters. In CVPR, 2019. 8
Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang. Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In CVPR, 2021. 8
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In ICLR, 2020. 8
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In CVPR, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. ICCV, 2017. 2,5
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019. 3, 13, 14
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. In ICLR, 2018. 2
Francesco Mezzadri. How generate random matrices from the classical compact groups. arXiv preprint math-ph/0609050, 2006. 1,3
P. Molchanov, S. Tyree, and T. Karras. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017. 2
Pavlo Molchanov, Arum Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. CVPR, 2019. 2, 8, 9
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICMI, 2013.
11

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019. 5, 13
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In ICMI, 2019. 8
R. Reed. Pruning algorithms a survey. IEEE Transactions on Neural Networks, 4(5):740-747, 1993. 1,2
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020. 3,
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chia Lang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 8,8
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. In ICLR, 2014. 2, 15
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. In NeurIPS, 2020. 2,
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: tutorial and survey. Proceedings of the IEEE, 105((2);2295-2322, 2017, 2
Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997. 1,
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In ICML, 2019a. 7
Huan Wang, Xinyi Hu, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruning for efficient convolutional neural networks via incremental regularization. IEEE Journal of Selected Topics in Signal Processing, 14(4):775-788, 2019b. 5,
Huan Wang, Can Qin, Yue Bai, and Yun Fu. Dynamical isometry: The missing ingredient for neural network pruning. arXiv preprint arXiv:202i.059// 2021a. 1, 6, 7, 8,9
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Emerging paradigms of neural network pruning. arXiv preprint arXiv:2103.064
2021b. 2
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In ICLR, 2021c. 2. 6, 8, 13
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In CVPR. 2020. 3, 6. 8, 13
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. NeurlPS, 2016.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In CVPR, 2017. 3, 7.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the mmaller-nomm foom-less-ine dess-inforrmatvee assumption in channel pruning of convolution layers. In ICLR, 2018. 2,
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aven -wwrre channel pruning for deep neural networks. In NeurIPS, 2018.
12

---

Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
A.1 CODE REFERENCE
We mainly refer the following code implementations in this work. They are all open-licensed.
Official PyTorch ImageNet example: examle::: http:///hhhh..oooooooocemeeeeeee
(Wang al., 2021c): http:///ghhttooooo//unnSSu//rrrr
Pruning;
OrthConv (Wang et al., 2020): http::/gtttccoooosammommllooo/ot
Neural-Networks;
Rethinking the value of network pruning (Liu et al., 2019): http://github.com/Eiic networkk -ewwarr-prunnngggiienne hwtp://uuuuuuiiee////e////maaaaa mingiic/rethinking-neee
noom-prunnng..
Data split. All the datasets in this paper are public datasets with standard APIs in PyTorch (Paszke al., 2019). We employs these standard APIs for the train/test data split keep fair comparison with other methods.
A.2 TRAINING SETTING
The specific training settings are summarized in Tab. For the in our OPP method: regularization granularity A, regularization ceiling r and regularization update interval K,, we summarize them in Tab. 5. These hyper-parameters mainly refer to the official implementation of GReg-1 (Wang et al., 2021c) since we tap into similar growing regularization scheme as it does.
that are introduced
Table 4: Training setting summary. For the SGD solver, in the parentheses are the momentum and weight decay. For LR schedule, the first number is initial LR: the second (in brackets) is the epochs when LR is decayed by factor 1/10; and #epochs stands for the total number of epochs.
Dataset
MNIST
CIFAR 10/10
ImageNet
Solver
SGD (0.9, le-4)
SGD (0.9, 5e-4)
SGD (0.9, le-4)
Batch size
100
CIFARIO: 128: CIFARI00, ImageNet: 256
LR schedule (scratch) le-2, [30,60], #epochs:90
le- 1e-1, [100,150], #epochs:200 le-1, [30,60], #epooch::90
LR schedule (prune)
Fixed (le-3)
I.R schedule (finetune) le-2, [30,60], #epochs:90
Ic-2, [60,90] #epochs:
1c-2, [30,60,75], #cpochs:90
Table 5: Hyper-parameters of our methods,
Datasct
MNIST
CIFARI0/100
ImageNet
Regularization granularity A
le44
Regularization ceiling т
Regularization update interval K₁
10 iterations
5 iterations
A.3 HARDWARE AND RUNNING TIME
We conduct all experiments using 4 NVIDIA V100 GPUs (16GB memory per GPU). It takes roughly 41 hrs to prune ResNet50 on ImageNet using our OPP method (pruning and 90-epoch finetuning both included). Among them, 12 hrs (namely, close 30%) are spent on pruning and 29 hrs are spent on finetuning (about 20 mins per epoch).
B LAYER-WISE PRUNING RATIOS
The layer-wise pruning ratios are pre-specified in this paper. For the ImageNet benchmark, we exactly follow 1/GReg2 (Wang et al., 2021c) for the layer-wise pruning ratios keep fair comparison to it. The specific numbers are summarized in Tab. 6. Each number is the pruning ratio shared by all the layers of stage in ResNet34/50.
Reason for using pre-defined pruning ratios. this paper consider pruning independent with the pruning ratio choosing. The main consideration is that pruning ratio is broadly believed to reflect
13

---

Under review as a conference paper at ICLR 2022
Table 6: Summary of layer-wise pruning ratios.
Dataset
Network
Speedup
Pruned top-1 accuracy (%)
Pruning ratio
ImageNet
ResNet34
1.32x
73.77
[0, 0.50, 0.60, 0.40, 0, 0J"
ImageNet
ResNet50
1.49x
76.44
[0, 0.30, 0.30, 0.30, 0.14, 0]
ImageNet
ResNet50
2.31
75.60
[0, 0.60, 0.60, 0.60, 0.21, 0]
ImageNet
ResNet50
2.56x
75.12
[0, 0.74, 0.74, 0.60, 0.21, 0]
ImageNet
ResNet50
3.06
74.51
[0, 0.68, 0.68, 0.68, 0.50, 0]
*In addition to the pruning ratios, skip several layers, following the setting of T1 (pruned-B) (Li
al., 2017) for fair comparison. Concretely, refer the implementation of (Liu al., 2019)
https://gitub.ccom//e hiapsiiuuubooooorr-iiinnni
the redundancy of different layers, which an inherent characteristic of the model, thus should not be coupled with the subsequent pruning algorithms. Thus, we choose to set pruning ratios before pruning. Exploring better pruning ratio is not our focus in this paper given the limited length.
In fact, showing the effectiveness of the pruning ratios is unnecessary to our paper as long as we use the same ratios as GReg-1/2 does keep a fair comparison. The reason is, if the pruning ratios we used turns out good, there is no concern about it; if they turn out bad, that's 's even better, because shows our method can achieve SOTA performance even if using such bad pruning ratio combination. Either case, it does not undermine the value of our method.
C SENSITIVITY ANALYSIS OF HYPER-PARAMETERS
Among the three hyper-parameters in Tab. 5, regularization ceiling works as a termination condi- tion. We only requires it to be large enough to ensure the weights are compressed to a very small amount. It does not have be 1. The final performance is also less sensitive it. The pruned performance seems to be more sensitive to the other two hyper-parameters, so here we conduct hypee-paramettr sensitivity analysis to check their robustness.
Results are presented in Tab. and Tab. 8. Pruning ratio 0.7 (for ResNet56) and 0.5 (for VGG19) are chosen here because the resulted sparsity the most typical (i.e., not too large or small). (1) For K,,, in general, a larger Ku tends to deliver a better result. This is no surprise since a larger K2 implies more time for the network to adapt and recover when undergoing the penalty. (2) For A, we do not see a clear pattern here: either small or large A can achieve the best result (for different networks). On the whole, when varying the hyper- parameerrseterrs within a reasonable range, the performance is pretty robust, no catastrophic failures. Moreover, note that the default setting is actually not the best for both K2 and A. This is because we did not hard-tune our hyper- parameters,, however, they still achieve encouraging performance compared the counterpart methods, have shown in the main text.
Table 7: Sensitivity analysis of K2 on CIFAR10/100 datasets with the proposed OPP algorithm. K. = 10 is the default setting. Pruning ratio 0.7 (ResNet56) and 0.5 (VGG19) are explored here. Each setting is randomly run for 3 times, mean and standard deviation accuracy reported. The best is highlighted with bold and the worst is highlighted with blue color.
K,
15
20
Acc. (%, ResNet56)
92.24
92.42
92.4310.20
92.50
92.31 40.16
Acc. (%, VGG19)
71.33.0.06
71.45
71.71 10.09
71.43
71.68
Table 8: Sensitivity analysis of A on CIFARI0/100 datasets with the proposed OPP algorithm. A Le 4 the default setting. Pruning ratio 0.7 (ResNet56) and 0.5 (VGG19) are explored here. Each setting is randomly run for 3 times, mean and standard deviation accuracy reported. The best is highlighted with bold and the worst highlighted with blue color.
A
lc-5
5c-5
lc-4
5c-4
1c-3
Acc. (%, ResNet56)
92.3740.12
92.294o.10
92.43...20
92.4040.15
92.44
Acc. (%, VGG19)
71.39
71.37+0.14
71.71+0.08
71.58+0.31
71.25-0.31
14

---

Under review as a conference paper at ICLR 2022
D MORE BACKGROUND OF DYNAMICAL ISOMETRY
In the main paper (Fig. 2), we present the mean JSV (0.0040) right after pruning and those after using OrthP (1.0000) and our OPP method (3.4875). We say "OPP achieves 3.4875, also comparable to OrthP". How to understand this "comparable"? By what standards? This section is meant to provide more background concerning these questions.
Dynamical isometry (DI) is defined by mean JSV close to I in (Saxe et al., 2014) (rigorously in (Saxe al., 2014). DI describes the distribution of all JSVs. Mean JSV is only an average sketch of the distribution. But this approximation is accurate enough for analysis in this paper, so we say "DI defined by mean JSV"). In other words, if network has mean ISV close 1, we can say this network has dynamical isometry. This is a fundamental basis of our paper.
Then, non-trivial technical question we need to answer is: When we deal with practical deep neural networks in the real world, how close is the so-called "close to 1 in the above definition? To our best knowledge, there is no outstanding theory to quantify this, so we resort to empirical analysis. In Tab. 9 below, we present the mean JSV of MLP-7-Lineat network on MNIST under different pruning ratios, along with their accuracies before and after finetuning.
Table 9: Mcan JSV and test accuracies (%) of MLP-7-Lincar on MNIST under different pruning ratios. Each result is randomly run for 3 times. We report the mean accuracy and (std). "ft." is short for finetuning.
Pruning 2.4987
mean JSV
1.7132
0.9993
0.5325
0,2711
0.1180
0,0452
0.0151
0.0040
0.0004
Acc. before ft. 92.77
91.35
78.88
62.21
9,74
9.74
9.74
Acc. after ft.
92.82 (0.05)
92.80 (0.04) 92.80 (0.01) 92.77 (0.01) 92.77 (0.02) 92.77 (0.00) 92.78 (0.02) 91.37 (0.03) 87.82 (0.03)
As seen, there is a clear trend - larger pruning ratio, lower accuracy (before or after finetuning), smaller mean JSV. Of special note is the mean JSV range where the pruned network can be finetuned back to the original accuracy (92.77%) [0.0151, 4987). This means, for networks with mean JSV from 0.0151 2.4987, in spite that their immediate accuracies (without finetuning) can be distinct (e.g., 91.35% vs. 9.74%), intrinsically, they are equivalently potential after finetuning. By this "equivalently potential" rule, all the mean JSVs the range [0.0151, 2.4987] can regarded as comparable (although the two extremes have a gap of two orders of magnitude). That is, two networks, as long as their accuracies after finetuning are comparable, we deem their mean JSVs as comparable.
In Fig. 2, OPP achieves 92.87/92.77 test accuracy, as comparable to 92.79/92.77 when using OrthP. Also, 3.4875 is close to 2.4987 in the same order of magnitude. Thus, we say OPP is comparable to OrthP in Fig. 2.
15


Reviews:
[{"review_id": "zPoiTRmG5py", "reviewer": "Reviewer_8kct", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Clear motivation and writing.\n2. Well-established pipeline.\n\nCons:\n1. Comparisons with the state-of-the-art methods are insufficient, such as [1].\n2. Experiments on ImageNet are insufficient and results are not comparable.\n3. Moderate novelty. This kernel orthogonality idea has been employed in deep networks in previous methods.\n\n\n[1] ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting, Ding et al., ICCV 2021", "summary_of_the_paper": "This paper proposes a structured pruning method based on kernel orthogonality analysis. The method is straightforward and well-investigated in the network pruning field.", "summary_of_the_review": "This paper has moderate novelty with the well-studied kernel orthogonality method. Besides, the experiment design is not sufficient. The performance gain in ImageNet is incremental and not sufficient compared to the state-of-the-art methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "haOc2vHzaj", "reviewer": "Reviewer_ipz6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1. Orthogonality preserving structured pruning is interesting.\n2. The results demonstrate the merits of the method.\n3. Overall the paper is well written.\n\n# Weaknesses\n\n1. The role of BN regularization is not clear to me. It is not clear how the BN parameters of pruned filters would make a difference in dynamical isometry (DI) and how the proposed regularization alleviates this issue? More clarity is required.\n\n2. It is not clear that the method indeed improves DI. Considering Fig.2 c, mean JSV increases and is higher than 1. So not clear the method indeed improves DI. \n\n3. The notation inside the Frobenius norm is not clear in Eq.3. Probably a type, please comment.", "summary_of_the_paper": "The paper proposes a dynamical isometry preserving approach for structured pruning by introducing an orthogonality preserving regularization for weight matrices in each layer. The experiments on cifar and imagenet datasets show improvements over comparable methods.", "summary_of_the_review": "Overall the paper is interesting. The connection to BN regularization and DI could be explained better to improve clarity.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XCGvSsUyKL", "reviewer": "Reviewer_idn1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written in general, and the proposed methods are intuitive. In the experimental analyses, the proposed methods outperform baseline on various benchmark datasets.\n\nHowever, there are few major issues with the paper:\n\n- There are three main claims in the paper; achieving structured pruning, orthogonality and dynamic isometry. However, these claims are not explored in detail;\n\n-- Structured pruning is only mentioned as a synonym for filter pruning or channel pruning in the review of the related work. This is achieved in the proposed method by user defined parameters that identify the indices of parameters to be pruned. In the analyses, selection of this set of indices affects the accuracy as well. That is, there is not a major contribution for structured pruning.\n\n-- Orthogonality is aimed to be achieved by regularization on gram matrices of kernels. However, how well/much orthogonalization is obtained is not analyzed. In addition, regularization of batch normalization parameters has much more effect on the accuracy compared to orthogonalization. \n\n-- The paper claims to achieve dynamic isometry by orht. However, it is also stated “We also propose to regularize batch-normalization parameters for better preserving dynamical isometry for the whole network.” That is, either orth. does not enable dynamic isometry by itself, or dynamic isometry does not affect the accuracy remarkably. This claim should be examined theoretically or experimentally in detail.\n\nTo summarize, the relationship between these three proposed major claims should be analyzed in detail. Otherwise, the scope and claims of the paper should be updated accordingly.\n\nAdditional questions:\n\n-\tHow did you calculate speedup? Could you please provide an example of actual running times for training and test for an experiments, such for the VGG19 + Cifar100 case, in comparison with the baseline?\n\n-\tHave you employed regularization of parameters of BN of baseline methods to check how this method also affects accuracy of baseline?\n", "summary_of_the_paper": "This paper proposes regularizing learnable parameters of NNs to maintain the dynamical isometry during pruning and improve their accuracy. In the experimental analyses, the proposed OPP outperforms baseline methods on benchmark datasets.\n\nThe paper is well written in general, and the proposed methods are intuitive. In the experimental analyses, the proposed methods outperform baseline on various benchmark datasets.\n", "summary_of_the_review": "The paper is well written in general, and the proposed methods are intuitive. However, the proposed claims are not explored well. More precisely, the relationship between the proposed major claims should be analyzed in detail. Otherwise, the scope and claims of the paper should be updated accordingly. To this end, the paper proposes regularizing learnable parameters of NNs using different heuristics. Therefore, the novelty and scope can be redefined according to provided analyses and results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K-CGW9WDc4", "reviewer": "Reviewer_QYz3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\nThe authors propose a nice way to maintain dynamic isometry property during pruning: just doing a back-propagation considering the orthogonal constraint. The method is quite intuitive and simple, which I think it’s good.\n\nCons:\nI think the imagenet performance is not quite better than previous methods. The improvement is in general 0.1 points except ResNet50 (3.06x) where the improvement is 0.5 (still not significant). If the authors want to claim their method has larger advantage in larger pruning ratio, there should be more experiments to support such a claim.\n\nThe authors argue to use “difference between performance under different learning rate” as a metric to measure whether dynamic isometry is preserved, which in my opinion is not a good measure. Can the author measure the jacobian of each layer?\n\nQuestions:\n\nAbout the claim: “SGD training in finetuning can help recover it”, I’m confused what is the goal of recovery when looking at figure 2. The model seems not to converge to JSV 1, then what information this figure conveys?\n\nWhat is the L1+KernOrth in Table 1 differs from the proposed method? Is the difference just whether regularize BN parameter? Moreover, there is no explanation of what is the difference between L1 + X v.s. X + L1, which is confusing.\n\nWhy this paper only use kernel orthogonality instead of convolutional orthogonality?\n\nComments:\n\nI think the content would be much easier to read if you can create a 2D plot with the x-axis being the speedup and the y-axis being the accuracy. In its current form, all I can see is the proposed method does not have a clear advantage over the previous method.\n\nRegarding maintaining approximately dynamic isometry, this paper is also related:\n[1] deep isometric learning for visual recognition. ICML 2020.", "summary_of_the_paper": "Dynamic isometry is shown to be a useful property that enable effective gradient propagation through the forward/backward. However, pruning will largely damage such a structure. This paper studies how to maintain the “dynamic isometry” property during pruning. Specifically, after getting an initial assessment of filter importance, the algorithm will maintain the partially kernel orthogonality of the important filters. They also propose to regularize the BN parameters to future boost the performance.", "summary_of_the_review": "I like the idea of maintaining dynamic isometry during pruning. But there is still great room to improve in both the presentation and analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a pruning approach that regularizes the gram matrix of convolutional kernels to encourage kernel orthogonality among the important filters meanwhile driving the unimportant weights towards zero. While the reviewers found the proposed method well-motivated and intuitive, they believe that the proposed claims are of limited novelty and are not supported well by the experiments. Analyzing and explaining the effect of different parts of the proposed method, i.e., orthogonalization and regularization of batch normalization parameters, on the accuracy of the pruned models would significantly improve the manuscript.
Decision: Reject