Trainability Preserving Neural Pruning
Authors: Huan Wang
Yun Fu
Venue: ICLR 2023 poster (2023)
Keywords: neural network structured pruning
trainability
kernel orthogonalization

Abstract: Many recent works have shown trainability plays a central role in neural network pruning -- unattended broken trainability can lead to severe under-performance and unintentionally amplify the effect of retraining learning rate, resulting in biased (or even misinterpreted) benchmark results. This paper introduces trainability preserving pruning (TPP), a scalable method to preserve network trainability against pruning, aiming for improved pruning performance and being more robust to retraining hyper-parameters (e.g., learning rate). Specifically, we propose to penalize the gram matrix of convolutional filters to decorrelate the pruned filters from the retained filters. In addition to the convolutional layers, per the spirit of preserving the trainability of the whole network, we also propose to regularize the batch normalization parameters (scale and bias). Empirical studies on linear MLP networks show that TPP can perform on par with the oracle trainability recovery scheme. On nonlinear ConvNets (ResNet56/VGG19) on CIFAR10/100, TPP outperforms the other counterpart approaches by an obvious margin. Moreover, results on ImageNet-1K with ResNets suggest that TPP consistently performs more favorably against other top-performing structured pruning approaches. Code: https://github.com/MingSun-Tse/TPP.

Published as a conference paper at ICLR 2023
TRAINABILITY PRESERVING NEURAL PRUNING
Huan Wang's Yum Ful.2
Northeestern University, Boston, USA wang. heasterr. edu
*AInnovation Labs, Inc. yunfuêece new..eneeu.... edu
ABSTRACT
Many recent works have shown trainability plays a central role in neural network pruning unattended broken trainability can lead severe under-performance and unintentionally amplify the effect of retraining learning rate, resulting in bi- ased (or even misinterpreted) benchmark results. This paper introduces trainabil- ity preserving pruning (TPP), a scalable method to preserve network trainability against pruning, aiming for improved pruning performance and being more robust to retraining srper-parameters (e.g., learning rate). Specifically, we propose to penalize the gram matrix of convolutional filters to decorrelate the pruned filters from the retained filters. In addition to the convolutional layers, per the spirit of preserving the trainability the whole network, we also propose to regularize the batch normalization parameters (scale and bias). Empirical studies on linear MLP networks show that TPP can perform on par with the oracle trainability re- covery scheme. On nonlinear ConvNets (ResNet56/VGG19) on CIFAR10/100, TPP outperforms the other counterpart approaches by an obvious margin. More- over, results on ImageNet-1 with ResNets suggest that TPP consistently per- forms more favorably against other top-performing structured pruning approaches. Code: http://giuuu..oomMminnnnnn-----e
INTRODUCTION
Neural pruning aims remove redundant parameters without seriously compromising the perfor- mance. It normally consists of three steps (Reed, 1993; Han et al., 2015; 201 Li et al., 2017; Liu al., 2019b; Wang et al., 202 Gale et al., 2019; Hoefler et al., 2021; Wang et al., 2023): pre- train a dense model; prune the unnecessary connections to obtain a sparse model; retrain the sparse model to regain performance. Pruning is usually categorized into two classes, unstructured pruning (a.k.a. element-wise pruning fine-grained pruning) and structured pruning (a.ka. filter pruning or coarse- -raried pruning). Unstructured pruning chooses a single weight as the basic pruning el- ement; while structured pruning chooses a group of weights (e.g., 3d filter or 2d channel) as the basic pruning element. Structured pruning fits more for acceleration because of the regular sparsity. Unstructured pruning, in contrast, results in irregular sparsity, hard to exploit for acceleration unless customized hardware and libraries are available (Han et al., 2016a; 2017; Wen et al., 2016).
Recent papers (Renda et al., 2020; Le & Hua, 2021) report an interesting phenomenon: During retraining, a larger learning rate (LR) helps achieve significantly better final performance, empow- ering the two baseline methods, random pruning and magnitude pruning, more complex pruning algorithms. The reason behind is argued (Wang et al., 202 a; 2023) to be re- lated to the trainability neural networks (Saxe al., 2014; Lee al., 2020: Lubana Dick, 2021). They make two major observations to explain the LR effect mystery (Wang et al., 2023). (1) The weight removal operation immediately breaks the network trainability or dynamical isometry (Saxe et al., 2014) (the ideal case of trainability) of the trained network.
match or beat many
(2) The broken trainability slows down the optimization in retraining, where a greater LR aids the model converge faster, thus a better performance is observed earlier using a smaller LR can actually do well, but needs more epochs.
Although these works (Lee et al., 2020; Lubana & Dick, 2021; Wang et al., 2021a; 2023) provide plausibly sound explanation, more practical issue is how to recover the broken trainability or maintain it during pruning. In this regard, Wang et al. (2021a) proposes to apply weight orthogo- nalization based on QR decomposition (Trefethen & Bau III, 1997; Mezzadri, 2006) to the pruned

---

Published as a conference paper at ICLR 2023
anall-maguitude (unimportant) filters and their BN params: Apply TPP regularkzation Eq. (3), Eq. (5)
Large-magnitude (important) filters and their BN params: Not regularized
filter 1,4,6
filter 2,3,5,7,8
Skip Connection
BN
Wo
W
[6-11)
F(1-2)
Residual Block
Figure I: Illustration of the proposed TPP algorithm typical residual block. Weight parameters are classified into two groups as a typical pruning algorithm does: important (white color) and unimportant (orange blue color), right from the beginning (before any training starts) based the filter Lo-nmm.. Then only the unimportant parameters are enforced with the proposed TPP regularization terms, which is the key to maintain trainability when the unimportant weights are eventually eliminated from the network. Notably, the critical part of a regularization-base algorithm lies in its specific regularization term, i.e. Eqs. (3) and (5), which we will show perform more favorably than other alternatives (see Tabs. and 10).
pruning
model. However, their method is shown to only work for linear MLP networks. On modern deep convolutional neural networks (CNNs), how maintain trainability during pruning still elusive.
We introduce trainability preserving pruning (TPP), a new and novel filter pruming algorithm (see Fig. 1) that maintains trainability via a regularized training process. By our observation, the primary cause that pruning breaks trainability lies in the dependency among parameters. The primary idea of our approach is thus to decorrelate the pruned weights from the kept weights so as "cut off" the dependency, so that the subsequent sparsifying operation barely hurts the network trainability.
Specifically, we propose to regularize the gram matrix of weights: All the entries representing the correlation between the pruned filters (i.e., unimportant filters) and the kept filters (i.e. important filters) are encouraged to diminish to zero. This is the first technical contribution of our method. The second one lies in how treat the other entries. Conventional dynamical isometry wisdom suggests orthogonality, namely, self-correlation and 0 cross-correlation, even among the kept filters, while find directly translating the orthogonality idea here is unnecessary or harmfull because the too strong penalty will constrain the optimization, leading to deteriorated local minimum. Rather, propose not to impose any regularization on the correlation entries of kept filters.
Finally, modern deep models are typically equipped with batch normalization (BN) (loffe & Szegedy, 2015). However, previous filter pruning papers rarely explicitly take BN into account (ex- cept two (Liu et al., 2017; Ye et al., 2018); the differences of our work from theirs will be discussed in Sec. 3.2) mitigate the side effect when it removed because its associated filter is removed. Since they are also part of the whole trainable parameters in the network, unattended removal of them will also lead to severely crippled trainability (especially large sparsity). Therefore, BN parameters (both the scale and bias included) ought to be explicitly taken into account too, when we develop the pruning algorithm. Based on this idea, we propose to regularize the two learnable parameters of BN to minimize the influence of its absence later.
Practically, our TPP easy to implement and robust to hyper-parameter variations. On ResNer50 ImageNet, TPP delivers encouraging results compared to many recent SOTA filter pruning methods.
Contributions. (1) We present the first filter pruning method (trainability preserving pruning) that effectively maintains trainability during pruning for modern deep networks, via a customized weight gram matrix as regularization target. (2) Apart from weight regularization, a BN regularizer is introduced to allow for their subsequent absence in pruning this issue has been overlooked by most previous pruning papers, although it is shown to be pretty important to preserve trainability, especially in the large sparsity regime. (3) Practically, the proposed method can easily scale to
2

---

Published as a conference paper at ICLR 2023
modern deep networks (such ResNets) and datasets (such as ImageNet- 1K (Deng et al., 2009)). It achieves promising pruning performance in the comparison to many SOTA filter pruning methods.
2 RELATED WORK
Network pruning. Pruning mainly falls into structured pruning (Li et al., 2017; Wen et al., 2016; He et al., 2017; 2018; Wang et al., 202 20) and unstructured pruning (Han al., 2015; 2016b; LeCun et al., 1990; Hassibi & Stork, 1993; Singh & Alistarh, 2020), according to the sparsity structure. For more comprehensive coverage, we recommend surveys (Sze et al., 2017; Cheng et al., 2018; Deng et al., 2020; Hoefler et al., 2021; Wang et al., 2022). This paper targets structured pruning (filter pruning, to be specific) because is more imperative to make modern networks (e.g., ResNets (He et al., 2016)) faster rather than smaller compared to the early single-branch convolutional networks.
is noted that random pruning of a normally-sized (i.e., not severely weerparameterized) network usually leads to significant performance drop. We need to cleverly choose some unimportant pa- rameters to remove. Such a criterion for choosing is called pruning criterion. In the area, there have been two major paradigms to address the pruning criterion problem dating back to the 1990s: regularization-bue based methods and importance- based (a.k.a daliency-basede methods (Reed, 1993).
Specifically, the regularization based approaches choose unimportant parameters via sparsity- inducing penalty term (e.g., Wen et al. (2016); Yang et al. (2020); Lebedev & Lempitsky (2016); Louizos et al. (2018); Liu et al. (2017); Ye et al. (2018); Zhang et al. (2021a; 2022; 2021b)). This paradigm can be applied to a random or pretrained network. Importance-bant bassed methods choose unimportant parameters via an importance formula, derived from the Taylor expansion of the loss function (e.g., LeCun et al. (1990); Hassibi & Stork (1993); Han et al. (2015; 201 Li et al. (2017); Molchanov et al. (2017; 2019)). This paradigm majorly applied to a pretrained network. Despite the differences, it is worth noting that these two paradigms are not firmly unbridgeable. We can develop approaches that take advantage of both ideas, such as Ding et al. (2018); Wang et al. (2019; 2021b)- these methods identify unimportant weights per certain importance criterion; then, they utilize a penalty term to produce sparsity. Our TPP method in this paper is also in this line.
Trainability, dynamical isometry, and orthogonality. Trainability describes the easiness of opti- mization of a neural network. Dynamical isometry, the perfect case of trainability, is first introduced by Saxe et al. (2014), stating that singular values of the Jacobian matrix are close to 1. It can be achieved (for linear MLP models) by the orthogonality of weight matrix at initialization. Recent works on this topic mainly focus on how to maintain dynamical isometry during training instead of only for initialization (Xie et al., 2017; Huang et al., 2018; Bansal et al., 2018; Huang et al., 2020; Wang et al., 2020). These methods are developed independent of pruning, thus not directly related to our proposed approach. However, the insights from these works inspire us to our method (see Sec. 3.2) and possibly more in the future. Several pruning papers study the network trainability is- sue in the context of network pruning, such as Lee et al. (2020); Lubana & Dick (2021); Vysogorets & Kempe (2021). These works mainly discuss the trainability issue of a randomly initialization network. In contrast, we focus on the pruning case of pretrained network.
3 METHODOLOGY
3.1 PREREQUISITES: DYNAMICAL ISOMETRY AND ORTHOGONALITY
The definition of dynamical isometry that the Jacobian of network has as many singular values (JSVs) as possible close to 1 (Saxe et al., 2014). With it, the error signal can preserve its norm during propagation without serious amplification or attenuation, which in turn helps the convergence of (very deep) networks. For a single fully-connected layer W, sufficient and necessary condition to realize dynamical isometry is orthogonality, Le.,, W W =,
y = Wx, x,
||y|| vy y v Wx ||x||
if W W I,
where I represents the identity matrix. Orthogonality of weight matrix can be easily realized by matrix orthogonalization techniques such as QR decomposition (Trefethen & Bau III, 1997; Mez- zadri, 2006). Exact (namely all the Jacobian singular values are exactly 1) dynamical isometry
3

---

Published as a conference paper at ICLR 2023
Target gram mutrix WWT
Target grum mutrix WWT
Target gram mutrix WWT
413
A31
:
N
(a) Keree orthogonality
(b) Kernel orthogonality for pruning
(6) De-comelate pruned from kept
Figure 2: Regularization target comparison between the proposed scheme (c) and similar counter-
parts (a) and (b). Green part stands for zero entries. Index to N denotes the filter indices. (b, c), filter 2 and N are the unimportant filters to be removed. (a) Regularization target of pure kernel orthogonality (an identity matrix), no pruning considered. (b) Regularization target of directly ap- plying the weight orthogonality to filter pruming. (c) Regularization target of the proposed weight de-correlation solution in TPP: only regularize the filters to be removed, leave the others uncon- strained. This scheme maintains trainability while imposing the least constraint on the weights.
can be achieved for linear networks since multiple linear layers essentially reduce single 2d weight matrix. In contrast, the convolutional and non-linear cases are much complicated. Previous work (Wang et al., 2023) has shown that merely considering convolution ton, 2010) renders the weight orthogonalizattin method much less effective in terms recovering dynamical isometry after pruning, let alone considering modern deep networks with BN (Ioffe & Szegedy, 2015) and residuals (He et al., 2016). The primary goal of our paper is to bridge this gap.
ReLU (Nair & Hin-
Following the seminal work of Saxe et al. (2014), several papers propose to maintain orthogonality during training instead of sorely for the initialization. There are primarily two groups of orthogo- nalization methods for CNNs: kernel orthogonality (Xie et al., 2017; Huang al., 2018; 2020) and orthogonal convolution (Wang et al., 2020),
KKT = I Lorth KKT
kernel orthogonality
(2)
KK = I Lorth KKT
orthogonal convolution
Clearly the difference lies in the weight matrix vs. K: (1) denotes the original weight matrix in convolutional layer. Weights of CONV layer make up 4d tensor R NxCxxXXW (N stands for the output channel number, C' for the input channel number, H and W for the height and width of the CONV kernel.. Then, K is reshaped version of the 4d tensor: K E R NXXHHW (if N CHW; otherwise, K E RCHW xN). (2) In contrast, K E N HCHH,, stands for the doubly block- Toeplitz representation of K (H stands for the outpul feature map height, H for the input feature map height. Wo and W can be inferred the same way for width).
Wang et al. (2020) have shown that orthogonal convolution is more effective than kernel orthogo- nality (Xie et al., 2017) in that the latter is only a necessary but insufficient condition of the former. In this work, we will evaluate both methods to see how effective they are in recovering trainability.
3.2 TRAINABILITY PRESERVING PRUNING (TPP)
Our TPP method has two parts. First, we explain how we come up with the proposed scheme and how it intuitively is better than the straight idea of directly applying orthogonality regularization methods (Xie al., 2017; Wang et al., 2020) here. Second, batch normalization regularizer is introduced given the prevailing use of BN standard component in deep CNNs nowadays.
(1) Trainability vs. orthogonality. From previous works (Lee et al., 2020; Lubana & Dick, 2021; Wang et al., 2021a; 2023), we know recovering the broken trainability (or dynamical isometry) im- paired by pruning is very important. Considering orthogonality regularization can encourage isome- try, a pretty straightforward solution is to build upon the existing weight orthogonality regularization schemes. Specifically, kernel orthogonality regularizes the weight gram matrix towards an identity matrix (see Fig. 2(a)). In our case, we aim remove some filters, naturally we can regularize the weight gram matrix to be close to a partial identiry matrix, with the diagonal entries at the pruned filters zeroed (see Fig. 2(b); note the diagonal green zeros).
4

---

Published as a conference paper at ICLR 2023
The above scheme is simple and straightforward. However, it not in the best shape by empirical observation. It imposes too strong unnecessary constraint on the remaining weights, which will in turn hurt the optimization. Therefore, we propose to seek weaker constraint, not demanding the perfect trainability (ie. exact isometry realized by orthogonality), but only a benign status, which describes state of the neural network where gradients without being interrupted. Orthogonality requires the Jacobian singular values to be exactly 1; in contrast, benign trainability only requires them not to be extremely large or small so that the network can be trained normally. To this end, we propose to decorrelate the kept filters from the pruned ones: in the target gram matrix, all the entries associated with the pruned filters are zero; all the other entries stay as they are (see Fig. 2(c)). This scheme will be empirically justified (Tab. 3).
flow effectively through the model
Specifically, all the filters in a layer are sorted based on their I, Inonms... Then, we consider those with the smallest L1 norms as unimportant filters (the S below) (so the proposed method also falls into the magnitude-bae based pruning method group). Then, the proposed regularization term is,
L
|W:,, @ (1 mm D) m, = -0 if j E St, else 1,
(3)
where W refers to the weight matrix; represents the matrix full of 1; m is a 0/1-valued column mask vector; @ is the Hadamard (element-wise) product; and || ||r denotes the Frobenius norm.
(2) BN regularization. Per the idea of preserving trainability, BN is not ignorable since BN layers also trainable. Removing filters will change the internal feature distributions. If the learned BN statistics do not change accordingly, the error will accumulate and result in deteriorated performance (especially for deep networks). Consider the following BN formulation (loffe & Szegedy, 2015),
f W X + 3,
V +
where stands for convolution; 2 refers to the running mean/variance; small number, is used for numerical stability. The two learnable parameters are 7 and B. Although unimportant weights enforced with regularization for sparsity, their magnitude can barely be exact zero, making the subsequent removal of filters biased. This will skew the feature distribution and render the BN statistics inaccurate. Using these biased BN statisties will be improper and damages trainability. To mitigate such influence from BN, we propose to regularize both the and / of pruned feature map channels to zero, which gives us the following BN penalty term,
L2 = Σ
Σ
33.
(5)
1= 1ES
The merits of BN regularization will be justified in our experiments (Tab. 4). To sum, with the proposed regularization terms, the total error function is
== > (fi + L2),
2
where L't means the original classification loss. The coefficient grows by predefined constant A per K₂ iterations (up to a ceiling 7) during training to ensure the pruned parameters are rather close to zero (inspired by Wang al. (2019; 2021b)). See Algorithm in Appendix for more details.
Prior works (Liu et al., 2017; Ye et al., 2018) also study regularizing BN for pruning. Our BN regularization method starkly different from theirs. (1) In terms the motivation or goal, Liu et al. (2017); Ye et al. (2018) regularize to learn unimportant filters, namely. regularizing BN to indirectly decide which filters are unimportant. In contrast, in our method, unimportant filters are decided by their -nomms. We adopt BN regularization for a totally different consideration to mitigate the side effect of breaking trainability, which is not mentioned at all in their works. (2) In terms of specific technique, Liu et al. (2017); Ye et al. (2018) only regularize the multiplier factor y (because it is enough to decide which filters are unimportant) while we regularize both the learn- able parameters because only regularizing one still misses few trainable parameters. Besides, employ different regularization strength for different parameters (by the group of important filters vs. unimportant filters), while Liu al. (2017); Ye et al. (2018) simply adopt uniform penalty
Discussion.
5

---

Published as a conference paper at ICLR 2023
3.0
Mean JSV
Moan 18V
Mean JSV
2.5
Ir=0, .011
2.0
Ir 001
Ir=0, 001
1r=0.01
2.5
fr=0.00 01
1r-0.01
60
60
60
Test accunccy (96
80
92
20
Ir 001
91
Ir=0, 001
Ir=0, 001
1r=0.01
20
20
40
60
80
0
40
60
60
60
80
Epoch
Epoch
Epach
91.36 / 90.54/ 0.0040
92.79/ 92.77 / 1.0000 (b) L1tOrthP
92.82 / 92.77 3.4475 (c) TPP (ours)
(a) L1
Figure Mean Jacobian singular value (JSV) and test accuracy during retraining with differ- ent setups (network: MLP-7-Linear, dataset: MNIST). Below each plot are, in order, the best accuracy of LR 1e-2, the best accuracy of LR 1e-3, and the mean ISV right after prun- ing (i.e,, without retraining). LR 1e-2 and 1e-3 are short for two retraining LR sched- ules: (init LR 1e-2, decay at epoch 30/60, epochs: 90), (init LR 1e-3, decay at epoch 45, epochs: 90). The accuracies averaged by 5 random runs. For reference, the unpruned model has mean JSV 2.4987, test accuracy 92.77.
strength for all parameters this is another key difference because regularizing all parameters (in- cluding those that are meant to be retained) will damage trainability, which is exactly what we want avoid. In short, in terms of either general motivation or specific technical details, our proposed BN regularization is distinct from previous works (Liu et al., 2017; Ye et al., 2018).
4 EXPERIMENTS
Networks and datasets. We first present some analyses with MLP-7-Linear network on MNIST (LeCun et al., 1998). Then compare our method to other plausible solutions with the ResNet56 (He et al., 2016) and VGG19 (Simonyan & Zisserman, 2015) networks. on the CIFARIO and 100 datasets (Krizhevsky. 2009), respectively. Next we evaluate our algorithm on ImageNet- 1K (Deng et al., 2009) with ResNer34 and ResNet50 (He et al., 2016). Finally, we present ablation studies show the efficacy of two main technical novelties in our approach. On ImageNet, we use public torchvision models (Paszke et al., 2019) as the unpruned models for fair comparison with other papers. On other datasets, we train our own base models with comparable accuracies reported in their original papers. See the Appendix (Tab. 5) for concrete training settings.
Comparison methods. We compare with Wang et al. (2021a), which proposes method, OrthP, to recover broken trainability after pruming pretrained models. Furthermore, since weight orthogonal- ity is closely related to network trainability and there have been plenty of orthogonality regulariza- tion approaches (Xie et al., 2017; Wang al., 2020; Huang et al., 2018; 2020; Wang et al., 2020), straightforward solution is to combine them with Ia pruning (Li et al., 2017) to see whether they can help maintain or recover the broken trainability. Two plausible combination schemes are easy see: 1) apply orthogonality regularization methods before L1 pruning, 2) apply orthogonality regulariza- tion methods after L1 pruning, i.e., in retraining. Two representative orthogonality regularization methods are selected because of their proved effectiveness: kemel orthogonality (KernOrth) (Xie al., 2017) and convolutional orthogonality (OrthConv) (Wang et al., 2020), so in total there are four combinations: I + KernOrth, I,- + OrthConv, KernOrth + L,, OrthConv LT.
Comparison metrics. (1) We examine the final test accuracy after retraining with the similar FL.OPs budget- this is currently the most prevailing metric to compare different filter pruning methods in classification. Concretely, we compare two settings; a relatively large retraining LR (1e-2) 2) and a small one 3). We introduce these settings because previous works (Renda et al., 2020; Le & Hua, 2021; Wang et al., 2021a; 2023) have showed that retraining LR has a great impact on the final performance. From this metric,
see how sensitive different methods (2) We also compare the test accuracy before retraining - from this metric, we will see how robust different methods are in the face of weight removal.
the retraining LR.

---

Published as a conference paper at ICLR 2023
Table 1: Test accuracy (%) comparison among different isometry maintenance recovery methods on ResNet56 on 10. Scratch stands for training from scratch. KernOrth means Kernel Or- thogonalization (Xie et al., 2017); OrthConv means Convolutional Orthogonalization (Wang al., 2020). Two retraining LR schedules are evaluated here: initial LR 1e-2 vs. 1e-3. Acc. diff. refers to the accuracy gap of LR 1 against LR 1e-2.
ResNet56 on CIFARIO: Unpruned acc. 93.78%,
Params: 0.85M, FLOPs: 0.25G
Layerwise PR
0.3
0.5
0.7
0.9
0.95
Sparsity/Speedup
31.14% 45x 49.82%/1.9
x 70.57%
90.39% 14x1 15.19%
Initial retraining
LR 1e-2
Scratch
93.16 (0.16)
92.78 (0.23)
92.11 (0.12)
88.36 (0.20)
84,60 (0.14)
L, (Li et al., 2017)
93.79 (0.06)
93.51 (0.07)
92.26 (0.17)
86.75 (0.31)
83.03 (0.07)
L1 + OrthP (Wang et al., 102
93.69 (0.02)
93.36 (0.19)
91.96 (0.06)
86.01 (0.34)
82.62 (0.05)
41 + KernOrth (Xie ct al, 2017)
93.49 (0.04)
93.30 (0.19)
91.71 (0.14)
84.78 (0.34)
80.87 (0.47)
11 + OrthConv (Wang et al, 2020)
92.54 (0.09)
92.41 (0.07)
91.02 (0.16)
(0.27)))27)
80.23 (1.19)
KernOrth (Xie ct al., 2017) + L1
93.49 (0.07)
92.82 (0.10)
90.54 (0.25)
85.47 (0.20)
79.48 (0.81)
OrthConv (Wang ct al., 2020)+ + L1
93.63 (0.17)
93.28 (0.20)
92.27 (0.13)
86.70 (0.07)
83,21 (0.61)
TPP (ours)
93.81 (0.11)
93.46 (0.06)
92.35 (0.12)
89.63 (0.10)
85,86 (0.08)
Tnitial
retraining
L.R 1e-3
11 (I.i et al,,
93.43 (0.06)
93.12 (0.10)
91.77 (0.11)
87.57 (0.09)
83. 10 (0.
TPP (ours)
93.54 (0.08)
93.32 (0.11)
92.00 (0.08)
89.09 (0.10)
85,47 (0.22)
Acc. diff. (L,)
-0.38
-0.40
-0.50
+0.82
+0.07
Acc. diff. (TPP)
-0.27
-0.14
-0.35
-0,54
-0.39
4.1
ANALYSIS: MLP-7-LINEAR ON MNIST AND RESNET56 ON CIFAR10
is a seven-layer linear MIP. It adopted Wang et al. (2021a) for analysis because linear MI.P is the only network that can achieve exact dynamical isometry (all JSVs are exactly 1) so far. Their proposed dynamical isometry recovery method, OrthP (Wang et al., 202 is shown to achieve exact isometry on linear MLP networks. Since we claim method TPP can maintain dynamical isometry too, conceivably, our method should play a similar role to OrthP in pruning. To confirm this, prune the MLP-7-Linear network with method.
TPP can perform as well as OrthP on linear MLP. In Fig. 3, (b) is the one equipped with OrthP, which can exactly recover dynamical isometry (note its mean JSV right after pruning is 1.0000), so it works as the oracle here. (1) OrthP improves the best accuracy from 91.36/90.54 to 92.79/92.77, Using TPP, we obtain 92.81/92.77. Namely, in terms of accuracy, our method is good as the oracle scheme. (2) Note the mean JSV right after pruning - the L1 pruning destroys the mean JSV from 2.4987 0.0040, and OrthP brings back 1.0000. In comparison, TPP achieves 3,4875, at the same order of magnitude of 1 1.0000, also as good as OrthP. These demonstrate, in terms of either the final evaluation metric (test accuracy) or the trainability measure (mean JSV), our TPP performs as well as the oracle method OrthP on the linear MLP.
Loss surface analysis with ResNet56 on CIFARIO. We further analyze the loss surfaces (Li et al., 2018) 1) of pruned networks (before retraining) by different methods. Our result (due to limited space, we defer this result to Appendix; see Fig. 4) suggests that the loss surface of our method is flatter than other methods, implying the loss landscape is easier for optimization.
4.2 RESNET56 ON CIFARIO / VGG19 ON CIFARIO0
Here we compare our method to other plausible solutions on the CIFAR datasets (Krizhevsky, 2009) with non-linear convolutional architectures. The results in Tab. 1 (for CIFARIO) and Tab. 10 (for CIFAR100, deferred to Appendix due to limited space here) show that,
(1) OrthP does not work well - L1 + OrthP underperforms the original L1 under all the five pruning ratios for both ResNet56 and VGG19. This further confirms the weight orthogonalization method proposed for linear networks indeed does not generalize to non-lineas CNNs.
(2) For KemOrrth vs. OrthConv, the results look mixed OrthConv is generally better when applied before the L1 pruning. This is reasonable since OrthConv is shown more effective than KernOrth in enforcing more isometry (Wang et al., 2020), which in turn stand more damage of pruning.
(3) Of particular note is that, none of the above five methods actually outperform the L, pruning or the simple scratch training. It means that neither enforcing more isometry before pruning nor
7

---

Published as a conference paper at ICLR 2023
Table Comparison on ImageNet- IK validation set. "Advanced training recipe (such as cosine LR
schedule) is used; we single them out for fair comparison.
Method
Model
Unpruned top- (%) Pruned lop-1 (%) Top-I drop (%) Speedup
Li (pruned-B) (Li al., 2017)
73.23
72.17
1.06
1.32x
I1 (pruned-B, reimpl.) (Wang al., 2023)
73.31
73.67
-0.36
1.32x
Taylor-FO (Molchanov et al,, 2019)
ResNet34
73.31
72.83
0.48
1.29x
GReg-2 (Wang et al., 202
73.31
73.61
-0.30
1.32x
TPP (ours)
73.31
73.77
-0.46
1.32x
ProvablcFP (Liebenwein al.,
76.13
75.21
0.92
1.43x
MetaPruning (Liu et al, 2019a)
76.6
76.2
0.4
1.37x
ResNet50
GReg-1 (Wang et al., 202
76.13
76.27
-0.14
1.49x
TPP (ours)
76.13
76.44
1.49
IncReg al,
75.60
72.47
3.13
2.00x
(Ho 2018)
76.15
74.61
1.54
1.72x
HRank (Lin al, 2020)
76.15
74.98
1.17
1.78x
Taylor-FO (Molchanov
2019)
76.18
74.50
1.68
1.82x
Factorized (Li ct 2019)
76.15
74.55
1.60
2.33x
DCP (Zhuang et al., 2018)
ResNe150
76.01
74.95
1.06
2,25x
CCP-AC (Peng et al., 2019)
76.15
75.32
0.83
2.18x
GReg-2 (Wang et al., 2021b)
76.13
75.36
0.77
2,31X
CC (Li 2021)
76.15
75.59
0.56
2.12x
MetaPruning (Liu et al., 2019a)
76.6
75.4
1.2
2,00x
TPP (ours)
76.13
75.60
0,53
2.31x
LFPC ( at,,
76.15
74.46
1.69
2.55%
GReg-2 (Wang et al., 2021b)
76.13
74.93
1.20
2,56x
ResNet50
CC (Li 202))
76.15
74.54
1.61
2.68x
TPP (ours)
76.
75.12
1.01
2.56x
IncKeg ( (Wang et al." 2019)
75.60
77:007
4.53
3,00X
Taylor-FO (Molchanov et al., 2019)
76.18
71.69
4.49
3,05x
ResNe150
GReg-2 (Wang et al,, 202
76.13
73.90
2.23
3.06x
TPP (ours)
76.13
74.51
1.62
3.06
Method
Network
Top-I (%)
FL.P (G)
CHEX (Hou al., 2022) CHEXX* (Hoou al., 2022)
77.4
ResNe150
TPP (ours)
77.75
TPP' (ours)
76.52
compensating isometry after pruning help recover trainability. In stark contrast, TPP method outperforms L1 pruning and scratch consistently against different pruming ratios (only one exception pruning ratio 0.7 on ResNet56, but our method is still the second best and the gap to the best is only marginal: 93.46 vs. 93.51). Besides, note that the accuracy trend in general, with larger sparsity ratio, TPP beats L1 or Scratch by a more pronounced margin. This is because, ar larger pruning ratio, the trainability is impaired more, where our method can help more, thus harvesting more performance gains. We will see similar trends many times.
(4) In Tabs. 1 and 10, we also present the results when the initial retraining IRR is 1e-3. 3. Wang al. (2021a) argue that if the broken dynamical isometry can be well maintained/recoovrrdd, the final performance gap between LR le-2 and 1e-3 should be diminished. Now that TPP is claimed to be able to maintain trainability, the performance gap should become smaller. This is empirically verified in the table. In general, the accuracy gap between LR 1e-2 and LR 1e-3 of TPP is smaller than that of L2 pruning. Two exceptions are PR 0.9/0.95 on ResNet56: LR 1e-3 is unusually better than LR 1e for Li pruning. Despite them, the general picture is that the accuracy gap between LR 1e-3 and 1e-2 turns smaller with TPP. This is a sign that trainability is effectively maintained.
4.3 IMAGENET BENCHMARK
We further evaluate TPP on ImageNet- 1K (Deng et al., 2009) in comparison to many existing filter pruning algorithms. Results in Tab. show that TPP is consistently better than the others across different speedup ratios. Moreover, under larger speedups, the advantage of our method is usually more evident. E.g. TPP outperforms Taylor-FO (Molchanov et al., 2019) by 1.15% in terms of the top-1 acc. drop at the 2.31 speedup track; at 3.06 speedup. TPP leads Taylor-FO (Molchanov et al., 2019) by 2.87%. This shows TPP is more robust more aggressive pruning. The reason is easy to see more aggressive pruning hurts trainability more (Lee et al., 2019; Wang et al., 2023), where method can find more use, line with the observations CIFAR (Tabs. and 10).
8

---

Published as a conference paper at ICLR 2023
Table 3: Test accuracy (without retraining) comparison between two plausible schemes diagonal vs. decorrelate in our TPP method.
ResNet56 on CIFARIO: Unpruned acc. 93.78%,
Params: 0.85M, FLOPs: 0.25G
Layerwise PR
0.3
0.5
0.7
0.9
0.95
TPP (diagonal)
92.67 (0.29)
91.97 (0.02)
90.21 (0.23)
23.23
(5.19)
14.23 (1.42)
TPP (decorrelate)
92.74 (0.16)
92.07 (0.05)
89.95 (0.26)
30.35 (4.69)
17.33 (0.50)
VGGT9 on CIFARTO0: Unpruned acc. 74.02%, Params: 20.08M, FLOPs: 0.80G
Layerwise PR
0.1
0.3
0.5
0.7
0.9
TPP (diagonal)
68.70 (0.18)
64.55 (0.14)
55.66 (0.73)
13.76 (0.53)
1.00 (0.00)
TPP (decorrelate)
72.43 (0.12)
69.31 (0.11) 62.59 (0.14)
18.97 (1.25)
1.00 (0.00)
Table 4: Test accuracy (without retraining) comparison w.E.f. the proposed weight gram matrix reg- ularization and BN regularization. PR stands for layerwise pruning ratio.
ResNet56 on CIEARIO: Unprumed acc.
93.78%. Params: 0.85M, FLOPs: 0.25G
Gram Reg
BN Reg
PR = 0.3
PR PR5005"
PR = 0.7
PR 009
PR= 0.95
/
92.94 (0.14)
92.48 (0.19)
90.48 (0.09)
70.53 (1.69)
23.05 (2.61)
X
92.79 (0.03)
92.23 (0.08)
90.46 (0.21)
44.25 (2.46)
16.52 (0.43)
92.40 (0.30)
91.95 (0.04)
90.26 (0.23)
26.79 (2.19)
10.50 (0.63)
VGGI9 on CIFARIOO: Unpruned acc.
74.02%, Params:
20.08M, FLOPs: 0.80G
Gram Reg
BN Reg
PR 0.1
PR 0.3
PR 0.5
PR = 0.7
0.9
/
73.44 (0.07)
71.61 (0.12)
69.28 (0.25)
65.15 (0.20)
2.84 (1.13)
X
73.01 (0.13)
71.26 (0.19)
68.67 (0.10)
61.70 (0,46)
1.75 (0.38)
X
/
71.97 (0.23)
70.26 (0.61)
68.40 (0.30)
2.10 (0.27)
1.02 (0.03)
We further compare to more strong pruning methods. Notably, DMCP (Guo et al., 2020), LeGR (Chin al., 2020), EagleEye (Li al., 2020), and CafeNet (Su et al., 2021) have been shown outperformed by CHEX (Hou et al., 2022) (see their Tab. 1) with ResNet50 on ImageNet. Therefore, here we only compare to CHEX. Following CHEX, we employ more advanced training recipe (e.g., cosine LR schedule) referring to TIMM (Wightman et al., 2021). Results in Tab. 2 suggest that our method surpasses CHEX at different FL.PSs
4.4 ABLATION STUDY
This section presents ablation studies to demonstrate the merits of TPP's two major innovations: (1) We propose not over-penalize the kept weights in orthogonalization (i.e., (c) vs. (b) in Fig. 2). (2) We propose to regularize the two learnable parameters in BN.
The results presented in Tabs. 3 and 4, where compare the accuracy right after pruning (i.e., without retraining). We have the following major observations: (1) Tab. 3 shows using decorrelate (Fig. 2(c)) is better than using diagonal (Fig. 2(b)). generally speaking. Akin Tabs. I and 10, at greater sparsity ratio, the advantage of decorrelate is more pronounced, except for too large sparsity (0.95 for ResNet56, 0.9 for VGG19) because too large sparsity will break the trainability beyond repair. (2) For BN regularization, in Tab. 4, when it is switched off, the performance degrades. It also poses the similar trend: BN regularization is more helpful under the larger sparsity.
CONCLUSION
Trainability preserving is shown to be critical in neural network pruning, while few works have re- alized it on the modern large-scale non-linear deep networks. Towards this end, we present new filter and novel pruning method named trainability preserving pruming (TPP) based on regulariza- tion. Specifically, we propose an improved weight gram matrix as regularization target, which does not unnecessarily over-penalize the retained important weights. Besides, we propose to regularize the BN parameters to mitigate its damage to trainability. Empirically, TPP performs as effectively as the ground-truth trainability recovery method and is more effective than other counterpart ap- proaches based on weight orthogonality. Furthermore, on the standard ImageNet-1K benchmark, TPP also matches even beats many recent SOTA filter pruning approaches. As far we are concerned, TPP is the first approach that explicitly tackles the trainability preserving problem in structured pruning that easily scales to the large-scale datasets and networks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regular- izations in training deep networks? In NeurIPS, 2018. 3
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. In ICML, 2022. 21
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35 (1):126-136, 2018.
Ting-Wa Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model com-
pression via learned global ranking. In CVPR, 2020. 9
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 3, 6,
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. In AAAI, 2018. 3
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.094 09574, 2019.
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmep: Differentiable markov channel pruning for neural networks. In CVPR, 2020. 9
S. Han, X. Liu, and H. Mao. EIE: efficient inference engine on compressed deep neural network. ACM Sigarch Computer Architecture News, 44(3):243-244, 201
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. In NeurlPS, 2015. 1,3
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. In ICLR, 2016b. 1,
Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, et al. ESE: Efficient speech recognition engine with sparse 1stm on fpga. In ACM/SIGDA International Symposium on Field-Pron Programbogee Gate Arrays, 2017. 1
B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NeurlPS, 1993. 3
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 3, 6
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In ISCAI, 2018, 3,
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In CVPR, 2020. 8
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. JMLR, 22 1241):1-124, 2021. 1,3
Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen- Chen, Rong Jin, Yuan Xie, and Sun- Yuan Kung. Chex: Channel exploration for enn model compression. CVPR, 2022. 8,
10

---

Published as a conference paper at ICLR 2023
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In AAAI, 2018. 3, 4, 6
Lei Huang, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, and Ling Shao. Controllable orthog-
onalization in training dnns. In CVPR, 2020. 3, 4, 6
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 2, 4,5
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 6,
Duong H Le and Binh-Son Hua. Network pruning that matters: A case study on retraining variants. In ICLR, 2021. 1,6
V. Lebedev and Lempitsky. Fast convnets using group-wise brain damage. In CVPR, 2016.
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In NeurlPS, 1990. 3
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2288-2324, 1998. 6
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019. 8
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. In ICLR, 2020. 1, 3,4
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In ECCV, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017. 1, 3, 6, 7, 8, 16, 17, 18, 19, 20, 21
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. NeurlPS, 2018.
Tuanhui Li, Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convo- lutional neural networks via factorized convolutional filters. In CVPR, 2019. 8
Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In CVPR, 2021. 8
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In ICLR, 2020. 8
Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang. Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In CVPR. 2020. 8
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In ICCV, 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017. 3. 5,
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019b. 1, 13
Christos Louizos. Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1 regularization. In ICLR, 2018. 3

---

Published as a conference paper at ICLR 2023
Ekdeep Singh Lubana and Robert P Dick. A gradient flow framework for analyzing network pruning. In ICLR, 2021. 3,4,
Francesco Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math-ph/0600050, 2006. 1,
Molchanov, S. Tyree, and T. Karras. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, 2019. 3, 8, 18, 20
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurlPS, 2019. 6, 13
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In ICML, 2019. 8
R. Reed. Pruning algorithms survey. IEEE Transactions Neural Networks, 4(5):740-747, 1993. 3,3
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020. 1,
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. In ICLR, 2014. 3,
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 6
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. In NeurIPS, 2020. 3
Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. In ICLR, 2021.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: tutorial and survey. Proceedings of the IEEE, 105(12):2295-232) 2017. 3
Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997. 1,
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. arXiv preprint arXiv:2107.02306, 2021. 3
Huan Wang, Xinyi Hu, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruning for efficient convolutional neural networks via incremental regularization. IEEE Journal of Selected Topics in Signal Processing (JSTSP), 14(4):775-788, 2019. 5,8
Huan Wang, Can Qin, Yue Bai, and Yun Fu. Dynamical isometry: The missing ingredient for neural network pruning, arXiv preprint arXiv:2021.08 005966, 2021a. 1,4, 6, 7, 8, 16
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In 5, 13, 14, 16, 19
ICLR, 2021b.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Recent advances on neural network pruning at initialization. In IJCAI, 2022. 3
Huan Wang, Can Qin, Yue Bai, and Yun Fu. Why is the state of neural network pruming fusing? on the faimess,, comparison setup. and trainability in network pruning. arXiv preprint arXiv:2301.00 005219 2023. 1, 4,4, 6, 8, 18, 20
12

---

Published as a conference paper at ICLR 2023
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In CVPR, 2020. 3, 4, 6, 7, 13, 16
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. NeurIPS, 2016. 1,3, , 18,
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint 1xXiv:2100000466. 2021. 9
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In CVPR, 2017. 3, 4, 6, 7,
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differen- tiable scale-invariant sparsity measures. In ICLR, 2020. 3, 18, 20
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the
assumption in channel pruning convolution layers. ICLR, 2018. 2, 3,
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. In NeurlPS, 2021a.
Yulun Zhang, Huan Wang. Can Qin, and Yun Fu. Learning efficient image super-resolution networks via structure-regularried pruning. In ICLR, 2022. 3
Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In ICMI., 2021b. 3
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awarc channel pruning for deep neural networks. In NeurIPS, 2018.
A IMPLEMENTATION DETAILS
Code reference. We mainly refer to the following code implementations in this work. They all open- open-licensed.
Official PyTorch ImageNet example';
(Wang al., 2021b)?;
OrthConv (Wang et al., 2020)3;
Rethinking the value of network pruning (Liu al., 2019b)
Data split. All the datasets in this paper are public datasets with standard APIs in PyTorch (Paszke et al., 2019). We employs these standard APIs for the train/test data split to keep fair comparison with other methods.
Training setups and hyper-parameters. Tab. summarizes the detailed training setups. For the hyper-parameters that introduced in TPP method: regularization granularity A, regulariza- tion ceiling 7 and regularization update interval K,,. we summarize them in Tab. 6. We mainly refer the official code of GReg-1 (Wang al., 2021b) when setting up these syper-parameters since we tap into a similar growing regularization scheme as GReg-l does.
For small datasets (CIFAR and MNIST), each reported result averaged by at least random runs, mean and std reported. For ImageNet-IK, we cannot run multiple times due to our limited resource budget. This said, in general the results on ImageNet have been shown pretty stable.
htusss/wwillllooooogggsssseieuer Pruninn
13

---

Published as a conference paper at ICLR 2023
Table 5: Summary of training setups. In the parentheses of SGD are the momentum and weight decay. For LR schedule, the first number is initial LR; the second (in brackets) is the epochs when
LR is decayed by factor 1/10; and #epochs stands for the total number of epochs.
Datasct
MNIST
CIFARTO/100
ImageNet
Solver
SGD (0.9, lc-4)
SGD (0.9, 5c-4)
SGD (0.9, lc-4)
Batch size
100
CIFARIO: 128, others: 256
LR schedule (scratch)
le-2, [30,60], Hepochs:90
1e-1, [50), Hepochs:200
le-1, (30,60)0000 #epochs:90
LR schedule (prune)
Fixed (Ie-3)
LR schedule (retrain)
le 1e-2,[30,60], #epochs:90
Te-2, [60,90] #epochs: 120
le-2, [30,60,75], #epochs:90
Table 6: Hyper-parameters of our methods,
Dataset
MNIST
CIFARI0/100
ImageNet
Regularization granularity A
lc-4
Regularization ceiling r
Regularization update interval K,.
10 iterations
5 iterations
Hardware and running time. We conduct all our experiments using 4 NVIDIA V GPUs (16GB memory per GPU). It takes roughly 41 hrs to prune ResNet50 on ImageNet using our TPP method (pruning and 90-epoch retraining both included). Among them, 12 hrs (namely, close to 30%) are spent on pruning and 29 hrs are spent on retraining (about 20 mins per epoch).
Layerwise pruning ratios. The layerwise pruning ratios are pre-specified in this paper. For the ImageNet benchmark, we exactly follow GReg (Wang et al., 2021b) for the layerwise pruning ratios keep fair comparison it. The specific numbers are summarized in Tab. 7. Each number the pruning ratio shared by all the layers of the same stage in ResNet34/50. On top of these ratios, some layers are skipped, such as the last CONV layer in residual block. The best way to examine the detailed layerwise pruning ratio is to check the code at: http://ihuuub.omMMiigggg/
Table 7: A brief summary of the layerwise pruning ratios (PRs) of ImageNet experiments.
Model
Speedup
Pruned top- acc. (%)
PR
ResNer34
1.32x
73.77
[0, 0.50, 0.60, 0.40, 0, 0]
ResNet50
1,49x
76.44
[0, 0.30, 0.30, 0.30, 0.14. 0]
ResNet50
2.31x
75.60
[0, 0.60, 0.60, 0.60, 0.21, 0]
ResNet50
2,56x
75.12
[0, 0.74, 0.74, 0.60, 0.21, 0]
ResNet50
3.06x
74.51
[0, 0.68, 0.68, 0.68, 0.50,
B SENSITIVITY ANALYSIS OF HYPER-PARAMETERS
Among the three syper-parameters in Tab. 6, regularization ceiling 7 works as a termination condi- tion. We only requires it to be large enough ensure the weights are compressed to a very small amount. It does not have to be 1. The final performance is also less sensitive to it. The pruned performance seems be more sensitive the other two syper-parameters, so here we conduct byper-parameter sensitivity analysis to check their robustness.
Results are presented in Tab. and Tab. 9. Pruning ratio 0.7 (for ResNet56) and 0.5 (for VGG19)
are chosen here because the resulted sparsity is the most representative (i.e., not too large or small). (1) For Ka, general, larger K2, tends deliver a better result. This is no surprise since larger K2 allows more iterations for the network to adapt and recover when undergoing the penalty. (2) For A, we do not see a clear pattern here: either a small or large A can achieve the best result (for different networks). On the whole, when varying the hyper-parameters within reasonable range, the performance is pretty robust, no catastrophic failures. Moreover, note that the default setting is actually not the best for both K,, and A. This is because we did not heavily search the best hyper-pane ppprmttrraeters; however, they still achieve encouraging performance compared to the counterpart methods, have shown in the main text.
14

---

Published as a conference paper at ICLR 2023
Table 8: Robustness analysis of K the CIFARIO and 100 datasets with TPP algorithm. In
default, Ku 10. Layerwise PR = 0.7 for ResNet56 and 0.5 for VGG19. The best is highlighted in red and the worst in blue.
K
5
10
20
Acc. (%, ResNet56)
92.2440.03
92.42+0.14
92.35+o.12
92.50
92.31 20.16
Acc. (%, VGGI9)
71.33
71.4540.21
71.61+0.08
71.4340.21
71.68+0.18
Table 9: Robustness analysis of A on the CIFARIO and 100 datasets with our TPP algorithm. In default, A le 4. Layerwise PR = 0.7 for ResNet56 and 0.5 for VGG19, The best is highlighted in red and the worst in blue.
A
lc-5
5e-5
lc44
5e-4
lc-3
Acc. (%, ResNet56)
92.37
92.2940.10
92.35
92.40
92.44
Acc. (%, VGGI9)
+1.39
71.37+0.14
71.61 +0.08
71.58+0.31
71.25+0.31
C ALGORITHM DETAILS
The details of our TPP method is summarized in Algorithm 1.
Algorithm
T'rainability Preserving Pruning (TPP)
1: Input: Pretrained model O layerwise pruning ratio ri of 1-th layer, for 21, L).
2: Input: Regularization ceiling т, penalty coefficient update interval Ku,, penalty granularity A.
0. X, for all filter j. Set pruned filter indices Si by L -o-morm sorting. 4: while Ay S T, for j E S do
3: Init: Iteration
5:
if' i % K2
then
6:
\ = xy + A for j E St.
Update regularization co-efficient in
Eq.
(6)
7:
end if
8:
Network forward, loss (Bq. (6) backward, parameter update by stochastic gradient descent.
9:
Update iteration: i = 1.
10: end while
11: Remove filters in St to obtain smaller model O'
12: Retrain @' to regain accuracy.
13: Output: Retrained model O'
D RESULTS OMITTED FROM THE MAIN TEXT
Loss surface visualization. The loss surface visualization of ResNet56 on CIFAR10 is presented Fig. 4.
VGG19 on CIFAR100. The results of VGG19 on CIFARIO0 is shown in Tab. 10.
Examination of the early retraining phase. To further understand how pruning hurts trainability and how our TPP method maintains it. in Tab. 11, we list the mean JSVs of the first 10-epoch retraining (at pruning ratio 0.9). Note the obvious mean JSV gap between LR 0.01 and LR 0.001 without OrthP: LR 0.01 can reach mean JSV 0.65 just after epoch of retraining while LR 0.001 takes over 8 epochs. When OrthP is used, this gap greatly shrinks. We also list test accuracy of the first 10-epoch retraining in Tab. 12. Particularly note that the test accuracy correlates well with the mean JSV trend under each setting, implying that the damaged trainability primarily answers for the under- performmnee of LR 0.001 after pruning.
Then, when TPP used in place of OrthP, we can see after 1-epoch retraining, the model can achieve mean JSV above 1 and test accuracy OrthP. These re-iterate that the proposed TPP method can work just as effectively as the ground-truth trainability recovery method OrthP on this toy setup.
90%, which are particularly similar to the effect of using
15

---

Published as a conference paper at ICLR 2023
144
539
642
3.0
53
485
-356
-025
660
659
3
-15
-8.5
-850
-455
0.
038
1.0
.1.0
-106
-03
06
01 160
(a) L1 (Li et al., 2017)
(b) GReg-2 (Wang et al., 2021b)
(c) TPP (ours)
Figure 4: Loss surface visualization of pruned models by different methods (w/o retraining).
ResNet56 CIFAR10. Pruning ratio: 0.9 (coom in to examine the details).
Table 10: Test accuracy (%) comparison among different dynamical isometry maintenance or re- covery methods on VGG19 on CIFAR100. 100. Scrutch stands for training from scratch. KernOrth means Kernel Orthogonalization (Xie et al., 2017); OrthConv means Convolutional Orthogonaliza- tion (Wang al., 2020). Two retraining LR schedules evaluated here: initial LR 1e-2 vs. 1e-3. Acc. diff. refers to the accuracy gap of LR 1e-3 3 against LR 1e-2. 2.
VGGI9 on CIFAR100: Unpruned acc. 74.02%, Params: 20.08M, FLOPs: 0.80G
Layerwise PR Sparsity/Speddup
0.1
0.3
0.5
0.7
0.9
19.24% x 51.01% x
74.87% 90.98%/8.844 98.96%
Tnitial
retraining
1e-2
Scratch
72.84 (0.25)
71.88 (0.14)
70.79 (0.08)
(0.11)
54.37 (0.40)
L, (Li et 2017)
74.01 (0.18)
73.01 (0.22)
71.49 (0.14)
66.05 (0.04)
51.36 (0.11)
L, OrthP (Wang al., 2021a)
74.00(0.04
72.30 (0.49)
68.09 (0.24)
62.22 (0.15)
48.07 (0.31)
I. KerOOrth (Xie al., 2017)
73.72 (0.26)
72.53 (0.09)
71.23 (0.10)
(0.14))
50.75 (0.30)
I. + OrthConv (Wang al., 2020)
73.18 (0.10)
72.25 (0.31)
(0.11)
(4.51.00433
48.31 (0.18)
KermOrth (Xie et al., 2017) +
73.73 (0.23)
72.41 (0.12)
(0.19
50.72 (0.87)
OrthConv (Wang al., 2020) L
73.55 (0.18)
72.67 (0.09)
65.66 (0.10)
50.53 (0.46)
TPP (ours)
74.02 (0.24)
73.19 (0.07)
71.61 (0.08)
67.78
57.70 (0.37)
Initial
retraining
1e-3
L, (Li et al., 2017)
73.67 (0.05)
72.04 (0.12)
(0.02)
64.72 (0.17)
(0.44)
TPP (ours)
73.83 (0.02)
72.29 (0.07)
71.16 (0.12)
67.47 (0.17)
56.73 (0.34)
Acc. diff. (L,)
-0.34
-0.97
-1.28
-1.33
-2.93
Acc. diff. (TPP)
-0.19
-0.90
-0.45
-0.31
-0.97
E MORE ANALYTICAL RESULTS
In this section, we add more analytical results to help readers better understand how TPP works.
E. COMPARISON UNDER SIMILAR TOTAL EPOCHS
In Tab. 1, TPP takes few epochs for regularized training before the sparsifying action, while pruning is one-shot, taking no epochs. Namely, the total training cost of TPP is larger than those one-shot methods. It is of interest how the comparison will change if these one-shot methods are given more epochs for training.
First, note that the results of KernerOrrth, L1+Orthonnv, KemOrth+L4, and OrthConv+Li in Tab. 1 also involve training (the Kemorth/OrthCan is essentially a regularized training), which takes 50k iterations. We make following changes:
Our TPP takes 100k iterations based on the default hyper-parameter setup, so make fair comparison, we decrease the regularization update interval K2, from 10 to 5, making the regularization of TPP also take 50k iterations.
Meanwhile, we add 128 retraining epochs (50k 391 iters per epoch zs 128 epochs) to the L1 and Lr+OrthP methods (when their retraining epochs are increased, the LR decay epochs are proportionally scaled); plus the original 120 epochs, the total epochs are 248 now.
16

---

Published as a conference paper at ICLR 2023
Table 11: Mean JSV of the first 10 epochs under different retraining settings. Epoch 0 refers the model just pruned, before any retraining. Pruning ratio is 0.9. Note, with OrthP, the mean JSV is because OrthP can achieve exact isometry
Epoch
0
2
4
5
6
8
10
LR=10 w/o OrthP
0.00
0.65
0.89
1.01
0.97
1.10
1.14
1.27
1.33
1.29
1.42
LR=10-S, w/o OrthP
0.00
0.00
0.00
0.00
0.10
0.25
0.35
0.42
0,51
0.74
0.86
LR=10=2 w/ OrthP
1.00
1.23
1.42
1.43
1.40
1.47
1.51
1.56
1.60
1.65
1.68
LR=10 w/ OrthP
1.00
1,50
1.64
1.73
1.84
1.87
1.93
1.96
1.99
2.00
2.04
LR=10 ""
w/ TPP (ours)
2.98
22.5
2.01
1.67
1.97
2.10
1.97
2.08
2.05
2.06
2.06
LR=10-3, w/ TPP (ours)
2.98
2.96
2.95
2.99
2.99
3.01
3.21
3.19
3.05
3.04
3.02
Table 12: Test accuracy (%) of the first 10 epochs corresponding to Tab. 11 under different retraining settings. Epoch 0 refers to the model just pruned, before any retraining. Pruming ratio is 0.9.
Epoch
2
5
10
LR=10 w/o OrthP
9.74
64.34
80.01
80.23
81.77
85.80
85.82
86.21
86.35
86.60
86.15
LR=10-S w/o OrthP
9.74
9.74
9.74
11.89
21.34
27.75
32.96
35.38
49.66
64.89
68.97
LR=10-2 w/ OrthP
9.74
91.05
91.39
91.33
91.37
91.77
91.69
90.74
91.39
91:58
91.44
LR=10 3, w/ OrthP
9.74
90.81
91.59
91.77
91.85
92.04
92.12
92.22
92.12
92.33 92.25
TR=10 w/ TPP (ours)
89.21
91.54
91.00
91.45
91.83
91.56
90.89
91.33
90.68
91.54
91.21
LR=10-3 TPP (ours)
89.21
92.12
91.82
92.09
92.15
91.95
92.00
92.02
92.09
92.08 92.08
Now all the comparison methods in Tab. 1 have the same training cost (i.e, the same 248 total epochs). The new results of I, OrthhP and TPP under this strict comparison setup are presented below:
Table 13: Test accuracy comparison under the same total epochs (ResNet56 on CIFAR10).
Layerwise PR
0.3
0.5
0.7
0.9
0.95
L1 (Li al., 2017)
93.65 (0.14)
93.38 (0.16)
92.11 (0.16)
87.17 (0.26)
83.94 (0.45)
L, (Li et al., Orth)
93.58 (0.03)
93.30 (0.10)
91.69 (0.13)
85.75 (0.26)
82.30 (0.20)
TPP (ours)
93.76 (0.10)
93.45 (0.05)
92.42 (0.14)
89.54 (0.08)
85.98 (0.29)
We make the following observations:
For Li pruning, more retraining epochs do not always help. Comparing these results to Tab. 1, we may notice at small PR (0.3, 0.5), the accuracy drops little (this probably is due to overfitting when the PR is small, the pruned model does not need many epochs to recover; while too long training triggers overfitting). For larger PR (like 0.95), more epochs help quite significantly (improving the accuracy by 0.91%).
L1+OrthP still underperforms L1, same as in Tab.
Despite using fewer epochs, TPP is still pretty robust Compared to Tab. 1, the perfor- mance varies by very marginal gap ( 0.1%, within the std range, not statistically significant gap). In general, TPP is still the best among all the compared methods, and, the advantage is more obvious at larger PRs, implying TPP is more valuable in more aggressive pruning cases.
E.2 TPP Random BASE MODELS
Pruning is typically conducted pretrained model. TPP is also brought up in this context. This said, the fundamental idea of TPP, i.e., the proposed regularized training which can preserve train- ability from the sparsifying action, is actually independent of the base model. Therefore, we may expect TPP can also surpass the L, pruning method (Li et al., 2017) in the case of pruming a ran- dom model this is confirmed in Tab. 14. As expected, the damaged trainability problem does not
17

---

Published as a conference paper at ICLR 2023
only exit for pruning pretrained models, but also exits for pruning random models; so TPP performs better than L1 pruning, especially in the large sparsity regime.
Table 14: Test accuracy of applying TPP vs. L, pruning (Li et al., 2017) to random base model.
ResNet56 on CIFARIO: Unpruned acc. 93.78%, Params: 0.85M, FLOPs: 0.25G
Layerwise PR
0.3
0.5
0.7
0.9
0.95
L, pruning (Li et al., 2017) TPP (ours)
88,98 (0.13)
88.79 (0)
87.60 (0.21)
85.09 (0.09)
82.68 (0.32)
91.93 (0.13)
91.27 (0.16) 90.36 (0.16) 87.60 (0.09)
85.36 (0.18)
Moreover, in Fig. 5, we show the mean JSV and test accuracy when pruning random model with different schemes (L1, L1+OrthP, and our TPP). We observe that, in general, the JSV and test accuracy of pruning random model pose similar patterns to pruning pretrained model (Fig. 3):
Using L1, the LR=0.01 achieves "better" (not really better, but due to damaged trainability,, the performance of LR=0.001 has been underestimated, see Wang et al. (2023) for more detailed discussions) JSV and test accuracy; note the test accuracy solid line is above the dashed line by an obvious margin.
While using L Orthh or our TPP, the LR=0,001 can actually match LR=0.01. Same as the case of pruning a pretrained model, here, TPP behaves similarly the oracle trainability recovery method OrthP.
To summarize, the prainabiltty-rreeevving effect of TPP also generalizes to the case of prun- ing random networks.
35V
1.0
Monn
Inw0.001
I-eil E01
001 [m-2.01
[--6/0]
0.0
MM
50
1--0:01
1-1101
1y-0.01
40
50
20
43
80
10
Spuch
Epoch
Upoch
87.86/ 82.47 / 0.0002 (a) Ta
92.78 99.771 / 1.0000 (b) L +rtthP
92.82 / 2.36617
(c) TPP (ours)
Figure 5: Mean ISV and test accuracy during retraining with different methods (network: ML.P-7- Linear, dataset: MNIST) when pruning random model. Below each plot are, in order, the best accuracy of LR C. 01, the best accuracy of LR 0 001, and the mean JSV right after pruning (i.e. without retraining).
In Fig. 6, we show the mean JSV and test accuracy over the overall process (including the regularized training and retraining) of TPP applicd to a random model.
E.3 TPP MORE ADVANCED PRUNING CRITERIA
In the main text, we demonstrate the effectiveness of TPP, which employs the magnitude (Ly-norm) of filters to decide masks. It of interest whether the effectiveness of TPP can carry over to other more advanced pruning criteria. Here we combine TPP with other 3 more advanced crite- ria: SSL (Wen et al., 2016), DeepHoyer (Yang et al., 2020), and Taylor-FO (Molchanov et al., 2019). SSL and DeepHoyer are regularization-baseee layerwise pruned indices (as well as the pruning ratio) is not pre-spec,,eed, but "learned" by the reg- ularized training. As such, the layerwise pruning ratios are not uniform (see Fig. 7 for example). Taylor-FO is a more complex pruning criterion than magnitude by taking into account the first-order gradient information.
pruning methods like ours; differently, their
18

---

Published as a conference paper at ICLR 2023
2
Mcan JSV
0.001
1r000
50
150
Test accuracy (%)
80
0.0000
70
1r-0001
50
100
150
Epoch
Figure 6: Mean ISV and test accuracy during regularized training and retraining with TPP (network: MLP-7-Linear, dataset: MNIST) when pruning a random model. The model spends 100 epochs on regularized training (LR schedule this stage the same, 0.001, fixed); then pruned by Lı pruning (Li et al., 2017); then retrained with different LR schedules (0.01 vs. 0.001). As seen, pruning the TPP regularized model does not make the mean JSV drop significantly point (epoch 100), namely, Irainability preserved. In stark contrast, when applying Li pruning to a normally trained (i.e., not TPP regularized) model, mean ISV drops from 2.4987 to 0.0040 (see Fig. 3), namely, trainability not preserved. Note, when trainability is preserved, retraining LR 0.01 and 0.001 do not pose obvious test accuracy gap; while trainability is not preserved, the gap would be obvious Fig. 5(a).
the pruning
Specifically, first these methods decide the layerwise pruned indices, given total pruning ratio. Then, we inherit these layerwise pruned indices when using our TPP method.
Results shown in Tab. 15. We observe, small PRs (0.3-0.7), TPP performs similarly I1- This agrees with Tab. 1, where TPP is comparable to LT- At large PRs (0.9, 0.95), the advantage of TPP starts expose more indices, which is statistically significant advantage as indicated by the std (and again, when PR larger, the advantage of TPP is generally more pronounced). This table shows the advantage of TPP indeed can carry over to other layerwise pruning ratios derived from more advanced pruning criteria.
PR 0.9/0.95, TPP beats I, by 0.9/1.44 with SSL learned pruned
E.4 TRAINING CURVE PLOTS: TPP vs. L1 PRUNING
In Fig. 8, we show the training curves of our TPP compared to L, pruning with ResNet56 on CIFAR10.
E.5 PRUNING RESNET5O ON IMAGENET WITH LARGER SPARSITY RATIOS
noted that our method only beats some of the compared methods marginally (<0.5% top-1 ac- curacy) in low sparsity regime (around 2x~ 3 spseedu see Tab. 2). This is mainly because when the sparsity low, network trainability is not seriously damaged, thus our cainability-preservvii method cannot fully expose its advantage. Here showease scenario that trainability is inten- tionally damaged more dramatically.
In Tab. 2, when pruning ResNet50, researchers typically do not prume all the layers the last CONV layer in residual block is usually spared (Li et al., 2017; Wang et al., 2021b), for the sake of performance. Here intentionally prune all the layers (only excluding the first CONV and last
19

---

Published as a conference paper at ICLR 2023
Table 15: Comparison between 11 pruning (Li et al., 2017) and TPP with pruned indices derived from more advanced pruning criterion (Taylor-FO (Molchanov et al., 2019)) or regular- ization schemes (SSL (Wen al., 2016), DeepHoyer (Yang al., 2020)). Network/Dataset: ResNet56/CIFAR1O 10: Unpruned acc. 93.78%, Params: 0.85M, FLOPs: 0.25G. Total PR repre- sents the pruning ratio (PR) of the whole network. Note, due the non-uniform layerwise PRs, the speedup below, which depends the feature map spatial size, can be quite different from each other, even under same total PR.
Total PR
0.5
0.9
0.95
Sparsity/Speedup
12.86%//1.66
47.09%
72.28%//.32x
92.49%
95.87%//.77x
Li (w/ SSL pruned indices)
20.07
93.47:0.05
92.76+0.15
89.00
84.75=0.21
TPP (w/ SSL layerwise pruned indices)
93.86g0.00
93.50go.15
92.81
89,90_a.r
86.19.0.22
Sparsity/Seecdup
26.87%
52.11%//195x
76.50%
93.21%
96.02%/9.60
L1 (w/ DeepHoyer pruned indices)
1
93.81
92.33 os
87.61_o15
84.27
TPP (w/ DeepHoyyrr pruned indices)
93.88
23.5920.01
92.58
88.76_0.17
85.64
Sparsity/Speedup
31.21%
49.94%/1999
70.75%/3.55
90.66%
95.41%
L1 (w/ Taylor-FO pruned indices)
#0.11
93. 20
92.24
87.28
83.31
(w/ Taylor-FO pruned indices)
=0.00
93.64+0.14
92.32+0.13
88.06=0.10
85.34
ResNet56, CIFARIO, PRO.3
ResNet56, CIFARIO, PRO.5
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
10
20
30
40
50
0
10
20
30
40
50
Layer index
Layer index
ResNet56, CIFARIO, PRO.7
ResNet56, CIFARIO, PRO.9
1.00
0.75
0.50
0.25
0.25
0.00
0.00
20
50
30
Layer index
Layer index
ResNet56, CIFARIO, PRO.95
1.00
20
30
50
Layer index
Figure 7: Layerwise pruning ratios learned by SSL (Wen et al., 2016) with ResNet56 on CIFARIO, given different total pruning ratios (indicated in the title of each sub-figure).
classifier FC) in ResNet50. For the L1 pruning method (Li et al., 2017) (we report a stronger version re-implemented Wang al. (2023)), different layers pruned independently since the layerwise pruning ratio has been specified. All the hyper- hyper-parameters parameters of the retraining process are maintained the same for fair comparison, per the spirit brought up in Wang et al. (2023).
20

---

Published as a conference paper at ICLR 2023
ResNet56, CIFARIO, PRO.3
ResNet56, CIFARIO, PRO.5
100.0
100.0
Train sccurac (%)
Train (9g
97.5
TPP
97.5
TPP
95.0
92.5
92.5
90.0
90.0
20
40
80
100
120
20
40
80
100
120
Epoch
Epoch
ResNet56, CIFARIO, PRO.7
ResNet56, CIFARIO, PRO.9
TPP
90
Thin
Train
60
LI
TPP
120
Epoch
Epoch
ResNet56, CIFARIO, PRO.95
Train accuracy
1.1
TPP
0
20
80
100
120
Epoch
Figure 8: Training curves during retraining with ResNet56 on CIFARIO different pruning ratios (PRs). We can observe that at large PRs (0.9, 0.95), TPP significantly accelerates the optimization the comparison L1 (Li et al., 2017), because of better trainability preserved before retraining.
The results in Tab. 16 show that, when the trainability is impaired more, our TPP beats L1 by 0.77 to 2.17 top-1 accuracy ImageNet, much more significant than Tab. 2.
Table 16:
Top-1 I accuracy comparison between TPP and L₁ pruning with larger pruning ratios (PRs). All layers but the 1st CONV and last FC layer (including the downsample layers and the 3rd CONV in residual block) are pruned. Uniform layerwise pruning ratio is used.
Layerwise PR Sparsity/Speedup
0.5
0.7
0.9
0.95
72.94%6/3.63
89.34%//8445
98.25%
99.34%
L, (Li al., 2017)
71.25
66.02
47.96
33.21
TPP (ours)
73.42 (+2.17)
68.16 (+2.14)
49.19 (+1.23)
33.98 (+0.77)
F MORE DISCUSSIONS
Can TPP be useful for finding lottery ticket subnetwork in filter level?
To our best knowledge, filter-level winning tickets (WT) are still hard find even using the original LTH pipeline. Few attempts in this direction have succeeded E.g., Chen al. (2022) tried, but they can only achieve bit marginal sparsity (0%) with filter-level WT (see their Fig. 3, ResNet50 on ImageNet) while weight-level WT typically can be found over 90% sparsity. This said. do think this paper can contribute that direction since preserving trainability is also a central issue in LTH, too.
21


Reviews:
[{"review_id": "fK9i-_Kn8vV", "reviewer": "Reviewer_LVfu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a regularization method to maintain the trainability of the model during the filter pruning process. The regularization is applied on both the weight and the BN parameters. Experiment results on multiple dataset and model architectures show promising performance comparing to other SOTA pruning methods.", "summary_of_the_review": "Generally the paper proposes an interesting and strong method for improving the finetuning trainability of the pruned model. The method is novel and solid. Meanwhile there are some concern on the behavior of the model during the regularized training phase and the fairness of the experiment. I would recommend a weak acceptance for now, expecting the author to clear my doubt in the rebuttal.\n\n## Post rebuttal\n\nI have read other reviews and all responses from the author. Most of my concerns are resolved. The paper provides a novel prespective for improving DNN pruning, where is shows small but consistent improvement in accuracy under practical sparsity, and larger accuracy improvement under high sparsity. I would keep my score as weak accept.", "strength_and_weaknesses": "## Strength\n1. The paper makes novel and significant contribution by introducing the concept of trainability preservation into the neural network pruning process.\n2. The proposed method is well motivated and theortically sound\n3. Thorough emperical evaluation is provided to prove the effectiveness of the proposed method\n\n## Weakness\n1. The paper only demonstrates the model performance during the finetuning process. As a regualrization paper, it would be interesting to show how the regularization perform during the regularized training process. Things like how does the regularization affect model accuracy before pruning, and how it changes the mean JSV. I would encourage the author to repeat Figure 3 with the regualrized training epochs before the pruning.\n2. To my understanding the regularizer requires additional training epochs before pruning to take effect, however there's no discussion on how many epochs is needed, and how it's decided. This may also cause unfair comparison for the results in Table 1 and 2, as some baselines may be trained with less epochs in total (both before and after pruning), and may benefit from more training epochs\n3. The paper limits the exploration on L1 megnitude-based pruning with fixed pruning precentage on each layer. However there are more advanced filter pruning techniques like Taylor importance-based global pruning[1], or dynamically determining the filters to be pruned during regularized training process[2,3]. It would be interesting to see if the proposed method works under these pruning schemes.\n\n[1] https://arxiv.org/abs/1906.10771\n\n[2] https://proceedings.neurips.cc/paper/2016/file/41bfd20a38bb1b0bec75acf0845530a7-Paper.pdf\n\n[3] https://arxiv.org/pdf/1908.09979.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_mqpb-Bv2y0", "reviewer": "Reviewer_p5ry", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on preserving the trainability of network after pruning. As proposed by the authors, the trainability of pruned networks can be affected with the phenomenon of being less robust to finetuning learning rate. To alleviate this issue, the authors proposes the trainability preserving pruning (TPP) that can maintain the trainability during pruning. The authors suggest that the dependency between weights may cause the issue. So in this paper, the authors propose to decorrelate the pruned weights and the kept weights. In detail, the authors propose to regularize the gram matrix of weights in the way that the correlation of between the kept and pruned weights approaches 0. In addition, the authors suggest that the parameters of BatchNormalization should be explicitly considered for pruning by regularizing the parameters. Experiments on MNIST, Cifar and ImageNet verifies the efficacy of the proposed methods. On ImageNet classification task  with ResNet50, the proposed TPP  illustrates significant improvement compared with other SOTA methods.", "summary_of_the_review": "please refer the comments listed above.", "strength_and_weaknesses": "Pros\nThis paper proposes a post process for pertained network to get  more trainable sparse weights for finetuning. Experiments verifies the trainability of the sparse weight via comparisons with different finetuning  learning rate.  Extensive experiments on several datasets illustrates the efficacy of the proposed method. In addition, the authors also consider the effect of batch normalization parameters and adds to the regularization term.\n\nCons\nThe authors illustrate the ablation with regularizing  BN weight and without regularizing BN weight. However, I do not find the results of using the regularization term of BN parameters alone. Could the authors show this result?\n\n\nQuestion\nAs shown in the paper, the proposed TPP method is a post-process algorithm for a pertained model. After the TPP process, the sparse weight is finetuned to get a sparse model with decent performance. I am wondering whether the TPP algorithm can be combined with the pretraining? In addition,  the TPP post process also takes the classification loss into consideration, so it can be considered as the part of finetuning. It would be better to have the experiments with similar finetuning cost to compare TPP with other methods in a more fair way. Whether the proposed TPP can be useful for finding lottery ticket subnetwork in a filter level?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RK-j3t29I7k", "reviewer": "Reviewer_Qf3U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a neural network pruning method which perserves trainability after pruning. The authors argue that the proposed approach will make the network easier to train after pruning therefore lead to a better finetuned model. First, the proposed method use L1 norm to select important and unimportant filters. Then,  a trainability regularization term is induced during the finetuning step which enforces the important filters to be orthogonal to unimportant ones. In addition, parameters in BN layers are regularized by L2 norm for those unimportant filters. The authors validate the proposed method on CIFAR-10/100 and ImageNet-1k.", "summary_of_the_review": "The majority concern is lacking novelty. Some improvments in the experiments would make this work more stronger and more convincing.", "strength_and_weaknesses": "[Strength]\n\n* The paper is overall well-written and easy to follow.\n* The authors compared their method to multiple strong baseline methods, such as OrthConv and KernOrth.\n* In the finetuning, the authors compared two different learning rates. All methods are trained under the same settings for fairness.\n* Experiment results on ImageNet are reported, in addition to the CIFAR-10/100, including a dozen baseline methods.\n* The accuracy improvement is significant on CIFAR-10/100\n\n[Weakness]\n* The novelty is incremental. Orthogonal regularization is not new in training deep network. Dating back to years ago, there were lots of discussions of weight regularization to make network more trainable, such as dynamic isometry constraint or nuclear norm regularizer. This work extends these ideas to network pruning, with some heuristic adaptations. For example, the correlation between selected important filters is not penalized in the proposed regularizer. BN norms of unimportant filters are considered. However, all these adaptations lack strong theoretical motivations. The authors did not provide any theoretical insight of why the proposed decorrelation plays a critical role in deep neural network pruning. From the numerical results, it seems that the improvement of the proposed method is also marginal on large-scale datasets.\n\n* Marginal improvement on large-scale datasets such as ImageNet. The difference sof the proposed method v.s. previous SOTA methods are around or less than 0.5% in most cases. This is not significant, especailly when the training setting in this work is not SOTA. For example, using a better training recipe in \"Resnet strikes back: An improved training procedure in timm“, it is possible to improve ResNet-50 from 76% to 80% on ImageNet-1k. The 0.5% improvement is hardly to say a significant number.\n\n* No trained long enough. In SOTA training setting, ResNet-50 on ImageNet requires 360 epochs of training. Training even longer can still improve accuracy a bit. In this paper, most training experiments are early-stopped. So it is hard to tell the final accuracies when all models are trained by 360 epoches.\n\n* Training curves not reported. As the key argument of this work is trainability, the convergence curve should be plotted. Again, please consider to train long enough such that all models achive their stable convergent points. Then please compare the convergence speed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QvsKhcYVFW", "reviewer": "Reviewer_aS72", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present trainability prerseving pruning (TPP), considering to maintain trainability for the pruned networks. The authors construct two regularization terms to achieve this goal. TTP decorrelates the pruned weights from the kept weights, thus achieves non-trivial improvement in pruning process. ", "summary_of_the_review": "The paper focuses on an interesting topic, and proposes a working solution. However, the motivation of the methods and some specific techniques are not very clear. Therefore, I think this paper should be weakly rejected.", "strength_and_weaknesses": "Strength:\n1. The trainability of pruned models is interesting, and may provide a new perspective to pruning, and even to the whole deep learning field.\n2. The proposed pruning method is easy to implement, and produces good experimental results.\n\nWeakness:\n1. The relationship between trainability and dynamical isometry is not clear. I find no evidence to indicate this point, i.e. how can trainability be expressed by the orthogonal property of channels?\n2. The necessity of contraining BN parameters is not shown. The reason in the paper is that \"Although unimportant weights are enforced with regularization for sparsity, their magnitude can barely be exact zero, making the subsequent removal of filters biased\". However, if the pruned weights are not zeros, how to achieve the speedup of models during  the inference phase?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster