Receding Neuron Importances for Structured Pruning
Authors: Mihai Suteu
Yike Guo
Venue: Submitted to ICLR 2023 (2023)
Keywords: structured pruning
regularization
sparsity
batchnorm
neuron importance

Abstract: Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under severe pruning.

Under review as a conference paper at ICLR 2023
RECEDING NEURON IMPORTANCES FOR STRUCTURED PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity- inducing regularisation on BatchNorm parameters, LI penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling param- eters, based on which we design novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neu- rons effectively recede, producing polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under severe pruning. Source code is available at: 4open. rer
INTRODUCTION
Modern deep neural network architectures (Simonyan & Zisserman 2014 He et al. 2016) achieve otate-of-the-art performance but require significant computational resources which makes their de- ployment onto edge devices difficult. Even though it has been shown that it is possible to train less over- ower-parametriseer models from scratch and obtain similar performance (Frankle & Carbin 2018), it remains a non-trivial task to actually find such a winning subnetwork.
In this work, we focus on structured one-shot pruning as a means to network compression which typically composed of three stages i) training large model to convergence, ii) removing parame- ters with low importance, and iii) fine-tuning the remaining network. Unstructured pruning, which works weight level (Yang al. 2019 Frankle & Carbin 2018), can remove much higher num- ber of parameters but produces sparse weight matrices which cannot be efficiently utilised without specialised hardware (Han et al. 2016). In contrast, by removing entire neurons structured pruning finds efficient structures akin to an implicit architecture search (Liu et al. 2018).
Structured pruning methods attribute an importance score each neuron, which enables their rank- ing and ultimately the decision of which to dispose (Li et al.] [016)] MMochanov et al.] (0166)). To this end, BatchNorm layers (Ioffe & Szegedy 2015)) become very appealing they explicitly learn parameters which uniformly scale the outputs of each neuron. This scaling parameter can be used as proxy for the importance the model attributes a neuron,
value of zero would effectively suppress an output. Furthermore, one can regularise these layers to obtain neuron level sparsity whilst maintaining classification performance (Liu al. 2017 Zhuang et al.) 2020). Such methods typically define neuron importance as the absolute value of its scaling parameter, an approach which limits the design of regularisers. Because the measure only half-bounded one cannot easily define levels of importance without looking at the overall distribution making it difficult target specific neurons. An example is the LI regulariser (Liu et al. (017) which shrinks all parameters with a constant gradient, creates sparsity by only shrinking unimportant neurons, leaving the others untouched.
ones with high importance. Ideally,
would design a regulariser which
In this work, create such regulariser and show it outperforms existing approaches at rate that increases with the amount of neurons pruned. Our contributions are two-fold: we first introduce simple variation of Normh which linearly transforms channels using bounded scalers. This

---

Under review as a conference paper at ICLR 2023
layer maintains the same performance as the original, while offering bounded importance score for neurons. Building on this measure we then define a novel regularisation, focused on shrinking only neurons with lesser weight, by having its gradient decay exponentially for higher importances. Our method significantly outperforms related approaches for VGG models, and we show that severe degradation be attributed to over-pruning early layers of the network.
RELATED WORK
Neural network compression through pruning is most commonly divided into structured and un- structured approaches. Unstructured pruning has gained a lot of attention in recent years (Frankle & Carbin 2018) it challenges conventional wisdom over the role of over- over-paraman parametrisation and weight initialisation in the optimisation of deep neural networks (Frankle et al. (020)). While these methods can achieve superior theoretical compression rates (Renda al. 2020). , their
remains impracti-
cal without specialised hardware that can take advantage of sparsity (Han et al.) (2016). Structured pruning on the other hand removes entire neurons from an architecture thus achieving real memory and computational efficiencies (Liu et al. 2018).
At the heart of structured pruning lies the task of identifying unimportant neurons remove from network. Quantifying importance can be based on numerous criteria including filter norms (Li
2016 He et al., 2018). reconstruction errors (He et al. al.
2017, Luo et al. 2017, Molchanov
et al.
2016b; Yu et al. 2018), redundancy (He et al. 2019
Suau et al.) 2020; Wang et al.) 2018)
and BatchNorm parameters (Liu et al., 2017] Zhuang et al.,
2020). Our work belongs the latter category as we focus on deriving an importance score solely based on channel scaling parameters.
In addition to defining importance measures, one can add regularisation during training to nudge networks into utilising their capacity more sparingly. Most relevant to our work are methods which apply sparsity regularisations on BatchNorm parameters (Liu et al. 2017 Zhuang et al. 2020). The most popular approach is Network Slimming (Liu et al. (017), which constrains the BatchNorm scaling parameters using the LI penalty. A drawback of this method is that shrinks all parameters with equal gradient irrespective of their importance. This issue is also addressed by Zhuang al.. (2200) who propose a regulariser that explicitly maximises the polarisation of the BatchNorm scaler distribution. While the method effectively increases the margin between important and unimportant neurons, it does so by both shrinking and expanding weights. Another method designed for non- linear shrinking is Yang et al. (2019), who propose the ratio of the LI and L2 norms a sparsity regulariser. Even though this method is not based on BatchNorm, but is targeted at filter weights, we include it in our comparison as has a similar motivation to our work.
In terms of pruning setup the most popular method is one-shot pruning et al. 2016 Zhuang et al., 2020 Yang et al., 2019) where all desired neurons are removed at once and the remaining network fine-tuned. Iterative approaches (Han et al., 2015) periodically prune and fine-tune until target ratio is met. Recent works aim to climinate the need of fine-tuning (Chen et al. 202T] altogether or Wang et al. 2020).
prune models after intialisation in data-free manner (Lee al. 2018
3 SIGMOID BATCHNORM
We introduce a variation of BatchNorm, which uses a single learnable parameter per channel and of- fers bounded importance score for filters. In its original formulation, BatchNorm (loffe & Szegedy 2013) first normalises each input channel r using batch statistics mean (B and standard deviation BB then applies an affine transformation using learnable parameters 7 and 3.
T.
PB
BN 1.3) =7
+ 8
Vo +
While BatchNorm has become ubiquitous in Deep Learning, the reasons behind its effectiveness are
not fully understood (Santurkar et al. 2018). The proliferation of BatchNorm variations (Ba et al.)
2016 Wu & He 2018: rather than the affine transformation following it. We empirically show that the transformation can be replaced to be linear without loss of performance.
al. 2016 suggests its benefits arise from normalising activations
2

---

Under review as a conference paper at ICLR 2023
8
VGG-I 19
ResNet-50
4
Density
0
-1
00.5
0.5
1.5
-1
0.5
0
0.5
1.5
Figure 1: Distribution of parameters across all BatchNorm layers pre-trained models available in PyTorch Passke et al. 2017). The scale stays in the interval [0, 1]. the offset / remains close to O.
We look at the empirical distributions of BatchNorm parameters to understand the effects of the affine transformation. While the scale " and offset 3 unbounded, in practice, these will be suppressed due to weight decay. We can observe this in pre-trained models; Figure [ shows the parameter distributions for all BatchNorm layers of VGG-19 (Simonyan & Zisserman (014)) and ResNet-50 (He et al. 2016) model. The BatchNorm parameters in both networks are well aligned as almost all y E [0, while a large proportion of are concentrated around 0.
Because BatchNorm operates on a channel level, it is of particular interest to structured pruning, it offers natural place look for measures that quantify filter importance. In line with our observation about the distribution of 8, previous approaches (Liu et al. 2017 Zhuang et al.) 2020) define filter importance as the magnitude of y and ignore the offset. While this has proven work in practice, one could argue it would be desirable to construct all the information available. Additionally, the utility such a measure could be greatly improved if it would be bounded offering better understanding by knowing the minimum and maximum importance filter could have.
importance score that incorporates
Given the aforementioned considerations, propose simple alteration to BatchNorm, termed a BN, which keeps the normalisation scheme, but changes the channel transformation to use a bounded scaling parameter. We remove the offset 3 and bound the scale parameter by applying the sigmoid function to y before multiplication:
a BN (77)
T uB
where a77) 1
Voz +
In Table 1 the performance of vanilla BatchNorm (BN) compared with variation oBN. Our variation performs comparable for VGG-16 and ResNet-56 on both CIFAR datasets. For completeness we also add the performance of BN and oBN with and without the bias term B. We can observe that the presence or absence of 3 does not have profound impact on accuracy. With aBN, we now have a normalisation layer that uses single, bounded parameter quantify the im- portance of a filter. We will use this property to decide which filters to suppress and ultimately prune.
CIFAR10 BN-3 BN aBN aBN+3
CIFARI00 BN-3 BN oBN
aBN+B
VGG-16 93.58
93.57
93.48
93.35
VGG-16
72.81
72.99
72.75
73.12
Re 92.87
93.34
93.11
93.33
Re
70.70
70.94 71.27
71.12
Table 1: No significant difference in classification accuracy can be seen between any BatchNorm variation, including toggles of the bias term for vanilla BN and oBN, Results are averaged over three runs and have standard deviation of z0... Normalisation layers are trained without regularisation.

---

Under review as a conference paper at ICLR 2023
(Rg)))
Neuron Importances
0.25
b=0
Counts
10a
LI
102
b=
0.2
b=2
10
0
pla
Milli A
0.15
b=3
0.2
0.4
0.6
0.8
1
1.2
hl
0.1
Counts
10
RNI
0.05
10
10a
0.0
0
1"
1
0.5
0.2
0.4
0.6
0.8
a (7)
(
Figure 2: Left: The gradient of the RNI loss with respect to to for different hyper-parameter choices. b shifts the range of neuron importances which the regulariser will affect. Right: Histograms of resulting neuron importances after using L1 or RNI regularisation.
RECEDING NEURON IMPORTANCE
Structured pruning aims to compress a network by identifying and removing unimportant neurons. We define 8 to be the parameters of a neural network and refer to the subset associated with Batch- Norm as (BN, Let ((0:) be a measure importance for neuron i, which can be computed from any model parameters associated with that neuron, such its filters in convolutional layers (Li et al., 2016 Yang et al., 2019). In BatchNorm based pruning,
derived only from 0BN, and is com- Invl (Liu et al. 2017 Zhuang et al. 2020). In this paper use the measure induced by our a BN layer such that 7(0,9)) (%:). As in previous work, we want to solve the following risk minimisation problem:
monly defined I(0P5)
min
L (0,X) + XR(O) + AsRR( # N
(3)
where X= (Ii, yi Wi is a labelled dataset with N training samples, and X, Às are scalar weight- ings for the regularisation terms R and Rs. R0)) is regularisation against over-fitting, such as weight decay, and is usually applied over all network parameters, including normalisation layers.
Rs is targeting only 0BN such that exhibits the following property during pruning:
C (0) x) C (O)(0)) x), if I(0.) I(0,)
Such property creates an ordering which ensures that removing a neuron 0; with lower importance will have less impact on model performance, than removing a more important 0j. We will assess the quality of such an ordering by looking at the model performance after pruning and fine-tuning. However, an ordering only partially evaluated when removing a single subset of neurons since the order of the remaining neurons is not taken into account. Therefore, for comprehensive estimate of the global ordering quality, is necessary prune model at multiple thresholds.
Generally we want Rs to induce sparsity over BN such that outputs of unimportant neurons are entirely suppressed during training. Pruning such network would result in less damage and con- sequently quicker recovery during fine-tuning. The typical choice of R, is the LI norm, which shrinks all scaling parameters with a constant gradient. Such a regularisation strategy simultane- ously subdues both important and unimportant neurons with equal weighting. From an optimisation perspective, LI will create a bias such that at convergence, any equilibrium would require the classi- fication loss to negate the constant gradient of the regulariser. Such a bias might lead to sub-optimal solutions, and can be mitigated by having the regularisation reduce its strength once a suitable, sparse distribution of importances has been found. We show that better regularisation term can be
4

---

Under review as a conference paper at ICLR 2023
designed by focusing on suppressing unimportant neurons, whilst leaving important ones untouched. Without an idea of bounds for I(@#N) it is difficult to decide on a threshold between important and unimportant neurons. Using the importance measure from our oBN layers we propose regulari- sation loss whose strength decays exponentially as T(0BN) reaches its maximum. We introduce the Receding Neuron Importance regularisation (RNI) on batch normalisation parameters 9
R2(1,b) = (7 + b)- loggo((( + b))), with (R = -oog(o ( b))
(5)
and V (R,)) -oo))==-o((((7>> + ()) + 6) a(r + +))
The hyper-parann ppper-parameter controls the range defining which neuron importances to target. Its effects can be visualised in Figure 2 (Left): b shifts the peak of the gradient after which an exponential decay occurs. As b increases only neurons with lesser importance are affected by the regularisation. With the weights of such neurons receding towards zero, the final distribution of importances will have a polarised bimodal shape- only important and zero-weight neurons will be left (Right).
EXPERIMENTS
In this section, we will evaluate our sparsity regularisation under one-shot pruning setting on the CIFAR10/100 and ImageNet datasets using VGG and ResNet models. In most structured pruning literature evaluation is performed only under relatively benign pruning ratios, which mostly preserve the performance of the original model and can make it difficult to compare related methods. In our evaluations we observe that the difference between approaches becomes noticeable when pruning ratios are increased. Under more inauspicious circumstances we show that our method outperforms existing state-of-the, ooe-the-rrt approaches and can, in some instances, significantly mitigate the detrimental effects of severe pruning. We also compare the pruned VGG/ResNet networks with similarly sized MobileNetV2 models and show that pruning outperforms these compact architectures.
5.1 EXPERIMENTAL SETUP
Pruning. Under a one-shot pruning scenario, a model is trained, then pruned and subsequently fine- tuned to recuperate performance. An appropriate regularisation strength is selected such that it does not impede training while also producing enough sparsity prune the desired amount of neurons. For practical reasons we train a model only once and then evaluate it under several pruning ratios. We prune networks in a global fashion, removing filters based on their importance score and without regard to the resulting distribution over the layers. To avoid the pruning of all neurons within a layer, a minimum of three filters will be preserved. While this avoids the extreme case of layer collapse, if a method disproportionately prunes the neurons of single layer it could still irreparably damage the network. From a technical perspective, pruning the filters of a layer changes its parameter count but also that of the following layer, since the expected input dimensionality has changed. While this has no further implications for VGG style architectures, it does pose some challenges for networks with residual connections. Similar related works, in order to avoid mismatched dimensions in ResNets, we do not prune skip connections or the last layer in residual connections.
Related Methods. We compare our approach with LI Slimming (Liu et al. (017). Polarization Regularisation (Zhuang et al. 2020), Deep Hoyer (Yang et al. 2019) and Uniform Channel Scaling (UCS) (Zhuang al.[ 2020). The same experimental settings are used for all methods during train- ing, pruning and fine-tuning. An exception is the UCS baseline, which does not use any sparsity regularisation and prunes in a local manner by removing the same percentage of filters across all layers. All networks of related approaches are built using vanilla BatchNorm, and with the same ini- tialisation scheme as Liu al. (2017). Models trained with o BatchNorm layers are initialised with y drawn from standard Normal distribution and will not use weight decay during training. The learning rate of o BatchNorm is set to be higher by factor of 10 than that the rest of the network found this to help with polarisation.
Hapeer-arameetrs. The same settings as Liu et al. (017) are used for the CIFAR1O and CIFAR100 datasets. VGG-16 and ResNet-56 models are both trained and fine-tuned for 160 epochs, with the same learning rate schedule and with a batch size of 64. Models are optimised using SGD with momentum, weight decay of 10-1 and initial learning rate of 10-1 which reduces stepwise by
5

---

Under review as a conference paper at ICLR 2023
factor of 10 at epochs 80 and 120. The respective sparsity regularisations applied all layers of the models. Datasets are augmented using only random crops and horizontal flips. On ImageNet, we evaluate methods prunable layers. Models are trained for 90 epochs and fine-tuned for 30, using a batch size of 512. The initial learning rate is 10-1 for training and 10-2 for fine-tuning, and is scheduled to decrease after 1/3 and 2/3 of the respective training time. For our RNI experiments the following hyper-parameters are used: Äs 10 b, 3 for VGG-16, As 10-4 b,b 0 for ResNet-56, and Às = 10 10-5,01000 b = 3 for ResNet-50.
the ResNet-50 bottleneck architecture and regularise only its
CIFARIO
CIFARI00
94%
70%
92%
60%
90%
Accuracy
88%
Slimming
50%
86%
Polarization Doeplloyer
40%
84%
UCS
82%
RNI
30%
50%
60%
70%
80%
90%
50%
60%
70%
80%
90%
Filters Pruned
Filters Pruned
Figure 3: Accuracy of fine-tuned VGG-16 models at varying pruning thresholds. Performance differences between methods become more pronounced at higher ratios. RNI degrades gracefully compared to related methods.
5.2
RESULTS
Apart from ImageNet, all methods are evaluated on three random seeds and their mean performance reported. For clarity, we will omit the standard deviation as we found the results be very similar across runs. Two scenarios are examined in more detail as they sit on opposite ends of the difficulty spectrum: pruning 50% and 90% of filters. The 50% mark is the default setting in the structured pruning literature and we expect all methods to recuperate most of their baseline performance. This pruning level also ensures a fair comparison between methods as it acts as an anchor for hyper- parameter choices. At 90% of filters pruned,
significant degradation in performance and far larger differentiation between methods and models.
In Figure 3 we have an overview of how VGG networks degrade as pruning ratios increase. On both datasets our method deteriorates gracefully compared to the abrupt declines of related approaches. While on 1I all methods have easier time maintaining their performance, degradation becomes immediately visible due to the increased difficulty of the dataset. Surprisingly, the locally pruning UCS baseline performs worst CIFAR10 10 but outperforms all methods, apart from ours, on CIFAR100. This shows that correctly identifying unimportant neurons in global manner becomes harder as the difficulty of the dataset increases. In our ablation studies we will show that this deterioration can be directly linked to excessive pruning of carly layers in VGGs.
CIFAR100
This task is relatively simple and expect excess model capacity to be prunable without much performance loss, as seen in T The only significant degradation happens for VGGs at the 90% mark, where UCS and DeepHoyer lose over 10% accuracy compared the baseline. ResNets are more robust and are lose less than 4% accuracy while reducing roughly 80% of FLOPs.
CIFARIO.
CIAFR100. Due to the increased difficulty, in Table 3 we see a larger deterioration for models trained on the CIFAR100 dataset. Here networks require more capacity, thus the number of nec- essary filters should be higher, and the identification of superfluous ones more difficult. With 90% of filters removed, VGG networks experience severe degradation as methods fail identify which neurons to keep. In this scenario our approach considerably outperforms state-of th-the- methods by 30% classification accuracy. The only exception is the UCS baseline, which by design maintains the stability of a model, as it will never prune excessive amounts of filters in any given layer. ResNets again prove be more robust to pruning and show less variation in performance between methods.

---

Under review as a conference paper at ICLR 2023
VGG-16
Baseline 50%
90%
Resnet-56
Baseline 50%
90%
Acc.
Acc.
FLOPs
Acc.
FLOPs
Acc.
Acc.
FLOPs
Acc.
FLOPs
Slimming
93.71
93.96
60.25
89.15
11.40
Slimming
93.68
93.45
54.07 89.90 19.55
Polarization
93.91
94.27
64.15
88.37
14.69
Polarization
93.34
93.50
52.69
89.43
18.89
DeepHoyer
93.66
93.96
61.32
80.82
8.09
DeepHoyer
93.44
93.51
51.07
89.94
20.84
UCS
93.84
93.02
25.19
83.69
1.13
UCS
93.78
93.33
50.39
88.78
14.79
RNI
93.53
93.64
48.24
90.96
8.40
RNI
93.53
93.01
56.13
90.26
21.54
Table 2: Results CIFARIO for different pruning ratios, showing absolute accuracy (%) and maining FLOPs (%) relative to the unpruned baseline. Due to the simplicity of the task, models can be significantly pruned without deterioration.
VGG-16
Baseline 50%
90%
Resnet-56
Baseline 50%
90%
Acc.
Acc.
FLOPs
Acc.
FLOPs
Acc.
Acc.
FLOPs
Acc.
FLOPs
Slimming
73.54
71.61
61.12
29.66
3.76
Slimming
71.94
70.06
56.97
63.92
15.29
Polarization
73.47
71.23
63.82
25.62
2.59
Polarization
71.80
70.08
40.78
62.42
15.47
DeepHoyer
73.19
71.66
67.23
25.42
4.98
DecpHoyer
71.88
71.14
49.22
63.49
19.51
UCS
73.80
70.84
25.19
54.39
1.13
UCS
71.90
70.72
50.40
63.65
14.79
RNI
72.28
72.46
36.72
57.92
6.31
RNI
71.30
70.65
48.24
63.95
16.34
Table 3: Results on CIFAR 100 for low and high ratios of filters pruned. Shown are the absolute classification accuracy (%) and remaining FL.OPs (%) relative the unpruned baseline. Due the increased complexity of the classification task, it becomes more difficult to remove excess capacity. The VGG architecture breaks down significantly after sever pruning.
ImageNet. On ImageNet in Table the results are similar, however we observe bigger losses due to the increased dataset diffieulty. Under these circumstances method still outperforms prior state-of-the-art, but is worse than the UCS benchmark. This again indicates that for more difficult datasets, global methods over-prune the wrong layers, something the UCS baseline will not do by design. Nevertheless, ResNets show remarkable robustness to pruming, which we attribute to the fact that their skip connections act fail-safe and limit the effect of over over-pruned layers.
Compact Networks. At higher pruning ratios the parameter count of models is significantly reduced, raising the question of whether they would outperform networks specifically designed to be lightweight. Compact networks such MobileNetV2 Sandler et al. (2018) have an ad- justable width multiplier allowing the instantia- tion of models with arbitrary number of parame- ters. We thus conduct additional baseline exper- iments comparing the RNI pruned VGG/ResNet models from Tables 2 and 3 non-sparse Mo- bileNets of the same size. For each of the RNI pruned models we find width multiplier such that the corresponding MobileNet has a similar number of parameters, and then optimise it using the same training setup, but without sparsity loss. From the results in Table 5 we can see that our pruned models can outperform similar sized MobileNets in almost every setting, proving the practical utility of higher pruning ratios.
ResNet-50
Baseline
50%
90%
Acc.
Acc.
FLOPs
Acc.
FLOPs
Slimming
75.74
72.56
41.12
51.74
16.50
Polarization
75.12
73.11
46.12
49.43
17.00
DeepHloyer
75.36
68.72
57.56
50.98
24.14
UCS
76.14
73.28
44.78
58.13
16.94
RNI
74.75
72.31
48.13
54.08
17.66
Table 4: ImageNet results showing absolute curacy (%) and remaining FLOPs (%) relative to the unpruned baseline.
Summary. Because of the evaluation over datasets with varying degrees of complexity, we can make several observations about how models degrade under a spectrum of pruning ratios. Firstly, at 50% of filters pruned differences between approaches are barely visible since all methods are able to regain most of their pre- pruning performance. Differences become obvious only at higher ratios, where our method clearly surpasses prior state-of-the-un Unsurprisingly, residual networks are more robust to pruning than VGG architectures, since their skip connections can propagate the signal even essential residual connections have been mistakenly over-pruned. This however comes at the cost

---

Under review as a conference paper at ICLR 2023
Pruning Ratio
Network
Dataset
50%
#P.
90%
#P.
VGG-16 [RNI]
CIFARIO
93.64
3,40M
90.96
167K
MobileNetV2
CIFARIO
93.38
3.43M
89.67
177K
VGG-16 [RNI]
CIFAR100
72:46
3.23M
57.92
158K
MobileNetV2
CIFAR100
75.16
3.12M
52.72
163K
ResNet-56 [RNI]
CIFARIO
93.01
401K
90.26
112K
MobileNetV2
CIFARIO
91.21
408K
88.55
124K
ResNet-56 [RNI]
CIFARI00
70.65
429K
63.95
128K
MobileNetV2
CIFARTOO
71.05
452K
52.72
163K
Table 5: Comparison between non-sparse MobileNetV2 models and RNI pruned VGG/ResNets on CIFAR datasets. For each setting the width multiplier has been chosen such that the MobileNet will have the same number of parameters as the pruned networks.
of pruning efficiency, as seen by the diminished reduction of FLOPs compared to VGG models. To surprise find that while UCS is not suited for simple classification tasks, it outperforms prior global pruning methods on more complex datasets. Because of its design, the UCS baseline avoids any risk of excessively pruning individual layers, making models less prone to sudden performance deterioration. This leads to the conjecture that the other related methods over-prune certain layers, which leads severe degradation VGG models. In our ablative analysis we show that this is indeed the case, and make the realisation that our method is the only one which does not significantly prune early layers a network. This offers an explanation why RNI noticeably outperforms similar approaches at high pruning ratios. Finally, from additional evaluations on compact architectures we see that pruned VGG/ResNet models outperform similarly sized MobileNets, which further proves the utility of pruning means to obtaining compressed networks.
As=10-3
As=10-4
Aec.
15
50%
Density
40%
10
30%
2
20%
As=10-3 -3
5
10%
As=10-1
0%
0
0.5
0.5
0
1
2
3
(())
(())
Figure 4: Effects of hyper-parameteen Às and for VGG-16 on CIFAR100. Left, Middle: Filter importance distributions after regularised training. Increasing Às or lowering b will produce sparser models. Right: One-shot performance a 90% pruning ratio. Both over- and under-regularissng can lead to performance degradation, however the search space for b is limited.
5.3 ABLATIVE ANALYSIS
In this section will perform two ablation studies better understand the behaviour of the Re- ceding Neuron Importance regulariser. We first analyse the effects of different choices neuron sparsity and one-shot pruning performance. Following that investigate how much each layer of VGG network will pruned under comparable approaches.
Sparsity. In addition to the regularisation strength Às, approach introduces the shifting param- eter We show that these two parameters are complementary in controlling the gradient of the regularisation loss and effective producing sparsity. While Äg scales the amplitude of the gradi-
8

---

Under review as a conference paper at ICLR 2023
ent, b will set the range which neuron importances start to recede. Figure (Left, Middle) shows how Às and choices create different distributions of neuron importances for a VGG model on CI- FARI00. Sparsity can be measured as the density around the zero importance limit, and increases with higher Àş or lower b. The interplay between these two parameters creates flexibility in the level of sparsity produced and the distribution of the remaining neuron importances. This ability create a high variety of distributions comes at the cost of relative hyper-parae parameter sensitivity with respect pruning accuracy, however this limitation only becomes visible at high pruning thresholds. Fig- ure 4 (Right) shows the one-shot pruning accuracy at 90% pruning ratio, of models trained with selected eyper-parameters We can observe that both under- and over-regularised models will suffer from severe pruning. Unsurprisingly, inducing strong degree of sparsity during training does not translate into a highly prunable model.
CIFARIO
CIFAR
Slimming DeepHoyer
100%
75%
50%
UCS
RNI
25%
0%
0
2
3
5
6
101112
0 3
4
6 7
9 101112
Figure 5: Percentage of filters removed from each layer when pruning a VGG-16 model at a 90% ratio. The deeper half of layers contain over 70% of the total amount of filters in the network. RNI stands out by not over-pruning layers in the first half.
Pruning Locations. Figure 5 shows what percentage of filters are removed from each layer during pruning. UCS prunes in local manner, removing the same fraction of filters from each layer, as can be seen from the unchanging colour-coding. It is worth reminding that in VGG-style architecture the number of filters periodically doubles as the network becomes deeper. This has the consequence that global pruning methods will achieve their target in large part through the amount of filters pruned from the deeper layers. These are less sensitive to pruning, as there will still be a high absolute number of filters left. Consequently, removing large percentage of neurons from the first layers will not add much to the global target, but will significantly limit their expressive power. Thus, is reasonable assume that filters in the early layers of network have higher importance. since the capacity of these layers is already limited.
we can see that deeper layers are indeed pruned more heavily than ones closer to the input. There are however some exceptions: on the CIFARIO dataset, the only method pruning significant amount of filters from the first layer DeepHoyer which also achieves the lowest accuracy based on the results from Table 2 On the CIFAR100 dataset Network Slimming and Neuron Polarization start excessively pruning from the 5th layer onward. RNI is the only approach that starts to heavily prune only in the latter half of the network. Our method, along with UCS, are the only ones that do not suffer catastrophic damage after pruning 90% of the total number of filters.
From Figure 5
CONCLUSION
In this work we introduce RNI, a novel sparsity regularisation for structured pruning, under which only the weights of unimportant neurons is designed to recede. To this end, we propose Batchhorm, a BatchNorm variation with bounded scaling factors, enabling the construction of such targeted regularisation functions without sacrificing performance. Our method consistently out- performs state-of-the-art methods for VGG/ResNet architectures on CIFAR and ImageNet datasets. We show that accurately comparing methods requires evaluation differences become more pronounced at higher levels. For VGG architectures our method signif- icantly reduces the performance degradation from severe pruning compared to prior art. In future work we would like to explore more applications of the RNI regulariser as well as a way to schedule its hyper-parameters to obtain pre-determined sparsity ratio.
spectrum of pruning ratios, as

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
This paper is entirely reproducible. All experimental details including setup and hyper-parameters
are described depth in Sections 5.1 and 5.2 All reported results have been run on publicly
available datasets. Furthermore, runnable Py Torch repository has been linked and attached as supplementary material.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 06450, 2016.
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: one-shot neural network training and pruning framework, Advances in Neural Information Processing Systems, 34, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: arXiv:/803.03663 1803.0363, 2.18.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why
missing the mark? arXiv preprint arXiv:2009.08576,
2020.
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. arXiv preprint wiXiv1106.006220 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Fie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-255 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint TrXiv:1808 06866, 2018.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference on computer vision, pp. 3389-1397, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340. 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087100 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference Computer Vision, pp. 2736-27444 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 18Xiv:1810.055 2018.
10

---

Under review as a conference paper at ICLR 2023
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprini crXiv:1611.06440 2016a.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:6611.06440 2016b.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003. 02280,9900,, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chii Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520 4520, 2018.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? arXiv preprint arXiv: 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Xavier Suau, Nicholas Apostoloff, et al. Filter distillation for network compression. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3129-3138. IEEE, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- 2016.
gredient for fast stylization. arXiv preprint
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376. 2020.
Dong Wang, Lei Zhou, Xueni Zhang, Xiao Bai, and Jun Zhou. Exploring linear relationship in feature map subspace for convnets compression. arXiv preprint arXiv: 1803.05729, 2018
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differen- tiable scale-invariant sparsity measures. arXiv preprint arXiv: 190.0:9908..0979, 2019.
Ruichi Yu, Ang Li, Chun Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching-
Yung Lin, and I Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194 9203, 2018
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33, 2020.
11


Reviews:
[{"review_id": "pYs6kfTEa2s", "reviewer": "Reviewer_SWo9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a channel-pruning method by regularizing the BatchNorm scaling parameters, and proposed a BatchNorm variation suitable for pruning. The authors point out that the previous BatchNorm-based pruning methods shrink all parameters with an equal gradient irrespective of their importance. Thus, the authors propose to remove the offset $\\beta$ in BatchNorm and bound the scale parameter $\\gamma$ by applying the sigmoid function before multiplication, and the Receding Neuron Importance regularization for pruning. However, the design is lacking in theoretical support and the authors didn’t prove the proposed method achieved better accuracy than previous methods.", "summary_of_the_review": "The paper is well-written, however, the proposed method seems incremental, the design is lacking in theoretical support and the experimental results are weak.", "strength_and_weaknesses": "Channel pruning is an important task in neural network compression and the effort towards better channel pruning methods is appreciable.\n\nExperimental results are weak. The results on the ImageNet dataset are much worse than the previous state-of-the-art results.\n\n[ref1] Soft filter pruning for accelerating deep convolutional neural networks\n\n[ref2] Pruning filter via geometric median for deep convolutional neural networks acceleration.\n\n[ref3] Metapruning: Meta-learning for automatic neural network channel pruning\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gr5bBetYU_", "reviewer": "Reviewer_LW7m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way to measure the 'importance' of a neuron, by re-parameterizing the scaling parameters of the batchnorm layers. The proposed method is to use a sigmoid on the scaling parameter, to ensure that it lies in [0,1], and the value of the sigmoid reflects the importance. Using this value (value closer to 1 means important), the 'unimportant' neurons are pruned in a one-shot manner. The paper also proposes a new regularizer for this scaling parameter, to control the amount of neurons to be pruned. The results on simpler image classification tasks like CIFAR-10, -100 suggest that this method indeed outperforms other methods, particularly at higher pruning levels. However, on more complex task like Imagenet, it seems to perform worse than the UCS method. Further, the algorithm is only evaluated on image tasks (CIFAR and ImageNet), and two architectures (ResNet and VGG).", "summary_of_the_review": "I think the proposed method to measure the 'importance' of a neuron is more principled than previous methods that use magnitude, but I think the method can be more refined to deal with the problem of over-pruning the wrong layers for complex tasks, and by adding more experiments to show how the method performs for different architectures and tasks.", "strength_and_weaknesses": "Strengths:\n1. The proposed measure and regularizer seem to be a more principled magnitude based pruning method than existing methods that directly work on the magnitude and apply an L1 regularizer. \n\n2. The results on CIFAR-10, 100 suggest that indeed this method has potential, since it outperforms existing methods on higher pruning ratios.\n\nConcerns:\n1. The proposed method is only tested on 2 architectures: ResNet and VGG, and on image tasks CIFAR and ImageNet. \n           \n- Can the method be applied to Transformer architecture on language tasks? \n- The paper applies the method only for one-shot pruning. Can this be applied in multiple iterations, pruning a certain fraction of neurons in each round?\n- Can this method be used to prune pre-trained networks? For example, take a pretrained network and fine-tune if for a few epochs on a downstream task, and then apply this pruning method.\n\n2. Although the method performs well on simpler tasks like CIFAR, it performs worse than UCS on difficult tasks like ImageNet. The paper says that this is because \"This again indicates that for more difficult datasets, global methods over-prune the wrong layers\". I think this is a major limitation. However, I think this might be remedied, for example, by a per-layer re-normalization of the importance scores, so that a similar ratio of neurons are pruned in every layer.\n\n3. (Minor) The paper promotes the proposed method by claiming that it measures the 'importance' of neurons by bounding the scaling parameter of batchnorm in the range [0, 1]. However, I am not fully convinced by this. Consider the case when a neuron has a low scaling parameter, but in its outgoing weights to the next layer, a few (say 10%) of the weights are large. That means that even though the output magnitude of that neuron is low, some neurons in the next layer give high weightage to its output. In that case, this neuron can still be important. In short, what I mean to say is that just the magnitude of a neuron's output might not necessarily capture its 'importance'. While this method still seems more principled than previous methods at using the scaling parameter's magnitude for pruning, I do not agree that it captures the importance of the neuron. I would suggest clarifying this in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WW4ypMIPNE", "reviewer": "Reviewer_uAwb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple variation of batch normalization with a bounded scaling parameter to estimate the importance of each channel. To only suppress unimportant units while preserving important units, the authors design a regularization loss on these bounded scaling factors, with additional hyperparameters. The proposed approach is validated on CIFAR and ImageNet with VGG, ResNet, and MobileNet models.", "summary_of_the_review": "In general, I believe this work did a good job in terms of designing interesting sparsity-inducing regularization for structured pruning. Although the in-depth comparison with Polarization Regularizer is necessary and some results are questionable, the experiments and analyses are quite good and thorough.", "strength_and_weaknesses": "Strength\n- The idea that shrinks only the outputs of unimportant neurons while maintaining the others is well-motivated.\n- The sigmoid batchnorm is sufficiently simple for people to build on, although the proposed regularization loss involves additional hyperparameters.\n- I believe the experimental validation is quite strong and extensive; various architectures with different capacities were used, and the baselines are recently proposed methods and competitive.\n- The authors provide detailed analyses on the hyperparameters and where to be pruned, which help understanding the network behavior. \n- The paper is generally quite thorough and generally feels complete: I believe it is generally ready for publication if it is decided by the reviewers that significance is sufficient to warrant publication.\n\nWeakness\n- As the authors described, this paper and Polarization Regularizer (Zhuang et al., 2020) share the same motivation and take a similar approach in terms of optimizing batchnorm scaling factors in a desired way. Although the authors briefly mentioned the difference in the related work section (i.e., While the method effectively increases the margin between important and unimportant neurons, it does so by both shrinking and expanding weights.), I am not sure why the proposed method should be better than Polarization Regularizer. Could the authors provide the merits of your work over Polarization Regularizer in more detail? \n- I am not sure whether the proposed method is effective for ResNets and is generally applicable to resblock-equipped networks. For instance, in the right part of Table 2, RNI slightly outperforms the baselines but with more FLOPs at 90% pruning ratio. In the right part of Table 3, at 90% pruning ratio, the accuracies are very similar. Furthermore, in Table 4, the superiority of RNI is more questionable: Slimming and Polarization yield better results with less FLOPs at 50% ratio.\n- I think the comparison in Table 5 should be improved; matching the number of parameters may be too naive. I would recommend the authors to add FLOPs to Table 5 and perform additional experiments using the network with similar FLOPs or latency. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject