Learned Threshold Pruning
Authors: Kambiz Azarian
Yash Sanjay Bhalgat
Jinwon Lee
Tijmen Blankevoort
Venue:  (2021)
Keywords: Efficiency
Model Compression
Unstructured Pruning
Differentiable Pruning

Abstract: This paper presents a novel differentiable method for unstructured weight pruning of deep neural networks. Our learned-threshold pruning (LTP) method learns per-layer thresholds via gradient descent, unlike conventional methods where they are set as input. Making thresholds trainable also makes LTP computationally efficient, hence scalable to deeper networks. For example, it takes $30$ epochs for LTP to prune ResNet50 on ImageNet by a factor of $9.1$. This is in contrast to other methods that search for per-layer thresholds via a computationally intensive iterative pruning and fine-tuning process. Additionally, with a novel differentiable $L_0$ regularization, LTP is able to operate effectively on architectures with batch-normalization. This is important since $L_1$ and $L_2$ penalties lose their regularizing effect in networks with batch-normalization. Finally, LTP generates a trail of progressively sparser networks from which the desired pruned network can be picked based on sparsity and performance requirements. These features allow LTP to achieve competitive compression rates on ImageNet networks such as AlexNet ($26.4\times$ compression with $79.1\%$ Top-5 accuracy) and ResNet50 ($9.1\times$ compression with $92.0\%$ Top-5 accuracy). We also show that LTP effectively prunes modern \textit{compact} architectures, such as EfficientNet, MobileNetV2 and MixNet.

Under review as a conference paper at ICLR 2021
LEARNED THRESHOLD PRUNING
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper presents novel differentiable method for unstructured weight pruning of deep neural networks. Our learned-threshon pruning (LTP) method learns per- layer thresholds via gradient descent, unlike conventional methods where they are set as input. Making thresholds trainable also makes LTP computationally efficient, hence scalable deeper networks. For example, it takes 30 epochs for LTP prune ResNet50 on ImageNet by factor of 9.1. This is in contrast to other methods that search for per-layer thresholds via computationally intensive iterative pruning and fine-tuning process. Additionally, with novel differentiable Lo regularization, LTP is able to operate effectively on architectures with muihh-normaliztionn This is important since L1 and L2 penalties lose their regularizing effect in networks with saacch-nrmalizztion. Finally, LTP generates a trail of progressively sparser networks from which the desired pruned network and performance requirements. These features allow L.TP to achieve competitive compression rates ImageNet networks such Alex (26.4x compression with 79.1% Top-5 accuracy) and ResNet50 (9.1 x compression with 92.0% Top-5 accuracy). We also show that LTP effectively prunes modern compact architectures, such as EfficientNet, MobileNetV2 and MixNet.
be picked based sparsity
INTRODUCTION
Deep neural networks (DNNs) have provided state-of-the-art solutions for several challenging tasks in many domains such as computer vision, natural language understanding, and speech processing. With the increasing demand for deploying DNNs on cesource-oonstrained edge devices, has become even more critical to reduce the memory footprint of neural networks and also to achieve power-efficient inference on these devices. Many methods in model compression Hassibi et al. (: LeCun et al. (1989); Han et al.
(2015b): [hang et al. (2018). 1), model quantization Jacob et al. (2018) 8); Lin et al.
(0166) Zhou et al.
(2017) Faraone et al. (2018) and neural architecture search Sandler et al. (2018):
Tan & Le (2019)) Cai al. (2018): Wu et al. (2019) have been introduced with these goals mind.
Neural network compression mainly falls into two categories: structured and unstructured pruning. Structured pruning methods, c.g., He et al. (2017): Li al. (2017); Zhang et al. (2016); : He et al. (2018), change the network's 's architecture by removing input channels from convolutional layers by applying tensor decomposition to the layer weight matrices whereas unstructured pruning methods such as Han et al] (2035)); Frankle & Carbin (011); Zhang et al.) (2018) rely on removing individual weights from the neural network. Although unstructured pruning methods achieve much higher weight sparsity ratio than structured pruning, unstructured is thought to be less hardware friendly because the irregular sparsity is often difficult to exploit for efficient computation Anwar et al. (017)). However, recent advances in AI accelerator design Ignatov et al. (2018) have targeted support for highly efficient sparse matrix multiply-ame and and--adddueee accumulate operations. Because of this, is getting increasingly important to develop state-of-ther oftt--o--he-atttte algorithms for unstructured pruning.
Most unstructured weight pruning methods are based on the assumption that smaller weights do not contribute as much to the model's performance. These pruning methods iteratively prune the weights that are smaller than a certain threshold and retrain the network to regain the performance lost during pruning. A key challenge in unstructured pruning is to find optimal setting for these pruming thresholds. Merely setting the same threshold for all layers may not be appropriate because the distribution and ranges of the weights in each layer be very different. Also, different layers may have varying sensitivities to pruning, depending on their position in the network (initial layers versus final layers) their type (depth-wise separable versus standard convolutional layers). The

---

Under review as a conference paper at ICLR 2021
best setting of thresholds should consider these layer-wise characteristics. Many methods Zhang et al. (2018): Ye et al. (2019); Manessi [ et al. (2018) propose a way to search these layer-wise wise thresholds but become quite computationally expensive for networks with large number of layers, such as ResNet50 or Efficcientee.
In this paper, we propose Learned Threshold Pruning (LTP) to address these challenges. Our proposed method uses separate pruning thresholds for every layer. We make the layer-wise thresholds trainable, allowing the training procedure to find optimal thresholds alongside the layer weights during finetuning. An added benefit of making these thresholds trainable is that it makes LTP fast, and the method converges quickly compared to other iterative methods such as Zhang et al. (2018); Ye et al.] (2019). LTP also achieves high compression on newer networks Tan & Le (2019)); Sandler eet al. (2018): Tan & L.e (2019b) with squeeve-excite Hu et al. (0088) and depth- wise convolutional layers Chollet (2017).
Our key contributions in this work are the following:
We propose gradient-base based algorithm for unstructured pruning, that introduces learnable threshold parameter for every layer. This threshold is trained jointly with the layer weights. We use soft-pruning and soft Lo regularization to make this process end-to-end trainable.
We show that making layer-wise thresholds trainable makes LTP computationally very efficient compared to other methods that search for per-layer thresholds via an iterative pruning and finetuning process, e.g., LTP pruned ResNet50 to 9.1 1x in just 18 epochs with 12 additional epochs of fine-tuning, and MixNet-S to 2x in 17 epochs without need for further finetuning.
We demonstrate state-of-the compression ratios on newer architectures, i.c., 1.33 3x and 2 for MobileNetV2, EfficientNet-Bi and MixNet-S, respectively, which are already optimized for efficient inference, with less than 1% drop in Top-1 accuracy.
The proposed method provides a trace of checkpoints with varying pruning ratios and accuracies. Because of this, the user choose any desired checkpoint based on the sparsity and performance requirements for the desired application.
2 RELATED WORK
Several methods have been proposed for both structured and unstructured pruning of deep networks. Methods like He et al. (2017): Li et al. (2017) use layer-wise statistics and data to remove input channels from convolutional layers. Other methods apply tensor decompositions on neural network layers, Denton et a (2014); ladebbrrg et al. (2014); /Zhag et al. (2016) apply SVD to decompose weight matrices and Kim et al. 2015): Lebedev et al. (2014) apply tucker and cp-dccompositions to compress. An overview of these methods can be found in Kuzmin et al. (2019). These methods
are all applied after training a network and need fine-tuning afterwards. Other structured methods change the shape of neural network while training. Methods like Bayesian Compression Louizos let al. 2017), VIBnets Dai et al. (2018) and Ll/Ll--regulaiization Srinivas et al. (2017); Louizos et al. (2018) add trainable gates to each layer to prune while training.
In this paper we consider unstructured pruning, i.e. removing individual weights from a network. This type of pruning was already in use in 1989 in the optimal brain damage LeCun et al. (189) and optimal brain surgeon Hassibi et al. (1993) papers, which removed individual weights in neural networks by use of Hessian information. More recently, Han al. (2015a) used the method from
Han et al. 2015b) as part of their full model compression pipeline, removing weights with small magnitudes and fine-tuning afterwards. This type of method is frequently used for pruning, and has recently been picked up for finding DNN subnetworks that work just as well as their mother network in Frankle & Carbin (2119) Zhou et al. (2019). Another recent application of Han et al. (2015b) by Renda et al. (2020) where weight and learning-rai rewinding schemes are used to achieve competitive pruning performances. These methods, however, are very computationally extensive requiring many hundreds of epochs of re-training. Finally, papers such as Molchanov et al. (2117); Ullrich et al. (2017) apply a variational Bayesian framework on network pruning.
Other methods that similar to work are Zhang et al. (2018) and Ye et al. (211)). These papers apply the alternating method of Lagrange multipliers to pruming, which slowly coaxes a network into pruning weights with 1.2-regularization-ikk term. One problem of these methods is that they
2

---

Under review as a conference paper at ICLR 2021
time-intensive, another is that they need manual tweaking of compression rates for each layer. In our method, we get rid of these restrictions and achieve comparable compression results, at fraction the computational burden and without any need for setting per-layer pruning ratios manually. Kusupati et al. 2020) and Manessi et al. (2018) learn per-layer thresholds automatically using soft thresholding operator close variant of it. However they rely on I1 and/or T22 regularization, which as shown in section 3.2 is inefficient when used in networks with nutch-normalization Ioffe & Szegedy (2015). He al. (2018) use reinforcement learning set layer-wise prune ratios for structured pruning, whereas we learn the pruning thresholds in the fine-tuning process.
3
METHOD
LTP comprises two key ideas, soft-pruming and soft Lu regularization, detailed in sections 3.1 and respectively. The full LTP algorithm is then presented section 3.3
3.2
3.1
SOFT PRUNING
The main challenge in learning per-layer thresholds during training is that the pruning operation is not differentiable. More precisely, consider an N-layyer DNN where the weights for the I-th convolutional or fully-con colleceendeeeed layer are denoted by {wwr]. and let index the weights within the layer. In magnitude-based pruning Han et al. (2015b) the relation between layer I's uncompressed weights and pruned weights is given by:
Ukl
WHl X step(wÁr 7),,
(1)
where 77 denotes the layer's pruming threshold and step(.) denotes the Heaviside step function. We name this scheme hard-pruning. Since the step function is not differentiable, (T) cannot be used to learn thresholds through back- back-propagation. To get around this problem, during training LTP replaces (I) with soft-pruning
W kl T
TI
Ukl
WH sigm
where sigm(.) denotes the sigmoid function and T is temperature hyper-parameter As result of (2) being differentiable, pack-propagation can now be applied to learn both the weights and thresholds simuitaneously.
Defining soft-pruning as in 2 has another advantage. Note that if whi is much smaller than т (i.e., 7 wf, T), Wki's 's soft-pruned version is almost zero and it pruned away, whereas if it is much larger (i.e., Wig 7 >> T), WhI R Ukt. Weights falling within the transitional region of the sigmoid function (i.e., W 7ा T), however, may end up being pruned or kept depending their contribution to optimizing the loss function. If they are important, the weights are pushed above the threshold through minimization of the classification loss. Otherwise, they are pulled below the threshold through regularization. This means that although LTP utilizes pruning thresholds similar to previous methods, it is not entirely a magnitude- based pruning method, it allows the network to keep important weights that were initially small and removing some of the unimportant weights that were initially large, c.f., Figure (lett.
Continuing with equation
2 it follows that
dea Om
oz (Wa) and
Ovil sigm( Owni
2 W/ T
7)
+ wKl ar(wa),
2
with
2wT sigm(
w - ml x (1- sigm(
2 - ml
(www)) T
(4)
T
T
The (.).) function also appears in subsequent equations and merits some discussion. First note that (r(ww) as given by (4) is the derivative of sigm(( m)/T) with respect to waa- Since the latter approaches the step Tunction (located at w 72) in the limit as T + 0, it follows that the former, i.e. or (ww) would approach a Dirac delta function, meaning that its value approaches zero everywhere except over the transitional region where it is inversely proportional to region's 's width, i.e.,
oz (ww) ~ T
for Iw - nl ~
(5)
3

---

Under review as a conference paper at ICLR 2021
3.2 SOFT Lo REGULARIZATION
In the absence of weight regularization, the per-layer thresholds decrease to zero if initialized otherwise. This is because larger thresholds correspond to pruning more weights away, and unless these weights are completely spurious, their removal causes the classification loss, i.e., L, to increase. Loosely speaking.
ac θn Among the different weight regularization methods, Ly-norm regularization, which targets minimiza- tion of the number of non-zero weights, ie.,
O, unless is small.
La.
befits pruning applications the most. This is because directly quantifies the size memory or FLOPS needed during inference. However, many works use L1 L2 regularization instead, due to the Lo-norm's 's lack of differentiabilltyy. Notably, HHan et al. 20135) utilizes L1 and Lz regularization to push redundant weights below the pruning thresholds.
L, or L2 regularization methods may work well for pruning older architectures such as AlexNet and VGG. However, they fail properly regularize weights in networks that utilize batch-normalizzttion layers yan Laarhoven 2017). Hoffer et al. (2018). This includes virtually all modern architectures such as ResNet, EfficientNet, MobileNet, and MixNet. This is because all weights in layer preceding a nalhaiaormaiization layer can be re-scaled by an arbitrary factor, without any change in batch-nom norm outputs. This uniform re-scaling prevents I, or T-2 penalties from having their regularizing effect. To fix this issue, van Laarhoven (2017) suggests normalizing the Lz-norm of a layer's weight tensor after each update. This, however, not desirable when learning pruning thresholds as the magnitude of individual weights constantly changes
result of the normalization. Hoffer al. (2018), on the other hand, suggests using L1 or Loo natch-normalization instead of the standard scheme. This, again, is not desirable focus on Lg regularization, which does work well with normalization..
does not address current architectures. Consequently, in this work,
As the case with hard- pruming in ( the challenge in using Lo regularization for learning per-layer pruning thresholds is that it is not differentiable, i.e.,
Lo Σ step(wEk m);
This motivates our soft Lo norm definition for layer i.e.,
Loji Σ sigm(
w TI
(6)
T
which is differentiable, and therefore can be used with back-propagation, i.e.,
0Lo.1 oT(wa)) and oLo.1 Owns
1
or(ww)
(7)
On
2
Wil
k
where (ww)) is given by Inspecting reveals an important aspect of Lo.. namely that only weights falling within the sigmoid transitional region, i.e., |w2 7y ~ T, may contribute to any change in Lo, This because other weights are either very small and completely pruned away, or very large and unaffected by pruning. The consequence is that if a significant fraction of these weights, e.g., as a result of back propaaaatann update, are moved out of the transitional region, Lo,1 becomes constant and the pruning process stalls. The condition for preventing the premature termination of pruning, when using To,, can then be expressed as
0Cr
T, for 1 ni
Own
where Lr denotes the overall objective function comprising both classification and soft Lo regular- ization losses, i.e.,
C
Σ Lo.1-
Note that the left hand side of Eq. g is the displacement of Wkz as a result of weight update. So states that weight displacement should not be comparable transition region's width (~ T).
8
4

---

and Under review as a conference paper at ICLR 2021
3.3 LEARNED THRESHOLD PRUNING
LTP is a magnitude based pruning method that learns per-layer thresholds while training or finetuning. Specifically, LTP adopts framework of updating all network weights, but only using their soft-pruned versions in the forward pass. Gradients for thresholds and weights can be computed using
0C
ac Oog
ac
aC Ookl
Σ
and
(10)
deg
Dwa
Orks
Own.
where Uka is the soft-pruned version of the weight wai as defined by 2 LTP uses D. 7 to update the per-layer thresholds:
(T))
aLo.) OC
An = - On Ön
(11)
Updating weights needs more care, in particular, minimization of Li with respect to ww is subject to the constraint given by (8) Interestingly, /Cr/Oww as given by (9) 110 and i.e.
0CT sigm(
W
7)
Oc
+ (wRE
OC
+ x)
or or(ww);
(12)
Owkt
T
Oek
includes (wk)) which result of 5 could violate ( for T
<> (a requirement for setting
T, c.f., (15) and Table []. There are two simple solutions to enforce
8 The first approach is to
compute TL /uwa given by [1... but clamping it based on (8. The second, and arguably simpler,
approach is to use
OCr Oec
WHI nI
ac Oori
s sigm
(13)
T
To appreciate the logic behind this approximation, note that for the vast majority of weights that are outside the sigmoid transitional region, equations (13) and (12) give almost identical values. On the other hand, although values given by (13) and (I2). after clamping, do differ for weights within the transitional region, these weights remain there for a very small fraction of the training time (as n moves past them). This means that they would acquire their correct values through mack-propagation once they are out of the transitional region. Also note that (13) is equivalent to only using the classification loss L (and not Lo.1) for updating weights, i.e.
Oc
dull Dwkt
7
Auga R -1) Dwit
and
sigm(
(14)
T
instead of (3). i.c. treating the sigmoid function in (3) as constant. These adjustments are necessary for preventing premature termination of LTP. For example Figure (right) depicts the scatter plot of the pruned model's 's w&, (y-axis) vs. those of the original one (x-axis) for layer3.2.conv' of ResNer20 on Cifar100 when (3) is used instead of (14. Note how the formation of a gap around the threshold (the red line) causes the pruning process to terminate prematurely with a small threshold. Finally,
after training finished LTP uses the learned thresholds to hard-prune the network, which can further be finetuned, without regularization, for improved performance.
4 EXPERIMENTS
4.1 CHOICE OF PAPEE--RRAMETERS
LTP has three main reproduce the results reported in this paper. LTP uses soft-pruning during training learn per-layer thresholds, but hard-pruning to finally remove redundant weights. Selecting a small enough T ensures that the performance of soft-pruned and hard-pruned networks are close. Too small T', on the other hand, is undesirable as it makes the transitional region of the sigmoid function too narrow. This could possibly terminate pruning prematurely. To set the pay-layer Ti in this paper, the following equation was used:
T, 7h, and X. Table provides the hypee-parameter values to
(15) While one could consider starting with a larger To and anneal during the training, a fixed value of To 1e-3 provided us with good results for all results reported in this paper. One important
T To Twal
5

---

Under review as a conference paper at ICLR 2021
odde srunce mode
2.5
0.8
50
15.8
15.2
0.0
00
2.5 2.5
Figure I: Scatter plot of pruned/original weights for layer3.2.conv2 ResNet20 (red lines indicate thresholds). Upper-leti/lowerrright squares show kept/pruned weights that were initially small/large (left). Using (3) instead of (4) stalls pruning by creating a gap around the threshold (right).
Network
Dataset
To
7m/11
1
X
ResNet20
Cifar100
le-3
le-5
2c
AlexNet
ImageNet
le-3
le-7
le-7
ResNet50
ImageNet
le-3
le-7
3e-7
MobileNetV2
ImageNet
le-3
le-7 le-7
EfficientNet-BB
ImageNet
le-3
5e-7
1e-6
MixNet-S
ImageNet
le-3
le-5 5e-8
Table 1:
values used produce results reported in this paper.
consideration when choosing 1h /n is given by equation (10). namely, that the gradient with respect to the pruning threshold gets contribution from the gradients all weights in the layer. This means that 0L/0rt can potentially be much larger than typical åL/ especially if values of à ddw are correlated. Therefore, to prevent changes in Ti that are orders of magnitude larger than changes in Ukts ///1 should be small. While Table summarizes the values used for 7h for producing results reported in this paper, any value between le-5 and le-7 would work fine. Finally, À is the determining the sparsity levels achieved by LTP. Our experiments show that to get the best results, À must be large enough such that the desired sparsity is reached sooner rather than later (this is likely due to some networks tendency to overfit, trained for too long), however too aggressive À may be disadvantagcous the first pruned model may have poor performance without any subsequent recovery.
primary
4.2 ABLATION STUDY
Figure 2 provides an ablation study of LTP with respect to various regularization methods for ResNet20 on Cifar100. As the figure shows, the absence of regularization, LTP only achieves modest sparsity of 94% after 100 epochs of pruning (upper-left). La regularization achieves good sparsity. but it provides poor performance (upper-right). This, as explained in section 3..2) is due to the unison and unbound fall of layer weights which deprives Lg loss of its regularizing effect (lower-right). Finally, the keep ratio plot indicates that Lo regularization provides LTP with a natural exponential pruning schedule shown in Zhu & Gupta (017) to be very effective.
4.3 IMAGENET PRUNING RESULTS
In this section we perform an evaluation LTP on the ImageNet ILSVRC-2015 dataset (Russakovsky al. (2015)). LTP is used prune wide variety of networks comprising AlexNet (Krizhevsky et al.
(201 7), ResNet50 (He et al. (2016)). MobileNet- V2 (Sandler et al. (008)). EfficientNet-BB (Tan & Le (2019a) and MixNet-S lan Le (2019b)).
Table 2 gives LTP's ResNet50 pruning results on ImageNet, where it matches the performance achieved by Ye et al. (2019), i.e. 9.11x compression with Top-5 accuracy of 92.0%. LTP also

---

Under review as a conference paper at ICLR 2021
1.0
Koop ratlo
0.9
0.8
0.6
70
10
20
E(U 026-3)
Iteration
regurantration
10
15
20
30
Itoration 0x183)
Figure 2: Ablation study with various regularization methods for ResNet20 Cifar100. Keep-ratio
(upper-left), training Top-1 (upper-right) and squared-weighte for layer3.2.conv2 (lower-left).
Method
Top1
Top-5
Rate
Original (CaffeNet)
75.17%
92.4%
x
Mao et al. 2017
N/A
92.3%
2.6x
et al.
2019
N/A
92.0%
9.16x
Original (Torch
76.15%
92.9%
1 9.31
Renda et al.
75.35%
N/A
LTP
73.71%
92.0%
9.11x
Original (used by STR)
77.01% 74.01%
N/A
1x 10.58x
Kusupati et al. (STR)
N/A
Table 2: ResNet50 pruning results.
matches Kusupati et al. (2000) performance, considering STR's higher baseline. We note that the iterative weight rewinding method introduced by Renda al. (2020) provides 1.6% higher top-1 accuracy at compression rate of 9.31 x, however it requires 900 epochs of re- re-training compared to LTP's 30. In fact as Table 3 shows, LTP is very computationally efficient, pruning most networks on ImageNet in less than 100 epochs. This is in sharp contrast to, c.g., Renda et al. (2020), Ye et al.) (2019). Han al. (2015a) He et (2018), etc., which typically require a few hundred epochs of training. Finally, Figure 3 provides LTP's top-1 trace (without finetuning) and error-bar bars across 10 independent runs on ResNet50. As the figure shows, LTP enjoys a low variability in terms of top-1 accuracy of pruned models across different runs.
Table 4 provides ITPP's AlexNet performance results on ImageNet, where achieves a compression rate of 26.4x without any drop in Top-5 accuracy. It is noteworthy that TorchVision's AlexNet implementation is slightly different from CaffeNet's. While both implementations have 56M weights in their fully-connected layers, Torch Vision model has only 2.5M weights in its convolutional layers compared to 3. 3.75M of CaffeNet's. As a result of being slimmer, the Torch Vision uncompressed model achieves 1.1% lower Top-1 accuracy, and conjecture can be compressed less.
To the best of our knowledge, it is the first time that (unstructured) pruning results for MobileNerV2, EfficientNet-B0 and MixNet-S are reported, c.f., Table This is partially because LTP, in contrast to
Method
Scenario
Epochs
Rate
He et al. 2018) Renda et al. 2020
ResNet50 ResNet50
376
5.13x
900
9.31 x
ResNet50
30
9.11x
MobileNetV2
101
3x
LTP
EfficientNet-BB
52
3x
MixNet-S.
25
Table 3: Computational complexity results.
7

---

Under review as a conference paper at ICLR 2021
Method
Top-5
Rate
72.4
Original (CaffeNet)
80.3%
1x
Han et al. 201
80.3%
9x
Accuracy
Manessi et al 2018
79.3%
12x
73.0
Zhang et al. 2018
80.2%
21
10p
Ye al.
2019)
80.2%
30x
72.8
Original (Torch Vision)
79.1%
1x
LTP LTP
79.1%
26.4x
0:00
0:11
0.12
0.13
0.24
0:15
78.7%
29.5x
Koop ratio
Figure 3: ResNet50 top-1 trace (without finetun- ing) and error-bars over 10 runs.
Table 4: AlexNet pruning results.
MobileNetV2
EfficientNet-B0
MixNet-S
Method
Top-1
Top-5
Rate
Top-1
Top-5
Rate
Top-1
Top-5
Rate
Uncompressed
71.8%
90.4%
1x
76.1%
93.0%
1x
76.0%
92.8%
1x
71.2%
90.1%
1.33x
75.7%
92.7%
2.22x
74.5%
91.6%
1.67 x
Global Pruning
68.1%
88.3%
2x
75.6%
92.6%
2.5x
74.4%
91.6%
1.85x
59.6%
83.0%
3x
75.1%
92.3%
74.5%
91.6%
2x
71.1%
90.1%
1.33x
76.1%
93.0%
2.22x
75.7%
92.1%
1.67
LTP
70.0%
89.2%
2x
75.9%
92.9%
2.5x
75.3%
92.2%
1.85x
68.9%
88.7%
3x
75.2%
92.4%
3x
75.1%
92.0%
2x
Table 5: MobileNetV2, EfficientNet-BO (without Swish) and MixNet-S pruning results.
many other methods such as, e.g., Han et al. (0155)) Zhang et al. (2018) and Ye et al. 2019), does not require preset per-layer compression rates, which is non con-trivial given these networks' large number of layers (50 ~ 100), parallel branches and novel architectural building blocks such as squeeze- and-excite. This, along with LTP's computational efficiency and batch-normalization compatibility, enables it be upplied to such diverse architectures out-of-the-boo In the absence of pruning results in the literature, Gelobal-Pruning as described and implemented in Ortiz et al. | (2019), was used to produce baselines. In particular, Top-1 advantage over one compressed by Global-Pruning. Finally, note that LTP can be used to compress MobileNetV2, EfficientNet-BO, and MixNet-S, which are architecturally designed be efficient, by 1.33 x, 3 x and 2x, respectively, with less than 1% drop in Top-1 accuracy.
that L.TP's 3x compressed MobileNetV2 provides 9%
5 CONCLUSION
In this work, we introduced Learned Threshold Pruning (LTP), novel gradient-based algorithm for unstructured pruning of deep networks. We proposed framework with soft Lo regularization and soft-pruning mechanisms to learn the pruning thresholds for each layer in an end-to-end manner. With an extensive set experiments, we showed that LTP an out-of-the-bor method that achieves remarkable compression rates on traditional (AlexNet, ResNet50) as well as modern (MobileNetV2. EfficientNet, MixNet) architectures. Our experiments also established that LTP gives high compres- sion rates even in the presence of batch normalization layers. LTP achieves 26.4x compression on AlexNet and 9.1x compression on ResNet50 with less than 1% drop top-5 accuracy on the Ima- geNet dataset. We are also the first to report compression results efficient architectures comprised of depth-wise separable convolutions and saueeze-anddexeiie blocks, e.g. LTP achieves 1.33x, 3x and 2x compression Mob.leNetV2. EfficientNet-BO and MixNet-S respectively with less than 1% drop in top-1 accuracy on ImageNet. Additionally, LTP demonstrates fast convergence characteristics, e.g. prunes ResNet50 in 18 epochs (plus 12 epochs for finetuning) compression factor of 9.1.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-18, 2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Bin Dai, Chen Zhu, and David Wipf. Compressing neural networks using the variational information bottleneck. arXiv preprint arXiv: 2018.
Emily L. Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 1269-1277 2014.
Julian Faraone, Nicholas J. Fraser, Michaela Blott, and Philip H. W. Leong, SYQ: learning symmetric quantization for efficient deep neural networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4300-4309 2018. doi: 10.1109/CVPR.2018.. 000452.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint
201 5a.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. CoRR, abs/1506. 506.0666.. 2015b.
Babak Hassibi, David G. Stork, and Gregory J. Wolff. Optimal brain surgeon: Extensions and performance comparison. In Advances in Neural Information Processing Systems 6, [7th NIPS Conference, Denver, Colorado, USA, 1993], pp. 263 263-270, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 109/CVPR.20166900
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 1398-1406, 2017. doi: 10. 109/CCC
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: automl for model compression and acceleration on mobile devices. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-1 2018, Proceedings, Part VII. pp. 815-832, 2018. 3-00--0133-----------00- 48.
doi:
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 2164-2174, 2018
Kristen Grauman, Nicolò
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitaae conference on computer vision and pattern recognition, pp. 7132-7141, 2018
networks, In Proceedings of the IEEE
Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley, and Luc Van Gool. Ai benchmark: Running deep neural networks on android smartphones. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018.

---

Under review as a conference paper at ICLR 2021
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, 2015.
Benoit Jacob, Skinmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticoonff inference. In The IEEE Conference on Computer Vision and Putlern Recognition (CVPR),June 2018.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In British Machine Vision Conference, BMVC 2014, Nottingham, UK, September 1-5, 2014, 2014.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang. and Dongjun Shin. Com- pression deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv: 1tti:::5111.0555.. 7.06300, 2015.
Alex Krizhevsky, lya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Commun. ACM, 2017.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity, 2020.
Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, and Max Welling. Taxonomy and evaluation of structured compression of convolutional neural networks. arXiv preprint 191X:::9/2200802, 2019.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems 2, (NIPS Conference, Denver, Colorado, USA, November 27-30, 1989), pp. 598-605, 1989.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Leurning Volume 48, ICML'16, pp. 2849-2858. JMLR.org. 2016.
Christos Louizos, Karen Ulirich, and Max Welling. Bayesian compression for deep learning. In Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3288-3298. Curran Associates, Inc., 2017.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018.
Franco Manessi, Alessandro Rozza, Simone Bianco, Paolo Napoletano, and Raimondo Schettini. Automated pruning for deep neural network compression. In 24th International Conference on Pattern Recognition, ICPR 2018, Beijing, China, August 20-24, 2018, pp. 657-664, 2018. doi: 10. N0.1109/CPP.2018888444
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J. Dally. Exploring the granularity of sparsity in convolutional neural networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26. 2017, pp. 1927-1934. IEEE Computer Society, 2017. doi: 21110/CVVR..201..22
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6- August 2017, pp. -498-2507, 2017.
10

---

Under review as a conference paper at ICLR 2021
Jose Javier Gonzalez Ortiz, Davis W. Blalock, and John V. Guttag. Standardizing evaluation of neural network pruning. In Workshop on Al Systems ar SOSP, 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 13222, 2015, doi: 10.1007/s 1263-111-1---11111111
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Suraj Srinivas. Akshayvarun Subramanya, and Venkatesh Babu. Training sparse neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 455 455-462, 2017. doi: 10. 109//VPRR.2017.6..
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 195:1:990. 2019a.
Mingxing Tan and Quoe V Le. abs/1907.09595, 2019b.
Mixconv: Mixed depthwise convolutional kernels. CoRR,
Karen Ultrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
Twan van Laarhoven. abs/1706.05350, 2017.
L2 regularization versus batch and weight normalization. CoRR,
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: wardware-award efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Shaokai Ye, Xiaoyu Feng, Tianyun Zhang, Xiaolong Ma, Sheng Lin, Zhengang Li, Kaidi Xu, Wujie Wen, Sijia Liu, Jian Tang, Makan Fardad, Xue Lin, Yongpan Liu, and Yanzhi Wang. Progressive DNN compression: A key to achieve ultra-high weight pruning and quantization rates using ADMM. CoRR, abs/ 1906.9979.,, 2019.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic DNN weight pruning framework using alternating direction method of multipliers. In Computer Vision ECCV 2018- 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII, pp. 191-207, 2018. doi: 10. 10.10077997-33000--02222
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10); 2016.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quan- tization: Towards lossless cnns with low-precision weights. arXiv preprint arxiv:170..03044 abs/1702.030444, 2017.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 3592-3603 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
11


Reviews:
[{"review_id": "hSVwnM8IWs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes soft pruning and soft l0 regularization. \n\nThe soft pruning learns a binarization function based on a sigmoid. I am curious to understand how the algorithm can let some low magnitude weights not be pruned and some large magnitude weights be pruned in the end. The paper claims the key is updating all the weights while maintaining the masks for inference. That seems to me ineffective compared to L1 or similar methods. Also, even in the case the weights are not masked, eventually, those gradients should become zero (as they are masked in the forward pass) and therefore preventing them from having large values, isn't it? That needs some clarification. \n\nAs the paper then focuses on L0 regularization, I missed comparisons to related work on that regard. This is not the first paper aiming at using L0 and proposing a differentiable approach. How this compares to others?  For instance, a quick search led to LEARNING SPARSE NEURAL NETWORKS THROUGH L0 REGULARIZATION in ICLR 2018. \n\n\n\nI am confused with one of the contributions is to \"provides a trace of checkpoints with varying pruning ratios and\naccuracies. Because of this, the user can choose any desired checkpoint based on the sparsity\nand performance requirements for the desired application\"\nWhy is this different from any other approach? As soon as the code saves the checkpoint (which most do) then, the user has access to the same flexibility, right?\n\n\nThe section about the hyperparameters is confusing. How are the hyperparameters determined? Each architecture is using a different hyperparameter, how a user could set these?\n\nThe experimental results do not seem to support the novelty and the text is kind of misleading. LTP is below Renda and Kusupati. Text suggests Kusupati uses a better baseline, would it be possible to show results of LTP on that baseline? if not, why not?\nFor Renda, seems like the key difference is the training process. Would be good to see the benefits of a longer training process for LTP. It is not clear to me that LTP can get better results even training for longer. \n\nResults on more compact networks compared to the self-implementation of Global pruning seem promising for larger compression rates. \n\nThe paper also claims the benefit of learning the threshold per layer, however, provides no result on the distribution of those parameters. Would be interesting to see how these values are distributed in each architecture to reinforce the value of not using a single value for all layers and architectures. \n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cPQPVWuQITT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper introduces a new type of soft threshold operator in conjunction with appropriate weight regularization that can be used in the context of neural network pruning to obtain sparse, performant networks from pre-trained, dense networks. The main idea is to replace the Heaviside step function that occurs in \"hard threshold\" pruning, which is non-differentiable, by a sigmoid function that can be differentiated and thus enables the efficient training/optimization of relevant pruning parameters. Pruning is hereby performed on a per-layer basis by training a regularized per-layer threshold. \n\n## Score\n\nI am quite intrigued by the method and I think it has potential. It seems easy enough to implement while providing decent improvement over competing methods, at least judging from the sparse experimental results that were presented. This brings me to the big weakness and the reason for my score. The experiment section does not provide enough evidence in my opinion to justify acceptance since I cannot say with full confidence that the method reliably performs well. And even when the approach does not outperform existing methods in all aspects, at least I would like to be able to judge what are the scenarios where the method performs well. More details are provided below. \n\n## Ways to Improve My Score\n\nMainly, please address the points I mentioned in the \"Weaknesses\". Some of them can be addressed by updating the writing. However, the major concern of mine is the experiment section. I think it requires a full overhaul including more comparison methods, standardized experiment settings, better organized presentation of the results, and clear description of the hyperparameter choices. \n\n## Strengths\n\n* The concept of the introduced soft threshold operator is easy to follow and intuitive. I appreciate the detailed description of the resulting derivatives in Section 3 and the provided intuition. As such the method is well-described and the benefits are clear. \n\n* The various aspects of their ablation studies are interesting, c.f. Figure 1, Figure 2. It helps better understanding some of their design choices. \n\n* The presented experimental evidence seems to hint at very decent performance, especially at the ImageNet scale. This is encouraging to see and underscores the intuition behind the method. \n\n## Weaknesses\n\n* To me, the biggest weakness is the presented experimental evidence. It seems scattered, the presentation is confusing, and most of all it makes it extremely difficult to assess the performance gains of the presented method in comparison to existing methods. Some points that I would like to list specifically are below: \n  1. There is no single figure that allows me to assess the prune-accuracy trade-off across a large range of prune ratios and for various comparison methods. Something like Figure 4 in the paper by Kusupati et al. 2020 (https://arxiv.org/pdf/2002.03231.pdf) is, in my opinion, necessary in order to more reliably assess the resulting performance. \n  2. The authors only present a selected set of prune ratios and resulting accuracies in their Tables 2-5. Also the results are presented inconsistently. Table 3 only reports compression rate. Table 4 reports Top-5 accuracy and compression rate but not Top-1 accuracy. Table 2, 5 present Top-1 and Top-5 accuracy but only a few selected comparison methods that differ from the one presented in Table 4. \n  3. More comparison methods: Unstructured pruning has seen quite a few advancements over the last couple of years and as such I believe it is crucial to compare to many more pruning methods. This is particularly important since standardized comparisons are missing and so simply presenting some results gathered from other papers is not enough in my opinion. \n  4. Were experiments repeated multiple times? I can see that Figure 3 was based on 10 repeated runs. What do the error bars represented? Also, what about the other experiments? Could the authors clarify how the other numbers were generated and also report mean and standard deviation. The experiments should be repeated at least a couple of times. \n\n* A more thorough comparison to STR (Kusupati et al. 2020, https://arxiv.org/pdf/2002.03231.pdf). STR shares a lot of similarities with this work in the sense that STR also introduces a per-layer threshold for pruning that can be efficiently optimized using a differentiable soft threshold operator. There is only one comparison point in Table 2, which however seems to be based on a different implementation thus resulting in a different baseline accuracy. There is also no discussion how the soft threshold operators of STR and LTP differ and what makes one better than the other.  \n\n* The experimental hyperparameters are not fully listed and the ones that are listed are scattered throughout the paper. Also, the authors did not provide code; so I couldn't check their implementation either. In particular:  \n  1. The contributions list in the introduction mentions the number of pruning and fine-tuning epochs for some experiments but the experiment section doesn't provide a full overview of pruning+fine-tuning epochs. Table 3 provides some of these numbers but not all details. Also what are the particular reasons for the choices? ResNet50 seems to require 30 total pruning epochs, while MobileNetV2 requires 101 epochs. Why? \n  2. What does $\\sigma^2_{|w_{kl}|}$ in equation 15 refer to? \n  3. What about training parameters? Are those the same as in the original paper? What about an overview table with all training parameters? \n  4. How were the comparison methods implemented? Were they even implemented or were the results taken from the respective papers? \n\n* I am not convinced that $L_1$ and $L_2$ regularization don't work for pruning in general as the authors claim in Sections 2 and 3.2. I understand their point and in their case their approximation to $L_0$-regularization indeed seems to play a central role but I wonder how true this statement is in general. In particular, while batch normalization (BN) may allow for arbitrary re-scaling of layers, the _relative magnitude_ of weights may still be impacted by $L_1$-regularization. Moreover, after all they don't use $L_0$-regularization either, just another differentiable approximation to $L_0$-regularization (just like $L_1$ is a differentiable approximation to $L_0$). \n\n## Other Minor Feedback\n\n* I find the introduction and related work interesting and it serves as an appropriate motivation for the work. However, I find it somewhat disingenuous and/or misleading not to mention global magnitude-based pruning in the first sections. The authors cite a lot of related work that requires manual and/or more complicated approaches to identifying per-layer sparsity patterns when the most obvious solution is to perform global magnitude pruning (GMP). Since this usually works quite well as baseline and the authors also compare to GMP in their experiment section, I believe this merits a longer discussion where the authors compare to GMP. \n\n* Section 3.3 could be simplified. I was pretty confused since a lot of prior quations are cited in the explanation between newly introduced equations. That makes it pretty hard to read and I believe the introduced concepts and resulting equations could be streamlined. E.g. you could introduce all relevant equations in a continuous paragraph instead of jumping between equations, which results in the heavy use of equations citations. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bbp-JccEu-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a new method to prune a neural network. The method is interesting, innovative and effective. It makes it possible to learn tunning parameter via back propagation, hence learn together with network's weights. \nThe work is well motivated. \nThe paper is well structured, the writing is clear and easy to follow.\nThe conducted experiments are thorough and clearly show the efficiency of the proposed method. The paper contains enough information to replicate the experiments.\n\nThe work would be beneficial for others if the code is published open.\n\nA question for clarification: When hard prunning the network (section 4.1), we just replace sigmoid(x) by step(x)?\n\nPost-discussion update: I have read the updated paper and other reviews, especially from reviewer #2. While I am still positive about the approach/methodology, I am not confident about the technical details of the experiments, without which, it's very hard to justify the effectiveness of the method. I share other reviewers' views regarding inconsistencies, e.g. Tables 2-5, that have not been fixed in the updated paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AvgrA-0gNoE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "My overall feeling about this paper is that I really liked it. I felt it was clearly written, nice pacing, didn't bombard us with things we already know, nor did it skip over things we don't know about. I like the approach of using the sigmoids to be able to learn the thresholds and the weight pruning. I really like this approach. In practice, there are still several hyper-parameters to tune, which are apparently model-specific (table 1), and it seems unclear how to directly set the sparsity (we can only influence it using lambda, not set it directly, as far as I can see?). Nevertheless I do really like this approach and direction, ie of learning things as much as possible. The approach seems to me analagous to using eg Adam instead of SGD.\n\nI feel the experiments section could be written a bit more clearly so as to make the assertions in the conclusion and introduction stand out as really obvious. For example, the assertion that LTP can be used in the presence of batch normalization was not made apparent in the experiments section I felt, nor was it shown as a benefit compared to other baselines, I felt.\n\n\n\nDetails:\n\n'computationally extensive' => 'computationally expensive'\n\nfigure 1 really far from where it is first referenced. had to hunt for it....\n\nwriting really clear. Good exposition of related papers. Pacing very nice. Very easy to understand.\n\nGood choice of which information to present. (cf papers that present lots of well-known knowledge, or skip over some key advanced concepts).\n\nKind of a detail, but I like the use of $sigm$ to denote sigmoid, (cf many papers will write the sigmoid out in full, which makes the equations much harder to read)\n\nPersonally, I would prefer that $\\sigma_T(w_{kl}) = \\frac{\\partial sigm((w^2_{kl}-\\tau_l)/T)}{\\partial w_{kl}}$ is defined before equation 3. Otherwise I first read equation 3, wondering what is $\\sigma_T$, and then realize it is defined underneath.\n\nTo be honest I'm not a fan of the notation $\\sigma_T(\\cdot)$, since when I first read it I parsed it as $sigm_T(\\cdot)$. I would prefer either using some other symbol, or perhaps writing out the full partial derivative, though that seems probably long, without some other symbol, or perhaps adding a $'$, like $\\sigma_T'(\\cdot)$.\n\nI'd also prefer that the definition of $\\sigma_T(w_{kl})$ includes the derivative in this definition, ie:\n\n$$\n\\sigma_T'(w_{kl}) = \\frac{\\partial}{\\partial w_{kl}} sigm\\left(\\frac{w^2_{kl} - \\tau_l}{T}\\right) = ... etc ....\n$$\n\nI like the paragraph that describes the behavior of $\\sigma_T'$\n\nI like the exposition of the various regularization methods. Concise and yet easy to understand.\n\n$\\eta$, in equation 8 is not obviously defined anywhere. I went hunting for it, but couldn't find it. Please define it before using it.\n\nEquation 8 comes out of nowhere, with no explanation of what it means, or why it is. The rest of the paper explains things very well, but equation 8, I'd have to think a lot, wasn't immediately obvious to me where it comes from, what it means.\n\nI'm not sure what the syntax $\\sim T$ means here. It normally means 'is distributed as', but that meaning doesn't seem to make sense here? It looks like it's being used to mean $\\approx$?.\n\nOk, I had to go all the way back to equation 2 and the paragraph after it. Looks like $\\sim$ is being used here to mean $\\approx$. I think that using $\\approx$ would be more standard, and easier to understand? Ok, I googled $\\sim$, and it turns out that it can often be used to mean 'is approximately the same order of magnitude as', eg https://math.stackexchange.com/a/2177014/45703  But I personally found it confusing because it is very often used to mean 'is distributed as', so personally I would prefer to have a short explanation like 'where $\\sim$ means \"is approximately the same order of magnitude as\"'\n\nI guess the other reason I find it confusing though is this equation implies to me that $|w^2_{kl} - \\tau_l| = 0$ would not be in the region, but in fact the region is I feel something like:\n\n$$\n|w^2_{kl} - \\tau_l| \\lesssim T\n$$\n\nPersonally I think I would prefer the conditions for the transitional region written in this way; would be less confusing for me; I think.\n\nSimilarly equation 5 would be:\n\n$$\n\\sigma_T(w_{kl}) \\approx \\frac{1}{T}, \\text{ for } |w^2_{kl}-\\tau_l| \\lesssim T\n$$\n\nand then equation 8 becomes:\n\n$$\n\\eta \\cdot \\left| \\frac{\\partial \\mathcal{L}^T}{\\partial w_{kl}}\\right| \\ll T , \\text{ for }  |w^2_{kl}-\\tau_l| \\lesssim T\n$$\n\nI'm not sure though why this derivative is in this constraint? Isnt the constraint simply that\n\n$$\n\\sum_{l=1}^L \\sum_{k=1}^K I\\left[ |w^2_{kl} - \\tau_l| \\right] > m\n$$\nwhere $I[\\cdot]$ is an indicator function, and $m$ is some positive integer?\n\nie, at least some points need to be in the transitional region. I'm not sure I follow why we need a derivative in the constraint. Please can you add some description around equation 8 so I can follow what is going on :)\n\nAnd also trying to work through equation 8, it seems like it is saying that we want to make the derivatives as close to zero as possible, relative to T. But isnt this the opposite of what we want? Dont we want to have a reasonable number of derivatives that are not near zero?\n\nOk, after equatino 9, we get some explanation for equation 8 :) But I think the explanation could be moved forward somewhat :)))\n\nOk, based on this explanation, ie the one after equation 9, $\\eta$ is probably learnin rate. But please define $\\eta$, near equation 8 :)\n\nFrom the explanation, I'm not sure that equation 8 is a condition that actually *prevents* premature pruning, so much as a heuristic to minimize weights moving too quickly out of the transitional region. Preference to be clearer about this, since it would certainly have helped me to understand equation 8 more easily and quickly :)\n\nequation 11, the brackets could be nicer looking if use \"\\left(\" and \"\\right)\", I feel (so they are as large vertically as the derivative fractions they contain)\n\n$\\lambda$ in equation 11 is not defined. From section 4.1, we can see it is a hyper-parameter to be tuned, but that is not stated in equation 11. Preference to state at equation 11 that $\\lambda$ is a hyper-parameter, and $\\eta_{\\tau_l}$ is the learning rate for the threshold of layer l. Hmmm, does this mean that each layer has its own learning rate to tune for its threshold? If there is one single learning rate for the thresholds, I think it might be clearer to represent it as $\\eta_\\tau$? If there are per-layer learning rates, then this seems to be to contradict the implied promises in the introduction that we don't have per-layer hyper-parameters to set?\n\nOk, looks like $\\lambda$ was first used in equation 9. But still wasnt defined there I think?\n\nI'd also prefer that equation 9 was defined before presenting equation 8, on the whole. This way I can read sequentially, not have to skip forwards and backwards.\n\nThe sentence just before and after equation 12 is very complex and hard to take in. Please consider breaking into smaller simpler sentences. eg \n\n\"$\\partial L_T/\\partial w_{kl}$is given by 9, 10, 3 and 7 as:\n\n(equation goes here)\n\nThis includes $\\sigma_T(w_{kl})$, which from equation (5) $\\approx 1/T$, and will become large for $T \\ll 1$. This means that the gradient will become large, and constraint (8) will be violated.\n\"\n\nFigure 1 I feel needs a lot more explanation.\n- why are they so symmetrical about the y=x line? doesnt this imply that the number of weights less than theshold before pruning and the number of weights less than theshold after pruning is similar?\n- why is the y-axis labeled 'w'? I thought the pruned weights are 'v'?\n- why make the plot? What is the motivation of this plot?\n- why is the scale of the axes radically different between left and right (0-16 vs 0-3) ?\n- why is the left hand plot preferable to the right hand plot?\n- why is the gap around threshold in the right hand plot a bad thing?\n- why is the proportion of weights below threshold similar in both left and right?\n\n\n4. experiments\n\nI like the table of hyper-parameters in table 1. (cf many papers skip over which hyper-parameter settings were used, making reproducing the work challenging)\n\nAppreciate the explanation of the significance of $\\lambda$ as the primary hyper-paramter determining the sparsity levels. I feel that this explanation could be moved back to equation 9.\n\nAppreciate the observations on how to set $\\lambda$, and $\\eta_{\\tau_l}$.\n\nI wouldnt really call figure 2 an 'ablation study'. It's more like a comparison study of various baselines and approaches I feel? An ablation study I feel would be more like:\n\n- no regularization\n- no drop second term in 12 (and simply not use any clamping etc in its place)\n\nI think the ablation study should go after the imagenet pruning results. (I mean, I think it's traditional to put ablation studies after the section of results vs other baselines/sota models)\n\nTable 2 is very unclear to me\n- why is your method not at the bottom of the table\n- I think your own method should be in bold and say \"(ours)\" after the name\n- looking at the table, it's not very clear why we should choose LTP?\n   - the highest rate and top1 looks to be Kusupati et al?\n- in the text description, it says that Kusupati uses a stronger baseline, ie STR\n   - why don't you use STR too?\n- in the text it says that Renda et al needs more training\n   - why not put the amount of training required as an additional column in the table?\n\nJumping to table 3 mid-paragraph is I think jarring. I think first talk about table 2 in one paragrpah, then talk about table 3 in the next. Or at least don't mix and match across tables, at least without having first presented each table on its own first. I feel. Like the sentence 'In fact as table 3 shows' cannot I feel precede the clause 'Finally, figure 3 provides ...', which presents what is table 3. Oh, that's figure 3, not table 3. Anyway, I think table 3 needs some introduction, please.\n\nYes, so, it seems to me that the resnet50 results from table 3 could be added to table 2 perhaps?\n\nFigure 3 should be in a separate paragraph, since it is not a comparison with baselines/other models. It's just an obseration about LTP itself. And really, without any comparison with how other models/pruning strategies are, I'm not sure it is very meaningful to me? Like, for all I know other models have smaller error bars?\n\nI think table 4 presentation should follow immedaitely table 2 presentation.\n\nThen table 3 presentation.\n\nFigure 3 might be best in a separate 'appendix'-y sub-section at the end of section 4, I feel. Since it's not comparing to other models, like table 2, 4 nad 3 are.\n\nTable 4. If torchvision gives worse results than caffenet, then why not use caffenet instead, or re-implement the caffenet version of alexnet in torch? Otherwise, we can see taht LTP results in table 4 dont match the baselines, and we cannot tell if this is because the torchvision baseline is weaker, and LTP is strong, or whether LTP is weaker than eg Ye et al.\n\nIn fact, Ye et al only drops 0.1% top-5 accuracy compared to original, whereas LTP drops 0.4%, compared to torchnet original, so I feel that justifying the lower top-5 error rate on the weaker baseline model is not entirely sufficient?\n\nTable 5 looks like the strongest table to me. Might be worth putting it first? I feel that it could be useful to highlight the top results in each column in each scenario in bold? I think that ideally each column should be a single scenario (whereas here each column is multiple scenarios), then it is easy to highlight the top in each column. For example you could put the different rates as different columns, and use eg top-5 accuracy throughout the table. (or put top 1 and top 5 accuracies as tuples perhaps?)\n\nI kind of think that table 3 should be folded into the other tables.\n\nI think it's not clear from these tables why we should use eg LTP instead of Kusupati et al, or Ye et al. I think that either make it clear in the table somehow, or perhaps put in the text. Like eg \"Our method needs considerably less hyper-parameter tuning than existing SoTA methods, whilst achieving nearly the same accuracies for similar levels of compression.\"\n\nIn the conclusion, you mention batch normalization, but this was not brought to the fore in the experiments section. Like, I would expect to see some models that can only be pruned using LTP, and other approaches fail to prune, but I don't remember this being shown clearly in the experiment section?\n\nBasically, I think the assertions in the conclusion are exciting, but aren't made clearly obvious in the experiments section. I think for each assertion in the conclusion there should be a single table or graph that shows this assertion very clearly, in comparison to other possible baselines.\n\n[post discussion edit]\n\nAfter discussion, I lowered my  score to 'marginally above acceptance threshold':\n- the theory section of the paper looks very interesting to me\n- I find it hard to see clearly from the experimental section the extent to which the method beats existing methods\n- I feel that the experiments could be made more rigorous to clearly show the benefit compared to other techniques\n- concretely, I feel taht the tables could be structured in such a way that one can glance at each single table, and see clearly in what way LTP is better than the baselines. Concretely, for the results tables:\n\n- table 2: LTP gives worse accuracy than Renda, and worse compression. The text mentions training epochs are fewer for LTP, but the table doesn't show this benefit (there is no column with number of training epochs)\n- table 3: this table is a little apples and oranges I feel. it shows that the number of training epochs is less for LTP than Renda, but the compression ratio is slightly less. I feel that you could have compressed a little more, to make the compression ratios comparable. In addition, I feel it is important to include the accuracy in the table. Without accuracy, then I feel it is not possible to compare.\n- table 4: I feel you could do whatever is needed to do to ensure that the baseline model you are using matches the baseline that other teams are using. This could mean porting NTP to caffe, or porting caffe network into torch. Currently, the LTP pruning is on a worse 'parent' model, and performs worse than the other baselines in terms of accuracy. I'm not sure it's sufficient to hand-wavingly just add/subtract the delta in performance between the baselines to the LTP results (which is not explicitly being done, but if one doesn't do that, then one would have to assume that LTP performs worse, I feel)\n- which only leaves table 5 that plausibly provides an apples-for-apples comparison, but only for a single baseline", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for differentiable pruning that replaces the hard thresholding of standard pruning, with a soft version that permits taking the gradients of the pruning threshold. The proposed benefits are an accuracy that is better or competitive with alternative methods as well. Moreover, the paper suggests the technique to be efficient.

The pros of this paper are that it is working in an interesting setting of differentiable pruning, with the hope of -- in some sense -- simplifying the pruning process or at least unifying the process with standard training.  The technique is plausibly justified in its technical development. The paper also follows with a significant number of experiments. 

The cons of this paper are that the conceptual framework -- beyond the initial idea -- is not fully clear. In particular, this paper does not elucidate a clear set of claims and hence, results in the difficulty on the Reviewers part in detangling the claims and identifying the appropriate comparisons.

For example, the paper doesn't take up a simple claim that it is state-of-the-art in accuracy vs parameter measures (and would seem not to given the results of Renda et al. (2020)).  It need not necessarily make this claim, but there are suggestions to such a claim early in the paper. If this is not an intended claim, then the paper can remove any suggestions to such (i.e., the claims around new SoTA for networks not evaluated in prior work). 

The paper has a somewhat tentative claim that it is more efficient (in the total number of epochs of training) versus other techniques (Table 3).  However, the presented results are only at a single-point versus other methods.  Renda et al. (2020) directly consider accuracy versus retraining cost trade-offs. Appendix E of that paper provides one-shot pruning results for ResNet-50 showing accuracy on par with that presented here.  The number of retraining epochs is also similar to here. This paper, however, only compares against the most expensive iterative pruning data point in the other paper.

In sum, my recommendation is Reject. This is promising work that needs only (1) to include a few testable claims and (2) to re-organize the results (and perhaps run a limited set of new results) to thoroughly explore those claims. For example, if the most important claim is accuracy vs retraining cost, then it needs to show a more complete trade-off curve of the two results.  Of course, this, in principle, opens the door to comparisons to many other techniques in the literature.

Decision: Reject