Dynamic Model Pruning with Feedback
Authors: Tao Lin
Sebastian U. Stich
Luis Barba
Daniil Dmitriev
Martin Jaggi
Venue:  (2020)
Keywords: network pruning
dynamic reparameterization
model compression

Abstract: Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and  (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate the method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models and further that their performance surpasses all previously proposed pruning schemes (that come without feedback mechanisms).

Published as a conference paper at ICLR 2020
DYNAMIC MODEL PRUNING WITH FEEDBACK
Tao Lin
Sebastian U. Stich
HPFL, Switzerland tao. wwineennel.chh
EPFL., Switzerland
sebastian. wtw.sstiaa..ssichcee......
Luis Barba
Daniil Dmitriev
EPFL & ETH Zurich, Switzerland ethz. ch
EPFL, Switzerland
daniil aniil. wwmirrieeeepl..e.
Martin Jaggi
EPFL, Switzerland
martin. www.ggapgggg...oo.
ABSTRACT
Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency inference. We propose novel model compression method that generates a sparse trained model without additional overhead: by al- lowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method CIFAR- 10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-an performance dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.
INTRODUCTION
Highly overparametrized deep neural networks show impressive results on machine learning tasks. However, with the increase in model size comes also the demand for memory and computer power at inference stage-tWe resources that are scarcely available on low-end devices. Pruning techniques have been successfully applied remove significant fraction of the network weights while preserv- ing test accuracy attained by dense models. In some cases, the generalization of compressed networks has even been found be better than with full models (Han et al. 2013 2017: Mocanu et al. 2018).
The sparsity of a network is the number of weights that are identically zero, and can be obtained by applying sparsity mask the weights. There several different approaches to find sparse models. For instance, one-shot pruning strategies find a suitable sparsity mask by inspecting the weights of pretrained network (Mozer & Smolensky 1989 LeCun al. [990] Han al. (017)) While these algorithms achieve a substantial size reduction of the network with little degradation in accuracy, they are computationally expensive (training and refinement
the dense model), and
they are outperformed by algorithms that explore different sparsity masks instead of a single one. In dynamic pruning methods, the sparsity mask is readjusted during training according to different criteria (Mostafa & Wang. 2019 Mocanu et al. 20. However, these methods require fine-tuning of many hyperparameters.
We propose a new pruning approach to obtain sparse neural networks with state-of-the-ae test accuracy. Our compression scheme uses new saliency criterion that identifies important weights in the network throughout training to propose candidate masks. As a key feature, our algorithm not only evolves the pruned sparse model alone, but jointly also a (closely related) dense model that used in natural way to correct for pruning errors during training. This results in better generalization properties on a wide variety tasks, since the simplicity of the scheme allows us further to study from a theoretical point of view, and to provide further insights and interpretation. We do not require

---

Published as a conference paper at ICLR 2020
sparsity
Pruning before training
Pruning after trsining
med
Incremental pruning during training
high
Dynamic pruning during training
thmee
Scheme
Gradient
Praning
Comments
Pruning before training
R(W;) pruned model)
at the beginning
reaches local minima
mask my fixed throughout
finding suitable masks is challenging
Pruming after training
g(w;) full model)
once at the end
does not reach local minima
mask my fixed
fine-tuning pruned model required
Incremental during training
g(W:) partially pruned m.)
several times during training
adaptive
incrementally, m m.
cannot recover from premature pruning
Dynamic during training
gt w (on pruned model)
dynamically udapting mask
every few iterations
can
adaptive
premature pruning
Figure I: Schematic view of different pruning methodologies and their properties.
tuning of additional hyperparameters, and no retraining of the sparse model is needed (though can further improve performance).
Contributions.
A novel dynamic pruning scheme, that incorporates an error feedback in natural way and finds a trained sparse model in one training pass.
Sec.
Sec.
We demonstrate sate-of-the-ar performance (in accuracy and sparsity), ourperforming all previously proposed pruning schemes.
Sec.
Sec.
We complement our results by an ablation study that provides further insights. and convergence analysis for convex and non-convex objectives.
Sec.
Sec.
RELATED WORK
Previous works on obtaining pruned networks can (loosely) be divided into three main categories.
Pruning after training. Training approaches to obtain sparse networks usually include three stage of dense model, one-shot pruning and fine-tuning e.g., (Han et al.. (005)). Their results (i.e., moderate sparsity level with minor quality loss) made them the standard method for network pruning and led to several variations (Guo et al. 2016) Carreira-Perpincae & Idelbayev 2018).
Pruning during training. Zhu & Gupta (017) propose the use of magnitude-base based pruning and to gradually increase the sparsity ratio while training the model from scratch. A pruning schedule deter- mines when the new masks are computed (extending and simplifying (Narang al. (017)). He et al.) (2018) (SFP) prune entire filters of the model at the end of each epoch, but allow the pruned filters to be updated when training the model. Deep Rewiring (DeepR) (Bellec et al. 2018) allows for even more adaptivity by performing pruning and regrowth decisions periodically. This approach is computation- ally expensive and challenging to apply to large networks and datasets. Sparse evolutionary training (SET) (Mocanu et al., 2018) simplifies prune-regrowth cycles by using heuristics for random growth the end of each training epoch and NeST (Dai al. 2019) by inspecting gradient magnitudes.
Dynamic Sparse Reparameterization (DSR) (Mostaaa & Wang 2019) implements a prune- redistibuuterregrowth cycle where target sparsity levels are redistributed among layers, based on loss gradients (in contrast to SET, which uses fixed, manually configured, sparsity levels). Sparse Momentum (SM) (Dettmers & Zettlemoyer, 2019) follows the same cycle but instead using the mean momentum magnitude of each layer during the redistribute phase. SM outperforms DSR on ImageNet for unstructured pruning by a small margin but no performance difference on CIFAR experiments. Our approach also falls in the dynamic category but we use error compensation mechanisms instead hand crafted cehistribute-rgrrooe
cycles.
Pruning before training. Recently-spurred by the lottery ticket hypothesis (LT) (Frankle & Carbin 2019) methods which try to find a sparse mask that can be trained from scratch have attracted increased interest. For instance, Lee et al. (2019) propose SNIP to find pruning mask by inspecting connection sensitivities and identifying structurally important connections in the network for a given task. Pruning is applied at initialization, and the sparsity mask remains fixed throughout training.
2

---

Published as a conference paper at ICLR 2020
high
Wi
Loss
Ul w
One-shol pruning
function
value
((w)
Our algorithm
low
Figure 2: Left: One-shot pruning (red) computes a stochastic gradient at w and takes step towards the best. dense model. In contrast, [DF] computes a stochastic gradient at the pruned model w (here obtained by smallest magnitude pruning), and takes step that hest suits the compressed model. Right: One-shot pruning commits single sparsity mask and might obtain sparse models that generalize poorly (without retraining). DPFeeplores different available sparsity patterns and finds better sparse models.
Note that Frankle & Carbin (001)); Frankle et al. (2019) do not propose an efficient pruning scheme to find the mask, instead they rely iterative pruning, repeated for several full training passes.
Further Approaches. [Srinivas et al. (2017); Louizos et al. (2018) learn gating variables (e.g. through to regularization) that minimize the number of nonzero weights, recent parallel work studies filter pruning for pre- pre-trained models (You et al.] 2019). Gal et al. (2017); Neklyudov et al. (2017);
Molchanov al. (2017) prune from Bayesian perspectives to learn dropout probabilities during training to prune and sparsify networks as dropout weight probabilities reach 1. G.Ge et al. (2019) extensively study recent unstructured pruning methods large-scale learning tasks, and find that complex techniques (Molchanov et al., 2017 Louizos et al., (0018) perform inconsistently. Simple
magnitude pruning approaches achieve comparable better results (Zhu & Gupta 2017).
3 METHOD
We consider the training of non-convex loss function f: Rd R. We assume for weight vector w E IRd to have access to a stochastic gradient g(w) E IRd such that Elg(w)] = ((().. This corresponds to the standard machine learning setting with g(w) representing a (mini- gradient of one (or several) components of the loss function. Stochastic Gradient Descent (SGD) computes a sequence of iterates by the update rule
W W++1 == W. 7:g(w:))
(SGD)
for some learning rate "r. To obtain sparse model, a general approach is to prane some of the weights of Wt, i.e., set them zero. Such pruning can be implemented by applying a mask m E {0, 1) ]ad to the weights, resulting in a sparse model w, = m @ w,, where @ denotes the entry-wise (Hadamard) product. The mask could potentially depend on the weights w, (c.g., smallest magnitude pruning), or depend on & (c.g., the sparsity is incremented over time).
Before we introduce our proposed dynamic pruning scheme, we formalize the three main existing types of pruning methodologies (summarized in Figure DT. These approaches differ in the way the mask is computed, and the moment when is applied
Pruning before training. A mask mo (depending e.g. the initialization wo the network architec- ture f' is applied and (SGD) is used for training on the resulting subnetwork o f(w) := ff f(mo o w) with the advantage that only pruned weights need to be stored and updated" and that by training with SGD a local minimum of the subnetwork f (but not of the original training target) can be reached. In practice however, remains challenge efficiently determine good mask mo and wrongly chosen mask at the beginning strongly impacts the performance.
'The method introduced in Section typically follow one of these broad themes loosely, with slight variations in detail. For the sake of clarity omit technical and detailed discussion here.
When training on f(w), it suffices to access stochastic gradients of f(w), denoted by g( (w), which can potentially be cheaper he computed than by naively applying the mask to g(w) (note g(w) mo g(w)).
3

---

Published as a conference paper at ICLR 2020
Pruning after training (one-shot pruning). A dense model is trained, and pruning is applied to the trained model wy. As the pruned model Wr = my @ WT is very likely not at a local optimum of f. fine-tuning (retraining with the fixed mask my) is necessary to improve performance.
Pruning during training (incremental and dynamic pruning). Dynamic schemes change the mask m, every (few) iterations based on observations during training (i.e. by observing the weights and stochastic gradients). Incremental schemes monotonically increase the sparsity pattern, fully dynamic schemes can also reactivate previously pruned weights. In contrast to previous dynamic schemes that relied on elaborated heuristics to adapt the mask m,, we propose a simpler approach:
Dynamic pruning with feedback (DPF) Algorithm I.
Our scheme evaluates stochastic gradient at the prumed model W, = my w and applies it to the (simultancously maintained) dense model wy:
We+1 = W. vv>-gg(mm
7g m @ wx) = w, - -gg(- (g((w))
(DPF)
Applying the gradient the full model allows recover from "errors", i.e. prematurely masking out important weights: when the accumulated gradient updates from the following steps drastically change a specific weight, can become activated again (in contrast to incremental pruning approaches that have to stick to sub-optimal decisions). For illustration, observe that (DPF) can equivalently be written as
W++1 W, 7xg(w/ + ee);
where e, We- w, is the error produced by the compression. This provides a different intuition of the behavior of (DPF), and connects it with the concept of error-feedback (Stich et al. 2018 Karimireddy et al., 2019) We illustrate this principle in Figure 2 and give detailed pseudocode and [cF]scheme scheme can also be seen as an instance of more general class of schemes that apply (arbitrary) perturbed gradient updates the dense model. For instance straight-through gradient estimators (Bengio et al. (2033) that are used to empirically simplify the backpropagation can be seen as such perturbations. Our stronger assumptions on the structure of the perturbation allow to derive non-asymptotic convergence rates in the next section, though our analysis could also be extended to the setting (Yin et al. 2019) if the perturbations can be bounded.
further implementation details in Appendix IA. The [PF
4 CONVERGENCE ANALYSIS
We now present convergence guarantees for (DF)). For the purposes of deriving theoretical guar- antees, we assume that the training objective is smooth, that is [[V Vf(v)l S L ||w - v/l. Vw, v E ıRd, for a constant L > 0, and that the stochastic gradients are bounded E [lg(w:)]l < G2 for every pruned model W; m,, 3 W,. The quality of this pruning defined as the parameter 5, E [0, 1] such that
6, = |: w.ll [[w,]]
Pruning without information loss corresponds to w. = W,, i.e., 8, = O. and general 6, s 1.
Convergence on Convex functions. We first consider the case when f is in addition j-strongly convex, that is (V fV)), w v) < f(w f(v) - 2/2 [|l-|vll vvl?, Vw, E Rd. While it is clear that this assumption does not apply to neural networks, it eases the presentation as strongly convex functions have unique (global) minimizer w* arg min f(w).
Theorem 4.1. Let I be j-strongly convex and learning rales given as " -0(42 Then for a randomly chosen prumed model ü of the iterates (Wu...., r of DPF concretely u = w. with prob- ability Pt- (T-1)(1+2) it holds hhtt-in expectation over the stochasticity and the selection of ü:
2(t+1)
G2 WT + LE
Ef(ū] - f(w")(())))) =
8. [w..
The rightmost term in (2) measures the average quality of the pruning. However, unless 5; -> 0 or ||w/|l + 0 for f + xc, the error term never completely vanishes, meaning that the method converges only to a neighborhood of the optimal solution (this not only holds for the pruned model, but also for
'Our variable We corresponds to x; in the notation of Karimireddy et al. (0119). Their error-fixed SGD algorithm evaluates gradients perturhed iterates x; := x, + ee,, which correspond precisely to w, w, + our notation. This shows the connection of these two methods.
4

---

Published as a conference paper at ICLR 2020
the jointly maintained dense model, will show in the appendix). This behavior is expected, as the global optimal model w* might be dense and cannot be approximated well by a sparse model.
For one-shot methods that only prune the final (SGD) iterate Wr at the end, have instead:
EB((W) - f(w*) < 2B (J(r)) f(w")) + L 2 = LG 12T + LE or [
as we show in the appendix. First, we from this expression that the estimate is very sensitive to 8T and Wy, i.. the quality the pruning the final model. This could be better or worse than the average of the pruning quality of all iterates. Moreover, one looses also factor of the condition number &. in the asymptotically decreasing term, compared to ( This is due to the fact that standard convergence analysis only achieves optimal rates for average of the iterates (but not the last one). This shows
j
slight theoretical advantage of DPF over rounding at the end.
Convergence Non-Convex Functions to Stationary Points. Secondly, consider the case when f is non-convex function and show convergence (to a neighborhood) of a stationary point.
Theorem 4.2. Let learning rate be given 7
for c [(wo)- ((w))
Then for praned concretely := wi
LG
model u chosen uniformly at random from the iterates (wo...., WT of DPF
with probability pa T4I it holds in expectation over the stochasticity and the selection of ú.
( ( f(w")) + L 2 6g [wwll?]
EE v 12
VT
Extension to Other Compression Schemes. So far we put our focus on simple mask pruning schemes to achieve high model sparsity. However, the pruning scheme in Algorithm [ could be replaced by an arbitrary compressor C: Rd + RRd, i.e., We C(w,). Our analysis extends to compressors as e.g. defined (Karimireddy al. 2019 Stich & Karimireddy. 2019). whose quality is also measured in terms of the de parameters as in For example, if our objective was not to obtain a sparse model, but produce a quantized neural network where inference could be computed faster on low-precision numbers, then we could define C as quantized compressor. One variant of this approach implemented in the Binary Connect algorithm (BC) (Courbariaux et al. 2015) with prominent results, see also (Li et al. 2017) for further insights and discussion.
5 EXPERIMENTS
We evaluated DPF together with its competitors on wide range of neural architectures and sparsity levels. DPF exhibits consistent and noticeable performance benefits over its competitors.
5.1 EXPERIMENTAL SETUP
Datasets. We evaluated DPF on two image classification benchmarks: (1) CIFAR- 10 (Krizhevsky & Hinton 2009) (50K/10K training/test samples with 10 classes), and (2) ImageNet (Russakovsky
2015) (1 (1.28M/50K training/validation samples with 1000 classes). We adopted the standard
data augmentation and preprocessing scheme from He et al. (2016a) Huang et al. (0016); for further details refer to Appendix A.2
Models. Following the common experimental setting in related work on network pruning (Liu al. 2019 Gale et al. 2019 Dettmers & Zettlemoyer 2019 Mostafa & Wang 2019). our main experiments focus on ResNet (He et al. (2166a) and WideResNet (Zagoruyko & Komodakis (016). However, DPF can be effectively extended to other neural architectures, e.g., VGG (Simonyan 2015) DenseNet (Huang et al. (017). We followed the common definition in He et al. (0066)):Zagoruy
Komodakis (2016) and used ResNet-a, oe represent neural network with a layers and width factor b.
Baselines. We considered the state-of-the-art model compression methods presented in the table below our strongest competitors. We omit the comparison to other dynamic reparameterization methods, as DSR can outperform DeepR (Bellee et al. 2018) and SET Mocanu et al. 2018 by noticeable margin (Mostafa & Wang 2019).
Implementation DPF Compared to other dynamic reparameterization methods (e.g. DSR and SM) that introduced many extra hyperparameters, our method has trivial hyperparameter tuning
5

---

Published as a conference paper at ICLR 2020
Scheme
Reference
Pruming
How the maskis) are found
Lotkery Ticket (LT)
2019
FDRC
10- successive rounds of (Tull truining proming).
before training
SNIP
2019
LAT
By inspecting puperties/encativity of the network.
One-shot+ finc-tuning (One-shot P+FT)
2015
HPDT
after training
Saliency criterion (prunes smallest weights).
Incresmental pruning line turing (Incrermental)
2017 101750
ZG
incremental
Saliency criterion. Sparsily is gradually incremented.
Dynamic Sparse Reparameterization (DSR)
2019
MW
aame--rodirriltte-rrgroww cycle.
Sparse Momentum (SM)
2019
D7.
dynamic
Prune redistribute regrowit mean momentum.
DPF
ONPS
Reperameterization via error feedback.
overhead. We perform pruning across all neural network layers (no layer-wise pruning) using magnitude-based unstructured weight pruning (inherited from Han et al. (2015)). We found the best preformance when updating the mask every 16 iterations (see also Table 11) and we keep this value fixed for all experiments (independent of the architecture or task).
Unlike our competitors that may ignore some layers (e.g. the first convolution and downsampling layers in DSR), we [PP (as well as the One-shot P+FT and Incremental baselines) to all convolutional layers while keeping the last fully-connected laye biases and batch normalization layers dense. Lastly, our algorithm gradually increases the sparsity St of the mask from 0 to the desired sparsity using the same scheduling as in (Zhu & Gupta, 2017 see Appendix A.2
Training schedules. For all competitors, we adapted their open-sourced code and applied consistent (and standard) training scheme over different methods to ensure a fair comparison. Following the standard training setup for CIFAR-10, 10, we trained ResNet-a for 300 epochs and decayed the learning rate by 10 when accessing 50% and 75% of the total training samples (He et al. 2016a. Huang et al., 2017); and we trained WideResNet-cre as Zagoruyko & Komodakis (2016 6) for 200 epochs and decayed the learning rate by 5 when accessing 30%, 60% and 80% of the total training samples. For ImageNet training, we used the training scheme in (Goyal et al.) 2017) for 90 epochs and decayed learning rate by 10 at 30, 60, 80 epochs. For all datasets and models, wc used mini-batch SGD with Nesterov momentum (factor 0.9) with fine-tuned learning rate for DPF We reused the tuned (or recommended) hyperparameters for baselines (DSR and SM), and fine-tuned the optimizer and learning rate for One-shot P+FT, Incremental and SNIP. The mini-batch size is fixed to 128 for CIFAR-I0 and 1024 for ImageNet regardless of datasets, models and methods.
5.2 EXPERIMENT RESULTS
CIFAR-10. Figure 3 shows a comparison of different methods for WideResNet-28. For low sparsity level (e.g. 50%). DPF outperforms even the dense baseline, which is in line with regularization
properties of network pruning. Furthermore, DPF can prune the model up to a very high level (e.g. 99%), and still exhibit viable performance. This observation is also present in Table [ where the results of training different state-of-the-ar DNN architectures with higher sparsities are depicted. sPFshows shows reasonable performance even with extremely high sparsity level on large models (c.g. WideResNet-28-4 8 with sparsity ratio 99.9%), while other methods either suffer from significant quality loss or even fail converge.
DPF
Because simple model pruning techniques sometimes show better performance than complex tech- niques (Gale et al. 2019), we further consider these simple models in Table While[DPFJoutperoomm them in almost all settings, it faces difficulties pruning smaller models to extremely high sparsity ratios (e.g. ResNet-20 with sparsity ratio 95%). This seems however to be an artifact of fine-tuning, as DPF with extra fine-tuning convincingly outperforms all other methods regardless of the network size. This comes to no surprise as schemes like One-shot P+FT and Incremental do not benefit from extra fine-tuning. since it is already incorporated in their training procedure and they might become stuck in local minima. On the other hand, dynamic pruning methods, and in particular DPF work on different paradigm, and can still heavily benefit from fine-tuning 5
Figure 13 (in Appendix A.3.4 3. depicts another interesting property of DPF
When we search for a subnetwork with (small) predefined number of parameters for fixed task, it is much better to run
*The last fully- fullecco layer normally makes up only a very small faction of the total MACs, e.g. 0.05% for ResNet-50 on ImageNet and 0.0006% for WideResNet-255 28-2 on CIFAR-10. Beesddes that large fraction of the mask elements already converge during training (see e.g. Figure 4 heloow) all mask elements converge. Thus DPF can still benefit from fine-tuning on the fixed sparsity mask.

---

Published as a conference paper at ICLR 2020
Dorse madel
021
B51
Top-
2.5
0.6
0.7
2.6
0.3 10 0.2 02 2.6 6.
02
52
3A
26
06
1.
L2
CA 16
(a) Sparsity ratio top-1 I test acc. (b) # params v.s. top-1 test
(c) MAC v.s. top-1 test acc.
Figure 3: Top-1 test accuracy of NeteResNet-28-2 CIFAR-10 for unstructured weight pruning. The original model has 1.47M parameters with 216M MACs (Multiplier-ACe ACeuuiraaCC We varied the sparsity ratio from 50% to 99%. The complete numerical test accuracy values refer Table in Appendix [.33.]] The lower of params and MACs the model has, the higher sparsity ratio it uses. All results are averaged over three runs. Note that different methods might consider different types of layers and thus the same pruning sparsity ratio might result in the slight difference for both of # of params and MACs. The DSR cannot converge when using the extreme high sparsity ratio (99%).
Table I: Top-1 test accuracy of SOTA DNNs on CIFAR-10 unstructured weight pruning, We considered unstructured pruning and the indicates the method cannot converge. The results are averaged for three runs. The results we presented for each model consider some reasonable pruning ratios (we prune more aggressively for deeper and wider neural networks), and readers refer to Table 4 (in Appendix A.3.T for complete overview.
Methods
Baseline on model
SNIP
SM
DSR
Target Pr.
Model
(L* [2019
(DZ, 2019
(MW,2019 2019
DPF
VGG16-D
93.74 0.13
93.04 0.26
93.59 0.17
93.87
0.15
95%
91.10 0.22
91.98 0.01
+ 92.00
92 42 -
70%
ResNet-2D
92.48 0.20
90.53 #: 0.27
91.54 0.16
91.78 :: 0.28
92.17 : 0.21
80%
88.50 +: 0.13
89.76 0-40
87.88 + 0.04
90.88 + 0.07
90%
0.74
95%
ResNet-32
93.83 0.12
90.40 0.26
91.54 + 0.18
91.41 + 0 0.23
92 42 0.18
90%
87.23 : 0.29
88.68 + 0.22
81.12 = 0.32
90.94 = 0.35
95%
ResNet-56
94.51 +: 0.20
91.43 +: 0.34
92.73 0.21
93.78 = 0.20
93.95 : 0.11
90%
90.96 0.40
92.37
92.74
95%
92.58 + 0,22
93.41 0.22
93.88 = 0.08
94.36 = 0.24
90%
WideResNet-28-2
95.01 + 0.04
90.80 0.04
92.24 0.14
92.74 + 0.17
93.62 + 0.05
95%
83.45 0.38
85.36 0.80
99%
93.62 + 0.17
94.45 +: 0.14
94.63 = 0.08
95.38 + 0.04
95%
WideResNet-28-4
95.69 0.10
92.06 0.38
93.80 +: 0.24
93.92 = 0.16
94.98 = 0.08
97.5%
89.49 0.20
0.04
92.50
99%
95.49 0.21
96.67 0.14
95.81 + 0.10
96.08 + 0.15
90%
94.92 1 0.13
95.64 0.07
95.55 = 0.12
95.98 : 0.10
95%
WideResNet-284
96.06 0.06
96.31
92.01 : 0.11
94.38 0.12
91.10 = 0.12
95. 63 + 0.16
99%
74.50
2.23
88.65 = 0.36
91.76 0.18
99.9%
DPF on a large model (e.g. WideResNet-28.. than on smaller one (e.g. WideResNet-28-2), That
1s, DPF performs structural exploration more efficiently in larger parametric spaces.
Table 2: Top-1 test accuracy of SOTA DNNs CIFAR-10 for unstructured weight pruning via some simple pruning techniques. This table complements Table [ and evaluates the performance of model compression under One-shot P+FT and Incremental, as well how extra fine-tuning (FT) impact the performance Incremental and our DPF Note that One-shot P+FT prunes the dense model and uses extra fine- tie-tuning itself. The Dense, In- cremental and DPFitrain with the same number of epochs from scratch. The extra fine-tuning procedure considers the model checkpoint the end of the normal training. the number of training epochs (60 epochs in case) with tuned optimizer and learning Detailed hyperparameters tuning procedure refers Appendix A.2
and
Methods
Model
Baseline on dense model
One-shat (H1, (2015
Incremental (7G. (7G_2011)
Incromental
DPF
Target pc.
ritio
ResNet-20
12 48 0.20
86.91 à 0.16
90.18 0.12
90.55 2: 0.38
80.21 2: 0.10
$0.24 à: 0.28
90.54 0.25
90.88 0.07
88.01 1:01::000 0.30
90.34 =: 0.31
91. 2: 0.12
95%
90%
ResNet-32
83.83
89.31 0.18
90.86 0.17
90.33 0.18
00.94 0.35
0.14
ResNet-56
B4.51 0.20
91.61 0.07
93.26 0.06
92.14
93.14 0.23
82..50 0.25
90.09 0.16
93.
92.74 0.08
0.11
93.95 d: 0.17
01.15
95%
90%

---

Published as a conference paper at ICLR 2020
ImageNet. We compared DPF to other dynamic reparameterizatioo methods well as the strong Incremental baseline in 3 For both sparsity levels (80% and 90%), [PF] shows a significant improvement of top-1 test accuracy with fewer or equal number of parameters.
Table 3: Top-1 I test accuracy of ResNet-50 on ImageNet for unstructured weight pruning. The # of parameters for the full model is 25,56 M. We used the results of DSR from Mostafa & Wang (2019) as we use the same (standard) training/data augmentation scheme for the same neural architecture. Note that different methods prune different types layers and result
different # of parameters for the same target pruning ratio. We also directly (and fairly) compare with the results Incremental (7hu & Guptal (2077) reproduced and fine-tuned by Gale et al. (2019), where they consider layer-wise sparsity ratio and fine-tune both the sparsity warmup schedule and label- for better performance.
Top-1 accuracy
Top-5 accuracy
Pruning ratio
Method
Dense
Pruned
Difference
Difference
Target
Reached
semaining params
Incremental (2G 2017
75.95
74,25
-1.70
92.91
91.84
-1.07
80%
73.5%
6.79 M
DSR (MW. 2019
74.90
73.30
-1.60
92.40
92.40
80%:
71.4%:
7,30
SM (ZZ 2019
75.95
74.59
-1.36
92.91
92.37
-0.54
808
72.4%
7,06
DPF
75.95
75.48
-0,47
92.91
92.59
-0.32
80%
73.5%
6,79 M
Incremental
Gale et al. 2019
76.69
75.58
-1.11
80%
79.9%
5.15
DPF
75.95
75.13
-0.82
92.91
92.52
-0.39
80%
79.9%
5.15 M
Incremental (ZG. 2017
75.95
-2.59
92.91
DSR (MW 2019
74.90
71.60
-3.30
92.40
90.50
-1.90
90%
80.0%:
5.10 M
(b7 2019
75.95
72.65
-3.30
92.91
91.26
-1.65
90%
82.0%
4.59
DPF
75.95
74.55
-1.44
92.91
92.13
-0.78
90%
82.6%
4.45
DISCUSSION
Besides the theoretical guarantees, straightforward benefit DPF over one-shot pruning in practice its fine-tuning free training process. Figure 12 in the appendix (Section A.3.3 demonstrates the trivial computational overhead (considering the dynamic reparameterization cost) of involving DDF to train the model from scratch. Small number of hyperparameters compared to other dynamic reparameterization methods (e.g. DSR and SM) is another advantage of DPFF and Figure 11 further studies how different setups [PF]jmpaat the final performance. Notice also that for[DPF] inference is done only at sparse models, aspect that could be leveraged for more efficient computations,
Empirical difference between one-shot pruning and DPF From the Figure one can see that DPFitends to oscillate among several local minima, whereas one-shot pruning, even with fine tuning, converges to a single solution, which is not necessarily close to the optimum. We believe that the wider exploration of DPF) helps to find a better local minima (which can be even further improved by fine-tuning, as shown in Table@]. We empirically analyzed how drastically the mask changes between each reparameterization step, and how likely it is for some pruned weight to become non-zero in the later stages of training. Figure 4 shows at what stage of the training each element of the final mask becomes fixed. For each epoch, we report how many mask elements were flipped starting from this epoch. As an example, we see that for sparsity ratio 95%, after epoch 157 (i.e. for 43 epochs left), only 5% of the mask elements were changing. This suggests that, except for small percentage of weights that keep oscillating, the mask has converged early in the training. In the final epochs, the algorithm keeps improving accuracy, but the masks are only being fine-tuned. similar mask convergence behavior can be found in Appendix (Figure 7 for training ResNet-20 on CIFAR-10.
% of params not converged
100
Zoom in, opochs 120-200
75
10%
50
0.8
0.96
0.8
0
25
50
75
100
125
150
175
200
Epoch
Figure 4: Convergence of the pruning mask m. of DPF for different target sparsity levels (see legend). The y-axis represent the percentage of mask elements that still change after certain epoch (z-axis). The illustrated example are from WideResNet-228 on CIFAR-10 10. We decayed the learning rate at 60,120,160 epochs.
8

---

Published as a conference paper at ICLR 2020
DPF does not find lottery ticket. The LT hypothesis (Frankle & Carbin 2019) conjectures that for every desired sparsity level there exists a sparse submodel that can be trained to the same or better performance the dense model. In Figure we
we show that the mask found by DPF is not LT, iee, training the obtained sparse model from scratch does not recover the same performance, The (expensive) procedure proposed in Franklee & Carbin (20T9); Frankle et al.) (2019) finds different
masks and achieves the same performance as DPF for mild sparsity levels, but DPF is much better
for extremely sparse models (99% sparsity).
lottery ticket extra full training rounds
-22x
Top test accuracy
92
train ginal Init (mask from DPF)
-15x
re-d ain wis original init (mask from DIF))
90-
ticket training
12x
88-
9x
-xx -x
0.50
0.60
0.70
0.80
0.90
0.95
0.99
Target sparsity ratio
Figure 5: Top-1 test accuracy for different target sparsity levels (on WideResNet-28-22 with CIFAR-10, structured pruning). [PF]reacher reaches comparable accuracy than the LT training method (and better 99% target sparsity), but involves much less computation (right y-axis, green). Training the sparse models found by DPF from scratch does not reach the same performance (hence our sparse models are not lottery tickets).
Extension to structured pruning. The current offte-of-the-art dynamic reparameterization methods only consider unstructured weight pruning. Structured filter pruning is either ignored (Bellec et al. 2018 Mocanu et al. 2018 Dettmers Zettlemoyer 2019) or shown to be challenging (Mostafa & Wang 2019) even for the CIFAR dataset. In Figure 6 below we presented some preliminary results on CIFAR-10 10 to show that our DPF can also be applied to structured filter pruning schemes. DPF outperforms the current filter-norm based state-of-the-ar method for structured pruming (e.g. SFP He et al. 2018) by a noticeable margin. Figure in Appendix [A.4.3 displays the transition procedure of the sparsity pattern (of different layers) for WideResNet-259 with different sparsity levels. DPF can be seen as a particular neural architecture search method, as it gradually learns prune entire layers under the guidance of the feedback signal.
We followed the common experimental setup as mentioned in Section 5 with f2 norm based filter selection criteria for structured pruning extension CIFAR-10. We do believe better filter selection scheme (Ye et al. 2018 8 He et al. [0199] Lym et al. 2019) could further improve the results but we leave this exploration for the future work.
Top 2aa
65.0
64.5
54.2
0.4
2.5
0.5
3.2
3-2
14
0.52
0.75
1.00
3.25
150
1.75
2,00
225
MAC
MAG
MAC
(a) 288
(b) 2844
(c) WideResNet-28-4
Figure 6: MAC v.s. top-1 test accuracy, for training WedeResNNt-288 (with different width) on CIFAR-10. The reported results are averaged over three runs. The WideResNet-228 has 216M MACs, WideResNet-255 has 848M MACs and WideResNet-28-8 has 3366M MACs. Other detailed information refers to the Appendix e.g., the # of params v.s. top-1 test accuracy in Figure [4] and the numerical test accuracy score in Table
A.4. A
Lym et al, (2019) consider structured filter pruning and recontigure the large (but sparse) model to small (but dense) model during the training for better training efficiency. Note that they perform model update on gradually reduced model space, and it is completely different from the dynamic reparameterization methods (e.g. DSR, SM and our scheme) that perform reparameterization under original (full) model space.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
We acknowledge funding from SNSF grant (00021_17779,, as well as Google Focused Research Award.
REFERENCES
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In ICLR International Conference Learning Representations, 2018. URL httpp:://ppprreiiissss net forum? id=BB__NOIC-- wND1C-
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3338.. 2013.
Miguel A Carreira-Perpinán and Yerlan Idelbayev. "leamnig--compressonn algorithms for neural net pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2018.
Matthieu Courbariaux, Yoshua Bengio, and Jean- David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurlPS Advances in Neural Information Processing Systems, pp. 3123-3131, 2015.
Xiaoliang Dai, Hongxu Yin, and Niraj Jha. Nest: A neural network synthesis tool based on grow-and-ping prune paradigm. IEEE Transactions on Computers, 2019.
Deng, W. Dong, R. Socher, L.-J. Li, Li, and Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPRO9, 2009.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04458 0..00040444 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR International Conference on Learning Representations, 2019. URL
httts:://ppenreve/ net/ 2teee//oorumm11==113333333
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin, Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.0161 2019.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. NeurIPS Advances in Neural Information Processing Systems, pp. 3581-3590, 2017.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv: 170.02770 2017.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In NeurlPS- Advances in Neural Information Processing Systems, pp. 1379-1387, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS Advances in Neural Information Processing Systems, pp. 1135 1135-1143, 2015.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Shijian Tang, Erich Elsen, Bryan Catanzaro, John Tran, and William Dally. DSD: regularizing deep neural networks with sease-spareeeeesse training flow. In ICLR International Conference on Learning Representations, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 20.
10

---

Published as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630-645. Springer, 2016b.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. International Joint Conference on Artificial Intelligence (IJCAI), pp. 2234-2222222 2240, 2018.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Sai Praneeth Karimireddy, Quentin Rebjock, Sehastian Stich, and Martin Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. In International Conference on Machine Learning, pp. 3252-3261, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Simon Lacoste-Julien, Mark W. Schmidt, and Francis R. Bach. A simpler approach to obtaining an (1/t)) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002. 2012.
Yann LeCun, John Denker, and Sara A Solla. Optimal brain damage. In NeurlPS Advances in Neural Information Processing Systems, pp. 598 (188605,111 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. SNIP: Single-shot network pruning based on connection sensitivity. In ICLR International Conference on Learning Representations, 2019.
Hao Li. Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In NeurIPS. Advances in Neural Information Processing Systems, pp. 581 1-8811, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR International Conference on Learning Representations, 2019.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through [_ regularization. In ICLR International Conference on Learning Representations, 2018. URL
htttps://openre///
net forum [.d-H188hh@@F
Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Mattan Erez, and Sujay Shanghavi. Prunetrain: Gradual structured pruning from scratch for faster neural network training. In Inter- national Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2019.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML International Conference Machine Learning, pp.
JMLR. org,
2017.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML International Conference on Machine Learning, 2019.
11

---

Published as a conference paper at ICLR 2020
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In NeurlPS Advances in Neural Information Processing Systems, pp. 107-115, 1989.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. In ICLR International Conference on Learning Representations, 2017.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In NeurlPS Advances in Neural Information Processing Systems, pp. 6775-6784, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)2111255 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR International Conference Learning Representations, 2015.
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1888-15, 2017.
Sebastian U. Stich and Sai Karimireddy. The crror-feedback framework: Better rates for SGD with delayed gradients and compressed communication. arXiv preprint arXiv:I909.05:0 05350, 2019, URL https: //ta:////xxvvv /ttp:///xxxxv.or/ab//o//// org/abs////ooooooooooo...
Sebastian Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with mem- ory. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), NeurlPS Advances in Neural Information Processing Systems, pp. 4447-4458. Curran Associates, Inc., 2018. URL http:://apeet: ni wwtpperrs.ips.cc/pppeep fied- sgd- woroddwwt--mmor-ppppe
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking
as- sumption in channel pruning convolution layers. ICLR- International Conference on Learning Representations, 2018. URL
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understand- ing straight-through estimator in training activation quantized neural nets. In ICLR International Conference Learning Representations, 2019.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In NeurIPS Advances in Neural Information Processing Systems, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy pruning for model compression. arXiv preprint
2017.
12

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1 ALGORITHM
Algorithm 1 The detailed training procedure of DPF
imput: uncompressed model weights w R² pruned weights: w, mask: E {0, 13" reparametrization period: p. training iterations: T.
1: for 0,..., do
2: if then
trigger mask update, per default every 16 iterations compute mask m m.(ws) by arbitrary pruning scheme (c.g. unstructured magnitude pruning) end
3:
5: W. < m
> apply (precomputed) mask forward/hackward pass with pruned weights w. via arbitrary optimizer (e.g. SGD with momentum)
6: compute (mini-batch) gradient g(w)
7: We41 gradient update B(W)
end for
output: Wr and Wr
We trigger the mask update every p = 16 iterations (see also Figure [T] and we keep this parameter fixed throughout all experiments, independent of architecture or task.
We perform pruning across all neural network layers (no layer-wise pruning) using magnitude- based unstructured weight pruning (inherited from Han et al. (2015). Pruning is applied to all convolutional layers while keeping the last fully-connected layer, biases and batch normalization layers dense.
We gradually increases the sparsity of the mask from 0) to the desired sparsity using the same scheduling as in Zhu & Gupta (2017); see Appendix A.2 below.
A.2 IMPLEMENTATION DETAILS
We implemented our DPF PyTorch (Paskke et al. 2017). All experiments Tesla V100 GPUs Sparse tensors our implementation are respresented as the dense tenors multiplied by the corresponding binary masks.
NVIDIA
Datasets evaluate all methods on the following standard image classification tasks:
Image classification for CIFAR-10 (Krizhevsky Hinton 2009). Dataset consists of training set of 50K and test set of 10K color images of 32 x 32 pixels, well as 10 target classes. We adopt the standard data augmentation and preprocessing scheme (He et al.
2016a; Huang et al.) 2016).
Image classification for ImageNet (Russakovsky et al. 2015). The ILSVRC 2012 classi- fication dataset consists of 1.28 million images for training. and 50K for validation, with 1K target classes. We use ImageNet- Ik (Deng et al.) 2009) and adopt the same data pre- processing and augmentation scheme in He et al. (2016a b): (Simonyan & Zisserman 2015).
Gradual Pruning Scheme For Incrementall bascline, tuned their automated gradual pruning [- scheme 8, = + (s, 8y) (1 nAr to gradually adjust the pruning sparsity ratio 8; for e {to...., to n.)). That is, in our setup, increased from initial sparsity ratio s; 0 to the desired target model sparsity ratio S over the epoch (n) when performing the second learning rate decay, from the training epoch to 0 and with pruning frequency At epoch. In our experiments, we used this gradual pruning scheme over different methods, except One-shot P+FT, SNIP, and the methods (DSR, SM) that have their fine-tuned gradual pruning scheme.
Hyper--parmeete tuning procedure We grid-searched the optimal learning rate, starting from the range of (0.05, 0.10. 0.15, 0.20]. More precisely, we will evaluate a linear-spaced grid of learning rates. If the best performance was ever at one of the extremes of the grid, we would try new grid points that the best performance contained in the middle of the parameters.
13

---

Published as a conference paper at ICLR 2020
We trained most of the methods by using mini-batch SGD with Nesterov momentum. For baselines involving fine-tuning procedure (e.g. 2 we grac-searehed the optimal results by tuning the optimizers (i.e. mini-bateh SGD with Nesterov momentum, Adam) and the learning rates.
The optimal hyper- sypee-parameters for DPF The mini-batch size is fixed to 128 for CIFAR- 10 and 1024 for ImageNet regardless of datasets and models.
For CIFAR-10, we trained ResNet-a and VGG for 300 epochs and decayed the learning rate by 10 when accessing 50% and 75% of the total training samples (He al. 201 6a Huang et al. 2017); and we trained WideResNet-a-b as Zagoruyko & Komodakis (2016) for 200 epochs and decayed the learning rate by 5 when accessing 30%, 60% and 80% the total training samples. The optimal learning rate for ResNet-a, WideResNet-a-b and VGG are 0.2, 0.1 and 0.2 respectively; the corresponding weight decays are le-4 5e and le me-4- respectively.
For ImageNet training, used the training scheme in Goyal al. (2017) for 90 epochs, where gradually warmup the learning rate from 0.1 to 0.4 and decayed learning rate by 10 at 30; 60; 80 epochs. The used weight decay is 4.
A.3
ADDITIONAL RESULTS FOR UNSTRUCTURED PRUNING
A.3.1 COMPLETE RESULTS OF UNSTRUCTURED PRUNING ON CIFAR-10
Table details the numerical results for training SOTA DNNs on CIFAR-10. Some results of reconstruct the Table I and Figure 3
Table 4: Top-1 I test accuracy for training (compressed) SOTA DNNs on CIFAR-10 from scratch. We considered
unstructured pruning and the indicates the method cannot converge. The results are averaged for three runs.
Methods
Baseline on dense model
SNIP [2119]
SM (DZ,
DSR
Target Pr.
Model
(DZ2 2019
(MW, 2019
DPF
VGG16-D
93.74 0.13
93.04
0.26
93.59 0.17
93.87 : 0.15
95%
ResNet-20
92.48
91.10
0.01
92.00
ResNet-20
92.48 + 0.220
90.53 : 0.27
91.54 0.16
91.78 : 0.28
92.17 : 0.21
80%
ResNet-20
92.48 0.20
88.50 0.13
89.76 0.40
87.88 + 0.04
90.88
0.07
90%
ResNet-20
92.48 0.20
0.25
83.03
95%
ResNet-32
93.83 0.12
90.40 0.26
91.54 0.18
91.41 + 0.23
92.42 + 0.18
90%
ResNet-32
93.83 + 0.12
87.23 :: 0.29
88.68 +: 0.22
84.12 = 0.32
90.94 = 0.35
95%
ResNet-56
94.51 + 0.22
91.43
0.34
92.73 0.21
93.78 = 0.20
93. : 0.11
90%
RexNet-56
94.51 :: 0.20
90.96 0.40
92.57 0.09
92.74 : 0.08
95%
WideResNet-28-2
96.01 + 0.04
94.67 + 0.23
94.73 + 0.16
94,80 = 0.14
95. 11 #: 0.06
50%
WideResNet-288
95.01 +: 0.04
94.47 0.19
94.65 0.16
94.98 : 0.07
94.90 0.06
60%
WideResNet-28-2
95.01 + 0.04
94.29
0.22
94.46 0.11
94,80 0.15
94.86 0.13
70%
WideResNet-28.2
95.01 - 0.04
93.56
1: 0.14
94.17 0.12
94.37 = 0.13
94.76 : 0.18
80%
WideResNet-28-2
95.01 0.04
92.58
93.88
90%
WideResNet-28-2
95.01 = 0.04
90.80 0.04
92.24 1: 0.14
92.74 = 0.17
93. 62 1: 0.05
95%
WideResNet-28.
0.04 95.01
83.45 0.38
0.80 85.36
88.92 0.29
99%
WideResNet-28-4
95.69 0.10
95.42 0.05
95.57 0.08
95.67 = 0.07
95.58 0.21
70%
95.69 0.10
95.24 0.07
96.27 0.02
95.49 + 0.04
95.60 + 0.08
80%
WideResNet-28-4
95.69 0.10
91.56 2: 0.11
95.01 0.05
95.30 = 0.12
95.65 - 0.14
90%
WideResNet-28.4
95.69
94.63
WideResNNet22--4
96.69 0.10
92.06 + 0.38
93.80 0.24
93.92 = 0.16
94.98 + 0.08
97.5%
WideResNet
95.09 0.10
89.49 0.20
92.18 0.04
92.50 + 0.07
93.86 + 0.20
99%
WideResNet-28-8
96.06 0.06
95.81 0.05
95.92 0.12
96.06 - 0.09
70%
WideResNet-28-8
96.06 - 0.06
95.86 0.10
96.97 0.05
96.05 = 0.12
80%
WideResNes-28-8
96.06 0.06
95.49 0.21
95.67 0.14
95.81 - 0.10
96.08 + 0.15
90%
WideResNet-28-8
96.06 0.06
0.13
95.61 0.07
95.55
95.98 0.10
95%
WideResNet-22-88
96.06 + 0.06
94.11 + 0.19
96.31 0.20
96.11 + 0.07
95.84 + 0.04
97.5%
WideResNet-28-8
96.06 1.0:: 0.06
92.01 0.11
94.38 0.12
91.10 0.12
95. 63 -
99%
WidbeResNet28888
96.06 + 0.06
74.50 2.23
88.65 0.36
91.76 0.18
99.9%
A.3.2 UNDERSTANDING THE TRAINING DYNAMICS AND LOTTERY TICKET EFFECT
Figure 7 and Figure complements Figure 4 and details the training dynamics (e.g. the converge of 8 and masks) of DPF from the other aspect. Figure 9 compares the training dynamics between DPF
14

---

Published as a conference paper at ICLR 2020
and Incremental (Zhu & Gupta 2017), demonstrating the fact that scheme enables drastical reparameterization over the dense parameter space for better generalization performance.
% of params not converged
20%
Zoom in, epochs 200-300
75
50
200
250
300
25
0.7
0.9
100
150
250
Epoch
Figure 7: Convergence pruning mask m, of DPF for different target sparsity levels (see legend). The y-axis represent the percentage of mask elements that still change after certain epoch (z-axis). The illustrated example are from ResNet-20 on CIFAR-10. We decayed the learning rate at 150 and 225 epochs.
1.00
Maak 1 veto
0.60
0.4
0.02
0.22
0.2
0.00
25
50
75
200
125
150
175
200
25
50
75
100
123
350
175
202
30
75
130
125
150
200
Boodh
(a) 5 v.s. epoch.
(b) Mask flip ratio v.s. epoch.
(c) IoU v.s. epoch.
Figure 8: Training dynamics of DPF (WideResNet-22 28-2 on CIFAR- 10) for unstructured pruning with different sparsity ratios. IoU stands for Intersection over Union for the non masked elements of two consecutive masks; the smaller value the more fraction of the masks will flip.
DPF
Incremental
DPF Incremental
mask not converged
100%
100%
0.5
0.8
40%
0.6
0.9
75%
75%
0.7
0.95
30%
50%
50%
20%
25%
10%
0%
0%
0%
0
50
100
150
200
50
100
150
200
Epoch
Epoch
Epoch
Figure 9: Convergence of the pruning mask m, of DPF V.S. Incremental (Zuu & Gupta 2017) for different
target sparsity levels (see legend). The y-axis represent percentage mask elements that still change after a certain epoch (æ--axis). The illustrated example are from WedcRessett28-22 on CIFAR-10. We decayed the lcarning rate at 60; 120; 160 epochs. These two schemes use the same gradual warmup schedule (and (yper--arameters) for the pruning ratio during the training.
Figure i0 addition the Figure (in the main text) further studies the lottery ticket hypothesis under different training budgets (same epochs or same total flops). The results of DfF also demonstrate the importance of training-time structural exploration as well as the corresponding implicit regularization effects. Note that we do not want to question the importance of the weight initialization or the existence of the lottery ticket. Instead, our DPF can provide an alternative training scheme to compress the model to an extremely high compression ratio without sacrificing the test accuracy. where most of the existing methods still meet severe quality loss (including Frankle & Carbin (2019)
Liu et al. (2019); Frankle et al. (2019))
A.3.3 COMPUTATIONAL OVERHEAD AND THE IMPACT OF HYPER-PARE PARAMMETRS
In Figure [T we evaluated the 1 test accuracy of a compressed model trained by DPF under dif- ferent setups, e.g., different reparameterization period p, different sparsity ratios, different mini-batch
15

---

Published as a conference paper at ICLR 2020
95.0
Top-1 test accuracy
94
Top-1 test accuracy
92
94.5
90
94.0
88
DFF
93.5
DPF
re-train original (mask from
train original (mask from DPF) re-trein w/o original init Imask from DPF)
re-train wio original init (mask from DPF)
93.0
0.50
0.60
0.70
0.80
0.90
0.95
0.99
0.50
0.60
0,70
0.80
Sparsity ratio
Sparsity ratio
(a) Retraining with same epoch.
(b) Retraining with same flops.
Figure 10:
Investigate the effect of lottery ticket for model compression (unstructured weight pruning for WidcResNet28-22 2 on CIFAR- 10). It complements the observations in Figure by retraining the model for the same amount of computation budget (i.c. flops).
sizes, as well as whether layer-wise pruning not. We can witness that the optimal reparameter- ization (i.e p = 16) is quite consistent over different sparsity ratios and different mini-batch sizes, and we used it in all our experiments. The global-wise unstructured weight pruning (instead of layer-wise weight pruning) allows our DPF more flexible to perform dynamic parameter reallocation, and thus provide better results especially for more aggressive pruning sparsity ratios. However, we also need to note that, for the same number of compressed parameters (layerwise or globalwise unstructured weight pruning), using global-wise pruning leads to slight increase in the amount of MACs, as illustrated Table 5
Reperaateiitatin trequancy
8.25 0.1---2---11
022 8:22 344
-02
A/
R0:01
R7.01
92.20
0.28 1.0--4.4--.008 0:18
10-00.2
50
91:05
8.20 020 0.04 0:00 0.00
0066
00:03
12
06
91
40,02 42.41 30.21
00.00
32
0.88
91.75
0012-0100 00:00:0
1:30
-1.0
12
0.5 0.5 0.7
0.8 0.9
0.96
Sparsily
Batch
(a)
Reparameterization period vs.
(b) Reparameterization period mini batch size. The heatmap dis-
(c) Reparameterization scheme (whether layerwise) vs. sparsity ratio. The heatmap displays the test accuracy. We use mini-batch size 128 w/ reparamcterization period p p= 16.
sparsity ratio. The heatmap value the test accuracy minus the best accu- racy of the corresponding sparsity. We use mini-batch size 1 28 w/o layerwise reparameterization.
plays the test accuracy. We use spar-
sify ratio 0.80.
Figure 11: Investigate how the reparameterization period/scheme and mini-batch size impact the generalization performance (lest top-1 accuracy), for dynamically training (and reparameterizing) compressed model from scratch (ResNet-20 with CIFAR-10).
Table 5: Investigate how the reparameterization scheme (layer-wise not) impact the MACs (for the same number of compressed parameters), for using DPF on ResNet-20 with CIFAR-I0.
Target sparsity
50%
60%
70%
80%
90%
95%
w/o layerwise reparameterization w/ layerwise reparameterization
22.80M
19.13M
15,18M
11.12M
6,54M
4.02M
20.99M
16.91M
12.83M
8.75M
4.67M
2.63M
Figure 12 demonstrates the trivial computational overhead of involving DPF to gradually train a compressed model (ResNet-50) from scratch (on ImageNet). Note that we evaluated the introduced reparameterization cost for dynamic pruning, which is independent of (potential) significant system speedup brought by the extreme high model sparsity. Even though work did not estimate the practical speedup. we do believe we can have a similar training efficiency as the values reported in Dettmers & Zettlemoyer 20099)
16

---

Published as a conference paper at ICLR 2020
Dense
60
Training loss
Test Test top-1
40
20
Dense
20000
600D0 80000 100000 120000
60000 80000 100000 120000
Time (s)
Time
The learning curve of our DPF (with unstructured magnitude pruning) and the standard mini-bater SGD for training ResNet50 ImageNet. Our proposed has DPFjas trivial computational overhead. We trained ResNet50 4 NVIDIA V100 GPUs with 1024 mini-batch size. The target sparsity ratio is 80%.
Figure 12:
A.3.4 IMPLICIT NEURAL ARCHITECTURE SEARCH
DPF can provide effective training- time structural exploration or even implicit neural network search. Figure 13 below demonstrates that for the same pruned model size (i.e. any point in the æ-axis), we can always perform "architecture search" to get a better (in terms of generalization) pruned model, from a larger network (e.g. WideResNet-288 rather than the one searched from a relatively small network (e.g. 2842)4.)
96
Top-1 test accuracy
WideResNet28-2 8-2
90
WideResNet2844
WideResNNet2--8
1.5
2.0
# of params (M)
top-1 accuracy vs. the compressed model size, training WideResNet-28 (with different widths) on CIFAR-I0. The compressed model is searched from WideResNet-28 (fixed depth) with different width (number of filters per layer).
Figure 13:
A.4 ADDITIONAL RESULTS FOR STRUCTURED PRUNING
A.4.1
GENERALIZATION PERFORMANCE FOR CIFAR-10
complements the results of structured pruning in the main text (Figure 6). and Table 6 details the numerical results presented both Figure and Figure [4
Figure 14
26.2
25.3
15.0
Top
24.3
(a) 2ideResNNe18-2..
(b) WideResNet28-4.
(c) WideResNer18-8.
Figure 14: of params v.s. top-1 test accuracy, for training WideResNet28 (with different width) on CIFAR-10. Structured filter-wise pruning used here. The reported results are averaged over three runs.
17

---

Published as a conference paper at ICLR 2020
Table 6: Performance evaluation DPFF (and other baseline methods) for training Resee) variants CIFAR-10. We use the norm-based criteria for filter selection (as in SFP (He al. 2018 to estimate output channel-wise pruning threshold. We follow the gradual pruning warmup scheme (as in Zhu & Gupta ( from epoch to the epoch when performing the second learning rate decay. Note that SFP prunes filters within Prrprees
the layer by given ratio while our DPF prunes filters across layers. Duc the difference between filters for different layers, the # of parameters pruned by DPF might slight different from the one pruned by SFP. The pruning ratio refers either prune filters within the layer or across the layers.
Methods
Baseline dense model
SEP
Target ratão
Model
(H* (H112008)
0.31
RosNes-20
92.48 + 0:20
91.12 0.20
91 190
0.06
20%
ReiNes-20
92.48 + 0.20
90.32 + 0.25
91. 07
0.40
30%
RosNe-20
ResNet32
93.52 +: 0:13
02.07 +: 0.22
92.
0.16
30%
RosNes-32
93.02 + 0:13
91.14 + 0.46
91. 1..60-1
0:21
40%
ResNet-56
94.51 1: 0.20
90.99 1: 0.27
94.53
1: 0.13
30%
ResNet-56
0:20
95.01 1: 0.01
94.02 :: 0.24
91.52
1: 0.08
40%
WideResNet-28. 28.2
95.01
+ 0.04
90.34 = 0.14
94.11 = 0.112
50%
WideResNet-266 28-2
95.01 0.0.4
92.07 0.09
93.74
0.25
60%
WideResNet-28-2
WideResNet-28.2
55.01
+: 0.04
86.00 =: 1.00
90.53 =: 0.11
80%
WideResNet-28. 4
95.69
+ 0.10
96.15 0.11
95. 50 =: 0.005
40%
WideResNet-222 28-4
95.09
4: 0.10
94186 1: 0.10
95.43 0.16
50%
WideResNet-28--
95.60
0.10
95, 0.005
WideResNett28-44
$5.89
+: 0.10
93.37 =: 0.18
94.67 :: 0.08
700
NetteResNe-28844
95.68
0.10
91.88 0.59
933. 038.79-11 + 0.00
80%
WideResNat-28-8
96.06 DOR
0.04
NideResset-28-9
0.06
0.00
96. 0.002
50%
Na1--8e
96.06 +: 0.06
96.40
0.14
95. 88 0.16
60'S
WideReeNNtt2884
96.06 + D.06
94.50 + 0.22
95.77 71.77- + 0.11
70%
WideResNet-28-8
0.21
A.4.2 UNDERSTANDING THE LOTTERY TICKET EFFECT
Similar to the observations in Section 6 (for unstructured pruning), Figure 15 instead considers structured pruning and again we found DPF does not find lottery ticket. The superior generalization performance DFF cannot be explained by the found mask or the weight initialization scheme.
Top-1 test accuracy
94
93
92
DPF
re-train w/ original init (mask from DPF)
re-train w/o original init (mask from DPF)
0.40
0.50
0.60
0.70
0.80
Sparsity ratio
Figure 15: Investigate the effect of lottery ticket for model compression (WideResNet28-22 with CIFAR-10 for structured pruning. We retrained the model with the mask from the model trained by DPF by using the same epoch budget.
A.4.3 MODEL SPARSITY VISUALIZATION
Figure below visualizes the model sparsity transition patterns for different model sparsity levels under the structured pruning. We can witness that due to the presence of residual connection, DPF gradually learns to prune the entire residual blocks.
18

---

Published as a conference paper at ICLR 2020
tosppit Sommens Lerre inkrervvee
Vagge foctore Spare Deraa anwoo
140 coreue
(a) Structured pruning with 40% sparsity.
(b) Structured pruning with 50% sparsity.
(c) Structured pruning with 60% sparsity.
(d) Structured pruning with 70% sparsity.
(e) Structured pruning with 80% sparsity.
Figure 16: The element- wise sparsity of each layer for WideResNet28-2 (trained on CIFAR-10 via DPF under
different structured pruning sparsity ratios. DPF for model compression (with structured pruning) performs implicit as a neural architecture search.
19

---

Published as a conference paper at ICLR 2020
B MISSING PROOFS
In this section we present the proofs for the claims Section
First, we give the proof for the stongly convex case. Here we follow Lacoste-Julien et al. (2012) for the general structure, combined with estimates from the error-feedback framework (Stich et al. 20118] Stich & Karimireddy 2019) to control the pruning errors.
Proof of Theorem
4.1 By definition of (DF)) We+1 = W. - -)g(w,), hence,
E
[
11 Wi
|w: w" 2%x (w, - w', "",))(w.)) + 2 (lg(w)))
I |W:- - -w'l1 -27: (wr - -w", (V(W)) + G2
ww w*1l² -27h (wr-w -w", Vs((w)) G
+ 2% (We We. V.((w)))
By strong convexity,
-2(W,- -w, (W:)))) S - I w01 -2((( - f(w")),
and with ||a + bll² s 2 ||a|| + ||| further
Iw: - w 1| 2
Wt w + w/ well?
and with (a. b) S 2a |a|| 2 + - |bb| for a,
Fd and a > 0.
Wt, (w.)) S 2L w. -w11? + 1
2 2(W:
f
2L
2L W well 2 + [V)(w)) - fl("))
2L
S 21 w/l 2 + f(w,) f(w"),
where the last inequality is a consequence of I-smmothhnss. Combining all these inequalities yields
E
|We+1
w01 we
s (1
|/w, w-1l2 Ih (J(W.) - f(w")) + G22
2
+ (2L + 14)
[: -wll?
1 1m))
2 l|w, w'11 7t - S(w")) + 722
+ +, I. || [w, - we
as / < L. Hence, by rearranging and multiplying with a weight A > 0:
(f(()( f(w*)) S
Ae(1 22//2) l| w+1
d
"
|e+1
w² + G2
7/
7t
+ +3 || W W. 2
By plugging in the learning rate, 7/- = u(t+2) and setting d = (t + 1) we obtain
AE (J(..) f(w"))
4 ((1 + 1)IE ||w. - w 12 - (1 1)(( 2)E| || W.+1 w w*]]²]
4(t 62 3(1 LE Iw will
pll + 2)
By summing from = to t = T' these we-ihttt inequalities, we obtain a telescoping sum:
T
(f(w)) - f(w")) < / 0 - (T + 1)(T + 2) | - w w-]] + 4(T + 1) 4
G2
4
T
+ 3L
X W 2
1=0
4(T + 1)
T
G2 +
3L
A w. -w.1"
w.l
1=0
20

---

Published as a conference paper at ICLR 2020
El d (T+1)T++2
Hence, for Ar
1
4(T + 1)
3L
(f - f(ww))
G
Xx W.
w -w/l
AT
nÂT
AT
t=0
1=0
G²
L
=0
N.
W/
NuT
ÀT
Finally, using | well2 = -r:l/
l/ by I shows the theorem.
Before giving the proof of Theorem 4.2 we first give justification for the remark just below Aaecorem4.11or the one-shot pruning of the final iterate.
We have by L-smoothness and (a, b) S 2 ||a|] + a |bb]² for a, b Rd and a > 0 for any iterate Wt:
f(wi) - f(ww) S f(w.) - f(w")( + (Vf(w,). We - ww)) + L || [. -well
2
$ f(w,) - f(ww) + 2L Vf((w))l + w w. 11
2((((w)) f(w*)) 5. |/w/l?
(4)
Furthermore, again by L-smoothness,
S(Wr)- - s(w") S 2 || w/ 1=1 =0 LG
122
as standard SGD analysis gives the estimate E. |||WT - w+1 O
nT
G²
see e.g.
Lacoste-Julien
et al. 2012). Combining these two estimates (with
WT) shows the claim.
Furthermore, we also claimed that also the dense model converges to a neighborhood of optimal solution. This follows by L-smoothness and (4): For any fixed model w, we have the estimate 4 hence for randomly chosen (dense) model u (from the same distribution as the sparse model in Theorem 4.1 we have
Ef(uu) f(w*)( (E f((")( }1 LE 5 |wa [2]
(Thm
O
G2 μT
LE
5e [wall()
Lastly, we give the proof of Theorem 4.2 following Karimireddy et al. (2019).
Proof of Theorem 4.2 By smoothness, and (a, b) < 1/2 ||a|| + 1/4 [b| for a, b E Rd.
To E Ell((w))l
E (f(we)) wi] S f(w:)
) Eg (w.)
T.Q
S f((w)) 7
+
2
+2² +22 LG2
= -(w,))
2
+7 + (ww,,) vs(W.))
< f(w)) - - + +, LG2
2
+ 7 [V((.) (.(w))l + || [VVf((:))l
2
$ f(w.) - ? |v(()))| + +2² I.G2
T.2
|| 1
2
and by rearranging
2 | S [Ef((w)) Ef(w + LL2 + I,²B [| 12,12
21

---

Published as a conference paper at ICLR 2020
Summing these inequalities from = to T gives
1 [
Σ
5
2
- [[((w++1)) +
T+1 0
(T
+11
1
T
E ||: - w.ll
T +
t=00
2 ( (wo))) - (((w))
L2 T + Ln + T+ Σ
Bl
7T++1)
Finally, using ||, - will = 5, |w, 1| by [1 and plugging in the stepsize y that minimizes the right hand side shows the claim.
22


Reviews:
[{"review_id": "S1la1ROTYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a novel model compression method that uses error feedbacks to dynamically allocates sparsity patterns during training. The authors provided a systematic overview of a good number of existing model compression algorithms depending on the relative order of pruning and training processes. The effectiveness of the proposed algorithm is illustrated by comparing its generalization performance with 6 existing algorithms (and their variants) with two standard datasets and various networks of standard structures. The authors also showed the convergence rate and the fundamental limit of the proposed algorithm with two theorems. \n\nThis paper is well-written and very pleasant to read. I would like to accept this paper. But since I have never actually done research in model compression, I would say this is my 'educated guess'. \n\nSome quick comments:\n1. I did not go through the proofs of the two theorems. But it seems that there is a typo in the definition of strong convexity on Page 4: '\\Delta f(w)' should be '\\Delta f(v)'. I assume that this is just a typo. \n2. Sorry again for not knowing the details of the baseline algorithms. According to Table 1 and Table 2, the proposed method (DPF) outperforms all the baseline algorithms, without a single exception, which looks suspicious for me. After reading the paper, I still don't understand why this should be the case. Is this due to some implementation details? Can you think of some scenarios that the proposed algorithm may not be the one to go with? In other words, when the experiment seems to show that one algorithm absolutely outperforms all the other existing algorithms, there should be some take-home message on why, or some known limitations of the proposed method. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HyxogpQvtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a simple pruning method that dynamically sparsifies the network during training. This is achieved by performing at fixed intervals magnitude based pruning for either individual weights or entire neurons. While similar methods have been explored before, this work proposes a slight twist; instead of updating the weights of the model by following the gradient of the parameters of the dense model, they update the parameters of the dense model according to the gradients of the sparse model. Essentially, this corresponds to a variant of the straight-through estimator [1], where in the forward pass we evaluate the compressed model, but in the backward pass we update the model as if the compression didn’t take place. The authors argue that this process allows for ``feedback” in the pruning mechanism, as the pruned weights still receive gradient updates hence they can be ``re-activated” at later stages of training. They then provide a convergence analysis about the optimization procedure with such a gradient, and show that for strongly convex functions the method converges in the vicinity of the global optimum, whereas for non-convex functions it converges to the neighbourhood of a stationary point. Finally, the authors perform extensive experimental evaluation and show that their method is better than the baselines that they considered.\n\nThis work is in general well written and conveys the main idea in an effective manner.  It is also a timely contribution as sparse models / compression are important topics for the deep learning community. The overall method seems simple to implement, doesn’t introduce too many hyper-parameters and seem to work very well. For this reason I tend towards recommending for acceptance, provided that the authors address /comment on a couple of issues I found in the draft. \n\nMore specifically:\n- The connection to the error feedback is kind of loose and not well explained. After skimming Karimireddy et al. I noticed that 1. it evaluates the gradient at a point (i.e. the current estimate of the parameters), 2. compresses said gradient, 3. updates the parameters while maintaining the difference of the original w.r.t. the compressed gradient. In this sense, it seems a bit different that DPF, as your notation at the first equation of page 4 implies that you take the gradient of a different point, i.e. w_t + e_t instead of w_t. I believe that expanding a bit more about the connection would help in making the manuscript more clear.\n- There seems to be a typo / error on your definition of an m-strongly convex function at the “convergence of Convex functions” paragraph. I believe it should be <\\nabla f(v), w-v> <= f(w) - f(v) - 0.5 m ||w - v||^2, instead of <\\nabla f(w), w-v> <= f(w) - f(v) - 0.5 m ||w - v||^2.\n- The proposed gradient estimator seems to be an instance of the STE [1] estimator, that, as the authors mention, has been using at the Binary Connect algorithm. It would be interesting to see some more discussion about this similarity perhaps also expanding upon recent work that discusses the STE gradient as a form of coarse gradient [2].\n- At section 5.2 the authors mention that “dynamic pruning methods, and in particular DPF, work on a different paradigm, and can still heavily benefit from fine-tuning”. This claim seems to contradict the results at Figure 4; there it seems that the masks have “converged” in the later stages of training, hence one could argue that the fine-tuning already happens thus it wouldn’t benefit DPF. I believe it would be interesting if the authors provide a similar plot as the one in Figure 4 but rather for the ResNet-20 network on CIFAR 10 (which seems to benefit heavily from FT). Do the masks still settle at the end of training (as it was the case for WideResNet-28-2) and if they do, why is fine-tuning still increasing the accuracy?\n- Minor: Try to use consistent coloring at Figure 6 as while (a), (b) share the same color-coding, (c) is using a different one hence could be confusing.\n\n[1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, Yoshua Bengio, Nicholas Léonard, Aaron Courville, 2013\n[2] Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets, Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, Jack Xin, 2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJenAkApFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nMain contribution of the paper\n- The paper proposes a new pruning method that dynamically updates the sparse mask and the network weight.\n- Different from the other works, the proposed method does not require post-tuning.\n- A theoretical explanation of the method is provided.\n\nMethods\n- In this method, the weight of the baseline network is updated not by the gradient from the original weight but pruned weight.\n- Here, pruning can be conducted by (arbitrary) a pruning technique given the network weight (Here, the author uses the magnitude-of-the-weight given method from Han.et.al).\n\n\nQuestions\n- See the Concerns\n\nStrongpoints\n- The author provides the simple and effective pruning method and verifies the performance with a sufficient amount of experiments.\n- The author argues that the method is applicable to various pruning techniques.\n\nConcerns\n- It seems that the paper omits the existing work (You.et.al - https://arxiv.org/pdf/1909.08174.pdf), which seems to share some contribution. The reviewer wants the author to clarify the differences and the strongpoints compared to the work.\n- The main pruning&update equation (DPF) does not seem to force the original network w to become sparse, such as by l1-regularization. So, the reviewer worried that the method might not produce sparsity if the initial weights are not that sparse.\nIf the reviewer missed the explanation about this, clarify this.\n- Regarding the above concern, what if we add regularization term in training the original network w? \n- As far as the reviewer knows, the proposed method improves the sparsity of the network, but most works choosing the strategy actually cannot meaningfuly enhance the operation time and just enhances the sparsity. Does the author think that the proposed method can enhance the latency? If so, a detailed explanation or experiment will be required.\n\nConclusion\n- The author proposes a simple but effective dynamic pruning method.\n- The reviewer has some concerns regarding the novelty, real speed up, and guarantee of the sparsity. \nHowever, the reviewer thinks that this work has meaningful observations for this field with a sufficient amount of verification, assuming that the author's answers for the concerns do not have much problem.\n\nInquiries\n- See the Concerns parts.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new, simple method for sparsifying deep neural networks.                                                      
It use as temporary, pruned model to improve pruning masks via SGD, and eventually                                                 
applying the SGD steps to the dense model.                                                                                         
The paper is well written and shows SOTA results compared to prior work.                                                           
                                                                                                                                   
The authors unanimously recommend to accept this work, based on simplicity of                                                      
the proposed method and experimental results.                                                                                      
                                                                                                                                   
I recommend to accept this paper, it seems to make a simple, yet effective                                                         
contribution to compressing large-scale models.                                                                                    
                             
Decision: Accept (Poster)