MaskConvNet: Training Efficient ConvNets from Scratch via Budget-constrained Filter Pruning
Authors: Raden Mu'az Mun'im
Jie Lin
Vijay Chandrasekhar
Koichi Shinoda
Venue:  (2020)
Keywords: Structured Pruning
Sparsity Regularization
Budget-Aware

Abstract: In this paper, we propose a framework, called MaskConvNet, for ConvNets filter pruning. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to a ConvNet architecture. MaskConvNet enjoys several advantages - (1) Flexible, the mask module can be integrated with any ConvNets in a plug-and-play manner. (2) Simple, the mask module is implemented by a hard Sigmoid function with a small number of trainable mask variables, adding negligible memory and computational overheads to the networks during training. (3) Effective, it is able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless of the datasets and ConvNet architectures used. (4) Fast, it is observed that the number of training epochs required by MaskConvNet is close to training a baseline without pruning. (5) Budget-aware, with a sparsity budget on target metric (e.g. model size and FLOP), MaskConvNet is able to train in a way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint simultaneously. Results on CIFAR-10 and ImageNet with several ConvNet architectures show that MaskConvNet works competitively well compared to previous pruning methods, with budget-constraint well respected. Code is available at https://www.dropbox.com/s/c4zi3n7h1bexl12/maskconv-iclr-code.zip?dl=0. We hope MaskConvNet, as a simple and general pruning framework, can address the gaps in existing literate and advance future studies to push the boundaries of neural network pruning.

Under review as a conference paper at ICLR 2020
MASKCONVNET: TRAINING EFFICIENT CONVNETS FROM SCRATCH VIA BUDGET-CONSTRAIINE FILTER PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose a framework, called MaskConvNet, for ConvNets filter pruning. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to a ConvNet architec- ture. MaskConyNet enjoys several advantages (1) Flexible, the mask module can be integrated with any ConvNets in a plug-and-pll manner. (2) Simple, the mask module implemented by hard Sigmoid function with small number of trainable mask variables, adding negligible memory and computational overheads to the networks during training. (3) Effective,
able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless the datasets and ConvNet architectures used. (4) Fast, it is observed that the number of training epochs required by MaskCon- vNet close to training a baseline without pruning. (5) Budget-aware, with sparsity budget target metric (e.g. model size and FLOP). MaskConvNet is able to train in a way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint simultaneously. Results on CIFAR- 10 and ImageNet with several ConvNet architectures show that MaskConVNet works competitively well compared to previous pruning methods, with budget- constraint well respected. Code is available at https: //c:://wwww /ww. dropbox. com/ s c4zzini 0ttriin77hbbbx111 111/mm1aaaaamm ht//maskkkoo wwwass-ooovoooo iccr co-doeeee.. zip?dl=0. We hope MaskConNNet, as a simple and general pruning framework, can address the gaps in existing literate and advance future studies to push the boundaries of neural network pruning.
INTRODUCTION
State-of-the-art Convolutional Neural Networks (ConvNets) achieve high accuracy in many applica- tions using very deep and wide networks, e.g., VGG Simonyan & Zisserman (2015) and ResNet He et al. (2016). Deep and wide networks are often sver-parameterized, which significantly increase the usage of memory (with tens of millions of weights) and compute (with tens of billions of MAC- - Multiplication and ACcumulation). It particularly challenging to deploy, in a naive manner, such large ConvNets on edge device that with limited memory and compute capacities.
To reduce the memory footprint and MAC of modern ConvNets, recent works propose either unstruc- tured pruning (a.k.a. weight pruning) to prune redundant weights LeCun et al. (1990); Han et al. (2015; 2016); Zhu Gupta (2017); Guo al. (2016); Louizos et al. (2017a;b); Frankle Carbin (2019); Lee et al. (2019); Gale et al. (2019) or structured pruning (e.g. filter-wise, chan depth-wise, etc.) to prune filter//chanees/laaeesss Li et al. (2016); Wen et al. (2016); Liu al. (2017); Ye et al. (2018); Neklyudov et al. (2017); Gordon et al. (2018); He et al. (2018); Véniat & Denoyer (2018); He et al. (2019); Lemaire et al. (2019); Li et al. (2019); Yang et al. (2019) from original over- perr-arrmeterizee deep networks, without incurring significant loss in accuracy. Both unstructured and structured pruning approaches support pruning from pre-trained models followed by fine-tuning and pruning while training from scratch.

---

Under review as a conference paper at ICLR 2020
Despite the impressive pruning rates achieved on target metrics such as weight variables and floating- point operations (FLOP), existing neural network pruning methods still suffer one or both of the weaknesses that (1) Additional (iterative) fine-tuning is often required to compensate the accuracy degradation, leading to slower convergence and thus more training epochs and (2) Most of existing pruning approaches are not capable of automatically and optimally allocating pruning rates across layers given a sparsity budget on target metric. On the other hand, pruning with budget constraints tai- lored for specific hardware platforms, as promising direction for hardware-softwer co-optimization. is becoming increasingly important especially for edge device.
In this work, we propose a new pruning framework, called MaskConvNet, address the above mentioned issues simultaneously. MaskConvNet provides elegant support for training budget- pruned networks from scratch, by adding a simple mask module to each bundled block of the target ConvNets. Fig. showcases an example of applying MaskConvNet to typical bundled block Conv/BN/ReLJU for filter pruning. The mask module is composed of a hard Sigmoid activation function with vector of trainable variables as input and the output is regularized with a sparsification loss, resulting in a sparse mask vector with many 0 elements and the rest are real values in (0, 1]. The pruned ConvNets is generated by element-wii wise multiplication between the mask vector and filters/BN parameters, in which filters/BN parameters with all zero are removed from the ConvNets.
We summarize the main contributions and advantages of MaskConvNet follows.
Flexible. The mask module can be integrated with any ConvNets in plug-and-play manner. Simple. Instead of directly regularizing filters, the mask module is defined separately and differ- entiable almost everywhere, and the number of mask variables equals the number of filters in the target ConvNets, implying that it adds negligible memory and computational overheads to the networks during training.
Effective. MaskConvNet is able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless of the datasets and ConvNets architectures used.
Fast. Our empirical observations on both CIFARIO and ImageNet show that the number of training epochs required by MaskConvNet is close to training baseline without pruning, mainly attributed to the fact that the mask module (1) enables threshold-free pruning, thanks to the hard Sigmoid activation function and (2) is shared within the bundled block structure (Fig. 1) which in turn addresses the mismatch problem between CONV and BN layers.
Budget-aware. With sparsity budget on target metric (in percentage), MaskConvNet is able to train in way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint si- multaneously. In contrast, previous budget-aware pruning methods suffer the iterative optimization process of alternating between training and evaluation of the target metric, which tedious and time-consuming.
In recent years, structured pruning attracts much more attention than its unstructured counterpart weight pruning, mainly because structured pruning doesn't 't require specific circuit design and thus more friendly to existing hardware. Thus, in this work we apply MaskConvNet one of most popular structured pruning methods, filter-wise pruning. It's worth noting that MaskConvNet, as a general pruning framework, can be easily extended to other structured pruning categories as well as weight pruning, by re-formatting the mask variables only. We would like to leave it for future work.
2 RELATED WORK
Both unstructured and structured pruning can be grouped into saliency-basee based LeCun et al. (1990); Han et al. (2015; 2016); Zhu & Gupta (2017); Guo et al. (2016); Frankle & Carbin (2019); Lee et al (2019); Li et al. (2016); He et al. (2018; 2019) and sparsity learning Louizos et al. (2017a;b); Wen al. (2016) Liu et al. (2017); Ye et al. (2018); Neklyudov al. (2017); Gordon et al. (2018); Lemaire et al. (2019); Li et al. (2019) approaches, where the former shrinks network size by well-defined thresholding criterion such as magnitude for weight variables Han et al. (2015; 2016); Zhu & Gupta (2017); Frankle& & Carbin (2019); Gale et al. (2019) or LI/L2 norm for filters Li et al. (2016); He et al. (2018; 2019) and the latter prunes networks by either adding regularization functions into training loss (c.g. L0/L1 Louizos et al. (2017a;b); Liu et al. (2017); Ye et al. (2018); Gordon et al. (2018); Lemaire et al. (2019), Group Lasso Wen et al. (2016)) learning dropout rates for weights/filters
2

---

Under review as a conference paper at ICLR 2020
Hard Sigmold
0.2
0.5
Fiters
Filter Madk
Scale
Muliipiization
Masked Scale
Maded
Maxbed Shit
BN
IF
Convelation
OF
OF
Figure 1: Overview framework of MaskConv for filter pruning typical bundled block structure composed of Conv - BN ReLU.
via variational dropout with the reparametrization trick Neklyudov al. (2017). MaskConvNet falls in the category of sparsity learning with specially designed regularization function added to the training loss.
Most of the existing pruning approaches, including pruning while training Zhu & Gupta (2017); Li al. (2016); Liu al. (2017); Ye et al. (2018); Gordon et al. (2018); He et al. (2018; 2019); Lemaire et al. (2019), require to perform additional (iterative) fine-tuning to compensate the accuracy degradation, mainly due hard pruning via thresholding and the mismatch between Batch Normal- ization and its corresponded pruned CONV/FC layer. This leads to slower convergence and thus more training epochs. For instance, with the method of gradual magnitude- during training proposed by Zhu & Gupta (2017) and pre-defined pruning rate 80% ResNet50, Gale et al. (2019) reported that it increases the number training epochs by 1.5x in order to achieve comparable ImageNet validation accuracy with the baseline model trained without pruning.
weight pruning
On the other hand, there is few work studying pruning mechanism to automatically and optimally allocate pruning rates across layers that fulfill budget specified for target metric. Existing works either only meet the budget by pruning using sub-optimal pruning rate pre-defined or empirically calculated Han et al. (2015); Li et al. (2016); Liu et al. (2017) can derive automatic pruning rate through sparsity learning methods which fail to fulfill the budget explicitly Louizos et al. (2017a;b); Wen et al. (2016); Neklyudov al. (2017).
MorphNet Gordon et al. (2018) and BAR Lemaire et al. (2019) are the most related work to MaskCon- vNet. while our work differs from them in several aspects. First, both MorphNet and BAR require pre-defined threshold for hard pruning, while MaskConvNet is threshold-free Second, MorphNet and BAR apply mask on y variables of BN and filters respectively, while MaskConvNet applies mask on both. Third, to meet the sparsity budget, both MorphNet and BAR alternate between training and evaluation of target metric with multiplier derived by grid search. MaskConvNet automates the process by adaptively shrinking or expanding the pruning ratio across layers in data-driven manner during training, avoiding the tedious iterative optimization process. Finally, BAR uses Knowledge Distillation Hinton et al. (2015) to facilitate the training of pruned networks for better accuracy, while NeskConvNet does not.
METHOD
3.1 FILTER PRUNING VIA THE MASK MODULE
Fig. I presents overview of the design of the mask module and how the mask module is applied for filter pruning. The mask module simply defined as hard Sigmoid activation function,
(m) min(max((m) + 0.5, 0), 1),
3

---

Under review as a conference paper at ICLR 2020
where input (m) is trainable mask variables tied to Conv filters ! For Conv layer with input channels, h output channels (filters) and k x l sized kemelss, the number of mask variables equals to h. Each element of the mask activation o(m) indicates the importance score of the associated filter. The same mask is applied to bias parameters for the Cony layer.
Each filter is reparameterized its weights multiplied by its mask activation via element-wiss multiplier. Concretely, given filter W IR and its mask activation ā (m) E R, the masked filter W is given by:
W(W m) W c (mm).
where @ is element-wise multiplication between a mask element and a 3D tesor with size w x l x k. It's easy know that mask element with importance score 0 means filter with w x k x A weight parameters is pruned from the network. The sparser the mask activation, the higher the pruning ratio.
It is worth noting that the zero-centered hard- Sigmoid ((m) clamps mask [0, 1]. allowing mask to span the whole range rather than binary. Thus, thresholding critiria is no longer required for hard pruning. The mask module is similar in spirit to the diversity networks Zelda Mariet (2016) in the sense that the pruned filters are "fused" into the remaining filters via non-zero mask elements rather than dropped completely, but our proposal is much simpler.
In our experiments, m is simply initialized with zeros, so that (m) 0.5, which is median of interval [0. 1]. One can initialize with random values, but such initialization could potentially give unknown prior bias to sparsification, which may give different structure with different random seed.
In the end of training, the pruned ConvNets is generated by Equation 2, where filters/BN parameters with all zero are removed from the ConvNets. One may note that the output feature maps of Conv layer i serve as the input to Conv layer i + 1. Thus, the input channels of layer i + 1 that corresponded to the pruned filters of layer i are also pruned.
Mask Sharing. In modern ConvNets, a Conv layer is often followed by a batch-norm layer which has "scale" weight parameter y and "shift" bias parameter 3 with dimension length equals to number of filters of its preceding Conv layer.
Previous pruning approaches often prune Conv filters with their successive Batch Normalization layer unchanged. This causes mismatch problem between pruned Conv and BN layers, given that by nature BN parameters highly depend the output feature maps from preceding Conv. To avoid the mismatch problem, we propose to share the mask module within the bundled block structure Conv-BN-ReLUU More specifically, BN scale and shift parameters are masked with the same mask activation for the corresponding Conv layer. Without mask sharing, we observed that the accuracy degrades severely after pruning. We speculate that this is due BN by default are unaware of pruned filters, i.e. pruned filters give O-valued output as input to BN, but BN considers O-valued input as if the filter is still alive (i.e not pruned).
Extending Mask to Residual Connections. In ConvNet architectures such as ResNet, residual connections involve element-wise addition operation where two operands must have the same dimensions. To extend the mask module to residual connections, one can share a single mask for all layers involved in residual connections. This is achieved by taking the mask for the last Conv layer in the last residual block as the base mask, and sharing the base mask with other Conv layers involved in other residual connections. Partially, the base mask is sliced to fit Conv layers with less number of filters.
3.2 SPARSITY REGULARIZATION ON THE MASK
With filters W reparameterized by the mask activation (m), the loss function for training budget- aware pruned ConvNets from scratch is given as:
L(W m) = LE (Wo o (mm)) (s(r(m)),
(3)
where LE is cross-entropy loss for classification (the "error"), and Ls is the sparsification loss defined on the mask.
We take Conv/FC example here as they account for over 99% of total operations in modern ConvNets. FC can he treated as special case of Conv layer where the kemal size is x 1.
4

---

Under review as a conference paper at ICLR 2020
Sparsification Loss. The sparsification loss is defined by sparsifying the mask:
o² d² (mm))
(s(ō(m)) Amm(((mm)) Au ((((m)))
(4)
where the goal is to minimize the mean ( and maximize variance-to-ma -eann o ((()))/u((((m In particular, the regularization multipliers Am and X introduced to precisely control regularization strength towards budget-aware pruning (see details the subsequent subsection).
Minimizing mean is basically LI Ll-regularization averaged over the total number of mask elements of the network, and maximizing variance is noft-binarization by dispersing mask elements to the either zero or one. Dividing variance by mean adaptively strengthen variance maximization when mean is becoming smaller.
Extension to Multi-metrics. Equation 3 can be extended to support sparsity regularization of the mask for multiple target metrics simultaneously, e.g. optimizing for parameter counts and FLOPs given by budget constraints for both metrics. A simple heuristic is to average the mean and variance-to-men tion and Equation 4 are rewritten as L(W, m) = CE(W @ ((m)) + Es (s: and ΣN (s)()(m)) = Ei= Ami (()) Σ
the mask (m) that is applied N target metrics. Accordingly, Equa-
(a(m) respectively. Di=l Api p(a(m))
Algorithm 1: Algorithm description of MaskConvNet for epoch.
Input: input-label pairs (X1. from N training mini-batches Output: Weights W, mask m Set target sparsity percentage s", base multipliers Xbaee and Abase, multiplier update interval multiplier EMA exponent o;;
for i in ... N do
(x1, + load mini batch(i);
// For every n mini-batches, evaluate sparsity and update multipliers;
if (i % = 0) then
8: f- compute computeesparsittee.eeeee sparsity(W W @ (m()). according to Eqn. 2
St 5 si-1 (1-a).
As + s" ---Sp St;
Am base As;
No bbase As:
end
Train on (i) with cross-entropy and mask regularization as in Eqn. 3;
end
3.3 BUDGET-AWARE MASKCONVNET: THE ALGORITHM
In previous works on grid search on sparsity regularization multiplier is performed, the multiplier is then used to find sparse network structure that could meet the budget. These approaches alternate between network training and sparsifying with the regularization multiplier until sparsification stops at arbitrary sparsity level. the model is undersparsed or oversparsed, the regularization multiplier value adjusted and training is repeated from scratch. This is very laborious and time-consuming.
based pruning Gordon (2018); Lemaire et al. (2019),
In contrast to previous works, NeskConvNet avoids the tedious iterative optimization process. With target sparsity budget (in percentage), MaskConyNet is able to train in way that the optimizer adaptively sparsifies the network and automatically maintains sparsity level, till the pruned network produces good accuracy and fulfills the budget constraint simultaneously.
The training procedure for one epoch is described in Algorithm 1. During training, the weight tensors reparameterized with masks according to Eqn. 2, both weight tensors and masks are jointly updated and sparsified via standard optimizer (e.g. SGD with momentum) according to Eqn. 3. Evaluating sparsity percentage at every mini-batch iteration can slow down training considerably,
5

---

Under review as a conference paper at ICLR 2020
hence evaluate and smooth the sparsity (with EMA) for every interval. In experiments, set n = 20 and EMA exponent a = 0.99.
As the regularization multipliers are periodically updated, sparsification is expanded stronger in early training, and becomes gradually weaker when reaching the target due to Ås approaches zero. On the other hand, when the pruned model is oversparsed, As becomes negative, and do the multipliers Am and A The optimizer will then shrink the mask sparsity (unsparsification) until As is positive again (See more detailed analysis in the next subsection).
3.4 How MASK UNSPARSIFICATION WORKS?
In case when Am and X are too big, too many mask elements will be zeroed and the model becomes oversparsified and misses the sparsity budget. If the masks elements are zeroed, how would the optimizer continue update these zeroed elements? How would the gradients flow and update the masks? We exploit three things to unsparse the masks: (1) Mask decay update; (2) Negative regularization multiplier; and (3) Positive mask parameter when hard-Sigmoid output is zero.
The SGD update rule for the regularization term (m) is given as m <- m - Am om 7jmm, where 1) the learning rate and is mask decay constant. Also recall the hard-Sigmoid function on mask is ā (m) min mmxx (m + 0.5, 0), 1).
0s
Am will be negative when sparsity difference As is negative. Also, when m 0.5,, then (m) and as O. However, is still non-zero negative. Hence, the term 77m is positive and thus Dm
the optimizer is able to increase m so that (mm) becomes non-zero, unsparsifying the mask.
The catch here that we cannot achieve exactly zero error target sparsity budget due zeroed ā (m) will always be bounced back to non-zero (unless we deliberately turn off mask decay). The model will always be undersparsfied and oversparsified continuously by small degree throughout training. The sparsity deviation depends on the value of mask decay constant applied to the mask. This will be discussed in experimental section.
Tea MoahCarfte
m
Sup-3, Mataii Faram seanily
Eosch
Figure 2: Testing/Validation accuracy (Top-1) and sparsity pattern as the function of training epochs for MaskConvNet with -edeResNet-28- 10 on CIFAR-IÓ (left) and ResNet-34 on ImageNet (right). For reference, the baseline accuracy included.
4 EXPERIMENTS
We evaluate MaskConvNet on both CIFAR-10 and ImageNet ILSVRC 2012 et al. (2009), with differ- ent ConvNets architectures including Simonyan Zisserman (2015); He al. (2016); Zagoruyko Komodakis (2016) as baseline architectures. For all CIFAR-10 experiments, we modified source code from Liu et al. (2018), and followed the same training setup and data augmentation. We used SGD with momentum of 0.9 as optimizer. Initial learning rate is set to 0.1 and divided by 10 at epoch 40 and epoch 80. All models are trained for 160 epochs. Am and Ao are updated every 20 mini-bant batches with EMA exponent all ImageNet experiments, we implemented the training code and hyper-parametes settings derived from source code for work by Banner et al. (2018).
0.99. Details on the settings of Am and X, are left in appendix section. For

---

Under review as a conference paper at ICLR 2020
VGG-19 Parameter Targets
VGG-19 FLOP Targeta
1000111 DMM mAaammi
pharmay
50
73
00
00
102
30
50
60
00
103
Facredet Sparcity
WRN-28-10 Parameter Targets
WRN-28-10 FLOP Targets
02:0004 17150))
$1 11100000
09.7 170077077
Pasmmaeer Sparsity
Figure 3: Results of MaskConvNet-P (parameter counts as target metric) and MaskConvNet-F FFLLO (FLOP as target metric) with various sparsity budgets on CIFAR-10 10, using VGG-19 and WideResNNet22-110 as baseline architectures. Text labels on plot points denote "Accuracy% (Actual / Budget Sparsity
Figure 4: Layer-wise parameter sparsities throughout training for VGG-19 (left) and WideResnet-28- 10 (right) on CIFAR-10, with warmup training till epoch 10. The brigher the color, the higher the sparsity.
4.1 MASKCONVNRT: IN-DEPTH ANALYSIS
In this section, we perform various experiments on CIFAR-10 and ImageNet explore the behaviors of NeskConvNet in-depth.
The convergence speed and sparsity pattern of ConvNet during training. Figure shows the testing/valinn validation accuracy (Top-1) 1) and sparsity pattern as the function of training epochs for on CIFAR-10 10 and ResNet-34 on ImageNet. The baseline
MaskConyNet with WidtResNet-28-- accuracy is included for reference. We observe that the convergence speed of MaskConv is highly similar the baseline architecture, indicating the effective training capability of Nask In addition, the sparsity pattern emerges at early training epochs and becomes stable as training going on, which consistent with the observation from the single-shot pruning work Lee al. (2019) that redundant neurons be identified in single step before training. One may note that the sparsity curve for ResNet-34 has small fluctuations. As shown in the next subsection as well as appendix, this issue can be solved by proper choice of mask decay.
Can Mask NenvNet meet the Budget?. Figure 3 presents the results of MaskConvNettP (parameter counts as target metric) and Net- (FL.OP target metric) with various sparsity budgets
7

---

Under review as a conference paper at ICLR 2020
CIFAR- 10, using VGG-19 and WideResNet-28-4 linearly decreases with increasing sparsity for both target metrics. More importantly, worth noting that the actual sparsities produced by both MaskConvNet-P and MaskConvNet-F are close to the budget, i.e. the error between actual and budget sparsity within 1 (absolute value) for most of the operating points.
baseline architectures. Accuracy almost
Looking into layer-wise parameter sparsities. We also investigate wise parameter sparsity patterns throughout training on CIFAR-I0, for VGG-19 trained with 50% FLOP sparsity budget and WideResnet-28-111 trained with 93.75% parameter sparsity budget. Figure 4 shows VGG-19 and WideResnet-28 to be sparsified, which matches the intuition in previous works such as Gordon et al. (2018). The hyper-thesis that early layers carry more information and can incur bigger accuracy loss if sparsified too much. In addition, we noted more uniform overall sparsity on WideResNet-259 10 than VGG-19 19. We speculate this be related to role of residual connections during sparsification process. This phenomenon is left for future investigation.
training results respectively. We observe that early layers are less likely
More results and analysis are introduced the appendix section, including sensititity analysis of base regularization multipliers Am and Ag visualization of mask histogram output by the hard- Sigmoid, inference speedup by measuring wall-clock latency of the baseline and pruned models on CPU core, and the effect of mask decay and warmup training accuracy well layer- sparsity pattern.
Table Comparison of MaskConvNet-F or MaskConWNet-P with Kate-of-the-art pruning approaches Network Slimming Liu et al. (2017), L1-Pruning Li et al. (2016) and BAR Lemaire et al. (2019), on CIFAR-10 with various ConvNet architectures.
Model
Accuracy (%)
FLOPs
Sparsity (%)
Parameters (M)
Sparsity (%)
VGG-16
Baseline
93.63
3.15
10a
0.00
14.99
0.00
LI-Pruning
93.40
2.06
10
34.20
5.40
64.00
MaskConvNet-FF
93.40
1.88
10
40.19
1.72
88.53
VGG-19
Baseline
93.67
7.97 10
0.00
20.04
0.00
Network Slimming
93.80
3.91
10
51.00
2.30
88.5
MaskConvNNet-F
93.03
3.66
10"
54.08
1.51
92.47
ResNet-56
Baseline
93.40
2.52 x 10
0.00
0.85
0.00
L1-Pruning
93.06
9.09
107
27.60
0.73
13.7
MaskConvNett-F
92.25
1.88 x 10
25.27
0.58
31.87
ResNet-164
Baseline
94.70
5.02 10
0.00
1.71
0.00
Network Slimming
94.92
3.81
10
23.70
1.44
14.9
MaskConvNet-F
94.28
3.46
10
30.96
0.79
53.59
WideResNett28--10
Baseline
95.30
1.19 10
0.00
36.48
0.00
BAR (16x reduction)
91.62
1.74
10
98.54
2.31
93.67
MaskConvNet-PP
94.28
3.50
10"
70.63
2.46
93.25
Table Comparison of MaskConvNett-P with both parameter counts and FLOP target metrics with Li et al. (2016) on ImageNet ILSVRC 2012 with ResNet-34 as bascline architecture.
Model
Accuracy (%)
FLOPs
Sparsity (%)
Parameters (M)
Sparsity(%))
Resnet-34
Baseline
73.27
3.69 x 10a
0.00
21.8
0.00
LI-Pruming
72.56
3.08
10
15.50
19.9
7.60
MaskConvNeet-F
72.56
3.29 x 10
10.75
17.7
19.00
8

---

Under review as a conference paper at ICLR 2020
4.2 COMPARISON WITH PREVIOUS WORKS
CIFAR-10. Table I compares NestC or MaskConvNet-PP with state-of-the-art pruning approaches Network Slimming Liu al. (2017), Pruning Filters for Efficient ConvNets Li al. (2016) denoted as LI-Pruning and Budget-Award Regularization (BAR) Lemaire et al. (2019), on CIFAR-10 with various ConvNet architectures including VGG-16, VGG-19, ResNet-56,, ResNet-164 and WideResNet-28-1U1 One can see that for almost all of the operating points, MaskConvNett-F performs comparably in terms of accuracy with Network Slimming and L.1 L.1-Prnnin, and achieves much higher sparsity on both FLOP and parameter counts. Compared to BAR, MaskConvNet.PP achieves comparable parameter sparsity with better accuracy, but the FLOP sparsity is much lower probably because the usage of Knowledge Distillation Hinton et al. (2015) in BAR as prior guiding the training process while our method does not use it.
ImageNet. Table 2 compares MaskConvNettFP with both parameter counts and FLOP target metrics with L1-Pruning Li et al. (2016), with ResNet-34 as the baseline architecture. In LI-Pruning paper, Resmet-34 is pre-trained, pruned, and fine-tuned for 20 epochs. Standard ResNet training takes 90 epochs. Compared to LI-Pruning, we managed to train ResNet-34 with fewer epochs (<90) that achieves the same accuracy and much higher parameter sparsity, but with slightly lower FLOP sparsity.
5 CONCLUSION AND FUTURE WORK
In this work, we proposed MaskConvNet, a general pruning framework where a model is trained and sparsified end-to-end with sparsity budget constraint on various target metrics. Promising results are reported on CIFAR-10 and ImageNet for the problem of filter pruning. We hope MaskConyNet can inspire future research including extension of the pruning framework (1) to other structured pruning categories well as unstructured pruning and (2) to other model structures beyond ConvNets, such as recurrent networks and self-attention networks.
REFERENCES
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, 2018.
J. Deng et al. ImageNet: A large-scale hicrarchical image database. In CVPR, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Morphnet: Fast & simple resource-constrained structure learning deep networks. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1586-1595. 2018.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances in Neural Information Processing Systems 29, pp. 1379-1387, 2016.
S. Han, H. M., and W. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmmnn coding. In ICLR, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. International Joint Conference on Artificial Intelligence (IJCAI), pp. 2234-2240, 2018

---

Under review as a conference paper at ICLR 2020
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598-598--0 198-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.
Carl Lemaire, Andrew Achkar, and Pierre-Marc Jodoin. Structured pruning of neural networks with budget-aware regularization. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 9108-9116, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1xXiv:100.008700 2016.
Jiashi Li, Qi Qi, Jingyu Wang, Ce Ge, Yujian Li, Zhangzhang Yue, and Haifeng Sun. Oiesr: Out- in-channel sparsity regularization for compact deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7046-7055, 2019.
the
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. 2018.
Christos Louizos, Karen Ullrich, and Max Welling, Bayesian compression for deep learning. In Advances in Neural Information Processing Systems 30, pp. 3288 3298, 2017a
Christos Louizos, Max Welling, and Diederik Kingma. Learning sparse neural networks through 10 regularization. arXiv preprint arXiv:1712.01331 2017b.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov, Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems 30, pp. 6775-6784, 2017.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Tom Véniat and Ludovie Denoyer. Learning úime/memory-efficien deep architectures with budgeted super networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Haichuan Yang, Yuhao Zhu, and Ji Liu. Ecc: Platform-independee energy-constrained deep neural network compression via bilinear regression model. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smallernnomm-ess-infommnii assumption in channel pruning convolution layers. ICLR (Poster), 2018
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Suvrit Sra Zelda Mariet. Diversity networks: Neural network compression using determinantal point processes. In ICLR, 2016.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. CoRR, 2017.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
PARAMETERS FOR SINGIE-METRIC AND MULTI-METRIC EXPERIMENTS
(,
Table Hyper-parameters for Single-metric experiments
Model
VGG-19
WideResnet-28-10
Target
Target
Target P
Target
Ame
Base
3,5 bace
6
1,2*
Table 4: Hyper-parament for Multi-metric experiments
MaskConvNet-P
MaskConvNet-F
MaskConvNettPF (Amie .Aol Ama base warmup)
(Aeaee Xbase, warmup)
(Am base warmup)
VGG-16
3,4, 20
5, 6, 20
3,4, 3,4,5,6 6, 20
VGG-19
3,4,20
3,4.20
3,4, 3, 4. 20
Resnet-56
3.
1, 50
1, 50
Resnet-164
1,5, 10
1,5, 10
1,5, 1,5,
Resnet-34
5, 20, 5, 20,
A.2 MASK HISTOGRAM
As shown in Figure 5, the mask values accumulate mostly at O, then 1, This observation is different compared to findings by Gordon et al. (2018), where the batch-norm scale parameter histogram resembles Gaussian distribution.
VISG-15
WIINOIB-11 Maek Viluee slat
2000
2030
2000
Figure 5: Mask Histogram after training.
A.3 SENSITIVITY ANALYSIS FOR BASE REGULARIZATION MULTIPLIERS
We train VGG-19 with target FLOP sparsity budget of 50%, with base regularization multipliers base and busse sampled from integers interval [0, 10). perform 10 trials. All models are trained for 160 epochs, and used late sparsity regularization (i.e. warmup) start epoch 10. Results are shown in Table 5, one can see that the accuracy is quite stable regardless of the base regularization multipliers used.tt
A.4 LATENCY REDUCTION
We measure wall-clock latency for forward pass of one mini-batch on Intel Xeon E5-2680 V4 Processor (Broadwell-EP,, 14 core, 2.4GHz) for CIFAR-10 models. We select checkpoint files for three models with different target FLOP sparsity budgets. Measurements are noisy, but overall we found latencies reduced compared to the baseline models (Table 6).
11

---

Under review as a conference paper at ICLR 2020
Table 5: Sensitivity analysis for base regularization multipliers Abase and Ahase with VGG-19 trained on CIFAR-10 with target FLOP sparsity 50%
A base
base
Accuracy (%)
FLOPs sparsity (%)
Parameter sparsity (%)
3
92.43
49.10
91.12
9
92.10
53.48
92.93
4
91.96
50.13
91.28
2
92.02
46.93
89.22
5
9
91.75
50.03
91.36
6
92.01
52.30
92.50
92.33
44.72
87.56
7
92.50
52.66
92.77
9
10
92.15
49.63
90.90
10
2
92.43
51.73
92.15
mcan + std
92.17 + 0.25
50.07 + 2.70
91.18 + 1.68
Table 6: Latency Measurements CIFAR-10
Model
Target FLOP sparsity (%)
Actual FLOP sparsity (%)
Latency (ms)
VGG-19
0 (Baseline)
0.00
22.45 + 1.17
50
51.91 = 2.46
12.94 + 0,42
60
62.35 2.92
11.42 + 0.28
70
69.52 = 0.01
10.81 + 0.06
WideResnet-28- 10
0 (Baseline)
0.00
64.15 + 0.21
50
54.59
25.43 + 1.09
60
58.54 2.5
22.79 + 0.94
70
72.58 = 4.2
19.92 + 0.98
A.5 EFFECT OF MASK DECAY AND WARMUP
Figure 6 shows different mask decays give different rate of unsparsification. Also with warmup training (late sparsification), accuracy is improved and the pruned structure becomes different compared to the one without warmup High contrast is visible between sparsity level between last layers and early layers. Last layers are sparsified more.
12

---

Under review as a conference paper at ICLR 2020
Cecral Sparstin
1.0%
DULLE spminco tMat away det.pecticiinnieoooo mereaiy
Aomay
14 Daniall Gpantinn quuntyy daliyaai
0..%%
Figure 6: Layer-wise parameter sparsities throughout training for VGG-19 on CIFAR-10, with different mask decay and warmup training. The brigher the color, the higher the sparsity.
13


Reviews:
[{"review_id": "ryel6kz0Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose a network pruning method to learn a pruned network during training. Specifically, they add a pruning mask for each layer and induce a sparisity loss on the mask variables during training. The pruned network is obtained by applying the learned mask to the networks.\n\nThe paper seems to be well contained. However, my assessment of this paper is weak reject. I am mainly concerned with the novelty of this method. Also i think some more evaluation is needed to fully understand the effectiveness of this method. My questions are summarized as follows:\n\nQ1: In the methods part, the authors said that “Previous pruning approaches often prune Conv filters with their successive Batch Normalization layer unchanged.” Can the authors give some reference here as to which pruning approaches?\n\nQ2: Did the authors compare the proposed approach to training the pruned networks from scratch as done in [1]? Also can the authors analyze the sparsity patterns of the pruned networks as done in section G in the appendix of [1]?\n\nQ3: What is the difference of your approach to [2]? They seem to be very similar. I think it is necessary to add some discussion in the related work. Is there any experimental results for comparison with [2]?\n\n[1] Rethinking the Value of Network Pruning. Liu et al. ICLR 2019\n[2] AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference. Luo et al. Arxiv, 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJeplPkCtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for training time filter pruning for convolutional neural networks. The main idea is to use a trainable soft binary mask to zero out convolutional filters and corresponding batch norm parameters.\n\nPros:\n+ The proposed method seems relatively easy to implement.\n+ The quantitative results in the paper indicate that MaskConvNet achieves performance competitive with previously proposed pruning methods.\n\nCons:\n- Writing of the paper could be significantly improved. See some examples below.\n- The main thing that bothered me about the method was the usage of hard sigmoid. If a mask component ever gets into one of the flat regions it won’t be able to escape. The authors propose a workaround which they call “mask decay update”. This approach looks quite hacky and I’m not sure how easy it is to make it work in practice.\n\nNotes/questions:\n* Abstract: “elegant support” -> “support”\n* Everywhere in the text: Back-to-back citations should have the form (citation1; citation2; …)\n* Section 1, paragraph 3: “suffer one” -> “suffer from one”\n* Section 1, paragraph 4: “above mentioned” -> “above-mentioned”\n* Figure 1: The figure would greatly benefit from a detailed description. What’s IF, OF and OF? The reader shouldn’t be guessing. \n* Section 2, paragraph 3: “corresponded” -> “corresponding”\n* Section 3.1, paragraph 2: “W \\in R” – W is probably not a scalar value therefore it’s in R^n. The same goes for the mask.\n* Section 3.1, paragraph 2: “It’s easy to know ...” – this sentence needs to be rewritten, e.g., “One can see that …”\n* Section 3.1, paragraph 2: “sparser” -> “more sparse”\n* Section 3.2, “Extension to Multi-metrics”: “FLOPs” are never defined in the paper. How is this quantity computed exactly? I’m also not entirely sure how useful it is to introduce multiple lambdas – it seems that this case corresponds to a single lambda which is a sum \\lambda_i.\n* Section 3.3, paragraph 1: “undersparsed”, “oversparsed” – not sure if these words exist. Maybe rephrase instead of introducing new terms?\n* Section 3.3, paragraph 1: “very laborious” -> “laborious”\n* Figure 3: Why not show points all the way to 0 sparsity?\n* Section 4.2, CIFAR-10: The authors mention that (Lemaire et al., 19) achieve better FLOP sparsity due to usage of Knowledge Distillation. From this sentence alone it’s not clear how exactly KD helps. Why can’t KD be applied in the proposed framework? I’d appreciate if the authors could elaborate on this.\n\nI must admit that I’m not an expert in the field of NN pruning but I’m surprised that training-time masking of filters has not been tried before. Even if it’s really the case I’m not entirely confident that the paper should be accepted: the “unsparsification” looks more like a hack than a principled approach and the overall quality of writing needs to be improved. I’m giving a borderline score for now but I’m willing to increase it provided that the rebuttal addresses my concerns.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eo1JwJ9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a masking process to improve the pruning of DNN. In addition, the algorithm proposes to automatically allocate the pruning rates over layers. \n\nOn the positive side:\n- I do believe the main contribution is automatically allocating the pruning rates over the network. \n\n\n\n\n\nRelated works:\n- How does this relate to methods using gating to prune in the presence of residual layers or BN?\n\n- Paper claims multiple times that related works need a train-prune-retrain process which is only valid for post-processing works. \n\n- I missed sparsity promoting works related to training from scratch where similar masks are implicitly used during training (even not explicitly stated) to maintain the zeros in the network. At least in [1,2] the training time is the same as the time used for training a network from scratch (same claim as in this paper). \n\n\nMethod:\n- The mask is trained using a sigmoid function and claims this will be representative of the relevance of the 'neuron' within the layer. How is this really related?\n- Why the initialization of the mask is to the mean? I think I am confused there. For initialization, I would argue all the neurons/parameters are relevant, right?\n\n- The claim that this method is 'much simpler' is a bit subjective. I do not see why. Please elaborate. \n- I am not sure if the claim of pruning filters and not considering the bn layer is correct. I would tend to think that, if pruning a neuron, the corresponding BN module should be modified (that is, propagating the zero to subsequent layers). \n\n- I do like the extension to residual connections. It would be great to have more and clearer details on how is this done. Would this also apply to the UNET type of architecture?\n\n- What is the intuition behind Eq 4 and how it is related to the relevance of a parameter? \n\n- The extension to multiple metrics is of interest, however, there is little detailed there. How is the automatic allocation done? This is repeated in section 3.3 but details are missing. \n\n\n- My understanding is that the regularization multiplier is affected by the learning rate, therefore their effect is lower as the training progresses. In this case, seems the opposite, right? (page 6 before 3.4). \n\n\n- The part with the sparsity budget is interesting. What guarantees are that the newly enabled neurons are actually useful? Could it be possible that the budget suggested is not the right for the task at hand and, therefore, the additional parameters are not really relevant / needed? \n\n\n\n\n\nExperiments:\n\n- There is loss with little sparsity when it comes to Imagenet (15 and 17% pruning) does not seem very promising even the training time is similar. \n\n- Seems like the experiments and comparisons to L1-pruning are not very surprising. It would be nice to have more comprehensive numbers. For imagenet, if using Resnet-50, would be easier to compare to other numbers. \n- In the imagenet comparison, L1-pruning is the version-A of the paper. What about the others or why that particular model?\n- How are the actual groups made? \n\nMinor details:\n- Please improve figure 1. It is not easy to understand. Same with Figure 3. What is seen in Figure 4. \n- I do believe the WideResNet-28-10 number of parameters for BAR is not correct. \n- Section 4.2 is a bit overselling. I do not see 'much-higher' parameter sparsity. The claims are mostly valid for VGG type of networks in this particular setting. \n\n\n\n\n[1] Learning the Number of Neurons in Deep Networks, NeurIps 2016\n[2] Compression-aware training of DNN, NeurIps 2017\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to learn a pruned convolutional network during conventional training.  Pruning the network has advantages (in deployment) of reducing the final model size and reducing the required FLOPS for compute.  The method adds a pruning mask on each layer with an additional sparsity loss on the mask variables. The method avoids the cost of a train-prune-retrain optimization process that has been used in several earlier papers.  The method is evaluated on CIFAR-10 and ImageNet with three standard convolutional network architectures.  The results show comparable performance to the original networks with the learned sparse networks. 

The reviewers made many substantial comments on the paper and most of these were addressed in the author response and subsequent discussion.  For example, Reviewer1 mentioned two other papers that promote sparsity implicitly during training (Q3), and the authors acknowledged the omission and described how those methods had less flexibility on a target metric (FLOPS) that is not parameter size.  Many of the author responses described changes to an updated paper that would clarify the claims and results (R1: Q2-7, R2:Q3).

However, the reviewers raised many concerns for the original paper and they did not see an updated paper that contains the proposed revisions.  Given the numerous concerns with the original submission, the reviewers wanted to see the revised paper to assess whether their concerns had been addressed adequately. Additionally, the paper does not have a comparison experiment with state-of the art results, and the current results were not sufficiently convincing for the reviewers.  Reviewer1 and author response to questions 13--15 suggest that the experimental results with ResNet-34 are inadequate to show the benefits of the approach, but results for the proposed method with the larger ResNet-50 (which could show benefits) are not yet ready. 

The current paper is not ready for publication.

Decision: Reject