Title: MASKCONVNET: TRAINING EFFICIENT CONVNETS FROM SCRATCH VIA BUDGET-CONSTRAINED FILTER PRUNING

Abstract: In this paper, we propose a framework, called MaskConvNet, for ConvNets filter pruning. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to a ConvNet architecture. MaskConvNet enjoys several advantages -(1) Flexible, the mask module can be integrated with any ConvNets in a plug-and-play manner. (2) Simple, the mask module is implemented by a hard Sigmoid function with small number of trainable mask variables, adding negligible memory and computational overheads to the networks during training. (3) Effective, it is able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless of the datasets and ConvNet architectures used. (4) Fast, it is observed that the number of training epochs required by MaskCon-vNet is close to training a baseline without pruning. ( 5) Budget-aware, with a sparsity budget on target metric (e.g. model size and FLOP), MaskConvNet is able to train in a way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint simultaneously. Results on CIFAR-10 and ImageNet with several ConvNet architectures show that MaskConvNet works competitively well compared to previous pruning methods, with budgetconstraint well respected. Code is available at https://www.dropbox.com/ s/c4zi3n7h1bexl12/maskconv-iclr-code.zip?dl=0. We hope MaskConvNet, as a simple and general pruning framework, can address the gaps in existing literate and advance future studies to push the boundaries of neural network pruning.

1. INTRODUCTION:

State-of-the-art Convolutional Neural Networks (ConvNets) achieve high accuracy in many applications using very deep and wide networks, e.g., VGG Simonyan & Zisserman (2015) and ResNet He et al. (2016) . Deep and wide networks are often over-parameterized, which significantly increase the usage of memory (with tens of millions of weights) and compute (with tens of billions of MAC -Multiplication and ACcumulation). It is particularly challenging to deploy, in a naive manner, such large ConvNets on edge device that with limited memory and compute capacities.

To reduce the memory footprint and MAC of modern ConvNets, recent works propose either unstructured pruning (a.k.a. weight pruning) to prune redundant weights LeCun et al. (1990) ; Han et al. (2015; 2016) ; Zhu & Gupta (2017) ; Guo et al. (2016) ; Louizos et al. (2017a; b) ; Frankle & Carbin (2019) ; Lee et al. (2019) ; Gale et al. (2019) or structured pruning (e.g. filter-wise, channel-wise, depth-wise, etc.) to prune filters/channels/layers Li et al. (2016) ; Wen et al. (2016) ; Liu et al. (2017) ; Ye et al. (2018) ; Neklyudov et al. (2017) ; Gordon et al. (2018) ; He et al. (2018) ; Véniat & Denoyer (2018) ; He et al. (2019) ; Lemaire et al. (2019) ; Li et al. (2019) ; Yang et al. (2019) from original over-parameterized deep networks, without incurring significant loss in accuracy. Both unstructured and structured pruning approaches support pruning from pre-trained models followed by fine-tuning and pruning while training from scratch.

Despite the impressive pruning rates achieved on target metrics such as weight variables and floatingpoint operations (FLOP), existing neural network pruning methods still suffer one or both of the weaknesses that (1) Additional (iterative) fine-tuning is often required to compensate the accuracy degradation, leading to slower convergence and thus more training epochs and (2) Most of existing pruning approaches are not capable of automatically and optimally allocating pruning rates across layers given a sparsity budget on target metric. On the other hand, pruning with budget constraints tailored for specific hardware platforms, as a promising direction for hardware-software co-optimization, is becoming increasingly important especially for edge device.

In this work, we propose a new pruning framework, called MaskConvNet, to address the above mentioned issues simultaneously. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to each bundled block of the target ConvNets. Fig. 1 showcases an example of applying MaskConvNet to a typical bundled block Conv/BN/ReLU for filter pruning. The mask module is composed of a hard Sigmoid activation function with a vector of trainable variables as input and the output is regularized with a sparsification loss, resulting in a sparse mask vector with many 0 elements and the rest are real values in (0, 1]. The pruned ConvNets is generated by element-wise multiplication between the mask vector and filters/BN parameters, in which filters/BN parameters with all zero are removed from the ConvNets.

We summarize the main contributions and advantages of MaskConvNet as follows.

• Flexible. The mask module can be integrated with any ConvNets in a plug-and-play manner.

• Simple. Instead of directly regularizing filters, the mask module is defined separately and differentiable almost everywhere, and the number of mask variables equals to the number of filters in the target ConvNets, implying that it adds negligible memory and computational overheads to the networks during training. • Effective. MaskConvNet is able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless of the datasets and ConvNets architectures used. • Fast. Our empirical observations on both CIFAR10 and ImageNet show that the number of training epochs required by MaskConvNet is close to training a baseline without pruning, mainly attributed to the fact that the mask module (1) enables threshold-free pruning, thanks to the hard Sigmoid activation function and (2) is shared within the bundled block structure (Fig. 1 ) which in turn addresses the mismatch problem between CONV and BN layers. • Budget-aware. With a sparsity budget on target metric (in percentage), MaskConvNet is able to train in a way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint simultaneously. In contrast, previous budget-aware pruning methods suffer the iterative optimization process of alternating between training and evaluation of the target metric, which is tedious and time-consuming.

In recent years, structured pruning attracts much more attention than its unstructured counterpartweight pruning, mainly because structured pruning doesn't require specific circuit design and thus is more friendly to existing hardware. Thus, in this work we apply MaskConvNet to one of most popular structured pruning methods, filter-wise pruning. It's worth noting that MaskConvNet, as a general pruning framework, can be easily extended to other structured pruning categories as well as weight pruning, by re-formatting the mask variables only. We would like to leave it for future work. 2019), require to perform additional (iterative) fine-tuning to compensate the accuracy degradation, mainly due to hard pruning via thresholding and the mismatch between Batch Normalization and its corresponded pruned CONV/FC layer. This leads to slower convergence and thus more training epochs. For instance, with the method of gradual magnitude-based weight pruning during training proposed by Zhu & Gupta (2017) and pre-defined pruning rate 80% on ResNet50, Gale et al. (2019) reported that it increases the number of training epochs by 1.5× in order to achieve comparable ImageNet validation accuracy with the baseline model trained without pruning.

2. RELATED WORK:

On the other hand, there is few work studying pruning mechanism to automatically and optimally allocate pruning rates across layers that fulfill a budget specified for target metric. Existing works either only meet the budget by pruning using sub-optimal pruning rate pre-defined or empirically calculated Han et al. (2015) ; Li et al. (2016) ; Liu et al. (2017) or can derive automatic pruning rate through sparsity learning methods which fail to fulfill the budget explicitly Louizos et al. (2017a; b) ; Wen et al. (2016) ; Neklyudov et al. (2017) .

MorphNet Gordon et al. (2018) and BAR Lemaire et al. (2019) are the most related work to MaskCon-vNet, while our work differs from them in several aspects. First, both MorphNet and BAR require pre-defined threshold for hard pruning, while MaskConvNet is threshold-free. Second, MorphNet and BAR apply mask on γ variables of BN and filters respectively, while MaskConvNet applies mask on both. Third, to meet the sparsity budget, both MorphNet and BAR alternate between training and evaluation of target metric with a multiplier derived by grid search. MaskConvNet automates the process by adaptively shrinking or expanding the pruning ratio across layers in a data-driven manner during training, avoiding the tedious iterative optimization process. Finally, BAR uses Knowledge Distillation Hinton et al. (2015) to facilitate the training of pruned networks for better accuracy, while MaskConvNet does not.

3. METHOD:

3.1 FILTER PRUNING VIA THE MASK MODULE Fig. 1 presents an overview of the design of the mask module and how the mask module is applied for filter pruning. The mask module is simply defined as a hard Sigmoid activation function,

EQUATION

where input (m) is trainable mask variables tied to Conv filters1 . For a Conv layer with w input channels, h output channels (filters) and k × k sized kernels, the number of mask variables equals to h. Each element of the mask activation σ(m) indicates the importance score of the associated filter.

The same mask is applied to bias parameters for the Conv layer.

Each filter is reparameterized as its weights multiplied by its mask activation via element-wise multiplier. Concretely, given a filter W ∈ R and its mask activation σ(m) ∈ R, the masked filter Ŵ is given by:

EQUATION

where is element-wise multiplication between a mask element and a 3D tesor with size w × k × k.

It's easy to know that mask element with importance score 0 means a filter with w × k × k weight parameters is pruned from the network. The sparser the mask activation, the higher the pruning ratio.

It is worth noting that the zero-centered hard-Sigmoid σ(m) clamps mask to [0, 1], allowing mask to span the whole range rather than binary. Thus, thresholding critiria is no longer required for hard pruning. The mask module is similar in spirit to the diversity networks Zelda Mariet (2016) in the sense that the pruned filters are "fused" into the remaining filters via non-zero mask elements rather than dropped completely, but our proposal is much simpler.

In our experiments, m is simply initialized with zeros, so that σ(m) = 0.5, which is median of interval [0, 1]. One can initialize with random values, but such initialization could potentially give unknown prior bias to sparsification, which may give different structure with different random seed.

In the end of training, the pruned ConvNets is generated by Equation 2, where filters/BN parameters with all zero are removed from the ConvNets. One may note that the output feature maps of Conv layer i serve as the input to Conv layer i + 1. Thus, the input channels of layer i + 1 that corresponded to the pruned filters of layer i are also pruned.

Mask Sharing. In modern ConvNets, a Conv layer is often followed by a batch-norm layer which has "scale" weight parameter γ and "shift" bias parameter β with dimension length equals to number of filters of its preceding Conv layer.

Previous pruning approaches often prune Conv filters with their successive Batch Normalization layer unchanged. This causes mismatch problem between pruned Conv and BN layers, given that by nature BN parameters highly depend on the output feature maps from preceding Conv. To avoid the mismatch problem, we propose to share the mask module within the bundled block structure Conv-BN-ReLU. More specifically, BN scale and shift parameters are masked with the same mask activation for the corresponding Conv layer. Without mask sharing, we observed that the accuracy degrades severely after pruning. We speculate that this is due to BN by default are unaware of pruned filters, i.e. pruned filters give 0-valued output as input to BN, but BN considers 0-valued input as if the filter is still alive (i.e not pruned).

Extending Mask to Residual Connections. In ConvNet architectures such as ResNet, residual connections involve element-wise addition operation where two operands must have the same dimensions. To extend the mask module to residual connections, one can share a single mask for all layers involved in residual connections. This is achieved by taking the mask for the last Conv layer in the last residual block as the base mask, and sharing the base mask with other Conv layers involved in other residual connections. Partially, the base mask is sliced to fit Conv layers with less number of filters.

3.2. SPARSITY REGULARIZATION ON THE MASK:

With filters W reparameterized by the mask activation σ(m), the loss function for training budgetaware pruned ConvNets from scratch is given as:

EQUATION

where L E is cross-entropy loss for classification (the "error"), and L S is the sparsification loss defined on the mask.

Sparsification Loss. The sparsification loss is defined by sparsifying the mask:

EQUATION

where the goal is to minimize the mean µ(σ(m)), and maximize variance-to-mean σ 2 (σ(m))/µ(σ(m)). In particular, the regularization multipliers λ m and λ v are introduced to precisely control regularization strength towards budget-aware pruning (see details in the subsequent subsection).

Minimizing mean is basically L1-regularization averaged over the total number of mask elements of the network, and maximizing variance is soft-binarization by dispersing mask elements to the either zero or one. Dividing variance by mean adaptively strengthen variance maximization when mean is becoming smaller.

Extension to Multi-metrics. Equation 3 can be extended to support sparsity regularization of the mask for multiple target metrics simultaneously, e.g. optimizing for parameter counts and FLOPs given by budget constraints for both metrics. A simple heuristic is to average the mean and variance-to-mean of the mask σ(m) that is applied to N target metrics. Accordingly, Equation 3 and Equation 4 are rewritten as 

L(W, m) = L E (W σ(m)) + N i=1 L Si (σ(m)) and N i=1 L Si (σ(m)) = N i=1 λ mi • µ(σ(m)) - N i=1 λ vi • σ 2 (σ(m)) µ(σ(m)) ,

; ∆s ← s * -s t ; λ m ← λ base m × ∆s; λ v ← λ base v

× ∆s; end Train on (x i , y i ) with cross-entropy and mask regularization as in Eqn. 3; end

3.3. BUDGET-AWARE MASKCONVNET: THE ALGORITHM:

In previous works on sparsity-regularization based pruning Gordon et al. (2018) ; Lemaire et al. (2019) , grid search on sparsity regularization multiplier is performed, the multiplier is then used to find a sparse network structure that could meet the budget. These approaches alternate between network training and sparsifying with the regularization multiplier until sparsification stops at arbitrary sparsity level. If the model is undersparsed or oversparsed, the regularization multiplier value is adjusted and training is repeated from scratch. This is very laborious and time-consuming.

In contrast to previous works, MaskConvNet avoids the tedious iterative optimization process. With a target sparsity budget (in percentage), MaskConvNet is able to train in a way that the optimizer adaptively sparsifies the network and automatically maintains sparsity level, till the pruned network produces good accuracy and fulfills the budget constraint simultaneously.

The training procedure for one epoch is described in Algorithm 1. During training, the weight tensors are reparameterized with masks according to Eqn. 2, both weight tensors and masks are jointly updated and sparsified via standard optimizer (e.g. SGD with momentum) according to Eqn. 3. Evaluating sparsity percentage at every mini-batch iteration can slow down training considerably, hence we evaluate and smooth the sparsity (with EMA) for every n interval. In our experiments, we set n = 20 and EMA exponent α = 0.99.

As the regularization multipliers are periodically updated, sparsification is expanded stronger in early training, and becomes gradually weaker when reaching the target due to ∆s approaches zero. On the other hand, when the pruned model is oversparsed, ∆s becomes negative, and so do the multipliers λ m and λ v . The optimizer will then shrink the mask sparsity (unsparsification) until ∆s is positive again (See more detailed analysis in the next subsection).

3.4. HOW MASK UNSPARSIFICATION WORKS?:

In case when λ m and λ v are too big, too many mask elements will be zeroed and the model becomes oversparsified and misses the sparsity budget. If the masks elements are zeroed, how would the optimizer continue update these zeroed elements? How the gradients flow and update the masks? We exploit three things to unsparse the masks: (1) Mask decay update; (2) Negative regularization multiplier; and (3) Positive mask parameter when hard-Sigmoid output is zero.

The SGD update rule for the regularization term L S (m) is given as m ← m -λ m η ∂S ∂m -λ m η m, where η is the learning rate and is mask decay constant. Also recall the hard-Sigmoid function on mask is σ(m) = min(max(m + 0.5, 0), 1). λ m will be negative when sparsity difference ∆s is negative. Also, when m = -0.5, then σ(m) = 0 and ∂S ∂m = 0. However, m is still non-zero negative. Hence, the term -λ m η m is positive , and thus the optimizer is able to increase m so that σ(m) becomes non-zero, unsparsifying the mask.

The catch here is that we cannot achieve exactly zero error to target sparsity budget due to zeroed σ(m) will always be bounced back to non-zero (unless we deliberately turn off mask decay). The model will always be undersparsfied and oversparsified continuously by small degree throughout training. The sparsity deviation depends on the value of mask decay constant applied to the mask. This will be discussed in experimental section. 

4. EXPERIMENTS:

We evaluate MaskConvNet on both CIFAR-10 and ImageNet ILSVRC 2012 et al. (2009) , with different ConvNets architectures including Simonyan & Zisserman (2015) ; He et al. (2016) ; Zagoruyko & Komodakis (2016) as baseline architectures. For all CIFAR-10 experiments, we modified source code from Liu et al. (2018) , and followed the same training setup and data augmentation. We used SGD with momentum of 0.9 as optimizer. Initial learning rate is set to 0.1 and divided by 10 at epoch 40 and epoch 80. All models are trained for 160 epochs. λ m and λ v are updated every 20 mini-batches with EMA exponent α = 0.99. Details on the settings of λ m and λ v are left in appendix section. For all ImageNet experiments, we implemented the training code and hyper-parameter settings derived from source code for work by Banner et al. (2018) . The convergence speed and sparsity pattern of MaskConvNet during training. Figure 2 shows the testing/validation accuracy (Top-1) and sparsity pattern as the function of training epochs for MaskConvNet with WideResNet-28-10 on CIFAR-10 and ResNet-34 on ImageNet. The baseline accuracy is included for reference. We observe that the convergence speed of MaskConvNet is highly similar to the baseline architecture, indicating the effective training capability of MaskConvNet. In addition, the sparsity pattern emerges at early training epochs and becomes stable as training going on, which is consistent with the observation from the single-shot pruning work Lee et al. (2019) that redundant neurons can be identified in a single step even before training. One may note that the sparsity curve for ResNet-34 has small fluctuations. As shown in the next subsection as well as appendix, this issue can be solved by proper choice of mask decay.

Can MaskConvNet meet the Budget?. Figure 3 presents the results of MaskConvNet-P (parameter counts as target metric) and MaskConvNet-F (FLOP as target metric) with various sparsity budgets on CIFAR-10, using VGG-19 and WideResNet-28-10 as baseline architectures. Accuracy almost linearly decreases with increasing sparsity for both target metrics. More importantly, it is worth noting that the actual sparsities produced by both MaskConvNet-P and MaskConvNet-F are close to the budget, i.e. the error between actual and budget sparsity is within 1% (absolute value) for most of the operating points.

Looking into layer-wise parameter sparsities. We also investigate layer-wise parameter sparsity patterns throughout training on CIFAR-10, for VGG-19 trained with 50% FLOP sparsity budget and WideResnet-28-10 trained with 93.75% parameter sparsity budget. Figure 4 shows VGG-19 and WideResnet-28-10 training results respectively. We observe that early layers are less likely to be sparsified, which matches the intuition in previous works such as Gordon et al. (2018) . The hyper-thesis is that early layers carry more information and can incur bigger accuracy loss if sparsified too much. In addition, we noted more uniform overall sparsity on WideResNet-28-10 than VGG-19. We speculate this to be related to role of residual connections during sparsification process. This phenomenon is left for future investigation.

More results and analysis are introduced in the appendix section, including the sensititity analysis of base regularization multipliers λ m and λ v , visualization of mask histogram output by the hard-Sigmoid, inference speedup by measuring wall-clock latency of the baseline and pruned models on a CPU core, and the effect of mask decay and warmup on training accuracy as well as layer-wise sparsity pattern. 

5. CONCLUSION AND FUTURE WORK:

In this work, we proposed MaskConvNet, a general pruning framework where a model is trained and sparsified end-to-end with sparsity budget constraint on various target metrics. Promising results are reported on CIFAR-10 and ImageNet for the problem of filter pruning. We hope MaskConvNet can inspire future research including extension of the pruning framework (1) to other structured pruning categories as well as unstructured pruning and (2) to other model structures beyond ConvNets, such as recurrent networks and self-attention networks. 

A.5 EFFECT OF MASK DECAY AND WARMUP:

Figure 6 shows different mask decays give different rate of unsparsification. Also with warmup training (late sparsification), accuracy is improved and the pruned structure becomes different compared to the one without warmup -High contrast is visible between sparsity level between last layers and early layers. Last layers are sparsified more. 

:

We take Conv/FC as example here as they account for over 99% of total operations in modern ConvNets. FC can be treated as a special case of Conv layer where the kernal size is 1 × 1.

Figures and Tables:

Figure 2: Testing/Validation accuracy (Top-1) and sparsity pattern as the function of training epochs for MaskConvNet with WideResNet-28-10 on CIFAR-10 (left) and ResNet-34 on ImageNet (right).For reference, the baseline accuracy is included.

Figure 3: Results of MaskConvNet-P (parameter counts as target metric) and MaskConvNet-F (FLOP as target metric) with various sparsity budgets on CIFAR-10, using VGG-19 and WideResNet-28-10 as baseline architectures. Text labels on plot points denote "Accuracy% (Actual / Budget Sparsity%)

Figure 6: Layer-wise parameter sparsities throughout training for VGG-19 on CIFAR-10, with different mask decay and warmup training. The brigher the color, the higher the sparsity.

Comparison of MaskConvNet-F or MaskConvNet-P with state-of-the-art pruning approaches Network Slimming Liu et al. (2017), L1-Pruning Li et al. (2016) and BAR Lemaire et al. (2019), on CIFAR-10 with various ConvNet architectures.

Table content: <table><tr><td>Model</td><td>Accuracy (%)</td><td>FLOPs</td><td colspan="3">Sparsity (%) Parameters (M) Sparsity (%)</td></tr><tr><td>VGG-16</td><td/><td/><td/><td/><td/></tr><tr><td>Baseline</td><td>93.63</td><td>3.15 × 10 8</td><td>0.00</td><td>14.99</td><td>0.00</td></tr><tr><td>L1-Pruning</td><td>93.40</td><td>2.06 × 10 8</td><td>34.20</td><td>5.40</td><td>64.00</td></tr><tr><td>MaskConvNet-F</td><td>93.40</td><td>1.88 × 10 8</td><td>40.19</td><td>1.72</td><td>88.53</td></tr><tr><td>VGG-19</td><td/><td/><td/><td/><td/></tr><tr><td>Baseline</td><td>93.67</td><td>7.97 × 10 8</td><td>0.00</td><td>20.04</td><td>0.00</td></tr><tr><td>Network Slimming</td><td>93.80</td><td>3.91 × 10 8</td><td>51.00</td><td>2.30</td><td>88.5</td></tr><tr><td>MaskConvNet-F</td><td>93.03</td><td>3.66 × 10 8</td><td>54.08</td><td>1.51</td><td>92.47</td></tr><tr><td>ResNet-56</td><td/><td/><td/><td/><td/></tr><tr><td>Baseline</td><td>93.40</td><td>2.52 × 10 8</td><td>0.00</td><td>0.85</td><td>0.00</td></tr><tr><td>L1-Pruning</td><td>93.06</td><td>9.09 × 10 7</td><td>27.60</td><td>0.73</td><td>13.7</td></tr><tr><td>MaskConvNet-F</td><td>92.25</td><td>1.88 × 10 8</td><td>25.27</td><td>0.58</td><td>31.87</td></tr><tr><td>ResNet-164</td><td/><td/><td/><td/><td/></tr><tr><td>Baseline</td><td>94.70</td><td>5.02 × 10 8</td><td>0.00</td><td>1.71</td><td>0.00</td></tr><tr><td>Network Slimming</td><td>94.92</td><td>3.81 × 10 8</td><td>23.70</td><td>1.44</td><td>14.9</td></tr><tr><td>MaskConvNet-F</td><td>94.28</td><td>3.46 × 10 8</td><td>30.96</td><td>0.79</td><td>53.59</td></tr><tr><td>WideResNet-28-10</td><td/><td/><td/><td/><td/></tr><tr><td>Baseline</td><td>95.30</td><td>1.19 × 10 10</td><td>0.00</td><td>36.48</td><td>0.00</td></tr><tr><td>BAR (16× V reduction)</td><td>91.62</td><td>1.74 × 10 8</td><td>98.54</td><td>2.31</td><td>93.67</td></tr><tr><td>MaskConvNet-P</td><td>94.28</td><td>3.50 × 10 9</td><td>70.63</td><td>2.46</td><td>93.25</td></tr></table>

Comparison of MaskConvNet-FP with both parameter counts and FLOP as target metrics withLi et al. (2016) on ImageNet ILSVRC 2012 with ResNet-34 as baseline architecture. Table1compares MaskConvNet-F or MaskConvNet-P with state-of-the-art pruning approaches Network SlimmingLiu et al. (2017), Pruning Filters for Efficient ConvNetsLi et al. (2016) denoted as L1-Pruning and Budget-Aware Regularization (BAR)Lemaire et al. (2019), on CIFAR-10 with various ConvNet architectures including. One can see that for almost all of the operating points, MaskConvNet-F performs comparably in terms of accuracy with Network Slimming and L1-Pruning, and achieves much higher sparsity on both FLOP and parameter counts. Compared to BAR, MaskConvNet-P achieves comparable parameter sparsity with better accuracy, but the FLOP sparsity is much lower probably because the usage of Knowledge DistillationHinton et al. (2015) in BAR as prior guiding the training process while our method does not use it.ImageNet. Table2compares MaskConvNet-FP with both parameter counts and FLOP as target metrics with L1-PruningLi et al. (2016), with ResNet-34 as the baseline architecture. In L1-Pruning paper, Resnet-34 is pre-trained, pruned, and fine-tuned for 20 epochs. Standard ResNet training takes 90 epochs. Compared to L1-Pruning, we managed to train ResNet-34 with fewer epochs (<90) that achieves the same accuracy and much higher parameter sparsity, but with slightly lower FLOP sparsity.

Table content: <table><tr><td>Model</td><td>Accuracy (%)</td><td>FLOPs</td><td colspan="3">Sparsity (%) Parameters (M) Sparsity(%)</td></tr><tr><td>Resnet-34</td><td/><td/><td/><td/><td/></tr><tr><td>Baseline</td><td>73.27</td><td>3.69 × 10 9</td><td>0.00</td><td>21.8</td><td>0.00</td></tr><tr><td>L1-Pruning</td><td>72.56</td><td>3.08 × 10 9</td><td>15.50</td><td>19.9</td><td>7.60</td></tr><tr><td>MaskConvNet-PF</td><td>72.56</td><td>3.29 × 10 9</td><td>10.75</td><td>17.7</td><td>19.00</td></tr></table>

Sensitivity analysis for base regularization multipliers λ base

Table content: <table><tr><td>m</td><td>and λ base v</td><td>with VGG-19 trained</td></tr></table>

Latency Measurements on CIFAR-10

Table content: <table><tr><td>Model</td><td colspan="3">Target FLOP sparsity (%) Actual FLOP sparsity (%) Latency (ms)</td></tr><tr><td>VGG-19</td><td>0 (Baseline)</td><td>0.00</td><td>22.45 ± 1.17</td></tr><tr><td/><td>50</td><td>51.91 ± 2.46</td><td>12.94 ± 0.42</td></tr><tr><td/><td>60</td><td>62.35 ± 2.92</td><td>11.42 ± 0.28</td></tr><tr><td/><td>70</td><td>69.52 ± 0.01</td><td>10.81 ± 0.06</td></tr><tr><td colspan="2">WideResnet-28-10 0 (Baseline)</td><td>0.00</td><td>64.15 ± 0.21</td></tr><tr><td/><td>50</td><td>54.59 ± 1.5</td><td>25.43 ± 1.09</td></tr><tr><td/><td>60</td><td>58.54 ± 2.5</td><td>22.79 ± 0.94</td></tr><tr><td/><td>70</td><td>72.58 ± 4.2</td><td>19.92 ± 0.93</td></tr></table>

Peer review (iclr2020):
This paper proposes a masking process to improve the pruning of DNN. In addition, the algorithm proposes to automatically allocate the pruning rates over layers. On the positive side: - I do believe the main contribution is automatically allocating the pruning rates over the network. Related works: - How does this relate to methods using gating to prune in the presence of residual layers or BN? - Paper claims multiple times that related works need a train-prune-retrain process which is only valid for post-processing works. - I missed sparsity promoting works related to training from scratch where similar masks are implicitly used during training (even not explicitly stated) to maintain the zeros in the network. At least in [1,2] the training time is the same as the time used for training a network from scratch (same claim as in this paper). Method: - The mask is trained using a sigmoid function and claims this will be representative of the relevance of the *neuron* within the layer. How is this really related? - Why the initialization of the mask is to the mean? I think I am confused there. For initialization, I would argue all the neurons/parameters are relevant, right? - The claim that this method is *much simpler* is a bit subjective. I do not see why. Please elaborate. - I am not sure if the claim of pruning filters and not considering the bn layer is correct. I would tend to think that, if pruning a neuron, the corresponding BN module should be modified (that is, propagating the zero to subsequent layers). - I do like the extension to residual connections. It would be great to have more and clearer details on how is this done. Would this also apply to the UNET type of architecture? - What is the intuition behind Eq 4 and how it is related to the relevance of a parameter? - The extension to multiple metrics is of interest, however, there is little detailed there. How is the automatic allocation done? This is repeated in section 3.3 but details are missing. - My understanding is that the regularization multiplier is affected by the learning rate, therefore their effect is lower as the training progresses. In this case, seems the opposite, right? (page 6 before 3.4). - The part with the sparsity budget is interesting. What guarantees are that the newly enabled neurons are actually useful? Could it be possible that the budget suggested is not the right for the task at hand and, therefore, the additional parameters are not really relevant / needed? Experiments: - There is loss with little sparsity when it comes to Imagenet (15 and 17% pruning) does not seem very promising even the training time is similar. - Seems like the experiments and comparisons to L1-pruning are not very surprising. It would be nice to have more comprehensive numbers. For imagenet, if using Resnet-50, would be easier to compare to other numbers. - In the imagenet comparison, L1-pruning is the version-A of the paper. What about the others or why that particular model? - How are the actual groups made? Minor details: - Please improve figure 1. It is not easy to understand. Same with Figure 3. What is seen in Figure 4. - I do believe the WideResNet-28-10 number of parameters for BAR is not correct. - Section 4.2 is a bit overselling. I do not see *much-higher* parameter sparsity. The claims are mostly valid for VGG type of networks in this particular setting. [1] Learning the Number of Neurons in Deep Networks, NeurIps 2016 [2] Compression-aware training of DNN, NeurIps 2017