Learning sparse DNNs with soft thresholding of weights during training

Abstract: This paper proposes a new and simple way of training sparse neural networks. Our method is based on a differentiation of the forward and backward paths: the weights in the forward path are a thresholded version of the weights maintained in the backward path. This decoupling allows for micro-updates, produced by gradient descent, to stack up, leading to the possible re-activation of weights that were set to zero in earlier training steps. At the end of training, links with zero weights are pruned away.

Additional critical specificities of our approach lie (i) in the progressive increase of the zeroed weight ratio along the training, and (ii) in the use of soft-thresholding rather than hard-tresholding to derive the forward-path weights from the ones  maintained in the backward path.
At constant accuracy, our approach reduces the number of training cycles to 1 compared to the state-of-the-art recursive pruning methods. At high pruning rates, it also improves the model accuracy compared to other single cycle pruning approaches (66.18% top-1 accuracy when training a ResNet-50 on ImageNet at 98% sparsity).


Reviews and author responses:

 We thank the reviewers for the time they devoted to our submission. Their constructive and valuable comments will definitely help us to strengthen our paper.

We indeed missed the method presented in [1], published a few months before our ICLR submission. After careful analysis, it appears that we need more time than the rebuttal period to run the experiments required to provide a fair comparison with this method. Indeed, the results presented in [1] cannot be directly and fairly compared, neither with the results presented in our submission, nor with the ones presented in most of the other papers addressing the pruning question in the literature. This is because the authors in [1] have run their training on twice more epochs than the 160 epochs conventionally adopted in the literature (as adopted in [2], and in most subsequent papers [4,6]). This difference is not mentioned in the CVPR paper, where the number of epochs considered to generate the results is simply not defined, but is attested by the code available at [3]. This makes the comparison unfair since it is known that the number of training epochs largely impacts the sparse model accuracy [2,5 (Table 2)]. Beyond the increased number of epochs, it is also worth noting that [1] further increases the training cost by multiplying the number of gradients to compute at each training iteration by a factor corresponding to the number of samples of the Gumbel distribution considered at each training step (see the sum on index i on line 9 in Algorithm 1, on page 3603 of their paper). **Hence, the plots provided in [1] have been generated with significantly increased computational resources** (presumably at least 4 times higher than in our case). More results are needed to position [1] with respect to the SoA, but the experiments provided in our submission demonstrate that all previous works adopting 160 number of epochs per training cycle end up in models with weaker accuracy than our proposal.

Regarding novelty, we respectfully disagree with the reviewer considering that the method is not novel because it only involves components (soft-thresholding, straight-through gradients, and progressive increase of the pruning ratio) that have been considered before, in a variety of contexts. Indeed, none of the previous works in the field has ever assembled those components in the way our paper is combining them. Consequently, none of those previous works has observed that sparse networks with best-in-class accuracy could be trained in a single training cycle, without resorting to differentiable proxies to approximate the discontinuity induced by pruning. This definitely makes our contribution original and valuable. We are sorry that our writing was not sufficiently explicit on that point.

Overall, as of today, **no previous work could achieve the accuracy we have reported in our submission with the same number of epochs per training cycle (even when more training cycles are run than in our approach)** , and we remain confident that, based on a fair comparison, i.e. at constant training cost, our approach will also achieve a similar or better accuracy than [1]. The fact that our solution reaches this accuracy without resorting to sophisticated differentiable approximation of the pruning operator is probably the most important lesson to draw from our paper, since it reveals that accurate optimization of the loss function with respect to the pruning variables is not required to reach sparse models with good generalization performance, thereby stressing once again the difference between optimizing the loss function and getting good generalization performance.

Thanks again to all reviewers for helping us to better position our work. We hope to rapidly present a consolidated version of our work.

[1] Xiao Zhou et al., Effective sparsification of neural networks with global sparsity constraint, IEEE CVPR 2021.

[2] J. Frankle et al., The early phase of neural network training, ICLR 2020.

[3] https://github.com/x-zho14/ProbMask-official

[4] Renda, A., Frankle, J., & Carbin, M. (2020). Comparing Rewinding and Fine-tuning in Neural Network Pruning, ICLR 2020

[5] Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2019). Rigging the Lottery: Making All Tickets Winners, ICLR2020

[6] Wang, H., Qin, C., Zhang, Y., & Fu, Y. (2020). Neural Pruning via Growing Regularization, 1–15. Retrieved from, ICLR2020 

The paper uses three techniques to "sparsify" a dense network over training. These techniques are: non-persistent pruning, which backpropagates the gradients and updates a copy of the dense weights of the network; soft thresholding, which replaces the hard threshold typically used while prunning neural networks with the soft threshold operator; and increasing the pruning ratio linearly while training. The paper shows that these three techniques can achieve good accuracy at a high sparsity levels on ImageNet and CIFAR-10, when compared to other approaches.  I enjoyed very much reading the paper, especially the "State of the Art" section which covers many relevant papers and explains existing approaches in detail. In addition, each subsection in the experiments and section 5, start with a brief summary of the main points of that section, which are very useful to keep track of the main points. Thus, congratulations for a wonderfully structured paper.

In terms of the content of the paper itself, my main criticism is lack of novelty. The three ideas that are used to "sparsify" the network have been proposed in the past. Namely, non-persistent pruning by Hubara et. al (2016) and Mellempudi et al. (2017); soft thresholding originally developed by Donoho (1995) and used in the context of sparse neural networks by Kusupati et al. (2020); and increasing the pruning ratio also used, for instance, by Kusupati et al. (2020) and Wang et al. (2020). Thus, the main contribution of the paper is the combination of these three existing techniques, and the experimentation on ImageNet and CIFAR-10.

The experiments were performed following standard practices: well-known datasets and architectures (ResNet-20 and VGG-13), results averaged over three independent runs initialized with different random seeds, and the test accuracy is reported at different sparsity levels (or equiv. fraction of remaining parameters after prunning).

The results of the experiments generally support the claims of the paper, but a comparison with more relevant and recent State of the Art prunning approaches is lacking (although the paper covers a broad literature in Section 2). The only published work that is directly compared is that of Kusupati et al. (2020) in Section 4.2, there are no other published baselines reported in the paper. 

The paper does perform several ablation studies in Section 4.3. In particular, the benefit of non-persistent pruning and soft-thresholding are assessed. Interestingly, the benefit of increasing the pruning ratio is not assessed in this section, instead the use of "LAMP or L1" is evaluated in Section 4.3.3 (which is somehow orthogonal to the ideas proposed in the paper). In addition, some of the conclusions drawn from the experiments are not clear to this reviewer:

- At the end of Section 4.3.1, the paper concludes that "a clear trend emerges" from Figure 3: "updating only the active weights during back-propagation leads to a lower accuracy compared to updating all weights". This sentence should be refined a bit, since this observation is only true when fraction of remaning weights is below 6.1.

- At the end of Section 4.3.3, the paper reports that "to the right of Figure 4 LAMP is shown to significantly improve the results". It is true that LAMP seems to be necessary for high sparsity levels (< 2% remaining weights) when hard thresholding and rewind are used, but in all other cases, if one compares the left (a global 11 pruning criterion) and right (LAMP) plots, for a given sparsity level, the accuracy reported in the left plot is higher than in the right. For instance, at 3.5% of remaining weights the left reports ~81.5% while the right reports ~80% for "Ours (hard th.)" ; for "Ours" left reports ~87.5% while the right reports ~86.5%; etc.

Finally, some additional minor comments:

- In the introduction, there seems to be some notes left from the paper draft: "to progressively increase the ratio of pruned **(inactive??)** weights".

- Table 1: I would suggest to keep a consistent order of the subcolumns for "Ours" and "Kusupati et al. (2020)". Currently, the first shows "Accuracy" followed by "Sparsity", while the other shows "Sparsity" followed by "Accuracy". This can confuse the reader.

- Soft-thresholding is not explained at all in the paper. One needs to check Kusupati et al. (2020) to understand what exactly the soft-thresholding does. I would suggest to add Eq. (1) from that paper, since it captures what soft-thresholding is. 

- Several figures in the paper (e.g. Figure 3, Figure 4) report the mean accuracy over three independent runs and the "variance". Is this a typo and the reported error bars are "standard deviation". Notice that reporting "mean +/ variance" does not make much sense since the variance doesn't even have the same units as the mean. Standard deviation or (even better) confidence intervals should be reported as error bars. Given the lack of novelty of the proposed approaches (all of them existed before), and the lack of a more in-depth comparison with existing state-of-the-art approaches, I recommend to reject the paper. 

In order to change my mind and vote to accept the paper as an "empirical paper", I believe additional experiments with a) existing baselines, b) additional datasets, and/or c) additional architectures, are required. This would increase the significance of the experimental results. 

In this paper, the authors proposed a training algorithm for sparse deep neural network. Their key idea is to decouple the forward and backward paths, that is, the weights used in the forward path are a threshold version of the weights maintained in the backward path. In this way, some pruned weights have possibility to be re-activated. The authors also conducted some experiments to evaluate the performance of their methods.  Firstly, I have some concerns about the novelty of this paper:

  --As the authors stated, one advantage of their proposed method is that their method allows the pruned weights to be re-activated, since the pruned weights are still involved in the backward propagation. However, I know that various existing methods have this property, such as [1] and [2]. The technique used in [1] is similar with this paper. I know [1] focuses on channel pruning, but I believe that it can be extended to weigh-level pruning easily. I suggest the authors to discuss and compare with them. 

  --The method Kusupati et al. (2020) also adopted the soft-threshold to stabilize the training process. 

  --Another main component of the proposed method, that is, the strategy of increasing the pruning ratio during training can also be found in the existing methods, such as Kusupati et al. (2020) and [2]. The authors need to give more discussion, about their differences and give empirical comparison among them.

Therefore, we can see that the three main components of the proposed can all be found in the existing methods. Therefore, this paper looks like a combination of several existing ideas. This is OK if one can achieve state-of-art in this way. Otherwise, the contribution and novelty of this paper is limited. However, comparing the empirical results (e.g., Table 1) of this paper with [2], I notice that the proposed method is worse than [2], especially when the sparsity is high. 

Moreover, there are several other methods, which can learn the weight redudancy during training, such as [4] and [2]. The authors need to discuss and compare with them. 

At last, I would like to point out that Kusupati et al. (2020) is not STATE-of-THE-ART now, I recommend the authors to discuss and compare their methods with other baselines such as [2] and [3]. 



[1] He, Yang, et al. "Soft filter pruning for accelerating deep convolutional neural networks." arXiv preprint arXiv:1808.06866 (2018).
[2] Zhou, Xiao, et al. "Effective Sparsification of Neural Networks with Global Sparsity Constraint." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[3] Sparse Weight Activation Training, NeurIPS 2020. 
[4] Yuan, Xin, Pedro Henrique Pamplona Savarese, and Michael Maire. "Growing Efficient Deep Networks by Structured Continuous Sparsification." International Conference on Learning Representations. 2020.
 1. The novelty of this paper seems limited. 
2. The proposed method cannot achive STATE-of-THE-ART performance. 
3. More empirical comparsion is needed to better evaluate the performance of the proposed method. 

The paper proposes a single training cycle method to train sparse networks. It adopts a soft-threshold pruning strategy and progressively increase the sparsity ratio of the network along the training iterations. Experiments show its advantage. Strengths:
This idea is simple and easy to understand. 
Weaknesses: 
The organization and the English usage are not well.  This paper is more like a draft and needs improvements for many aspects. There are some concerns below.
1. It is better to discuss the connection of the proposed method with l1-norm based methods. Actually, the used soft-threshold function is commonly used for solving l1-norm minimization. 
2. It is better to add a citation for the used soft-threshold function.
3. It is better to arrange the three subfigures of Fig. 2 in a single row.
4. It is better to compare with other SOTA methods, e.g., two-cycles training strategy.
5. The progressive increment of the ratio of weights depends on the total of epochs. It is better to study more on this issue. 
6. Listing 1 and 2 are important. It is better to move them to the main text and explain the core steps. 
7. At the end of the first paragraph of section 1, there is a "(\cite)". At the third paragraph of section 1, there is a "(inactive ??)".  "l1-based" should be "$l$-1 norm based". "How are the pruned weights selected?" should be "How to select the pruned weights?". This paper is more like a draft and needs improvements for many aspects. For example, the connection with l1-norm based methods and the   ablation study of the ratio of weights. So it is below the threshold of ICLR.

The authors propose a method for unstructured model pruning that applies soft-thresholding in the forward pass, while using the straight-through estimator in the backward pass. The latter allows for accumulation of gradients and rewiring of earlier-pruned weights. The authors show SOTA performance for the high sparsity-regime on ResNet-50 for Imagenet, and include ablation experiments on CIFAR10 with ResNet-20. 
 Strengths:
The proposed method allows for one-cycle unstructured pruning that facilitates rewiring during training. The paper is in general easy to follow. 

Weaknesses:
In general, some (discussion on) related works and more complete experiments are lacking.

More specific feedbacks:

- There are more related works that concern (unstructured) sparsity (and some of them rewiring), which are not mentioned. How does your approach for example relate to (among others) the following:
** Narang (2017) "Exploring Sparsity in Recurrent Neural Networks"

** Bellec (2018): "Deep Rewiring: Training very sparse deep networks"

** Mostafa (2019): "Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization"

** Wortsman (2019): "Discovering neural wirings"

In line with this, provide the reader in general with more background about differences in earlier methods. The main benchmark method from Kusupati for example is also not explained in detail. 

- last line page 1: Which graph in the work contributes to the statement that the proposed method significantly reduces the amount of training iterations? From other context I understood that you use the word iterations for training epochs, but in all experiments you use the same nr of epochs (=160). So do you mean training cycles, and are you referring to fig2-bottom here? 

- p4, section "increasing the pruning ratio": The authors claim that the pruning ratio is increased during training and that it guarantees an exact amount of sparsity in the network. It's not clear from the description how this exact ratio can be guaranteed? Because if the soft-thresholding parameter is just (deterministically) being increased, how can it guarantee an exact sparsity ratio?

- Figs .2-4 : It would be valuable to also have these graphs for the method from Kusupati et al., because now the reader has no insight in the performance of that method for this model and dataset. 

- Figs 2-4: It it more common to report standard deviation instead of variance, since these values are very small, the variance makes the shadings even smaller, which can lead to the (unfair) impression that methods are more distinct in performance than they truly are. 

- Why did the authors choose to only report the highest sparsities in Table 1? It makes me suspicious that the performance on lower sparsities is worse than that of Kusupati et al., and omitted on purpose. 

- p6: In the definition of OneCycle, it is not explained in which way the purning takes place. What type of pruning method is used from which reference? Similar for Rw and Rec. Rw, provide the corresponding citation as well. 

- 4.3: sEction 4.3.3 does not truly seem to answer the third research question of this section. Because the question is about the soft-thresholding operator (which you fix with a deterministic scheme if I understood that correctly), while section 4.3.3 is about the criterion to be used for deciding which weights to prune in a layer. 
What is the conclusion of section 4.3.3? Should the reader use your method but including LAMP? Because originally LAMP is not part of your method right?

- section 5 observation 1: This seems rather obvious, as you apply soft-thresholding on your weights, which is in fact the proximal operator of the L1-penalty. How do you actually prevent that not all weights collapse to zero at some point? As the soft-thresholding operator is constantly pushing the weights downwards. 

- fig. 5: Please provide axis labels. Readers unfamiliar with the tensorboard interface will not be able to understand these histograms. 
Also for me it's not super clear how to interpret these histograms. In section 5.1 it might help to at least address figures 5A-5D, so that it is easier for the reader to link the text to the right figure. But even then, the authors speak about double Gaussian distributions, which I find hard to find in the figures. 

- In general no details were provided about the training/validation/test splits of the Cifar-10 and imagenet datasets. For the Imagenet experiments, were the standard splits used? And which split was used to report performance? Is this in line with what Kusupati et al. reported? Otherwise it will be unfair to compare the results.

Minor things/typos:
- end first paragraph introduction: There is a "(cite)" still.
- intro: "consists of a full training procedure..." instead of "consists in a full training procedure..."
- p3, section "recursive pruning with rewinding": In the last sentence you mention 11 training cycles for 90% sparsity. About which model and dataset are we speaking here? 
- Fig. 2 and 3: In some cases it seems that the yellow line has no shading. Also in Fig. 2 the bottom image has other font sizes than the other two. 
- Table 1: Mention that it is the top-1 accuracy you are reporting. 
- fig. 4, caption: You mention "global l1 pruning". does that refer to your method? Because you didn't mention it before. 
- end p7: You mention that whole-layer pruning causes instability in the hard-threshold case. But also in the soft-threshold case, whole-layer pruning can be a similar issue. So how can this be an argument to explain the instability of the hard-threshold case?
- fig. 6:. How to interpret the y-axis? Does it mean that #weights nr of weights switched #switches nr of times in a certain training phase? Interpretability of the graph would be much more intuitive if the time axis would be on the x-axis. Now it's quite hard to verify the conclusions from the text. Also in section 5.2 you mention that growing the sparsity ratio leads to more weights jumping from inactive to active and vice-versa. But in my understanding, this graph does not inform us about the direction in which the weights switch (from active to inactive or vice-versa). 
- Second paragraph sec. 5.2: experience = experiment
What do the authors mean by "whole sparsity" and "eliminate the high learning rate factor"? Before this point a high learning rate factor was not mentioned in the paper. So why do we want to eliminate it?
And what do the authors mean by "real pruning that is desirable"?
- bibliography: Some of the papers in the list have been accepted in conferences/journals, while the authors only referred to the arxiv version. Please update the bibliography.

 The proposed method is simple and elegant. However certain things are lacking in the paper, e.g., a more thorough discussion about (technical) aspects of related work, some details about the performed experiments, and more complete experiments.  Only after making those improvements, this paper will meet the high standard of ICLR. 

Meta-review: This submission proposes a method for learning sparse DNNs which consists of three components: First, a "dense" network is maintained and updated in each backwards pass, but the forward pass is done via a sparsified version of the network; sparsification is done via "soft" thresholding; and the sparsity ratio is increased over the course of training. Reviewers noted that each of these components had been previously proposed, and that the state-of-the-art baselines are not actually state-of-the-art anymore. They also noted that the paper read more like a draft and needs substantial improvement. The consensus was therefore to reject.

Decision: Reject