Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a channel-level sparse training algorithm that can effectively utilize the channel sparsity to accelerate both forward and backward propagation process. They divide the sparse training process into weight updating step and mask updating step. And solve the problem separately. More specifically, they develop a variance-reduced policy gradient estimator to update the trainable pruning mask without the need for backpropagation, and hence a sparse computation on both forward and backward propagation.
main_review: Strength:
This paper shows an analysis that why previous parametric works are not able to utilize the sparsity to accelerate the training.
The authors evaluate their method on several representative DNNs and Datasets.
It is great that the authors also show actual training computational time rather than just using FLOPs to indicate the acceleration.
Their results seem decent when considering both accuracy and compression ratio.

Weakness:
In the paper, the authors claim that they are the first to achieve training acceleration in practice. However, that is not correct or not rigorous. The existing work PruneTrain [1], and ClickTrain [2] can also achieve training acceleration. The author should also cite those papers and compare with them.

For the result comparison part, the accuracy of the original (dense) models is too low. For example, the accuracy of the original model for ResNet50 on ImageNet is 76.1%. I understand this is a widely used official model. But that model is too old, and it was probably obtained by some legacy training recipe. Your model is trained with warmup, label smoothing, and maybe cosine lr scheduler, which may significantly boost the model accuracy. To avoid misleading the audience, it is more reasonable to show the dense model accuracy obtained by using the same training settings as yours.
For a similar concern, the reference pruning works that you choose are also too old, which cannot represent the SOTA results (e.g., MetaPruning [3], DMCP [4]).

Many references are missing publication information.

You mentioned the reference work NetAdapt in the caption of Table III, but you didn’t show the results.

The total training epochs and learning rate scheduler are missing. This may make your results less convincing.

It is good to see the accuracy and training speed if you directly train a small dense model with exactly the same model structure, layer width, and FLOPs as the remaining part (dense) of your sparse model. 

[1] PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration, SC’19
[2] ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning, ICS’21
[3] MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning, ICCV’19
[4] DMCP: Differentiable Markov Channel Pruning for Neural Networks, CVPR’20

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Paper proposes a variational based pruning method where, with 2 structurally sparse forward passes and 1 sparse backward, the training can be significantly faster. Description is clear and theoretically sounds, equations are easy to follow, code is provided, results are encouraging. The paper might be a great benchmark and starting point for faster pruning-while-training with efficient inference. 
main_review: References need more work. For example many papers do not have the venue(journal) where it was published. 

# On sparse neural network training

Authors provide an overview of existing methods. However almost all structured pruning algorithms can be applied after few echoes, dense model obtained by removing channels, and then the rest of the training will continue with the small model. This approach will speed training as well. Are authors aware of such methods? For example [R1-R2]. These methods finish pruning with only 30% of training done, making the last 70% to be truly sparse.  
Claiming to be the first in accelerating training via pruning might be challenged looking at previous work.  

# Approach

The method theoretically sound. Equations are clear to be followed, Algorithm 1 explains the implementation clearly. Explanation and theory behind VR-PGE is interesting and clearly explained to be motivated by mean field theory.

K (in line ) controls the pruning ration of the layer. If this is true then we have 1 extra parameter to finetune. 

# Results

Comparison shows better performance of the proposed approach over other pruning methods on CIFAR10/100. However, pruning network while training on CIFAR is not practical. Speeding up ImageNet training is more practical. 

When training ResNet-50 with accelerated strategy it will be nice to see a comparison with training a smaller model from scratch. For example, training ResNet-50 with uniform channel reduction initialization (only 50%, 75% of initial channels in the model should be 1.6x faster). One more obvious baseline is to finish training earlier for ResNet50. For example doing only half of training epochs. Cosine (or monotonic) learning rate decay will remove the need of hyper parameter search.

When comparing results with pruning methods more other approaches should be added. For example, R3 has 76.4% with 50% FLOPs reduction. This method doesn't make pruning faster, 

# Overall

The approach is interesting, clearly explained and theoretically sound. Reviewer is not too much familiar with variation methods for pruning and might miss some important references. The approach of pruning aware training that speeds up training is not novel, hovewer the method is novel.

Question if the pruning strategy is global (setting up K to be the neuron ration for the entire model, global pruning) or per-layer (setting K for every layer independently). In the former case cross-layer dependency might affect results and will need more hyper parameters. Code seems to indicate global pruning, just want to be sure.

Also additional baselines are required to see if the method outperform smaller model training from scratch in terms of training cost reduction. 
For example ResNet-50 with uniform scaling, finishing training earlier (would be easy to target a particular 1.6x speed up, ImageNet preferably)

Finally the baseline accuracy of ResNet50 on ImageNet is somewhat low. With the setting of cosine annulling, 4 GPUs we should expect ~77.2 accuracy of the full model. Did authors train the Original model with their setting or took from the model zoo. The latest might make comparison not correct. 

Providing code is great, however I found it a bit difficult to follow with too many arguments as input. Also for the final code release, would suggest to remove other than the proposed methods to make easy to follow. 



[R1] S. A. Aketi, S. Roy, A. Raghunathan, and K. Roy. Gradual channel pruning while training using feature relevance scores for convolutional neural networks. IEEE Access, 8:171924– 171932, 2020.

[R2] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2009–2018, 2020.

[R3] EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning, ECCV2020
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Network sparsity has historically targeted only inference.  Recent explorations in training with sparsity fall short of their potential, though, by not fully exploiting sparsity in the forward and backward propagation steps.  The authors decompose training a channel-wise sparse network into two parts: training the weights themselves (as usual) and learning the sparse structure.  Rather than using a chain-rule based gradient step for the structure, a unique policy gradient estimator with reduced variance is used; it is composed of two forward passes, eliminating the need for the chain rule to be used to learn the sparse structure.  Experiments show competitive accuracy for saved parameters and FLOPs for a number of networks, as well as actual time saved during training with the proposed technique for VGG19 on CIFAR-10.
main_review: # Originality

There's been lots of work in sparse training.  The authors present a nice survey in Section 2.2, but they've missed a number of works which do exploit sparsity in backpropagation [1-4].  It's likely that the recent works [3,4] could be trivially modified to support channel-pruning, rather than fine-grained sparsity.  What may be new here, and what the authors should clarify, is that the proposed approach is more theoretically grounded, without relying on heuristics to choose sparsity that isn't consistent between the forward and backwards passes.  Note: I'm not positive about this point; the burden remains on the authors to argue this convincingly!

I'm not familiar with any other work on reducing the variance of policy gradient estimation, though, so at least this contribution was new to me.

# Quality
Aside from the missing references that cast doubt on the claims of novelty, the experiments show that the technique does what it intends to do: variance of the gradients are reduced compared to the baseline PGE, accuracy is plausibly maintained as the number of nonzero parameters are reduced, and training time is reduced in practice.  In particular, reducing the training cost of ResNet50 on ImageNet1K by 1.6x is no small feat!  (In practice, this will be greatly reduced, of course.)

Remark 2 is potentially misleading.  It suggests that the authors of GrowEfficient used a dense backprop procedure, even for zero parameters.  Is this just for simplicity of implementation, though?  If so, it's no cause for penalization, as computing with zeros is a common way of testing an algorithm; exploiting the zeros often follows.

There's no discussion of hyperparameters p and a, how sensitive results are to their values, and suggestions on how to select them.  A step in the right direction would be to detail (in the main text) the settings used for the experimental results.

# Clarity
I found some parts of the manuscript confusing and hard to follow.
- Section 3 uses a fully-connected layer for simplicity (and it's understood that a convolution is a particular type of FC layer), but then Section 4 immediately switches to discussing in the context of convolutions.  Why the change?  Sticking with FC layers would help ease understanding.
- Line 122: "It is obvious that these methods cannot achieve significant speedup in training since they need to calculate the full gradient in backward propagation."  It's not obvious to me!  It sounds like the argument is that since the filters are not actually zero, neither are their gradients.  In the same sentence, though, it is said that "forward propagation could be sparsified if implemented carefully."  If these not-quite-zero values can be exploited for fprop, why couldn't the same happen in bprop?
- Similarly, the difference that gives true sparsity in the proposed method's backprop step, where prior efforts could not realize this sparsity, is not clear to me.  The authors suggest that since the output of the pruned channel is zero (Line 196), backprop can be sparse.  Is this not also the case in equation 1?  The term marked "forward" is zero for the pruned neuron, so the output should similarly be zero.

It'd be helpful if all equations were numbered for easy reference.

There were also a number of typos; another editing pass would be helpful to fix these (and, likely, others that I missed):
- Line 70: "… zero valued mask*s*"
- Line 74: "propagation*S*"
- Line 121: "filers" -> "filters"
- (Maybe not a typo, but a style thing) Figure 1's caption uses "3-rd," when "3rd" is more common, or, in this instance, "third" would also be appropriate.
- Algorithm 1, line 1: "p1" (is this supposed to be simply "p"?)
- Section 5.5 mentions Top-1 accuracies for MobileNetv2 on ImageNet, but I think these numbers (90%+) are for Top-5.  (Why switch between Top-1 and Top-5 for Figure 3's results and discussion?)

# Significance
Saving training time while still generating a network that maintains accuracy is an important topic.  This work is no magic bullet, though, because accuracy doesn't quite match the dense baseline accuracy, even for the more modest savings.  However, it does seem much more promising than the work to which it is compared, in both accuracy vs. parameters and achieved training time savings, so it may be a foundation on which future work improves.  However, there are some baselines missing both qualitative and quantitative comparisons, so the importance of this work may be reduced with the added context.

---

[1] "meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting," Sun et al.,  ICML 2017

[2] "Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training," Zhu et al., https://arxiv.org/abs/1806.00512

[3] "Sparse Weight Activation Training," Raihan and Aamodt, NeurIPS 2020

[4] "Top-KAST: Top-K Always Sparse Training," Jayakumar et al., NeurIPS 2020

limitations_and_societal_impact: There's no discussion of limitations in this work.  For example: 
- Can the proposed technique be applied to transient values, such as activations or attention layers (which have no learnable parameters), to reduce training (and inference) costs?  In recent thrusts for many domains, including image classification, such approaches are proving to be very successful.
- Are there limitations on optimization schemes that will and will not work with the proposed technique?

I'm sure the authors are better able to identify the limitations of their work than I am.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an efficient sparse training method to solves the problem of accelerating the training speed of deep neural networks and save memory usage.  The authors first formulate the training process as a continuous minimization problem under global sparsity constraints.  Then, separating the optimization process into two steps, corresponding to weight update and structure parameter update. A variance-reduced policy gradient estimator is proposed to update the structure parameter. Finally, the experimental results are solid and strong. The problem has lots of meanings for engineering implementation.  
main_review: Originality: The task is new. The method combinates some well-known techniques and also proposes its own technique part. It also makes a clear statement about the differences from previous contributions.

Quality: The submission is technically sound.  Most claims are well supported (e.g., by theoretical analysis or experimental results). This is a complete piece of work. The authors are careful and honest about evaluating both the strengths and weaknesses of their work.

Clarity: The submission is clearly written and well organized.

Significance: The results are important and others would like to use the ideas and build on them. 
The problem has lots of meanings for engineering implementation.  
The experimental results are solid and strong.
The analysis and the experiments of the proposed VR-PGE are solid and convincing.
limitations_and_societal_impact: The only limitation of this paper is the complexity of the method implementation, especially for the VR-PGE part. The attached code about the training is a little bit complicated. Do the authors have any plan to release the code or make it a component of PyTorch?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper addresses the topic of sparse training of neural networks. They target channel-level sparsity with the goal of realizing practical training speedups with existing software and hardware. The authors study the limitations of existing techniques in terms of their ability to exploit sparsity during training and show a wide array of empirical results with their proposed technique.
main_review: The framing of this paper and it’s “truly sparse neural network training” technique is sensational and does not acknowledge the contributions of prior work on this topic. 

Firstly, It is not correct to say that no existing method can effectively accelerate training. There is a body of literature on sparse training that has developed techniques for exactly this. For example: SET [5], DeepR [6], SNFS [7], RigL [2] and Top-KAST [3]. The authors cite some of these techniques but don’t seem to acknowledge their contributions.

Second, it is not correct to say that sparse matrix multiplication cannot be implemented efficiently on widely-used platforms. Recent work has shown success in doing so for sparse deep learning workloads [8, 9].

Third, when discussing channel-sparse algorithms, the authors state “It is obvious that these methods cannot achieve significant speedup in training since they need to calculate the full gradient in backward propagation”. This statement is an exaggeration. Even if the weight gradient computation needs to produce a dense gradient w.r.t. the sparsified weights the forward pass and backward gradient computation w.r.t. the input data to the linear operation can still take advantage of sparsity. That is to say 2/3rds of the FLOPs in the model still benefit from sparsity. This discrepancy leads me to believe that the authors could be underestimating the training-time FLOPs saving for other techniques in their experiments. 

Delving into this, the only existing technique for which the authors provide “training-cost savings” data in their experiments (Tables 2 & 3) is GrowEfficient. In their response, I’d like the authors of this work to clarify how they calculate the “training-cost savings” for GrowEfficient. Even if the weight gradients are calculated densely, the forward pass and half of the backward pass should benefit from sparsity.

On the topic of weight gradient computation, it’s not clear to me that GrowEfficient cannot exploit sparsity in these as well. As I understand it, GrowEfficient uses a similar Bernoulli random variable (q) on their weight matrices to induce sparsity. For cases where this value is 0, any gradient for the zeroed neuron should also be zero and thus GrowEfficient should also be able to exploit sparsity in their weight gradient computation. The authors of this work stated that “The authors of GrowEfficient [46] confirmed that actually they also calculated the gradient of qc w.r.t, sc in their Eqn.(6) via STE even if qc = 0.”. If this is true it would be correct to count the cost of computing a dense gradient for GrowEfficient. However, I was unable to confirm this from the GrowEfficient paper and basing conclusions on hearsay that is not documented in the literature feels tenuous. The clearest way to demonstrate this is true would be for the authors to re-implement the baseline technique, although I realize this is a considerable burden.

Other Issues
- In Table 1, many of these techniques are not “sparse training” algorithms, i.e. their aim is not to accelerate training with sparsity. For example, references 7, 50 and 28. Reference 19 is listed under both parametric and non-parametric techniques.
- In general, the taxonomy of “non-parametric” and “parametric” sparsification methods is unclear and appears to be limited in scope. For example, it does not take into account details of the algorithm captured by existing taxonomies like “dense-to-sparse” v. “sparse-to-sparse”. I’d recommend the authors take a look at the taxonomies used in recent literature relating to sparsity. For example, Gale et. al [1], Evci et. al [2], Jayakumar et. al [3], and Hoefler et. al [4]. As I understand it, “parametric” techniques use some form of sparsity inducing regularization and “non-parameteric” techniques do something else. I don’t think a new term is necessary to make this distinction.
- Ideally the authors would include training-cost savings for more techniques than GrowEfficient in their experiments. Given these theoretical savings rarely translate directly into runtime savings, it would improve the paper greatly if the authors included runtimes for more techniques than their own in Table 4.

References
[1] https://arxiv.org/abs/1902.09574
[2] https://arxiv.org/abs/1911.11134
[3] https://arxiv.org/abs/2106.03517
[4] https://arxiv.org/abs/2102.00554
[5] https://www.nature.com/articles/s41467-018-04316-3
[6] https://arxiv.org/abs/1711.05136
[7] https://arxiv.org/abs/1907.04840
[8] https://arxiv.org/abs/1911.09723
[9] https://arxiv.org/abs/2006.10901
limitations_and_societal_impact: I did not identify potential negative societal impacts in this work.
review_title: null

Meta-review: This paper proposes several effective schemes to ensuring the sparsity of a deep network being trained during backward and forward propagation. The reviewers recommendations are somewhat polarized and there are some serious disagreements between some reviewer and the authors regarding fair comparison with many existing sparsity promoting techniques. Despite some concerns, the proposed methods are relatively novel and effective and have been justified with adequate analysis and empirical evidences. The AC believes the quality of the paper is above the acceptance threshold. 
Decision: Accept (Poster)