Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a dynamic sparse pretraining method to reduce the computational cost of BERT pretraining. 
main_review: The authors propose to dynamically prune the transformer to improve the efficiency of BERT pre-training. To maintain the computational regularization, block sparsity is used in this work. The main technique of this paper is to add group LASSO regularization to induced sparsity and prune the network depending on the magnitude of the weights. The proposed method is simple yet effective. The resulting sparse pre-training method is nearly twice as efficient as the original BERT pre-training method. However, the novelty of this paper is limited because the proposed sparse pruning technique has been widely used to train convolutional neural networks. Moreover, it remains unclear whether the proposed method is tailored specifically for pre-training of sequence models. My additional concerns are listed below.

1) Comparison with other dynamic approaches: As discussed in this paper, other lines of research exist to explore sparse transformer architectures [1, 2]. However, the proposed method is not compare with these prior works in the paper.

2) Wall-clock evaluation: No wall-clock evaluation is provided. It is not clear whether the proposed method can provide real speedup on accelerator chips such as google TPU.

3) Finetuning results: The authors only show the MLM loss after pre-training as the quality metric. However, it is also important to show the performance of models after fine-tuning on GLUE / SQuAD datasets.

[1] Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
[2] Reducing Transformer Depth on Demand with Structured Dropout
summary_of_the_review: The novelty of the proposed method is limited and the empirical evidence is not strong enough to justify the method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper advocates a straightforward, dynamic sparse (DynSparse) pre-training approach for BERT for more efficient FLOP utilization while maintaining comparable accuracy.  This paper utilizes random reallocation of weights (instead of gradient-based reallocation) to achieve higher total explored degrees of freedom (DOF). This work also proposes a block-sparse DynSparse scheme for higher effective utilization of FLOPS on real hardware accelerators (like GPUs and IPUs).
main_review: **Positives**

* Reducing the training cost for language models like BERT is an important problem.

* This work covers both *structured* and *unstructured* sparse training for BERT. The proposed block-sparse DynSparse can improve hardware utilization.

* This work suggests minimum efficiency requirements for a time-to-train win for DynSparse over the conventional dense training.


**Concerns/Questions**

* This work in its current form lacks any quantitative comparison against other DynSparse work.  The authors argue for random re-allocation as a means to increase DOF. However, RigL [Evci et al. 2019] suggests that their gradient-based re-allocation outperforms SET, which also adopts random re-allocation, for both vision and NLP tasks (with RNNs, though). Is there any ground for random re-allocation to work better with BERT, or is it simply anecdotal? How would RigL perform with BERT?
  
* There is no evaluation of end-to-end time ("time-to-train"). Since the authors take into account the utilization in real accelerators by introducing block sparsity, I am wondering how much the proposed technique work reduce the end-to-end training time compared to the baseline dense training.
  
* Along the line of the previous one, the actual performance gains over dense training may be limited on commodity GPUs. According to Nvidia, cuSparse manages to deliver just about 50% of dense GEMM when using 32x32 blocks [R1]. Depending on the sparsity, the model may fall short of the minimum efficiency requirement of block sparse training to outperform dense training for BERT.

* Evaluation seems a bit too narrow to convince the generality of the proposed technique due to the usage of just one model, fixed sequence length, and lack of ablation study and sensitivity analysis. Can the proposed technique be effective to other models than BERT?  The technique seems quite generic and I am wondering how effective it would be to other models. Is there anything specific to BERT?

[R1] https://developer.nvidia.com/blog/accelerating-matrix-multiplication-with-block-sparse-format-and-nvidia-tensor-cores/

summary_of_the_review: This paper adapts DynSparse training to BERT to demonstrate its effectiveness. However, evaluation is somewhat narrow and it's not well justified why random re-allocation of weights works better than gradient-based re-allocation for BERT. Also, the performance gains in terms of end-to-end training time on the commodity accelerator like GPUs are still questionable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Overall, this paper proposed a dynamic sparse pre-training method for BERT language modeling which showed better performance in the number of FLOPs when compared to statically sparse and dense models across a large scale of network sizes. 

To be more specific, the proposed DynSparse has four contributions: 1) using random parameter re-allocation to alleviate network parameters collapse; 2) is scalable and FLOPs efficient; 3) showed effectiveness on structured training of BERT 4) structured regularization free while achieving Pareto improvements.
main_review: Strengths:
The motivation of using sparse training for BERT language modeling is clear and persuasive.
Extensive experiments and technique details are provided to support claims and demonstrate the effectiveness of the proposed method. 

Weaknesses:
It will be better if the author provides more details on the difference between zero and untrained in computation or the time and space complexity analysis of these two methods.
The author mentioned in the part of the contribution that the structured DynSparse training of BERT without structured regularization gives performance gains compared to dense BERT baseline. Structured regularization is used in the block-sparse DynSparse algorithm. Are these two in contradict with each other?
It will be better if the author gives more description of the techniques in the contribution in the part of the methodology.
The author mentioned that the collapse could be mitigated by reducing the influence of activations during training updates. It will help if the author provides more details about how the activations are reduced and how many are reduced compared to the dense baseline.

summary_of_the_review: Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. However, I am not fully persuaded by the methodology. More details like complexity analysis and latency reduction experiments could be given.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the feasibility to use the dynamic sparsity (DynSparse) method of continuously pruning and re-allocating network weights during training for large unsupervised (self-supervised) language models, namely BERT. Towards this end, the authors present several parameter studies (w.r.t. hyperparameters) of the original BERT training procedure (Mask Language Modeling and Next Sentence Prediction) to assess the Pareto-curve of training BERT with sparse vs dense. 
main_review: Sparse training for neural networks, especially very large transformer-based architectures, is a very  interesting and timely topic when discussing model efficiency. However, the presented paper does not provide much novelty or method advancement. The authors simply apply an already existing method to one type of network. While parameter-studies in general can be of interest, yielding new insights into the training procedure, my main concern is that the studies presented in this paper do not follow a  clear research question, but seem rather random. More importantly, no generalized “lessons-learned” from these studies are drawn, and the translation towards other network architectures is not discussed. As such, the paper lacks significant scientific impact.

Some further comments:
- It is not explained, why the algorithm is adapted to a block-sparse structure. What is the benefit of this, why is it needed?
- All results are reported in terms of reduced FLOPS, but actual training time (wall-clock time) is not considered. Especially on hardware specialized for dense/block structures (GPUs), accessing individual elements in a sparse pattern can, while requiring less operations, result in comparable compute times as their dense counterparts. Moreover, sub optimal usage of the device can thus negate the effects on efficiency, proclaimed by the sparse computation
- In Figure 2, the caption does not match or explain the Figure. How does this Figure illustrate the pruning and re-allocation step? What do the masks and the colours stand for?
- Equation 3: Where does this come from? This seems very empirical. What is the reasoning behind this?
- No details on the utilized hardware and training setting are given
summary_of_the_review: My main concern with this paper is that it is merely a parameter study of an existing method being applied to an existing architecture. While these kind of studies can still be of high scientific value, they need to provide new insight into the method or the architecture and yield some translation/generalization in terms of "lessons learned" for other applications in order to do so. I do not see this to be the case here.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper did an extensive investigation of applying dynamic sparse training for BERT. 
The main contributions of this work are several empirical observations:
1. The authors empirically show that random re-allocation of weights is better than
gradient-based approach (RigL) for unstructured DynSparse training of Bert. 
2. The authors investigated structured DynSparse training and found L1 norm is the
best metric to prune the weights.
3. For a specific range of size of BERT, the DynSparse training shows Pareto improvement (MLM validation loss vs. Non-embedding FLOPS) over the dense BERT. 
main_review: 
Strengths
- The paper shows a simple algorithm (DynSparse) that could help find good sparse
subnetwork in BERT, reduce FLOPS and achieve comparable MLM validation performance.
- The authors did intensive experiments on comparing several different design choices
such as the metric to re-allocate weights, prune the weights, etc.
- The sparse training tool developed is useful for the community.   



Weakness

- The novelty of this work is a bit limited. There is no new algorithm proposed to get better results for this sparse training of BERT. The discussions on what could be the reasons behind the design choice are not very convincing to me. For example, random re-allocation has more varieties of network topology explored than RigL, leading to better performance. If this understanding is correct, why would Figure 5 (a) have a convex curve?


- Evaluation of downstream tasks. 
Maybe I haven't done a good review of this field, but I am skeptical that MLM validation loss is a good indicator of the quality of pre-trained BERT. Could the authors find references supporting this metric? What if we fine-tune the DynSparse BERT and BERT Large under the similar resource constraint and compare their performance?

- Concern on the scale. 
I am not sure if the conclusions still hold when we 
scale to larger models. 
"DynSparse training can achieve more efficient utilization of FLOPs or network parameters at any scale." It seems a overclaim to me as the paper didn't show the comparison with BERT large. 
It would be helpful to label the point of BERT large and its DynSparse counterpart in Figure 3.

- Missing comparison with baseline. 
The conclusion from EarlyBert paper is that lottery ticket converges fairly fast and the subnetwork is identified during the early stage of training. This seems to conflict
with the findings in this work that network topology has to be updated sufficiently
to get good results. Could the author elaborate more on this point?
Regardless of always-sparse constraints, how does the quality of subnetwork found
by DynSparse compared to EarlyBERT, in terms of performance in downstream tasks.

summary_of_the_review: This work did a comprehensive empirical evaluation on applying DynSpare to BERT,
in both structured and unstructured settings. The novelty of algorithm and theoretical understanding is limited. The empirical results could be more thoroughly conducted, e.g., the scale of BERT, downstream task evaluation. 
As an initial step of exploring sparse training of BERT, it provides some promising results. The tool this work developed could benefit the community.
review_title: null

Meta-review: All of the reviewers believe the paper should not be accepted, and I concur with their recommendation for the reasons they mention.

Four of the reviewers (vEBH, idrP, KoFV, 5k4c) believe the technique proposed in this paper is not particularly novel. Rather, the novelty is that it is being used on a BERT model rather than the computer vision models that are typically the starting point for pruning work. They also argue that the paper was not particularly thorough in its comparison to other pruning techniques (specifically dynamic sparsity techniques), which is essential for pruning work given how crowded and noisy the space is. Finally, they rightfully note that the paper does not look at the real-world speedups attainable on conventional hardware (GPUs and TPUs), the latter of which has no support for sparsity and the former of which (NVIDIA Ampere) has limited support for specific kinds of sparsity and especially limited support for sparse training.

The reviewers also raised several more specific methodological issues with evaluation (e.g., using the MLM loss rather than fine-tuning as a basis for evaluation), but the above concerns alone were enough to convince me that the paper does not merit acceptance at this time.
Decision: Reject