Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work proposes a simple enough idea to speed up the training of BERT by progressively stacking new layers while fixing older layers. Empirically, with the same number of training steps (and less time), the proposed method can achieve a comparable performance to the original BERT. When the same amount of running time (more steps) is used, the proposed strategy can further improve the performance. 

One problem with the current paper is the empirical evaluation is only conducted on the GLUE benchmark, which is sequence-level and relatively simple. I think an experience on a slightly more difficult task such as SQuAD, which also requires token-level prediction, would be necessary to test the capacity boundary of the proposed approach.

Another question is what would happen or what the performance would be if the entire model is not jointly trained for the last 20% steps. This information will help to better understand this method.

In addition, the original motivation of the work comes from the fact that the attention patterns in the bottom layers do not change much after jointly trained with more higher layers. However, this does not mean the lower-layer attention patterns don't change much if the entire network is jointly trained from scratch. To truly establish the validity of motivation, it would be good to monitor and evaluate how much the lower-layer attention patterns change when jointly trained.
review_title: Interesting and neat idea but the experiments are limited

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a multi-stage layerwise training (MSLT) approach to reduce the training time of BERT.   Experimental results show that the proposed method can achieve  110%+ training speedup without significant performance degradation.

Overall the idea of multi-stage layerwise training is reasonable and the results look promising. 

My major concern about the work is the empirical comparisons. ALBERT, a closely related work, is not compared in this work.  This paper claims that "ALBERT has almost the same computational complexity as BERT, training an ALBERT model is still very time-consuming." 
I checked the ALBERT paper, which claims 1.7x training speedup. 
"An ALBERT configuration similar to BERT-large has 18x fewer parameters and can be trained about 1.7x faster. The parameter reduction techniques also act as a form of regularization that stabilizes the training and helps with generalization."
Seems the proposed method in this work (with 1.1x speedup) is not as good as ALBERT. I suggest to take ALBERT as backbone and check whether the MSLT can speed up the training of ALBERT.

Besides, it is better to also test on more complex downstream tasks, such as SQuAD1.1/2.0 and RACE.

review_title: Comparison with ALBERT

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a training strategy to progressively adding top transformer layers, which results in training time speedup. Usually when training transformers, all layers are updated simultaneously. The author found out doing a multi-stage layer-wise schedule helps convergence speed without significant performance degradation.

The paper is written very clearly. The idea is conveyed with sufficient background and related work. 
My biggest concern is from the originality and significance. It looks to me the approach that the paper proposed is a very straightforward extension from the work of (Gong et al., 2019).  Instead of updating all parameters in all transformer layers, this paper freezes bottom layers and only update top transformer layer (newly added). With experiments performed on base and large BERT models on GLUE dataset, this training strategy is approved to have slightly drop of quality but faster convergence speed. 

I would argue that this is a great investigation and experimentation but it does not meet the criteria of acceptance. In section 3.1, the author has used the attention distribution (before and after) to motivate the work in this paper. However, if the author could discuss/explains why this change affects the convergence speed in analytical/mathematical solution, that would definitely gives more credit to originality and significance.  I would be more than willing to re-evaluate my ratings if that happens. 

Minor comments/typos:

* abstract: "have has achieved" -> "have achieved".
review_title: Clear paper but lack of significance

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper proposes a simple method, ie multi-stage layerwise training (MSLT), to speedup BERT training. Specifically, the authors progressively stack layers. The bottom layers are fixed and only the new added top layers are trained. The proposed method can achieve more than 110% training speedup and achieve comparable (slightly worse) performance. Although compared with other speedup training methods like ELECTRA [1], the idea of this paper is not novel. However, the proposed method is simple and effective to some extend. Thus, I give a marginal score to this paper.

### Strengths
* The proposed training method is simple and easy to implement. 
* It can achieve 110% training speedup without significant performance degradation.

### Weaknesses and Questions
* The most related method to this paper is [2]. Thus, it is better to give more comparisons and discussions in the Experiment, so that the author can know the advantages of MSLT compared with [2].
* From Figure 4, it seems that BERT-base (baseline) needs to train around 80 hours. If training with MSLT using the same time as baselines (like train 2M steps), how about the performance compared with baseline? [3] shows that training with more steps can help improve performance. Does this phenomenon still remain in your method?




[1] Clark, Kevin, et al. "Electra: Pre-training text encoders as discriminators rather than generators." ICLR. 2020.

[2] Gong, Linyuan, et al. "Efficient training of bert by progressively stacking." ICML. 2019.

[3] Liu, Yinhan, et al. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019).
review_title: simple method to speedup BERT training

Meta-review: This paper proposed a variant of the existing training method "progressive stacking", and showed good empirical results comparing with the normal training procedure for BERT models. It contains some interesting points on the technical side, e.g, freezing the bottom layers when training newly added layers, but there are several concerns:  (1) This proposed method is called progressive stacking 2.0 but there is no comparison against the original progressive stacking in experiments. We had to check the empirical results in the original paper of progressive stacking, and did not notice any performance improvement of this new method over the original one; (2) The proposed method even introduced one more hyperparameter to tune: the number of top layers to copy.  This hyperparameter seems hard to choose. Different choices of this hyperparameter may dramatically impact the performance of this new method.  An ablation study on this should be conducted, e.g.,  what will the results look like if we only copy the last layer?  (3) The original progressive stacking method does not provide any practical guidance on how to determine the time to stack when the training goes. This severely limits the practical value of progressive stacking. If one stacks layers too early or too late, the stacking method may have no advantage at all. Unfortunately, this method called 2.0 still leaves this most critical issue away. 
Decision: Reject