Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary:

The authors propose a technique for reducing the computational requirements of training BERT early in training to reduce the overall amount of resources required.

Pros:

The paper is well written and clear for the most part. The authors do thorough experimental evaluation.

Cons:

I have two primary concerns about the paper and the proposed technique.
1. The positioning of the technique is not entirely clear to me. The authors pitch it as a technique for reducing the training time of BERT and use LayerDrop as a baseline technique that also removes network components. However, it feels like another baseline that should be considered is neural architecture search, which also seeks to automatically find a more efficient model to train. The difference here is that the authors find the model early in the training run, but it seems like the EarlyBERT procedure could be run once and the resulting model architecture could be saved and re-trained like NAS models are. 
2. I found the experimental results to be lacking detail and breadth necessary to establish the value of the technique. Firstly, the rough time estimates in Table 2 are very odd given the primary value of the proposed technique is to reduce training time. The accuracy of EarlyBERT is close enough to LayerDrop that accurate training cost numbers are needed to differentiate between the techniques. Secondly, quoting training time reductions over the dense baseline when the EarlyBERT mode does not achieve the same accuracy makes the comparison very difficult to make. This problem shows up quite commonly in the model compression literature [1] and I’d encourage the authors to show full accuracy-training time tradeoff curves so that the training time savings for a given accuracy can be more clearly established. Lastly, I found the use of reduced training epochs in EarlyBERT to be odd because you do not evaluate whether or not this can be done for the baseline models and there isn’t clear evidence as to why your model would be able to do this while others (e.g., DropLayer) cannot. Figure 2 also does not seem to corroborate that higher learning rates can be used with shorter training time to achieve better accuracy. The data in your figure shows that the best learning rate achieves the best model quality independent of the number of training epochs.

References:
1. https://arxiv.org/abs/2003.03033

review_title: Unclear experimental results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an approach to sparsifying BERT. What sets this work apart from prior work on model compression is that while prior works attempt to compress a pre-trained BERT, the authors in this work attempt to learn a sparsified BERT for the purpose of speeding up pre-training. The method essentially involves learning an independent bernoulli mask for all BERT heads along with Bernoulli masks for some later intermediate neurons. This model is then trained, along with the masks, for a few epochs. Then, heads/neurons corresponding to a small value of mask are pruned out. This results in a sparser BERT model, leading to faster pre-training.

#### Pros
The paper is well written and the presentation of the contribution is simple and well-motivated. 

#### Cons
1. Since one of the main contributions of this work is to make progress on improving the training / inference speed of large transformers, the authors could spend more time going over how the “Time Saved” column is computed. As of now, it seems casual and hand-wavy.
2. Experiment protocol:
  * How were the hyperparameters decided? Could we have uncertainty estimates for all results by reporting mean/std dev. across 3-5 runs (atleast for fine-tuning if doing this for pre-training is too compute intensive). This is especially useful since some of the numbers between EarlyBERT and Random appear very close in Table-1.
  * While a central argument is that most model distillation techniques still require expensive pre-training, it would still be useful to include some of those results in Table-2 since EarlyBERT is comparable to those techniques for the purpose of Table-2.
  * One way to contextualize how much time is saved during pre-training would be to report total time required for fine-tuning+pretraining on the entire glue benchmark. This would involve computing the pre-training time (time to learn BERT parameters on Wikipedia) + total fine-tuning time across all datasets (QQP/CoLA/MNLI etc) considered. This could then be compared against alternatives (such as DistilBert and DeeBERT). 
3. Baselines: Experiments on pre-training compare with no baselines. Some possibilities:
  * Using DeeBERT / other early exit approaches at pre-training time.
  * Training a BERT model for some epochs, and then distilling it into a smaller network for the rest of the training.


#### Misc Nitpicks
1. The phrase “structured sparsity’ is used in multiple places, but never defined. 
2. You et al. pioneers $\rightarrow$ You et al. pioneer
3. Could Prasanna et al. 2020 be extended to sparsify pre-training?
review_title: Interesting approach but needs some more experiments

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper uses the Lottery Ticket Hypothesis to compress BERT. More specifically, they adapt EarlyBird lottery tickets to the BERT setting in order to find winning configurations in early stages of training combine it with structured pruning methods to ensure the resulting network is more efficient to train. The method is a three-stage process: (1) searching - this phase involves training full BERT with some coefficient parameters to learn the mask, (2) drawing - use the mask to “draw a ticket” or select the sub-network to train, (3) training. Experiments show that performance isn’t that much worse when EarlyBERT is used for fine-tuning and for pre-training.

The goal of the paper is to find structured winning tickets for BERT in the early stages of training/fine-tuning.

Strengths:
1. Compressing BERT using the lottery ticket hypothesis has been getting a lot of attention recently, and doing so relatively efficiently is an exciting and interesting contribution.
2. Compared to previous and contemporary work, the combination of EarlyBird lottery tickets (You et al., 2020) to detect tickets early and network slimming (Liu et al., 2017) for structured pruning, is an interesting one. Prasanna et al., (2020) are doing structured pruning too, but via an iterative pruning method. 
3. Experiments for both pre-training (the first of its kind) and fine-tuning show that performance does not drop all that much for GLUE and Squad which are the main set of tasks BERT is typically evaluated on.  

Weaknesses:
1. The paper isn’t very clear in some places. It starts out explaining things well but then it gets harder to follow the details. Eg: need to add more information on the mask - it’s binarized at some point? The distance metric is still Hadamard like in EarlyBird? 
2. (More of a question/nit) Is it possible to also show what happens when a winning ticket for BERT fine-tuning is selected based on the pre-training objective? Chen et al., (2020) showed that these make for better tickets that are performant on many of the downstream tasks. It would be interesting to see if this holds true for EarlyBert and would add another layer of fine-tuning efficiency.

The work isn't terribly novel, but it's still interesting.

Questions and comments:
1. Why does the mask distance diverge for FC in pre-training (Figure 1b)? Does it somehow indicate that the training run is degenerate?
2. The ablation on the regularization parameter didn’t give a clear indication of how important selecting this is. It seems to not be that important, but would using separate values for attention and FC make a difference?
3. It would probably be helpful to expand the table/figure captions, make them a bit more detailed. 
4. Curious, why did you only use the largest tasks from GLUE?

review_title: Nice work

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main contribution of this work is to use Early Bird Lottery Tickets to reduce pre-training and fine tuning time for BERT. In order to reduce computation, the authors propose pruning the number of attention heads and neurons in the fully connected layers. They demonstrate that the technique works for BERT-Base and BERT-Large for GLUE and SQUAD tasks. 

The work is well-thought through and authors do a good job of explaining their approach and other existing works using lottery tickets.  The usage of lottery tickets during the pre training phase is the biggest strength of the paper since it can result in significant computational savings. The authors perform interesting ablation studies but they could be augmented by a few more experiments (see below). The paper could also be improved by comparison with relevant prior work. 

Here are some thoughts and questions that could help improve the paper: 

* Experiments in Section 4.2 are only during the fine tuning stage. How do these results compare with prior work for drawing lottery tickets in transfer learning for NLP (Chen et al, Prasanna et. al)? Particularly, it would be good to compare against Chen et al since their work is very relevant. 
* How long is the searching stage followed by the efficient training stage? Does this remain the same for all the tasks? Or did it require tuning? It would be interesting to see how soon we can switch from the searching stage to the efficient training stage.
* The paper proposes two different approaches for pruning: pruning attention heads and removing neurons from fully connected layers. It would be interesting to see how the accuracy of the approach changes if only neurons or attention heads are pruned. 
* In Section 4.3, the authors discuss reducing training time by reducing training steps for EarlyBERT. I think a similar analysis for BERT would be helpful to understand if the training time can be reduced for the baseline model as well. Also, pre-training techniques have tended to show improvements in downstreams tasks with longer pre-training as the long as the pre-training dataset is large enough. So, perhaps this reduction in pre-training steps might not be applicable for larger pre-training datasets.

Overall, I find the approach interesting and the authors show computational savings in the pre-training models. I recommend accepting the paper. 
review_title: Interesting approach to reduce pre-training approaches for large NLP models

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles a very important and under-studied problem: reducing the cost of training NLP models. The authors present a method that builds on the lottery ticket hypothesis (LTH). The authors first identify redundant structures early during training, then prune these structures, which leads to faster training. The authors experiment both with pre-training and fine-tuning of contextual models (BERT-{base,large}) and claim large reduction in training time, with reasonable loss in performance. 
Despite very encouraging results, several important methodological questions about the source of the efficiency gains and other aspects of the paper are left unanswered. I cannot recommend accepting this paper in its current form, but am looking forward to reading the authors' response which might clarify things.


Detailed comments:

Training (especially pretraining) costs have been going wild in AI and NLP more particularly, which leads to large costs ([1]) as well as potential environmental problems ([2]). Reducing these costs could have a very high impact on the field, allowing many more researchers to participate in state-of-the-art research [3].
As a result, this paper has a great potential, and its results seem very promising. 
Nonetheless, the current paper leaves too many open questions regarding the validity of the experiments.
First, much of the improvement (I think) comes from reducing the number of epochs and/or the number of steps. For fine-tuning, the authors run their model for 2.2 epochs, while their baseline model runs for 3 epochs, roughly 30% more which accounts for much of the reduction observed in Table 2. Similarly, for pretraining, the model runs 80% of the training steps (20% reduction), which accounts much of the training time reduction reported on section 4.3. Running a baseline model that runs *for the same amount of time* is essential to appreciate the contribution of this work (e.g., repeat the same analysis in Figure 3 for the vanilla BERT).
Second, the authors argue for a large reduction in runtime, but are very cryptic about how they actually measure this reduction (footnote 2), while reporting number in ranges of 5%. As the main contribution of this paper is the increased efficiency of the proposed approach, it must be clear how efficiency is measured. Finally, writing in general can be made clearer:

1. Last sentence of intro: "without scarifying accuracy" seems like an inaccurate description of the results presented in this paper. around 1% might be reasonable for 30-40% reduction in training time, but it is certainly a reduction in accuracy.
2. Some description of Network sliming would help
3. The term "intermediate neurons" (section 3.2) was unclear to me.
4. Section 3.3: how is mask difference defined?
5. Figure 1 was unclear to me. What do the axis represent? The authors say "the axes in the plots are the number of training steps finished." so why do you need two of them?
6. The "Non-trivial Sub-network" paragraph feels like it should be part of the Experiments section.
7. Implementation details are only given for the vanilla BERT Are they similar to the EarlyBERT model as well?
8. "Since we observe that the randomly pruned models do not competitive performance ...": how uncompetitive? I would have liked to see these results (also, please fix grammar in this sentence)
9. "Reducing it to 80% seems to be a sweet point with the best balance between performance and efficiency." -> I would disagree. The graph indicates that for MNLI and QNLI 60% seems like a better choice. 

Questions:
1. "We observe empirically that if pruned globally, the attention heads in some layers may be completely removed, making the network un-trainable.": in this case, couldn't the authors remove a full layer? 
2. The difference in the ablation results seem quite small (tables 3 and 4). Are they statistically significant?

Minor:
1. Missing period at the end of the first paragraph in the related work section.
2. Second paragraph of related work: McCarley et al. (2019) appears twice with different descriptions, is this intentional?
3. The authors say "we focus on larger datasets from GLUE (MNLI, QNLI, QQP and SST-2), as it is less meaningful to discuss efficient training", but then report and analyze results from other GLUE datasets as well.
4. "Hon downstream tasks with smaller learning rate" -> Do you mean smaller datasets?





References:
[1] Sharir, O., Peleg, B., and Shoham, Y. (2020). The cost of training NLP models: A concise overview. arXiv:2004.08900.
[2] Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proc. of ACL.
[3] Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. (2019). Green AI. arXiv:1907.10597.


review_title: Interesting and promising results, but important methodological unclarities.

Meta-review: It is important to develop efficient training methods for BERT like models since they have been widely used in real-world natural language processing tasks. The proposed approach is interesting. It speeds up BERT training via identifying lottery tickets in the early stage of training. We agree with the authors's rebuttal that autoML is not that related to the work here. Our main concern on this work  is its worse-than-BERT performance showed in Table 2. The performance gap is significant. Sufficiently more training steps would fill the performance gap but the proposed method may have no advantage any more over the normal training procedures. To make this work more convincing, we would like to suggest to include experiments on comparing different methods under similar prediction performance.  In addition, since the main claim of this work is for training efficiency,  it will be helpful to show the advantage of this method by directly presenting the training curves/ results of different methods.   Overall this paper is pretty much on the boundary. We encourage the authors to resubmit this work once these issues are well resolved. 
Decision: Reject