EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Authors: Xiaohan Chen
Yu Cheng
Shuohang Wang
Zhe Gan
Zhangyang Wang
Jingjing Liu
Venue:  (2021)
Keywords: Natural Language Processing
Lottery Tickets Hypothesis
Efficient Training

Abstract: Deep, heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks. However, their high model complexity requires enormous computation resources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NLP models, but only focus on reducing inference cost/time, while still requiring expensive training process. Other works use extremely large batch sizes to shorten the pre-training time at the expense of high demand for computation resources. In this paper, inspired by the Early-Bird Lottery Tickets studied for computer vision tasks, we propose EarlyBERT, a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. We are the first to identify structured winning tickets in the early stage of BERT training, and use them for efficient training. Comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks show that EarlyBERT easily achieves comparable performance to standard BERT with 35~45% less training time.

Under review as a conference paper at ICLR 2021
EARLYBERT: EFFICIENT BERT TRAINING VIA EARLY-BIRD LOTTERY TICKETS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep, heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many natural language processing (NLP) tasks. However, their high model complexity requires enormous computation re- sources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NL.P models, but only fo- cusing on reducing inference time while still requiring expensive training process. Other works use extremely large batch sizes to shorten the pre-training time, at the expense higher computational resource demands. In this paper, inspired by the Early-Bird Lottery Tickets recently studied for computer vision tasks, we propose EarlyBERT, a general computationally-effinn efficcent training algorithm applicable to both pre-training and fine-tuning of large-scale language models. By slimming the self-attention and fully-connected sub-layers inside a transformer, we are the first identify structured winning tickets in the early stage of BERT training. We apply those tickets towards efficient BERT training, and conduct comprehen- sive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks. Our results show that EarlyBERT achieves comparable performance to stan- dard BERT, with 35-455% less training time.
INTRODUCTION
Large-scale pre-trained language models (e.g. BERT (Devlin al., (018)) XLNet (Yang et al. 2019). T5 (Raffel et al. 2019)) have significantly advanced the state of the art in the NL.P field. Despite impressive empirical success, their computational inefficiency has become an acute draw- back in practice. As more and more transformer layers are stacked with larger self-attention blocks, model complexity increases rapidly. For example, compared to BERT-Large model with 340 million parameters, T5 has more than 10 billion to learn. Such high model complexity calls for expensive computational resources and extremely long training time.
Model compression is approach to alleviating this issue. Recently, many methods propose to encode large NLP models compactly (Sun et al.) (219)) et al. 2019 Sun et al.] 2020).. However, inference time, leaving the process of
the focus solely on reducing computational resources searching for the right compact model ever more costly. Furthermore, almost all model compression methods start with large pre-trained model, which in practice may not exist. Recent work (You) et al. 2020b) proposes to use large training batches, which significantly shortens pre-training time of BERT-Large model but demands daunting computing resources (1,024 TPUv3 chips).
In contrast, our quest is to find general resource-efficiet training algorithm for large NLP mod- els, which can be applied both pre-training and fine-tuning stages. Our goal is to trim down the training time, but also avoiding more costs of the total training resources (e.g., taking large-batch or distributed training). To meet this challenge demand, we draw inspirations from a recent work (You et al. 2020a) that explores the use of Lottery Ticket Hypothesis (LTH) for efficient training of computer vision models. LTH was first proposed in Frankle & Carbin (2019) as an exploration to understand the training process of deep networks. The original LTH substantiates trainable sparse sub-network at initialization, but it cannot be directly utilized for efficient training, since the sub- network itself has to be searched through tedious iterative process. In addition, most LTH works discussed only unstructured sparsity. The study of You et al. (2020a) presents new discoveries that structured lottery tickets emerge in early stage of training (i.e., Early-Bird Ticket), and there-

---

Under review as a conference paper at ICLR 2021
fore structurally sparse net-network can be identified with much lower costs, leading to practical
efficient training algorithms.
Inspired by the success of LTH and Early-Bird Ticket, we propose EarlyBERT, a general efficient training algorithm based on structured Early-Bird Tickets. Due to the vast differences between the architectures and building blocks of computer vision models and BERT, directly extending the method of You et al. 2020a) is not applicable to our work. By instead using network slimming (Liu al. 2017) on the self-attention and fully-connected sub-layers inside a transformer, we are the first to introduce an effective approach that can identify structured winning tickets in the early stage of BERT training, that are successfully applied for efficient language modeling pre- vre-training and fine- tuning. Extensive experiments on BERT demonstrate that EarlyBERT can save 35~45% training time without sacrificing accuracy, when evaluated on GLUE and SQuAD benchmarks.
RELATED WORK
Efficient NLP Models is well believed that BERT and other large NLP models are considerably
overparameterized (McCarley 2019 Sun et al. 2019). This explains the emergence of many model
compression works, which can be roughly categorized into quantization (Shen et al. 2020 Zafrir
al.
2019), knowledge distillation (Sun et al.
2019, Jiao et al., 2019; Sanh al. 2019 Sun al.,
2020 dynamic routing (Fan et al., 2019: Xin et al. 2020), and pruning (Li et al. 2020 Wang
ct al.
2019; McCarley| 2019 Michel et al. 2019). Almost all model compression methods focus on
reducing inference time, while their common drawback is the reliance on fully-trained and heavily- engineered dense models, before proceeding to their compact, sparse versions which essentially transplants the resource burden from the inference to the training stage
Pruning is the mainstream approach for compressing BERT so far. McCarley (2019) proposed to greedily and iteratively prune away attention heads contributing less to the model. Wang et al. (2019) proposed to structurally prune BERT models using low-rank factorization and augmented Lagrangian lu norm regularization. McCarley (2019) pruned less important self-attention heads and slices of MLP layers by applying to regularization to the coefficient corresponding to each head/MLP layer. Another line of works aim to reduce the training time of transformer- based models via large-batch training and GPU model parallelism (You et al. 2020b Shoeybi et al. (019). Our work is orthogonal to those works, and can be readily combined for further efficiency boost.
Lottery Ticket Hypothesis in Computer Vision Lottery Ticket Hypothesis (LTH) was firstly proposed in Frankle & Carbin (2019). which shed light on the existence of sparse sub-networks (i.e., winning tickets) initialization with non-trivial sparsity ratio that can achieve almost the same performance (compared to the full model) when trained alone. The winning tickets are identified by pruning fully trained networks using the so-called Iterative Magnitude- -asedd Pruning (IMP). However, IMP is expensive due to its iterative nature. Moreover, IMP leads to unstructured sparsity, which is known to be insufficient in reducing training cost or accelerating training speed practically. Those barriers prevent L.TH from becoming immediately helpful towards efficient training. [Moroos ctt al. 2019) studics the transfcrability of winning tickets between datasets and optimizers. [Zhou al. 2019) investigates different components in LTH and observes the existence of super-masks in winning tickets. Lately, You et al. (2020a) pioneers to identify Early-Bird Tickets, which emerge at the early stage of the training process, and contain structured sparsity when pruned with Network Slimming (Liu et al. 2017). Early-bird tickets mitigate the two limitations of IMP aforementioned, and renders it possible to training deep models efficiently, by drawing such tickets early in the training and then focusing on training this compact subnetwork only.
Lottery Ticket Hypothesis in NLP All above works evaluate their methods on computer vision models. For NLP models, previous work has also found that matching subnetworks exist in training on Transformers and LSTMs (Yu et al., 2019 Renda et al.) 2020). [vvi et al. (2020) derived an algorithm for training sparse neural networks according to LTH and applied it to a character-level language modeling on WikiText-1033 For BERT models, latest work (Chen et al. (020) found that the pre-trained BERT models contain sparse subnetworks, found by unstructured IMP at 40% to 90% sparsity. that are independently trainable and transferable to range of downstream tasks with no performance degradation. Another concurrent work (Prasanna et al., 2020) aims to find structurally sparse lottery tickets for BERT, by pruning entire attention heads and MLP layers. Their experiments turn out that all subnetworks ("good" and "bad"" have "comparable performance" when fined-tuned downstream tasks, leading to their "all tickets winning" conclusion.
2

---

Under review as a conference paper at ICLR 2021
Nevertheless, both works (Chen et al. 2020: Prasanna al. 2020) examine only the pre-trained BERT model, i.e., finding tickets with regard to the fine-tuning stage on downstream tasks. To our best knowledge, no existing study analyzes the LTH at the pre-training stage of BERT; nor has any work discussed the efficient BERT training using LTH, for either pre-training or fine-tuning stage. In comparision, our work represents the first attempt of introducing LTTH to both efficient pre-training and efficient fine-tuning of BERT. Our results also provide positive evidence that LTH and Early-Bird Tickets in NLP models are amendable structured pruning too.
THE EARLYBERT FRAMEWORK
In this section, we first revisit the original Lottery Ticket Hypothesis (LTH) (Frankle & Carbin 2019) and its variant Early-Bint Bird Ticket You al. 2020a), then describe our proposed EaT
3.1 REVISITING LOTTERY TICKET HYPOTHESIS
Denote [(;; 0) as deep network parameterized by 0 and as its input. A sub-network of ʃ can be characterized by a binary mask m, which has exactly the same dimension as 6. When applying the mask m to the network, we obtain the sub-network f(x; 0 O m). where @ is the Hadamard product operator. LTH states that, for a network initialized with do. an algorithm called Iterative Magnitude Pruning (IMP) can identify mask m such that the sub-network fa;; fo m) can be trained to have no worse performance than the full model f following the same training protocol. Such sub-network [2; 0o @ m), including both the mask m and initial parameters lo. called winning ticket. The IMP algorithm works as follows: (1) initialize m as an all-one mask; (2) fully train fa: 8g 3m) obtain a well-trained 0; (3) remove small portion of weights with the smallest magnitudes from 0 @ m and update m; (4) repeat (2)-(3) until a certain sparsity ratio is achieved.
Two obstacles prevent LTH from being directly applied efficient training. First, the iterative process in IMP is essential to preserve the performance of LTH; however, this is computationally expensive, especially when the number of iterations high. Second, the original LTH does not pursue any structured sparsity in the winning tickets. In practice, unstructured sparsity is difficult to be utilized for computation acceleration even when the sparsity ratio is high (Wen et al. 2016).
To mitigate these gaps, Early-Bird Tickets are proposed by You et al. (2020a), who discovers that when using structured mask m and a properly selected learning rate, the mask m quickly converges and the corresponding mask emerges as the winning ticket in the carly stage of training. The early emergence of winning tickets and the structured sparsity are both helpful in reducing computational cost in the training that follows. You et al. (2020a) focuses on computer vision tasks with convolu- tional networks such as VGG (Simonyan & Zisserman 2014) and ResNet (He et al. (016)) Inspired explore whether there are structured winning tickets in the early stage of BERT
by this, we set training that can significantly accelerate language model pre-training and fine-tuning.
3.2 DISCOVERING EARLYBERT
training framework consists of three steps: (1) Searching Stage: jointly train BERT and the sparsity-inducing coefficients to be used to draw the winning ticket; (ti) Ticket- drawing Stage: draw the winning ticket using the learned coefficients; and (iir) Efficient-tranning Stage: train EaTlyBERT for pre-training or downstream fine-tuning.
The proposed EarlyBERT
Searching Stage To search for the key sub-structure in BERT, we follow the main idea of Network Slimming (NS) (Liu et al. 2017). However, pruning in NS is based on the scaling factor y in batch normalization, which not used in most NL.P models such as BERT. Therefore, necessary modifications to the original NS so that it can be adapted to pruning BERT. Specifically, propose to associate attention heads and intermediate layers of the fully-connected sub-layers in a transformer with learnable coefficients. which will be jointly trained with BERT but with an additional e regularization promote sparsity.
make
Early BERT refers to the winning ticket discovered by the proposed Stage framework, which is equivalent to the resulting pruned BERT model after drawing the winning ticket. We also interchangeably use EarlyBERT the name of the proposed framework.
3

---

identify the winning tickets Under review as a conference paper at ICLR 2021
Some studies (iihhll et al., 2019 Voita al.) 2019) find that the multi-head self-attention module of transformer can be redundant sometimes, presenting the possibility of pruning some heads from each layer of BERT without hurting model capacity. A multi-head attention module (Vaswani al. 2017) is formulated as:
MultiHcad K, V) = Concat(head,,.... head,, WWo
where head; = Attention(QW: KKW? VW)),
(2)
where the projections Wo, W.O W W.Y are used for output, query, key and value. Inspired by Liu et al. (2017), we introduce set of scalar coefficients cf (i is the index of attention heads and means "head") inside head,;
head, = ci Attention(QW, KWY vw!).
(3)
After the self-attention sub-layer in each transformer layer, the output MultiHead((Q, K, V) will be fed into a two-layer fully-connected network, in which the first layer increases the dimension of the embedding by 4 times and then reduces it back to the hidden size (768 for BERT BASE and 1,024 for LARGE) We multiply learnable coefficients to the intermediate neurons:
FFN = =c max(0, xW1 2W,,
+ b, W2 + b2.
(4)
These modifications allow us to jointly train BERT with the coefficients, using the following loss:
((((=)),c) (o( + A||c||
(5)
where Lo is the original loss function used in pre-training or fine-tuning, is the concatenation of all the coefficients in the model including those for attention heads and intermediate neurons, and is the
that controls the strength of regularization.
Note that in this step, the joint training of BERT and the coefficients are still as expensive as normal BERT training. However, the winning strategy of EarlyBERT is that only need to perform this joint training for a few steps, before the winning ticket emerges, which is much shorter than the full training process of pre-training or fine-tuning. In other words, at a very low cost compared to the full training. Then, we draw the ticket (i.e., the FarlyBERT), reset the parameters and train EarlyBERT that is computationally efficient thanks to its structured sparsity. Next, we introduce how we draw EarlyBERT from the learned coefficients.
Ticket-drawing Stage
After training BERT and coefficients jointly, we draw EarlyBERT using the learned coefficients with a magnitude-based metric. Note that we prune attention heads and intermediate neurons separately, as they play different roles in BERT.
We prune the attention heads whose coefficients have the smallest magnitudes, and remove them from the computation graph. We also prune the rows in WO (see Eqn. 1 that correspond to the removed heads. Note that this presents design choice: should we prune the heads globally or layer-wisely? In this paper, we use layer-wise pruning for attention heads, because the number of heads in each layer is very small (12 for BERT BASE and 16 for BERTLARGEL We observe empirically that if pruned globally, the attention heads in some layers may be completely removed, making the network un-trriia.lle Furthermore, Raassauer al. 2220)) observes that attention heads in different layers exhibit different behaviors. This also motivates us to only compare importance of attention heads within each layer.
Similar to pruning attention heads, we prune intermediate neurons in the fully-connected sub-layers. Pruning neurons equivalent to reducing the size of intermediate layers, which leads reduced size of the weight matrices W1 and W2 in Eqn. (4). Between global and layer-wise pruming, empir- ical analysis shows that global pruning works better. We also observe that our algorithm naturally prunes more neurons for the later layers than earlier ones, which coincides with many pruning works on vision tasks. We leave the analysis of this phenomenon future work.
Efficient-training Stage We then train EarlyBERT that have drawn for pre- pre-training or fine- tuning depending on the target task. If we apply EarlyBERT to pre-training, the initialization 6n of BERT will be random initialization, the same setting as the original L.TH (Frankle & Carbin 2019)
4

---

Under review as a conference paper at ICLR 2021
and Early-Bird Tickets You et al. 2020a). If we apply EarlyBERT fine-tuning then 0g can be any pre-trained model. We can also moderately reduce the training steps in this stage without sacrificing performance, which is empirically supported by the findings in Frankle & Carbin (2019): You et al. (2020a) that the winning tickets can be trained more effectively than the full model. In practice, the learning rate also be increased to speed up training, in addition reducing training steps.
Different from unstructured pruning used in LTH and many other compression works (Frankle & Carbin 2019 Chen al., 2020), structurally pruning attention heads and intermediate neurons in fully-connected layers can directly reduce the number of computations required in the transformer layer, and shrink the matrix size of the corresponding operations, yielding a direct reduction in computation and memory costs.
3.3 VALIDATION OF EARLYBERT
Early Emergence Following similar manner in You al. (2020a), visualize the normalized mask distance between different training steps, to validate the early emergence winning tickets. In Figure the axes in the plots the number of training steps finished. We only one fully- connected sub-layer to plot Figure [ [d] due to high dimensionality. In both pre-training and fine-tuning, the mask converges in very early stage of the whole training process. Although observe an increase of mask distance in fully-connected layers during pre-training (in Figure [(b). this can be easily eliminated by early stopping and using mask distance the exit criterion.
Non-trivial Sub-network Here, by non-trivial we mean that with the same sparsity ratio in EarlyBERT, ran- domly pruned model suffers from sig- nificant performance drop. The perfor- mance drop happens even we only prune attention heads. We verify this by running fine-tuning experiments BERT BASE- Specifically, we prune 4 heads from each transformer layer in BERTRASE and EarlyBERT. We fine- tune BERT BASE for epochs with ini- tial learning rate 2 x 10-5, We run the searching stage for 0.2 epochs with À = x 10-1, draw EarlyBERT with pruming ratio p = 1/3, and then fine-tune EarlyBERT for 2 epochs with doubled initial learning rate. For the randomly pruned models, we randomly prune 4 heads in each layer and follow the same fine-tuning protocol as EarlyBERT. The reported results of randomly pruned models are the average of 5 trials with different seeds for pruning. The results on three tasks from GLUE benchmark (Wang et al. 2018) presented in Table show that randomly pruned model con- sistently under-performs EarlyBERT with significant gap, supporting our claim that EarlyBERT indeed identifies non-trivial structurss..
Table Comparison between sundomly-prunce models and EarlyBERT on GLUE tasks. We prune 4 heads in each layer.
Methods
MNLI
QNLI
QQP
SST-2
BERTBASE
83.16
90.59
90.34
91.70
EarlyBERTBASE
83.58
90.33
90.41
92.09
Random
82.26
88.87
90.12
91.17
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Backbone Models Following the official BERT implementation (Devlin et al. 2018 Wolf et al.)
2019), we use both BERT BASE (12 transformer layers, hidden size 768, 3,072 intermediate neurons, 12 self-attention heads per layer, 1 110M parameters in total) and BERT (24 transformer layers, hidden size 1,024, 4,096 intermediate neurons, 16 self-attention heads per layer, 340M parameters in total) for experiments.
Datasets We use English Wikipedia (2,500M words) as the pre-training data. For fine-tuning experiments and evaluation of models in the pre-training experiments, we use tasks from GLUE benchmark (Wang et al. 2018) and muestion-answering dataset SQuAD v1.1 (Rajpurkar et al. 2016). Note that as our goal is efficient pre-training and fine-tuning, we focus on larger datasets from GLUE (MNLI, QNLI, QQP and SST-2), less meaningful discuss efficient training
5

---

Under review as a conference paper at ICLR 2021
24
14
23
32
(a) Self-attention Pre-training
(b) FC Pre-training
35
34
(c) Self-attention Fine-tuning
(d) PC in Fine-tuning
Illustration of Mask Distance. Top: mask distance observed in pre-training. Bottom: mask distance observed in fine-tuning. The color represents the normalized mask distance between different training steps. The darker the color, the smaller the mask distance.
Figure 1:
on very small datasets. We the default training settings for pre-training and fine-tuning on both models. To evaluate model performance, we use Matthew's correlation score for CoLA, matched accuracy for MNLI, Fl-score for SQuAD v1.1, and accuracy in percentage for other tasks on GLUE. We omit % symbols in all the tables on accuracy results.
Implementation Details For the vanilla BERT, we fine-tune on GLUE datasets for epochs with initial learning rate 2 x 10-5, and for 2 epochs on SQuAD with initial learning rate x 10-5, we use AdamW (Loshchilov & Hutter, 2017) optimizer for both cases. For pre-training, we adopt LAMB optimization technique Yoou et al. 2020b), which involves two phases of training: the first 9/10 of the total training steps uses sequence length of 128, while the last 1/10 uses a sequence length of 512. Pre-training by default has 8,601 training steps and uses 64k/32k batch sizes and x 10-3/4 10-3 initial learning rates for the two phases, respectively. All experiments are run on 16 NVIDIA V100 GPUs.
4.2 EXPERIMENTS ON FINE-TUNING
The main results of EarlyBERT in fine-tuning are presented in Table 2 When drawing EarTyBERT, prune 4 heads in each layer from BERT BASE and heads from BERTLARGE and globally prune 40% intermediate neurons in fully-comne connected sub-layers in both models. We reduce the training epochs two on GLUE benchmark and scale up the learning rate by to buffer the effect of reduced epochs. For SQuAD dataset, we keep the default setting, as find SQuAD is more sensitive to number of training epochs. Ablation studies on the effects of the number of training epochs and learning rate included in the following section.
Several observations can be drawn from Table Firstly, in most tasks, EarlyBERT saves over 40% of the total training time without inducing much performance degradation. It can also outperform another strong baseline LayerDrop (Fan et al. 2019), which drops one third of the layers so that the number of remaining parameters comparable ours. Note that LayerDrop models are fine-tuned for three full epochs, yet EarlyBEr BERT is still competitive in most cases. Secondly, we consistently observe obvious performance advantage of EarlyBERT over randomly pruned models, which pro-

---

Under review as a conference paper at ICLR 2021
Table 2: Performance of EarlyBERT (fine-tuning) compared with different baselines.
Methods
MNLI
QNLI
QQP
SST-2
SQuAD
Time Saved
BERT BASE
83.16
90.59
90.34
91.70
87.50
BarlyBERTBASE
81.81
89.18
90.06
90.71
86.13
40~45%
RandoMBASE
79.92
84.46
89.42
89.68
84.47
45~50%
LayerDrop (Fan et al. 2019)
81.27
88.91
88.06
89.89
84.25
~33%
BERT LARGE
86.59
92.29
91.59
92.21
90.76
EarlyBERT,ARGE
85.13
89.22
90.64
90.94
89.45
35--40%
RandomLARGE
78.45
84,46
89.89
88.65
88.79
40~45%
LayerDrop (Fan et al.
2019
85.12
91.12
88.88
89.97
89.44
~33%
vides another strong evidence that EarlyBERT does discover nontrivial key sparse structures. Even though there still exists a margin between EarlyBERT and the baseline (like (You et al. (020a) also observed similarly in their tasks), the existence of structured winning tickets and its potential for efficient training highly promising. We leave as future work to discover winning tickets of higher sparsity but better quality.
Ablation Studies on Fine-tuning We perform extensive ablation studies to investigate important hyperparameter settings in EarlyBERT, using EarlyBERT BASE our testing bed. For all experi- ments, we use the average accuracy on the larger datasets from GLUE benchmark (MNLI, QNLI, QQP and SST-2) as the evaluation metric.
Number of training epochs and learning rate. We first investigate whether we can properly reduce the number training epochs, and scaling the learning rate can help compliment the negative effect caused by reducing training steps. Results in Figure[shoow that when we fine-tune EarlyBERT for fewer epochs on GLUE benchmark, up up-scaling learning rate first helps to recover performance, and then causes decrease again. We will use epochs and 4 x 10 rate for EarlyBERT on GLUE experiments.
learning
Regularization strength A proper selection of the regularization strength À decides the qual- ity of the winning ticket, consequently the performance of EarlyBERT after pre-training/fine- tuming. Results on different strength settings in Table 3 show that the regularization strength has marginal influence on EarlyBERT performance. We use A 10 4 that achieves the best performance in following experiments.
Pruning ratios p. We further investigate the effects of different pruning ratios as well as layer- wise/global pruning on the performance of FarlyBERT. As discussed in Sec. 3.2 we only consider layer-wise pruning for self-attention heads. Table 4 shows that the performance monotonically decreases when we prune more self-attention heads from BERT; however, and then a sharp decrease in accuracy, when the pruning ratio is raised for intermediate neurons in colleccondeeted sub-layers (40% pruning ratio seems to be the sweet spot). We also observe consistent superiority of global pruning over layer-wise pruning for intermediate neurons.
see slight increase
4.3 EXPERIMENTS ON PRE-TRAINING
We also conduct pre-training experiments and present the main results in Table 5 Similar to the settings in fine-tuning experiments, we prune 4 heads in each layer from BERT and 6 heads from BERTLARGES however, we prune slightly fewer (30%) intermediate neurons in fully-connected sub- layers in both models, since we empirically observe that pre-training is more sensitive to aggressive intermediate neuron pruning. In both phases of pre-training, we reduce the training steps to 80% of the default setting when training EarlyBERT (based on the ablation study shown in Figure 3) Other hyperparameters for pre-training follow the default setting described Sec. 4.T] All models are fine-tuned and evaluated on GLUE (Wang et al.) 2018) and SQuAD vI.1 datasets (Rajpurkar et al. (016)) with the default setting. Since we observe that the randomly pruned models do not competitive performance in fine-tuning experiment for all downstream tasks, in this section we focus on comparing the achievable performance EarlyBERT with full BERT baseline.
Thh time saving during training is roughly estimated using average time per mini-batch and thus is not very accurate and only serves refernece.

---

Under review as a conference paper at ICLR 2021
Table Ablation of regularization strength
- Epoch 2 Epoch
0.88
A
10
10-3
10
Accuracy (%)
Avg. Acc.
89.10
88.81
88.93
0.87
Table 4: Ablation of pruning ratios self-attention heads and intermediate neurons.
0.85
# Pruned Heads
5
6
26.5
46-5
66-5
8e-5
Learning
Layer-wise pruning
89.10
88.69
88.26
Pruned Neurons
30%
40%
50%
Figure 2: Effect of reducing training epochs and up-scaling learning rate for EarlyBERT in fine-tuning.
Layer-wise pruning Global pruning
88.33
88.48
87.91
88.54
88.70
88.01
Table 5: Performance of EarlyBERT (pre-training) compared with BERT baselines.
Methods
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
SQuAD
BERT BASE
0.45
81.40
84.07
89.86
89.80
60.29
90.48
87.60
FarlyBERT BASE
0.41
79.97
80.39
89.86
89.44
61.01
90.94
85.48
BERTLARDS LARGE
0.50
83,56
85,90
90.44
90.45
59.93
92.55
90.43
EarlyBERT LARGE
0.47
82.54
85.54
90.46
90.38
61.73
91.51
89.36
From the results presented in Table we can see that downstream tasks with larger datasets such as QNLI, QQP and SST-2 we can achieve accuracies that are close BERT baseline (within 1% accuracy gaps except for EarlyBERTBASE tasks with smaller learning rate, the patterns are not consistent: we observe big drops on CoLA and MRPC but improvement RTE. Overall, EarlyBERT achieves comparable performance while saving 30~335% training time thanks to its structured sparsity and reduction in training steps.
MNLI and SQuAD). However, on downstream
Reducing Training Steps in Pre-training We in- vestigate whether EarlyBERT, when non-essential heads and/or intermediate neurons are pruned, can train more efficiently, and whether we can reduce the number training steps in pre-training This can fur- ther help reduce training cost in addition to the effi- ciency gain from pruning. We use FarlyBERTpAsE BASE- Self (only self-attention heads are pruned when draw- ing the winning ticket) as the testing bed. Figure 3 shows the performance decreases more when duce the number of training steps to 60% or less. Re- ducing it 80% seems be sweet point with the best balance between performance and efficiency.
90.0
85.0
MNLI
Accuracy
MRPC
62.5
77.5
75.0
40%
60%
80%
100%
Train steps
Figure 3: Effect of reducing training steps in pre-training on EarlyBERTpASE.
5 CONCLUSION
In this paper, we present EarlyBERT, an efficient training framework for large-scale language model pre-training and fine-tuning. Based Lottery Ticket Hypothesis, FarlyBERT identifies structured winning tickets in an early stage, then uses the pruned network for efficient training. Experimental results on GLUE and SQuAD demonstrate that the proposed method is able achieve comparable performance to standard BERT with much less training time. Future work includes applying Early- BERT other pre-traine pre-trained language models and exploring more data-efficient strategies enhance the current training pipeline.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-traines bert networks, 2020.
Jacob Devlin, Ming-Woe Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 04805, 2018.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In MLSys, pp. 471-481 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint (xXiv::909.115555 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICI.R, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10Xx:1909.10351,, 2019.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gon- zalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05111, 2017.
J Scott McCarley. Pruning a bert-based question answering model. arXiv:1910.06366) 2019.
arXiv preprint
Paul Michel, Omer Levy, and Graham Neubig, Are sixteen heads really better than one? NeurIPS, 2019.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: gen- eralizing lottery ticket initializations across datasets and optimizers. In NeurlPS, pp. 1932-4942, 2019.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are win- ning. arXiv preprint arXiv:2005. 00561, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gru- ber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02211 02217. 2020.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint orXiv:2003.0 02389, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1910.0 2019.

---

Under review as a conference paper at ICLR 2021
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In AAAI, pp- 8815-8821, 2020.
M. Shoeybi, M. Patwary, Puri, P. LeGresley, J. Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint DrXiv:/4099 1556, 2014.
S. Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model com- pression. In EMNLP, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- BERT: a compact task-agnostic BERT for resource-limited devices. In ACL, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008,
2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: (00.:18844.7461, 2018.
Ziheng Wang. Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:1910.04733 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. NeurlPS, pp. 2074-2082, 2016.
Thomas Wolf, Lysandre Debut. Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: Statc-of-the-art natural language processing. ArXiv, pp. arXiv-1910, 2019.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In ACI, pp. 2246-2251, Online, July 2020. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurlPS, 2019.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In ICLR, 2020a.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In ICLR, 2020b.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari $ Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. In ICLR, 2019.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat, Q8bert: Quantized 8bit bert. arXiv preprint arXiv: 0888 2019.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In NewrIPS, pp. 3597-3607, 2019.
10


Reviews:
[{"review_id": "_QHZA65mG4K", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to sparsifying BERT. What sets this work apart from prior work on model compression is that while prior works attempt to compress a pre-trained BERT, the authors in this work attempt to learn a sparsified BERT for the purpose of speeding up pre-training. The method essentially involves learning an independent bernoulli mask for all BERT heads along with Bernoulli masks for some later intermediate neurons. This model is then trained, along with the masks, for a few epochs. Then, heads/neurons corresponding to a small value of mask are pruned out. This results in a sparser BERT model, leading to faster pre-training.\n\n#### Pros\nThe paper is well written and the presentation of the contribution is simple and well-motivated. \n\n#### Cons\n1. Since one of the main contributions of this work is to make progress on improving the training / inference speed of large transformers, the authors could spend more time going over how the “Time Saved” column is computed. As of now, it seems casual and hand-wavy.\n2. Experiment protocol:\n  * How were the hyperparameters decided? Could we have uncertainty estimates for all results by reporting mean/std dev. across 3-5 runs (atleast for fine-tuning if doing this for pre-training is too compute intensive). This is especially useful since some of the numbers between EarlyBERT and Random appear very close in Table-1.\n  * While a central argument is that most model distillation techniques still require expensive pre-training, it would still be useful to include some of those results in Table-2 since EarlyBERT is comparable to those techniques for the purpose of Table-2.\n  * One way to contextualize how much time is saved during pre-training would be to report total time required for fine-tuning+pretraining on the entire glue benchmark. This would involve computing the pre-training time (time to learn BERT parameters on Wikipedia) + total fine-tuning time across all datasets (QQP/CoLA/MNLI etc) considered. This could then be compared against alternatives (such as DistilBert and DeeBERT). \n3. Baselines: Experiments on pre-training compare with no baselines. Some possibilities:\n  * Using DeeBERT / other early exit approaches at pre-training time.\n  * Training a BERT model for some epochs, and then distilling it into a smaller network for the rest of the training.\n\n\n#### Misc Nitpicks\n1. The phrase “structured sparsity’ is used in multiple places, but never defined. \n2. You et al. pioneers $\\rightarrow$ You et al. pioneer\n3. Could Prasanna et al. 2020 be extended to sparsify pre-training?", "extra_scores": {}, "extra_text": {}}, {"review_id": "WJCsET4sSBq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper uses the Lottery Ticket Hypothesis to compress BERT. More specifically, they adapt EarlyBird lottery tickets to the BERT setting in order to find winning configurations in early stages of training combine it with structured pruning methods to ensure the resulting network is more efficient to train. The method is a three-stage process: (1) searching - this phase involves training full BERT with some coefficient parameters to learn the mask, (2) drawing - use the mask to “draw a ticket” or select the sub-network to train, (3) training. Experiments show that performance isn’t that much worse when EarlyBERT is used for fine-tuning and for pre-training.\n\nThe goal of the paper is to find structured winning tickets for BERT in the early stages of training/fine-tuning.\n\nStrengths:\n1. Compressing BERT using the lottery ticket hypothesis has been getting a lot of attention recently, and doing so relatively efficiently is an exciting and interesting contribution.\n2. Compared to previous and contemporary work, the combination of EarlyBird lottery tickets (You et al., 2020) to detect tickets early and network slimming (Liu et al., 2017) for structured pruning, is an interesting one. Prasanna et al., (2020) are doing structured pruning too, but via an iterative pruning method. \n3. Experiments for both pre-training (the first of its kind) and fine-tuning show that performance does not drop all that much for GLUE and Squad which are the main set of tasks BERT is typically evaluated on.  \n\nWeaknesses:\n1. The paper isn’t very clear in some places. It starts out explaining things well but then it gets harder to follow the details. Eg: need to add more information on the mask - it’s binarized at some point? The distance metric is still Hadamard like in EarlyBird? \n2. (More of a question/nit) Is it possible to also show what happens when a winning ticket for BERT fine-tuning is selected based on the pre-training objective? Chen et al., (2020) showed that these make for better tickets that are performant on many of the downstream tasks. It would be interesting to see if this holds true for EarlyBert and would add another layer of fine-tuning efficiency.\n\nThe work isn't terribly novel, but it's still interesting.\n\nQuestions and comments:\n1. Why does the mask distance diverge for FC in pre-training (Figure 1b)? Does it somehow indicate that the training run is degenerate?\n2. The ablation on the regularization parameter didn’t give a clear indication of how important selecting this is. It seems to not be that important, but would using separate values for attention and FC make a difference?\n3. It would probably be helpful to expand the table/figure captions, make them a bit more detailed. \n4. Curious, why did you only use the largest tasks from GLUE?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LxOVxNXxUNX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of this work is to use Early Bird Lottery Tickets to reduce pre-training and fine tuning time for BERT. In order to reduce computation, the authors propose pruning the number of attention heads and neurons in the fully connected layers. They demonstrate that the technique works for BERT-Base and BERT-Large for GLUE and SQUAD tasks. \n\nThe work is well-thought through and authors do a good job of explaining their approach and other existing works using lottery tickets.  The usage of lottery tickets during the pre training phase is the biggest strength of the paper since it can result in significant computational savings. The authors perform interesting ablation studies but they could be augmented by a few more experiments (see below). The paper could also be improved by comparison with relevant prior work. \n\nHere are some thoughts and questions that could help improve the paper: \n\n* Experiments in Section 4.2 are only during the fine tuning stage. How do these results compare with prior work for drawing lottery tickets in transfer learning for NLP (Chen et al, Prasanna et. al)? Particularly, it would be good to compare against Chen et al since their work is very relevant. \n* How long is the searching stage followed by the efficient training stage? Does this remain the same for all the tasks? Or did it require tuning? It would be interesting to see how soon we can switch from the searching stage to the efficient training stage.\n* The paper proposes two different approaches for pruning: pruning attention heads and removing neurons from fully connected layers. It would be interesting to see how the accuracy of the approach changes if only neurons or attention heads are pruned. \n* In Section 4.3, the authors discuss reducing training time by reducing training steps for EarlyBERT. I think a similar analysis for BERT would be helpful to understand if the training time can be reduced for the baseline model as well. Also, pre-training techniques have tended to show improvements in downstreams tasks with longer pre-training as the long as the pre-training dataset is large enough. So, perhaps this reduction in pre-training steps might not be applicable for larger pre-training datasets.\n\nOverall, I find the approach interesting and the authors show computational savings in the pre-training models. I recommend accepting the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "LjymxTRiccT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles a very important and under-studied problem: reducing the cost of training NLP models. The authors present a method that builds on the lottery ticket hypothesis (LTH). The authors first identify redundant structures early during training, then prune these structures, which leads to faster training. The authors experiment both with pre-training and fine-tuning of contextual models (BERT-{base,large}) and claim large reduction in training time, with reasonable loss in performance. \nDespite very encouraging results, several important methodological questions about the source of the efficiency gains and other aspects of the paper are left unanswered. I cannot recommend accepting this paper in its current form, but am looking forward to reading the authors' response which might clarify things.\n\n\nDetailed comments:\n\nTraining (especially pretraining) costs have been going wild in AI and NLP more particularly, which leads to large costs ([1]) as well as potential environmental problems ([2]). Reducing these costs could have a very high impact on the field, allowing many more researchers to participate in state-of-the-art research [3].\nAs a result, this paper has a great potential, and its results seem very promising. \nNonetheless, the current paper leaves too many open questions regarding the validity of the experiments.\nFirst, much of the improvement (I think) comes from reducing the number of epochs and/or the number of steps. For fine-tuning, the authors run their model for 2.2 epochs, while their baseline model runs for 3 epochs, roughly 30% more which accounts for much of the reduction observed in Table 2. Similarly, for pretraining, the model runs 80% of the training steps (20% reduction), which accounts much of the training time reduction reported on section 4.3. Running a baseline model that runs *for the same amount of time* is essential to appreciate the contribution of this work (e.g., repeat the same analysis in Figure 3 for the vanilla BERT).\nSecond, the authors argue for a large reduction in runtime, but are very cryptic about how they actually measure this reduction (footnote 2), while reporting number in ranges of 5%. As the main contribution of this paper is the increased efficiency of the proposed approach, it must be clear how efficiency is measured. Finally, writing in general can be made clearer:\n\n1. Last sentence of intro: \"without scarifying accuracy\" seems like an inaccurate description of the results presented in this paper. around 1% might be reasonable for 30-40% reduction in training time, but it is certainly a reduction in accuracy.\n2. Some description of Network sliming would help\n3. The term \"intermediate neurons\" (section 3.2) was unclear to me.\n4. Section 3.3: how is mask difference defined?\n5. Figure 1 was unclear to me. What do the axis represent? The authors say \"the axes in the plots are the number of training steps finished.\" so why do you need two of them?\n6. The \"Non-trivial Sub-network\" paragraph feels like it should be part of the Experiments section.\n7. Implementation details are only given for the vanilla BERT Are they similar to the EarlyBERT model as well?\n8. \"Since we observe that the randomly pruned models do not competitive performance ...\": how uncompetitive? I would have liked to see these results (also, please fix grammar in this sentence)\n9. \"Reducing it to 80% seems to be a sweet point with the best balance between performance and efficiency.\" -> I would disagree. The graph indicates that for MNLI and QNLI 60% seems like a better choice. \n\nQuestions:\n1. \"We observe empirically that if pruned globally, the attention heads in some layers may be completely removed, making the network un-trainable.\": in this case, couldn't the authors remove a full layer? \n2. The difference in the ablation results seem quite small (tables 3 and 4). Are they statistically significant?\n\nMinor:\n1. Missing period at the end of the first paragraph in the related work section.\n2. Second paragraph of related work: McCarley et al. (2019) appears twice with different descriptions, is this intentional?\n3. The authors say \"we focus on larger datasets from GLUE (MNLI, QNLI, QQP and SST-2), as it is less meaningful to discuss efficient training\", but then report and analyze results from other GLUE datasets as well.\n4. \"Hon downstream tasks with smaller learning rate\" -> Do you mean smaller datasets?\n\n\n\n\n\nReferences:\n[1] Sharir, O., Peleg, B., and Shoham, Y. (2020). The cost of training NLP models: A concise overview. arXiv:2004.08900.\n[2] Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proc. of ACL.\n[3] Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. (2019). Green AI. arXiv:1907.10597.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DfnIh_mRH-", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a technique for reducing the computational requirements of training BERT early in training to reduce the overall amount of resources required.\n\nPros:\n\nThe paper is well written and clear for the most part. The authors do thorough experimental evaluation.\n\nCons:\n\nI have two primary concerns about the paper and the proposed technique.\n1. The positioning of the technique is not entirely clear to me. The authors pitch it as a technique for reducing the training time of BERT and use LayerDrop as a baseline technique that also removes network components. However, it feels like another baseline that should be considered is neural architecture search, which also seeks to automatically find a more efficient model to train. The difference here is that the authors find the model early in the training run, but it seems like the EarlyBERT procedure could be run once and the resulting model architecture could be saved and re-trained like NAS models are. \n2. I found the experimental results to be lacking detail and breadth necessary to establish the value of the technique. Firstly, the rough time estimates in Table 2 are very odd given the primary value of the proposed technique is to reduce training time. The accuracy of EarlyBERT is close enough to LayerDrop that accurate training cost numbers are needed to differentiate between the techniques. Secondly, quoting training time reductions over the dense baseline when the EarlyBERT mode does not achieve the same accuracy makes the comparison very difficult to make. This problem shows up quite commonly in the model compression literature [1] and I’d encourage the authors to show full accuracy-training time tradeoff curves so that the training time savings for a given accuracy can be more clearly established. Lastly, I found the use of reduced training epochs in EarlyBERT to be odd because you do not evaluate whether or not this can be done for the baseline models and there isn’t clear evidence as to why your model would be able to do this while others (e.g., DropLayer) cannot. Figure 2 also does not seem to corroborate that higher learning rates can be used with shorter training time to achieve better accuracy. The data in your figure shows that the best learning rate achieves the best model quality independent of the number of training epochs.\n\nReferences:\n1. https://arxiv.org/abs/2003.03033\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: It is important to develop efficient training methods for BERT like models since they have been widely used in real-world natural language processing tasks. The proposed approach is interesting. It speeds up BERT training via identifying lottery tickets in the early stage of training. We agree with the authors's rebuttal that autoML is not that related to the work here. Our main concern on this work  is its worse-than-BERT performance showed in Table 2. The performance gap is significant. Sufficiently more training steps would fill the performance gap but the proposed method may have no advantage any more over the normal training procedures. To make this work more convincing, we would like to suggest to include experiments on comparing different methods under similar prediction performance.  In addition, since the main claim of this work is for training efficiency,  it will be helpful to show the advantage of this method by directly presenting the training curves/ results of different methods.   Overall this paper is pretty much on the boundary. We encourage the authors to resubmit this work once these issues are well resolved. 
Decision: Reject