Venue: NAACL 2022

# Towards Efficient NLP: A Standard Evaluation and A Strong Baseline

Xiangyang Liu\({}^{1,2,*}\), Tianxiang Sun\({}^{1,2,*}\), Junliang He\({}^{1,2}\), Jiawen Wu\({}^{1,2}\), Lingling Wu\({}^{1,2}\),

**Xinyu Zhang\({}^{3}\), Hao Jiang\({}^{3}\), Zhao Cao\({}^{3}\), Xuanjing Huang\({}^{1,2}\), Xipeng Qiu\({}^{1,2}\) +\({}^{1}\)School of Computer Science, Fudan University \({}^{2}\)Shanghai Key Laboratory of Intelligent Information Processing, Fudan University \({}^{3}\)Huawei Poisson Lab {xiangyangliu20,txsun19,xjhuang,xpqiu}@fudan.edu.cn  {zhangxinyu35,jianghao66,caozhaol}@huawei.com**

Equal contribution.Corresponding author.

###### Abstract

Supersized pre-trained language models have pushed the accuracy of various natural language processing (NLP) tasks to a new state-of-the-art (SOTA). Rather than pursuing the reachless SOTA accuracy, more and more researchers start paying attention to model efficiency and usability. Different from accuracy, the metric for efficiency varies across different studies, making them hard to be fairly compared. To that end, this work presents **ELUE** (**E**fficient **L**anguage **U**nderstanding **E**valuation), a standard evaluation, and a public leaderboard for efficient NLP models. ELUE is dedicated to depicting the Pareto Frontier for various language understanding tasks, such that it can tell whether and how much a method achieves Pareto improvement. Along with the benchmark, we also release a strong baseline, **ElasticBERT**, which allows BERT to exit at any layer in both static and dynamic ways. We demonstrate the ElasticBERT, despite its simplicity, outperforms or performs on par with SOTA compressed and early exiting models. With ElasticBERT, the proposed ELUE has a strong Pareto Frontier and makes a better evaluation for efficient NLP models.

## 1 Introduction

Driven by the large-scale pre-training, today's NLP models have become much more powerful Devlin et al. (2019); Yang et al. (2019); Lan et al. (2020); Raffel et al. (2020); Sun et al. (2020); Brown et al. (2020); Qiu et al. (2020). As a consequence of this drastic increase in performance, these pre-trained language models (PLMs) are notorious for becoming more and more computationally expensive due to the increasing number of parameters. Therefore, rather than pre-training a larger model to achieve a new state-of-the-art (SOTA) accuracy, most studies are pursuing improvement on other dimensions such as the number of parameters or FLOPs Gordon et al. (2020); Sanh et al. (2019); Jiao et al. (2020); Lan et al. (2020); Shen et al. (2020). For these works, the goal has shifted from simple SOTA to "Pareto SOTA". A Pareto SOTA model means that there is no other model is currently better than it on all the dimensions of interest. For example, a model may claim to be Pareto SOTA as long as it achieves the best accuracy under the same number of parameters or FLOPs. For these efficient models with fewer parameters or FLOPs, it is unfair to get them evaluated on the accuracy-centric benchmarks such as GLUE Wang et al. (2019), and ranked among many large-scale models.

The shifted goal has outpaced the existing benchmarks, which cannot provide a comprehensive and intuitive comparison for efficient methods. In the absence of a proper benchmark, measures of efficiency in different studies cannot be standardized, and different methods cannot be fairly compared. As a result, it is difficult to say _whether_ and _how much_ a method achieves Pareto improvement. To that end, we aim to build the Pareto frontier for various tasks with standard evaluation for both performance and efficiency. Our motivation can be briefly illustrated by Figure 1.

Need for a standard evaluationAs the goal has shifted, a new benchmark is urgently needed to

Figure 1: An illustration to show our motivation, that is, building the Pareto frontier can help recognizing whether and how much a method achieves Pareto improvement.



comprehensively compare the NLP models in multiple dimensions. Currently, this multi-dimensional comparison is done in the individual papers, resulting in the following issues: **(a) Incomprehensive comparison.** The comparison is usually point-to-point, e.g. comparing model performance under the same FLOPs. The comparison in a broader range is usually missed, especially for works in conditional computation where the model performance varies with FLOPs. **(b) Unaccessible results.** Even if the comprehensive line-to-line comparison is conducted, the results are usually presented in form of figure, in which the data points are not accessible for the following work. As a result, the following work has to reproduce or estimate the results (e.g. Xin et al. (2021) estimate values from the figures of Zhou et al. (2020)). **(c) Non-standard measurements.** Different works may adopt different metrics such as physical elapsed time, FLOPs, and executed model layers, making them hard to directly compare. Even if the adopted metrics are the same, there is no guarantee that they will be calculated in the same way (e.g. the hardware infrastructure, or the software to calculate FLOPs can be very different1). **(d) Inconvenience.** Recent studies usually choose GLUE Wang et al. (2019) as the main benchmark, which, however, is not suitable for dynamic methods due to its submission limitation that is designed to avoid overfitting on test sets.

Footnote 1: We find that the FLOPs of Transformers calculated by different libraries (thop, ptflops, and torchstat) can be different. And besides, all of them missed FLOPs in some operations such as self-attention and layer normalization.

Need for a strong baselineCurrently, there are roughly two branches of efficient methods in NLP: static methods (e.g. distillation, pruning, quantization, etc.) and dynamic methods (e.g. early exiting). **(a) Static models** are obtained given an expected number of parameters or inference latency. These methods often use the first few layers (to keep the same number of parameters or FLOPs) of some pre-trained model followed by a classification head as their baseline, which, however, is too weak to serve as a baseline. **(b) Dynamic models** usually add multiple internal classifiers to the pre-trained LMs, and therefore allow flexible inference conditioned on the input. Nevertheless, the injected internal classifiers introduce a gap between pre-training and fine-tuning. Training the internal classifiers on downstream tasks often degenerates the performance of the entire model Xin et al. (2021). Thus, static models need a strong baseline, and dynamic models need a strong backbone.

ContributionsIn this work, we address the above needs by contributing the following:

* a standard benchmark for efficient NLP models. (1) ELUE supports online evaluation for model performance, FLOPs, and number of parameters. (2) ELUE is also an open-source platform that can facilitate future research. We reproduce and evaluate multiple compressed and early exiting methods on ELUE. All of the results are publicly accessible on ELUE. (3) ELUE provides an online leaderboard that uses a specific metric to measure how much a model oversteps the current Pareto frontier. ELUE leaderboard also maintains several separate tracks for models with different sizes. (4) ELUE covers six NLP datasets spanning sentiment analysis, natural language inference, similarity and paraphrase tasks. The ELUE benchmark is publicly available at [http://eluebenchmark.fastnlp.top/](http://eluebenchmark.fastnlp.top/).
* a strong baseline (backbone) for static (dynamic) models. ElasticBERT is a multi-exit Transformer Vaswani et al. (2017) pre-trained on \({\sim}160\)GB corpus. The pre-training objectives, MLM and SOP Lan et al. (2020), are applied to multiple Transformer layers instead of only the last layer. Gradient equilibrium Li et al. (2019) is adopted to alleviate the conflict of the losses at different layers. For static models, ElasticBERT is a strong baseline that can reach or even outperform distilled models. For dynamic models, ElasticBERT is a robust backbone that closes the gap between pre-training and fine-tuning. We release the pre-trained model weights of ElasticBERT2 as well as code3. Footnote 2: [https://huggingface.co/fnlp](https://huggingface.co/fnlp)

Footnote 3: [https://github.com/fastnlp/ElasticBERT](https://github.com/fastnlp/ElasticBERT)

## 2 Related Work

NLP BenchmarksEvaluating the quality of language representations on multiple downstream tasks has become a common practice in the community. These evaluations have measured and 

pushed the progress of NLP in recent years. SentEval Conneau and Kiela (2018) introduces a standard evaluation toolkit for multiple NLP tasks. Further, GLUE Wang et al. (2019b) and SuperGLUE Wang et al. (2019a) provide a set of more difficult datasets for model-agnostic evaluation. Another line of work is multi-dimensional evaluations. EfficientQA Min et al. (2020) is an open-domain question answering challenge that evaluates both accuracy and system size. The system size is measured as the number of bytes required to store a Docker image that contains the submitted system. DynabenchKiela et al. (2021), an open-source benchmark for dynamic dataset creation and model evaluation, also supports multi-dimensional evaluation. In particular, Dynabench measures model performance, throughput, memory use, fairness, and robustness. Both EfficientQA and Dynabench require the user to upload the model along with the required environment to the server, which is costly for users to upload and also for the server to evaluate. In contrast, ELUE adopts a cheaper way to evaluate performance and efficiency of the model. Recently, Long-Range Arena (LRA) Tay et al. (2021) is proposed to evaluate models under the long-context scenario. Different from ELUE, LRA mainly focuses on Xformers Lin et al. (2021). Besides, some tasks included in LRA are not NLP tasks, or even not real-world tasks, while ELUE consists of common language understanding tasks. In addition, ELUE is also inspired by other well-known benchmarks, such as SQuAD Rajpurkar et al. (2016), MultiNLI Williams et al. (2018), DecaCNLP McCann et al. (2018), CLUE Xu et al. (2020b), HotpotQA Yang et al. (2018), GEM Gehrmann et al. (2021), etc.

Efficient NLP ModelsCurrent efficient NLP models can be roughly categorized as two streams: model compression (static methods) and conditional computation (dynamic methods). Model compression is to reduce the number or precision of model parameters to achieve faster training and inference. Currently, there are several ways to achieve model compression: (1) _Knowledge Distillation_, which is to learn a compact student model that learns from the output distribution of a large-scale teacher model Sanh et al. (2019); Jiao et al. (2020) (2) _Model Pruning_, which is to remove parts of parameters that are less important Gordon et al. (2020), (3) _Weight Sharing_ across different parts of the model Lan et al. (2020) is also a common technique to significantly reduce parameters, (4) _Quantization_, which is to use low bit precision to store parameter and accelerate inference with low bit hardware operations Shen et al. (2020), and (5) _Module Replacing_, which is to replace the modules of a big model with more compact substitutes Xu et al. (2020a). In contrast, conditional computation is to selectively execute only parts of the model conditioned on a given input Bengio et al. (2013); Davis and Arel (2014). As a representative, an end-to-end halting approach, Adaptive Computation Time (ACT) Graves (2016), is developed to perform input-adaptive computation for recurrent networks. The idea of ACT is later adopted in Universal Transformer Dehghani et al. (2019). Recently, as the rising of deep models for natural language processing, early exiting is widely used to speedup inference of transformer models Liu et al. (2020); Xin et al. (2020); Schwartz et al. (2020); Zhou et al. (2020); Elbayad et al. (2020); Liao et al. (2021); Xin et al. (2021); Sun et al. (2021); Zhu (2021); Li et al. (2021).

## 3 ELUE: A Standard Benchmark for Efficient NLP Models

ELUE aims to offer a standard evaluation for various efficient NLP models, such that they can be fairly and comprehensively compared. In Section 3.1, we list the design considerations to achieve this motivation. In Section 3.2, we describe the tasks and datasets included in ELUE. In Section A.1, we illustrate how to make a submission on ELUE, and how the submission is evaluated. In Section 3.3, we discuss the design of our leaderboard.

### Design Considerations

Now we enumerate main considerations in the design of ELUE to ensure that it meets the needs mentioned early.

Multi-dimensional EvaluationThe evaluation of ELUE should be multi-dimensional for comprehensive comparison. Instead of point-to-point comparison, methods can be compared in a line-to-line style in ELUE, where the "line" is a performance-efficiency trade-off curve.

Public AccessibleAll data points in ELUE should be publicly accessible such that the following work does not need to reproduce or estimate 

results from previous work. To facilitate future research, some representative methods should be reproduced and evaluated in ELUE.

Standard EvaluationThe measurement of model efficiency should be standardized in ELUE such that this line of methods can be fairly compared. Current studies usually use number of parameters Lan et al. (2020); Jiao et al. (2020), FLOPs Jiao et al. (2020); Liu et al. (2020); Li et al. (2021), actual inference time Sanh et al. (2019); Schwartz et al. (2020), or number of executed layers Zhou et al. (2020); Sun et al. (2021) to measure model efficiency. Among these metrics, measuring actual inference time is costly for both users and the server, and highly depends on the computation infrastructure and software implementation, while number of executed layers ignores the shape of input and hidden layers, therefore is inaccurate. Thus, ELUE adopts number of parameters and FLOPs as the metrics for model efficiency.

Easy-to-UseELUE should be friendly to users, which means that the submission should be as simple as possible. Roughly speaking, there are currently two ways of submissions: (1) submitting the trained model such as SQuAD Rajpurkar et al. (2016), Dynabench Kiela et al. (2021), and (2) submitting the predicted test files such as GLUE Wang et al. (2019), SuperGLUE Wang et al. (2019), and CLUE Xu et al. (2020). The submission of ELUE lies in the latter way. Nevertheless, to evaluate number of parameters and FLOPs, the submitted test files should conform to a specific format, and besides, a Python file to define the used model is also required. For more details about submission and evaluation, see Appendix A.1.

### Task and Dataset Selection

Following GLUE Wang et al. (2019), SuperGLUE Wang et al. (2019), and CLUE Xu et al. (2020), we collect tasks that can be formatted as single sentence classification or sentence pair classification. Since ELUE mainly focuses on efficient models, the difficulty of dataset is not a primary consideration. Instead, we collect tasks and datasets that are commonly used and publicly available in the community. The statistics of the collected datasets are listed in Table 1.

Sentiment Analysis Sentiment analysis, which is to classify the polarity of a given text, is a fundamental task in NLP. We select two well-known movie review datasets, Stanford Sentiment Treebank (SST) Socher et al. (2013) and IMDb Maas et al. (2011). For SST, we use the two-way class split, i.e. SST-2. Different from GLUE, SST-2 samples in ELUE are complete sentences instead of phrases. For IMDb, we randomly select 2.5k positive samples and 2.5k negative samples from training set to construct a development set.

Natural Language InferenceNatural language inference (NLI) is a task to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither. NLI is often formulated as a sentence pair classification task Devlin et al. (2019); Sun et al. (2021). We select two NLI datasets, SNLI Bowman et al. (2015) and SciTail Khot et al. (2018). SNLI is a crowd-sourced collection of sentence pairs with balanced labels: _entailment_, _contradiction_, and _neutral_. We use the spell-checked version of the test and development sets4. The hard samples, which do not have golden labels due to the disagreement of annotators, are removed from the dataset and left for model diagnostic. SciTail is a two-way (_entail_ or _neutral_) entailment classification dataset, which is derived from multiple-choice science exams and web sentences.

Footnote 4: [https://nlp.stanford.edu/projects/snli/](https://nlp.stanford.edu/projects/snli/)

Similarity and ParaphraseFor similarity and paraphrase tasks, we also select two datasets, Microsoft Research Paraphrase Corpus (MRPC) Dolan and Brockett (2005), and Semantic Textual Similarity Benchmark (STS-B) Cer et al. (2017), both of which are also included in GLUE. MRPC is a collection of automatically extracted sentence pairs, each manually-labeled with a judgment to indicate whether the pair constitutes a paraphrase. STS-B is a corpus of sentence pairs, each of which is labeled with a score from 0 to 5 to represent the degree to which two sentences are semantically equivalent.

\begin{table}
\begin{tabular}{l l c r r} \hline \hline
**Tasks** & **Datasets** & **|Train**i & **|Devl** & **Testl** \\ \hline Sentiment & SST-2 & 8,544 & 1,101 & 2,208 \\ Analysis & IMDb & 20,000 & 5,000 & 25,000 \\ \hline Natural Language & SNLI & 549,367 & 9,842 & 9,824 \\ Inference & SciTail & 23,596 & 1,304 & 2,126 \\ \hline Similarity and & MRPC & 3,668 & 408 & 1,725 \\ Paraphrase & STS-B & 5,749 & 1,500 & 1,379 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of datasets in ELUE.

 



### Leaderboard

Following prior work Yang et al. (2018); Wang et al. (2019, 2020), we also integrate a leaderboard in ELUE. For dynamic models that have multiple performance-FLOPs coordinates on each dataset, we need to sum up these coordinates as a score. A critical problem is to measure how good a coordinate is. In other words, to measure a coordinate \((p,f)\), where \(p\) is performance and \(f\) is FLOPs, we need a baseline performance under the same FLOPs. We choose ElasticBERT as the baseline curve. We evaluate different layers of ElasticBERT, and obtained 12 coordinates \((p_{i}^{EB},f_{i}^{EB})_{i=1}^{12}\), which are then used to interpolate to get a performance-FLOPs function \(p^{EB}(f)\). With the baseline curve at hand, we can score a submission curve as

\[\text{ELUEScore}=\frac{1}{n}\sum_{i=1}^{n}[p_{i}-p^{EB}(f_{i})]. \tag{1}\]

Note that the coordinates of ElasticBERT are separately interpolated on different datasets. The final ELUE score is an unweighted average of the scores on all the 6 datasets. Figure 2 gives an illustration of how ELUE score is computed. The ELUE score reflects the extent to which the submission oversteps the ElasticBERT.

In addition, following EfficientQA Min et al. (2020), ELUE leaderboard also maintains four additional separate tracks, corresponding to models below 40M, 55M, 70M, 110M parameters. Models in these tracks are ranked by the average performance on all the datasets.

## 4 ElasticBERT: A Strong Baseline for Efficient Inference

Despite the encouraging results achieved by existing efficient models, we argue that a strong baseline (backbone) is needed for both static methods and dynamic methods. Static methods often choose the first few layers of some pre-trained models as their baseline (e.g. Sun et al. (2019); Jiao et al. (2020)), which can be weak. Dynamic methods that enable early exiting by training multiple internal classifiers usually introduce a gap between pre-training and fine-tuning, and therefore hurt the performance of the entire model Xin et al. (2021). Thus, as illustrated in Figure 3, we present the ElasticBERT that bridges the gap between static and dynamic methods, and therefore can serve as a strong baseline for static methods and also a strong backbone for dynamic methods.

ElasticBERT is a multi-exit pre-trained language model with the following training objective:

\[\mathcal{L}=\sum_{l=1}^{L}(\mathcal{L}_{l}^{\text{MLM}}+\mathcal{L}_{l}^{ \text{SOP}}), \tag{2}\]

where \(L\) is the total number of layers, \(\mathcal{L}^{\text{MLM}}\) is the n-gram masked language modeling loss, \(\mathcal{L}^{\text{SOP}}\) is the sentence order prediction loss Lan et al. (2020). The two losses are applied to each layer of the model, such that the number of layers can be flexibly scaled on downstream tasks, and therefore it is named "ElasticBERT".

Bridge the Gap Between Static and Dynamic MethodsAs a baseline for static methods, the depth of ElasticBERT can be flexibly reduced on demand. Compared with the first \(l\) layer of BERT Devlin et al. (2019), the \(l\)-layered ElasticBERT is a complete model Turc et al. (2019); Li et al. (2021) and can achieve better performance. It is worth noticing that ElasticBERT can be regarded as a special instance of LayerDrop Fan et al. (2020) where the dropped layers are constrained to the top consecutive layers. As a backbone for dynamic methods, training classifiers injected in intermediate layers would be consistent with pre-training. Therefore, ElasticBERT can not only be used as a static complete model, but also be used as a backbone model of dynamic early exiting.

Gradient EquilibriumPre-training with the simply summed loss in Eq. (2) could lead to a _gradient imbalance_ issue Li et al. (2019). In particular, due to the overlap of subnetworks, the variance of the gradient may grow overly large, leading to unstable training. To address this issue, we follow Li et al. (2019) and adopt the gradient equilibrium

Figure 2: An illustration to show how ELUE score is computed.



(GE) strategy5 in the pre-training of ElasticBERT.

Footnote 4: The reader is referred to the original paper for more details. In brief, the gradients of \({\cal L}_{j}\) w.r.t. the parameters of the \(i\)-th layer (\(i<j\)) would be properly rescaled.

Grouped TrainingIn our preliminary experiments, we found that summing up losses at all layers could slow down pre-training and increase memory footprints. To alleviate this, we divide \(L\) exits into \(G\) groups. During training, we optimize the losses of the exits within each group by cycling alternately between different batches:

\[{\cal L}=\sum_{l\in{\mathcal{G}_{i}}}({\cal L}^{\text{MLM}}_{l}+{\cal L}^{ \text{SOP}}_{l}). \tag{3}\]

In Section B.3 we explore the performance of different grouping methods. As a result, we group the 12 exits of ElasticBERTBASE into \({\mathcal{G}}_{1}\)={1, 3, 5, 7, 9, 11, 12} and \({\mathcal{G}}_{2}\)={2, 4, 6, 8, 10, 12}, and group the 24 exits of ElasticBERTLMEG into \({\mathcal{G}}_{1}\)={1, 4, 7, ..., 22, 24}, \({\mathcal{G}}_{2}\)={2, 5, 8, ..., 23, 24}, and \({\mathcal{G}}_{3}\)={3, 6, 9, ..., 21, 24}. Our experiments demonstrate that grouped training can significantly speedup the process of pre-training without a loss in performance.

## 5 Experiments

### Experimental Setup

Pre-training SetupFollowing BERT Devlin et al. (2019), we train ElasticBERT in two different configurations: ElasticBERTBASE and ElasticBERTLMEG, which have the same model sizes with BERTBASE and BERTLARGE, respectively. The detailed description can be found in Appendix B.1.

Downstream EvaluationWe evaluate ElasticBERT on the ELUE benchmark, as a static model and as a dynamic model. As a static model, we evaluate different layers of ElasticBERT, denoted as ElasticBERT-\(n\)L. As a dynamic model, we inject and train internal classifiers in ElasticBERTBASE and adopt two strategies, entropy Xin et al. (2020) and patience Zhou et al. (2020), to enable early exiting, denoted as ElasticBERTentropy and ElasticBERTpatience. To compare with previous work, we also evaluate ElasticBERT on the GLUE benchmark Wang et al. (2019). The comparison results is shown in Appendix B.2. For static usage, we fine-tune ElasticBERT and our baseline models for 10 epochs with early stopping using AdamW optimizer Loshchilov and Hutter (2019) with learning rates of {1e-5, 2e-5, 3e-5} and batch size of 32, and warm up the learning rate for the first 6 percent of total steps. In dynamic usage, for the models using two-stage training methods, we train for 3 epochs for each stage, and we train for 5 epochs for other models. Other optimization configurations are the same as those in static scenario.

Figure 4: Comparison of the average performance on ELUE test sets between ElasticBERT and baselines.

Figure 3: ElasticBERT is pre-trained with multiple pre-training heads attached at the intermediate layers. For static usage (left), it can be pruned on demand while outperforming previous pre-trained models with the same size. For dynamic usage (right), it can serve as the backbone for early exiting methods, achieving better performance-efficiency trade-off than early exiting models with other backbones.

 

BaselinesWe compare ElasticBERT with three types of baselines: **(1)** Directly fine-tuning pre-trained models and their first \(n\) layers. We choose BERT Devlin et al. (2019), ALBERT Lan et al. (2020), RoBERTa Liu et al. (2019) and LayerDrop Fan et al. (2020) as our baselines. For the use of the first \(n\) layers, we simply add a linear classifier on top of the truncated model. **(2)** Compressed models. We choose two distilled models, DistilBERT Sanh et al. (2019) and TinyBERT Jiao et al. (2020), one pruned model, HeadPrune Michel et al. (2019), and one model obtained by using _module replacing_, BERT-of-Theseus Xu et al. (2020) as our baseline models. **(3)** Dynamic early exiting models. To verify the effectiveness of ElasticBERT as a strong backbone of dynamic early exiting methods, we also compare ElasticBERTentropy and ElasticBERTpatience which have the same early exiting strategy as DeeBERT Xin et al. (2020) and PABEE Zhou et al. (2020) with four representative early exiting models: DeeBERT Xin et al. (2020), FastBERT Liu et al. (2020), PABEE Zhou et al. (2020), and CascadeBERT Li et al. (2021).

### Evaluating ElasticBERT on ELUE

ElasticBERT and our baselines are evaluated on ELUE tasks. For the BASE version of ElasticBERT, BERT, ALBERT, RoBERTa and LayerDrop, we evaluate the first 3/4/6/12 layers. For the LARGE version of the models, we evaluate the first 6/8/12/24 layers. For dynamic methods, we fine-tune ElasticBERT along with the injected internal classifiers using the gradient equilibrium (GE) strategy Li et al. (2019), and adopt two different early exiting strategies: entropy-based strategy Xin et al. (2020) and patience-based strategy Zhou et al. (2020).

Results of Static ModelsThe performance of ElasticBERT and our baseline models on ELUE task test sets is shown in Table 2, where we find that ElasticBERTBASE and ElasticBERTLARGE outperform BERT and ALBERT with the same number of layers, but are slightly weaker than RoBERTBASE and RoBERTALARGGE. Besides, we find that the superiority of ElasticBERT over its baselines can be significant with fewer layers (See Figure 4 for the

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline
**Model** & **\#Params** & **\#FLOPs** & **SST-2** & **IMDb** & **MRPC** & **STS-B** & **SNLI** & **SciTail** & **Average** \\ \hline  & \multicolumn{7}{c}{_BASE Models_} & \\ \hline BERTLASE & 109M & 13399M & 85.1 & 93.0 & 83.1 & 84.2 & 90.4 & 93.2 & 88.2 \\ ALBERTASE & 12M & 13927M & 86.6 & 92.9 & 87.8 & 88.3 & 90.1 & 93.4 & 89.9 \\ RoBERTBASE & 12SEM & 13103M & 88.3 & **94.9** & 88.0 & **89.6** & 91.3 & 92.8 & **90.8** \\ LayerDropBASE & 12SEM & 13103M & 88.5 & 94.2 & **88.2** & 87.1 & 90.7 & 92.8 & 90.3 \\
**ElasicBERTBASE** & 109M & 13399M & **88.6** & 93.9 & 87.9 & 87.6 & **91.3** & **93.8** & 90.5 \\ \hline BERTBASE-6L & 67M & 6700M & 83.3 & 91.0 & 82.6 & 82.5 & 88.9 & 90.7 & 86.5 \\ ALBERTASE-6L & 12M & 6972M & 84.7 & 92.0 & 85.3 & 83.5 & 89.3 & 92.3 & 87.9 \\ RoBERTBASE-6L & 82M & 6552M & 86.8 & 92.6 & 86.7 & 84.5 & **90.2** & 91.3 & 88.7 \\ LayerDropBASE-6L & 82M & 6552M & 86.3 & **92.9** & 86.3 & 86.1 & 89.5 & 90.3 & 88.6 \\ HeadPrune-BERTBASE & 86M & 9249M & 84.8 & 84.7 & 77.8 & 74.8 & 87.8 & 88.3 & 83.0 \\ DistilBERT & 67M & 6700M & 84.8 & 92.0 & 83.8 & 81.7 & 89.2 & 89.7 & 86.9 \\ TinyBERT-6L & 67M & 6700M & 85.3 & 89.0 & 86.2 & 85.7 & 89.3 & 90.0 & 87.6 \\ BERT-of-Theseus & 67M & 6700M & 84.4 & 90.7 & 82.4 & 85.0 & 89.4 & 92.1 & 87.3 \\
**ElasticBERTBASE**–6L & 67M & 6700M & **87.0** & 92.7 & **87.3** & **86.9** & 90.1 & **92.5** & **89.4** \\ \hline  & \multicolumn{7}{c}{_LARGE Models_} & \\ \hline BERTLarge & 335M & 47214M & 87.9 & 94.0 & 85.9 & 86.7 & 90.8 & 93.9 & 89.9 \\ ALBERTLARGE & 18M & 48876M & 87.7 & 93.8 & 88.1 & 89.3 & 90.2 & 93.6 & 90.5 \\ RoBERTLAERG & 355M & 46042M & **90.5** & **95.7** & **89.9** & 90.5 & **91.6** & **95.8** & **92.3** \\ LayerDPLPAGE & 355M & 46042M & 90.4 & 95.3 & 89.5 & **91.0** & 91.4 & 95.2 & 92.1 \\
**ElasticBERTLARGE**– & 335M & 47214M & 89.8 & 95.0 & 89.8 & 90.9 & 91.4 & 95.7 & 92.1 \\ \hline BERTLMREG-6L & 108M & 11922M & 80.4 & 89.6 & 74.3 & 70.5 & 87.4 & 84.4 & 81.1 \\ ALBERTLARGE-6L & 18M & 12397M & 84.5 & 92.0 & 84.7 & 85.1 & 89.4 & 90.8 & 87.8 \\ RoBERTLABGRG-6L & 129M & 11664M & 83.5 & 91.7 & 77.9 & 72.7 & 88.6 & 84.7 & 83.2 \\ LayerDropLarge-6L & 129M & 11664M & 85.4 & 92.5 & 77.3 & 75.9 & 88.8 & 84.1 & 84.0 \\
**ElasticBERTLARGE**– & 6L & 108M & 11922M & **86.8** & **92.9** & **86.2** & **86.3** & **89.8** & **92.4** & **89.1** \\ \hline \hline \end{tabular}
\end{table}
Table 2: ElasticBERT and static baseline performance on ELUE task test sets. We report the mean of Accuracy and F1 for MRPC, Pearson and Spearman correlation for STS-B and Accuracy for other tasks. The reported FLOPs is the average over all the datasets.

 

results of 3/4 (6/8) layers of the BASE (LARGE) models.

Results of Dynamic ModelsWe compare ElasticBERTentropy and ElasticBERTpatience with four dynamic models: DeeBERT Xin et al. (2020), FastBERT Liu et al. (2020b), PABEE Zhou et al. (2020a), and CascadeBERT Li et al. (2021). The performance-FLOPs trade-off of the dynamic models on ELUE task test sets are shown in Figure 5, which demonstrates that ElasticBERT can achieve better performance-FLOPs trade-off.

Evaluating ELUE ScoresAccording to Eq. (1), we also evaluate the ELUE scores of these baselines. As shown in Table 3, the ELUE score of ElasticBERTBASE is natural to be zero on all tasks. Among the other baselines, we find that ElasticBERTpatience achieves the best ELUE score, while HeadPrune achieves the worst ELUE score. In addition, we find that dynamic models perform better than static models on average.

Figure 5: Performance-FLOPs trade-offs on ELUE task test sets. Because STS-B is a regression task, for which the entropy-based methods are not applicable, we only evaluate patience-based methods, i.e., PABEE and ElasticBERTpatience.

 

Conclusion and Future Work

In this work, we present ELUE, which is a public benchmark and platform for efficient models, and ElasticBERT, which is a strong baseline (backbone) for efficient static (dynamic) models. Both of the two main contributions are aimed to build the Pareto frontier for NLU tasks, such that the position of existing work can be clearly recognized, and future work can be easily and fairly measured.

Our future work is mainly in four aspects: (1) Including more baselines in ELUE, (2) Supporting the evaluation for more frameworks such as TensorFlow Abadi et al. (2016), (3) Supporting diagnostics for submissions, (4) Supporting the evaluation of more different types of tasks.

## Acknowledgment

This work was supported by the National Key Research and Development Program of China (No.2020AAA0106702) and National Natural Science Foundation of China (No.62022027).

## Ethical Considerations

The proposed ELUE benchmark aims to standardize efficiency measurement of NLP models. The collected datasets are widely used in previous work and, to our knowledge, do not have any attached privacy or ethical issues. Our proposed ElasticBERT is a pre-trained model to reduce computation cost and carbon emission. The pre-training data is public resources adopted in previous work, and therefore would not introduce new ethical concerns.

## References

* Abadi et al. [2015] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In _12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016_, pages 265-283. USENIX Association.
* Bengio et al. [2013] Yoshua Bengio, Nicholas Leonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. _CoRR_, abs/1308.3432.
* Bowman et al. [2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 632-642. The Association for Computational Linguistics.
* Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* multilingual and cross-lingual focused evaluation. _CoRR_, abs/1708.00055.
* Conneau and Kiela [2018] Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018_. European Language Resources Association (ELRA).
* Davis and Arel [2014] Andrew S. Davis and Itamar Arel. 2014. Low-rank approximations for conditional feedforward computation in deep neural networks. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings_.
* Dehghani et al. [2019] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreti, and Lukasz Kaiser. 2019. Universal transformers. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Dolan and Brockett [2005] William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In _Proceedings of the Third International Workshop 

on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005_. Asian Federation of Natural Language Processing.
* Elbayad et al. (2020) Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
* Fan et al. (2020) Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
* Gehrmann et al. (2021) Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanachi, Aremu Anuuluwapo, Antoine Bosselut, Khyathi Raghavich Chandu, Miruna-Adriana Cinciu, Dipnjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emeze, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madam, Mounica Maldela, Khyait Mahajnan, Saad Mahmood, Rohdiastura Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubango Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, Joao Sedoc, Thibault Sellam, Samaria Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. _CoRR_, abs/2102.01672.
* Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus.
* Gordon et al. (2020) Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: studying the effects of weight pruning on transfer learning. In _Proceedings of the 5th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2020, Online, July 9, 2020_, pages 143-155. Association for Computational Linguistics.
* Graves (2016) Alex Graves. 2016. Adaptive computation time for recurrent neural networks. _CoRR_, abs/1603.08983.
* Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling BERT for natural language understanding. In _Proceedings of EMNLP 2020_, pages 4163-4174. Association for Computational Linguistics.
* Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_, pages 5189-5197. AAAI Press.
* Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshai, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Wasseem, Ponuts Steentrop, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 4110-4124. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020._OpenReview.net.
* November 2, 2019_, pages 1891-1900. IEEE.
* Li et al. (2021a) Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021a. Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade. In _Findings of EMNLP_.
* Li et al. (2021b) Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2021b. Accelerating BERT inference for sequence labeling via early-exit. In _Proceedings of ACL/IJCNLP 2021_, pages 189-199. Association for Computational Linguistics.
* Liao et al. (2021) Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. 2021. A global past-future early exit method for accelerating inference of pre-trained language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 2013-2023. Association for Computational Linguistics.

 

Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2021. A survey of transformers. _CoRR_, abs/2106.04554.
* Liu et al. (2020a) Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020a. Fastbert: a self-distilling BERT with adaptive inference time. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 6035-6044. Association for Computational Linguistics.
* Liu et al. (2020b) Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020b. Fastbert: a self-distilling BERT with adaptive inference time. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 6035-6044. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA_, pages 142-150. The Association for Computer Linguistics.
* McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. _CoRR_, abs/1806.08730.
* Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 14014-14024.
* Min et al. (2020) Sewon Min, Jordan L. Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick S. H. Lewis, Yuxiang Wu, Heinrich Kuttler, Linguing Liu, Pasquale Minervini, Pontus Steentorp, Sebastian Riechel, Sobe Yang, Minjoon Seo, Gaudier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Onderje, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Sejer Schlichtkrull, Somal Gupta, Yashar Mehdad, and Wen-tau Yih. 2020. Neurips 2020 efficientqa competition: Systems, analyses and lessons learned. In _NeurIPS 2020 Competition and Demonstration Track, 6-12 December 2020, Virtual Event / Vancouver, BC, Canada_, volume 133 of _Proceedings of Machine Learning Research_, pages 86-111. PMLR.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoti Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 8024-8035.
* Qiu et al. (2020) Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. _SCIENCE CHINA Technological Sciences_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Shanan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 2383-2392. The Association for Computational Linguistics.
* Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. _CoRR_, abs/1910.01108.
* Schwartz et al. (2020) Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 6640-6651. Association for Computational Linguistics.
* Shen et al. (2020) Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-BERT: hessian based ultra low precision quantization of BERT. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, 

AAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 8815-8821. AAAI Press.
* Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. _CoRR_, abs/1909.08053.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hytat Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 1631-1642. ACL.
* Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 4322-4331. Association for Computational Linguistics.
* Sun et al. (2021a) Tianxiang Sun, Xiangyang Liu, Xipeng Qiu, and Xuanjing Huang. 2021a. Paradigm shift in natural language processing. _CoRR_, abs/2109.12575.
* Sun et al. (2020) Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge embedding. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_, pages 3660-3670. International Committee on Computational Linguistics.
* Sun et al. (2021b) Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021b. Early exiting with ensemble internal classifiers. _CoRR_, abs/2105.13792.
* Tay et al. (2021) Yi Tay, Mostafa Dehghani, Samira Abnar, Yakiang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena : A benchmark for efficient transformers. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. _CoRR_, abs/1908.08962.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Wang et al. (2019a) Alex Wang, Yada Prucksachatkun, Nikita Nangia, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 3261-3275.
* Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Demos_, pages 38-45. Association for Computational Linguistics.
* Xin et al. (2020) Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deepbert: Dynamic early exiting for accelerating BERT inference. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 2246-2251. Association for Computational Linguistics.
* Xin et al. (2021) Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. Bexirt: Early exiting for BERT with better fine-tuning and extension to regression. In _Proceedings of EACL 2021_, pages 91-104. Association for Computational Linguistics.
* Xu et al. (2020a)Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020a. Bert-of-theseus: Compressing BERT by progressive module replacing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 7859-7869. Association for Computational Linguistics 

* Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yingin Cui, Junyi Li, Jin Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020b. CLUE: A chinese language understanding evaluation benchmark. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_, pages 4762-4772. International Committee on Computational Linguistics.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLnet: Generalized autoregressive pretraining for language understanding. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 5754-5764.
* November 4, 2018_, pages 2369-2380. Association for Computational Linguistics.
* Zhou et al. (2020a) Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley, Ke Xu, and Furu Wei. 2020a. BERT loses patience: Fast and robust inference with early exit. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Zhou et al. (2020b) Wai Zhu. 2021. Leebert: Learned early exit for BERT with cross-level optimization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 2968-2980. Association for Computational Linguistics.
* Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_, pages 19-27. IEEE Computer Society.

## Appendix A Details of Evaluation

### Submission and Evaluation

ELUE supports two kinds of submissions: submitting test files, or submitting from a paper.

Submit test filesUsers are required to submit two kinds of files: (1) predicted test files, and (2) a model definition file in Python. The predicted test files can be multiple, each indicates the prediction under a certain efficiency. The submitted test files should be in the following format:

Different from traditional predicted test files as in GLUE, an additional column "modules" is required to indicate the activated modules to predict each sample. The numbers before each module represent the input shape of that module, e.g. the "(10)" before "emb" indicates that the input of "emb" is a sequence of length 10. Note that this format is also compatible with token-level early exiting methods Li et al. (2021b), where the sequence length is progressively reduced as the processing of layers.

Along with the test files, a Python file to define the model is also required. Figure 6 is an example Python file using PyTorch Paszke et al. (2019) and Transformers Wolf et al. (2020).

With the submitted Python file, ELUE is able to evaluate the average FLOPs on a dataset, and the number of parameters of the model.

In cases that the evaluation is not applicable, e.g. the programming language, or dependencies of the submitted Python file is not supported in ELUE, the user is allowed to evaluate FLOPs and number of parameters by themselves and upload their results along with the predictions to the ELUE website.

Submit from a paperInspired by Paper with Code6, we also expect that ELUE can serve as an open-source platform that can facilitate future research. Therefore, there is a track for the authors of 

published papers to share their experimental results on ELUE datasets.

Performance MetricsSince the classes in MRPC are imbalanced, we report the unweighted average of accuracy and F1 score. For STS-B, we evaluate and report the Pearson and Spearman correlation coefficients. For other datasets, we simply adopt accuracy as the metric.

## Appendix B Experimental Details and Additional Results

### Details of Training ElasticBERT

The parameters of ElasticBERT are initialized with BERT, and therefore it has the same vocabulary and tokenizer as BERT. ElasticBERT is pre-trained on \(\sim\)160GB uncompressed English text corpora, which is comprised of English Wikipedia (12GB), BookCorpus (4GB) Zhu et al. (2015), OpenWebText (38GB) Gokaslan and Cohen (2019), and part of the C4 corpus (110GB) Raffel et al. (2020). We use Adam optimizer Kingma and Ba (2015) to pre-train ElasticBERTBASE and ElasticBERTLARGE and other hyperparameters are listed in Table 4. Our implementation is based on Huggingface's Transformers Wolf et al. (2020) and the MegatronLM toolkit Shoeybi et al. (2019). ElasticBERT is trained on 64 32G NVIDIA Tesla V100 GPUs.

### Evaluating ElasticBERT on GLUE

To verify the effectiveness and the elasticity of ElasticBERT, we also evaluate ElasticBERT and our static baselines on the GLUE benchmark. We evaluate the first 6/12 layers of the BASE models, and the first 6/24 layers of the LARGE models.

Experimental results of ElasticBERT and our baseline models on GLUE are presented in Table 6, from which we find that ElasticBERT outperforms BERT and ALBERT with the same number of layers, but is weaker than RoBERTa in the 12/24 layers configuration. Compared with ElasticBERT that is trained for 125K steps with batch size of 4K, RoBERTa is trained for 500K steps with batch size of 8K, which makes its number of training samples 8 times larger than that of ElasticBERT. When using fewer layers (6 layers of BASE and LARGE models), ElasticBERT achieves the best performance among the static baselines, confirming its great elasticity.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Hyperparameter** & **ElasticBERTBASE** & **ElasticBERTLARGE** \\ \hline Adam \(\beta_{1}\) & 0.9 & 0.9 \\ Adam \(\beta_{2}\) & 0.999 & 0.999 \\ Peak Learning Rate & 2e-4 & 2e-4 \\ Warm Up Type & Linear & Linear \\ Warm Up Rate & 0.04 & 0.04 \\ Weight Decay & 0.01 & 0.01 \\ Batch Size & 4096 & 4096 \\ Training Steps & 125k & 125k \\ Number of Layers & 12 & 24 \\ Hidden Size & 768 & 1024 \\ Attention Heads & 12 & 16 \\ FFN Intermediate Size & 3072 & 4096 \\ \hline \hline  & & \\ \end{tabular}
\end{table}
Table 4: Hyperparameters for ElasticBERT pretraining

Figure 6: An example Python file for submission.

 



### Ablation Study

About the Training StrategyElasticBERT adopts the gradient equilibrium (GE) to alleviate the conflict between the losses at different exits. Here, we compare GE with two other existing training strategies, two-stage training Xin et al. (2020) and weighted training Zhou et al. (2020a). Two-stage training is that, first training the top classifier along with the backbone model, and then freeze the parameters of the backbone model and train the injected internal classifiers. By this, two-stage training maintains the performance of the top classifier. Weighted training is to weight the loss of each exit according to the corresponding layer, which is

\[{\mathcal{L}}=\frac{\sum_{l=1}^{L}l\cdot{\mathcal{L}}_{l}}{\sum_{l=1}^{L}l}. \tag{4}\]

Experimental results of ElasticBERT with the three training strategies are shown in Figure 7. It can be observed that training with GE strategy performs the best on both SST-2 and MRPC.

About the Grouped ExitsAs shown in Eq. (3), we divide the \(L\) exits into different groups to speedup the pre-training. Therefore, how to group these exits needs to be explored. Here we evaluate four different grouping methods, as described in Table 5. To keep the overall performance of the entire model, the exit classifier on the top of the model is included in each group. According to

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline
**Model** & **\#Params** & **\#FLOPs** & **CoLA** & **MNLI-m/mm** & **MRPC** & **QNLI** & **QQP** & **RTE** & **SST-2** & **STS-B** & **Average** \\ \hline  & \multicolumn{9}{c}{_BASE Models_} \\ \hline BERTBASE & 109M & 661STM & 56.5 & 84.6/84.9 & 87.6 & 91.2 & 89.6 & 69.0 & 92.9 & 89.4 & 82.9 \\ ALBERTBASE & 12M & 6861M & 56.8 & 84.9/85.6 & 90.5 & 91.4 & 89.2 & **78.3** & 92.8 & 90.7 & 84.5 \\ RoBERTBASE & 125M & 6727M & 63.6 & **87.5/87.2** & 90.8 & **92.7** & **90.3** & 77.5 & **94.8** & **90.9** & **86.1** \\ LayerDropBase & 125M & 6727M & **G4.5** & 86.4/86.5 & **91.6** & 92.2 & 89.9 & 71.1 & 93.7 & 88.6 & 84.9 \\
**ElasticBERBASE** & 109M & 661SEM & 64.3 & 85.8/59.9 & 91.0 & 92.0 & 90.2 & 76.5 & 94.3 & 90.7 & 85.6 \\ \hline BERTBASE-GL & 67M & 3308M & 44.6 & 81.4/81.4 & 84.9 & 87.4 & 88.7 & 65.7 & 90.9 & 88.1 & 79.2 \\ ALBERTBASE-GL & 12M & 3435M & 52.4 & 82.6/82.2 & 89.0 & 89.8 & 88.7 & 70.4 & 90.8 & 89.6 & 81.7 \\ RoBERTBASE-GL & 82M & 3364M & 44.4 & 84.2/**84.6** & 87.9 & 90.5 & **89.8** & 60.6 & 92.1 & 89.0 & 80.3 \\ LayerDropAge-GL & 82M & 3364M & 53.7 & 83.8/83.8 & 87.6 & 89.8 & 89.4 & 64.3 & 91.3 & 88.1 & 81.3 \\ HeadPrune-BERTBASE & 87M & 4744M & 48.7 & 71.0/79.7 & 80.2 & 86.1 & 84.7 & 62.5 & 89.4 & 85.2 & 76.4 \\ DistilBERT & 67M & 3308M & **S5.6** & 82.1/82.0 & 86.5 & 89.2 & 88.8 & 63.9 & 91.3 & 86.7 & 80.7 \\ TinyBERT-GL & 67M & 3308M & 46.3 & 83.6/83.8 & 88.7 & 90.6 & 89.1 & 73.6 & 92.0 & 89.4 & 81.9 \\ BERT-of-Theseus & 67M & 3308M & 45.1 & 81.4/81.9 & 88.1 & 88.1 & 88.9 & 70.1 & 91.4 & 88.8 & 80.4 \\
**ElasticBERBASE-GL** & 67M & 3308M & 53.7 & **84.3**/84.2 & **89.7** & **90.8** & 89.7 & **74.0** & **92.7** & **90.2** & **83.3** \\ \hline  & \multicolumn{9}{c}{_Test Set Results_} \\ \hline TinyBERT-GL & 67M & 3308M & 42.5 & 83.2/82.4 & 86.2 & 89.6 & 79.6 & **73.0** & 91.8 & 85.7 & 79.3 \\
**ElasticBERBASE-GL** & 67M & 3308M & **49.1** & **83.7/83.4** & **87.3** & **90.4** & **79.7** & 68.7 & **92.9** & **86.9** & **80.3** \\ \hline  & \multicolumn{9}{c}{_LABGR Models_} \\ \hline BERTLMAGE & 335M & 23446M & 61.6 & 86.2/86.8 & 90.1 & 92.2 & 90.1 & 72.9 & 93.5 & 90.4 & 84.8 \\ ALBERTLMAGE & 18M & 24296M & 60.1 & 86.6/86.1 & 90.4 & 91.6 & 89.6 & 83.0 & 95.2 & 91.4 & 85.9 \\ RoBERTLMAGE & 355M & 23840M & 66.4 & 89/89.6 & 91.6 & **94.2** & 90.7 & 86.6 & 95.4 & 92.3 & **88.4** \\ LayerDropLPARGE & 355M & 23840M & **66.6** & **89.7/89.6** & 91.2 & 93.9 & 88.5 & **86.6** & **95.5** & **92.6** & 88.2 \\
**ElasticBERTLANGE** & 335M & 23446M & 66.3 & 88/88.5 & **92.0** & 93.6 & **90.9** & 83.1 & 95.3 & 91.7 & 87.7 \\ \hline BERTLMAGE-GL & 108M & 5863M & 20.7 & 76.5/76.5 & 76.4 & 84.3 & 87.3 & 58.5 & 89.7 & 77.3 & 78.5 \\ ALBERTLMage-GL & 18M & 6083M & 51.7 & 82.2/82.9 & 86.5 & 89.4 & 88.6 & 66.4 & **92.2** & 89.4 & 81.0 \\ RoBERTLMAGE-GL & 129M & 5962M & 43.3 & 80.4/80.9 & 80.0 & 86.1 & 88.9 & 54.9 & 90.1 & 80.5 & 76.1 \\ LayerDropLog

the experimental results in Table 5, we choose the odd/even grouping method for ElasticBERTBASE. Similarly, our experiments demonstrate that grouping 24 exits into \({\mathcal{G}_{1}}\)={1, 4, 7, ..., 22, 24}, \({\mathcal{G}_{2}}\)={2, 5, 8, ..., 23, 24}, and \({\mathcal{G}_{3}}\)={3, 6, 9, ..., 21, 24} works well for ElasticBERTLARGE.

Effect of Gradient EquilibriumTo verify that GE can enhance performance, we pre-train ElasticBERTBASE with and without GE using Wikipedia and BookCorpus data. As shown in Table 7, ElasticBERT with GE outperforms that without GE in two different configurations.

Effect of Grouped TrainingIf we divide \(L\) exits into \(G\) groups, the number of samples used for training the remaining exits is \(1/G\) of the last exit. To verify that this does not degrade the performance of the internal exits, we compare the performances of pre-training ElasticBERTBASE with and without Grouped Training. As shown in Table 7, we can observe that ElasticBERT with Grouped Training and GE does not suffer much performance loss compared with that training with only GE. In addition, as shown in the Table 8, using Grouped Training reduces \(\sim\)43% training time compared with that without Grouped Training.

### Overall Comparison

We compare ElastiBERT with other large scale pretrained models in Figure 8, from which we find that ElasticBERT is more robust to depth reduction. As the number of layers decreases, ElasticBERT offers greater advantages over other pre-trained models.

## Appendix C ELUE Website

The ELUE website is built using Vue and Spring Boot. We use MySQL for data storage and our private servers to run the scoring script for each submission.

Figure 8: Comparison of average performance on ELUE test sets between the ElasticBERT and other pre-trained models.