SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiangyang Liu $^{1,2*}$ , Tianxiang Sun $^{1,2*}$ , Junliang He $^{1,2}$ , Jiawen Wu $^{1,2}$ , Lingling Wu $^{1,2}$ , Xinyu Zhang $^{3}$ , Hao Jiang $^{3}$ , Zhao Cao $^{3}$ , Xuanjing Huang $^{1,2}$ , Xipeng Qiu $^{1,2\dagger}$ $^{1}$ School of Computer Science, Fudan University
$^{2}$ Shanghai Key Laboratory of Intelligent Information Processing, Fudan University  $^{3}$ Huawei Poisson Lab
{xiangyangliu20,txsun19,xjhuang,xpqiu}@fudan.edu.cn
{zhangxinyu35,jianghao66,caozhao1}@huawei.com
Supersized pre-trained language models have pushed the accuracy of various natural language processing (NLP) tasks to a new state-of-the-art (SOTA). Rather than pursuing the reachless SOTA accuracy, more and more researchers start paying attention to model efficiency and usability. Different from accuracy, the metric for efficiency varies across different studies, making them hard to be fairly compared. To that end, this work presents ELUE (Efficient Language Understanding Evaluation), a standard evaluation, and a public leaderboard for efficient NLP models. ELUE is dedicated to depicting the Pareto Frontier for various language understanding tasks, such that it can tell whether and how much a method achieves Pareto improvement. Along with the benchmark, we also release a strong baseline, ElasticBERT, which allows BERT to exit at any layer in both static and dynamic ways. We demonstrate the ElasticBERT, despite its simplicity, outperforms or performs on par with SOTA compressed and early exiting models. With ElasticBERT, the proposed ELUE has a strong Pareto Frontier and makes a better evaluation for efficient NLP models.
Driven by the large-scale pre-training, today's NLP models have become much more powerful (Devlin et al., 2019; Yang et al., 2019; Lan et al., 2020; Raffel et al., 2020; Sun et al., 2020; Brown et al., 2020; Qiu et al., 2020). As a consequence of this drastic increase in performance, these pre-trained language models (PLMs) are notorious for becoming more and more computationally expensive due to the increasing number of parameters. Therefore, rather than pre-training a larger model to achieve a new state-of-the-art (SOTA) accuracy, most studies are pursuing improvement on other dimensions such
Figure 1: An illustration to show our motivation, that is, building the Pareto frontier can help recognizing whether and how much a method achieves Pareto improvement.
as the number of parameters or FLOPs (Gordon et al., 2020; Sanh et al., 2019; Jiao et al., 2020; Lan et al., 2020; Shen et al., 2020). For these works, the goal has shifted from simple SOTA to "Pareto SOTA". A Pareto SOTA model means that there is no other model is currently better than it on all the dimensions of interest. For example, a model may claim to be Pareto SOTA as long as it achieves the best accuracy under the same number of parameters or FLOPs. For these efficient models with fewer parameters or FLOPs, it is unfair to get them evaluated on the accuracy-centric benchmarks such as GLUE (Wang et al., 2019b), and ranked among many large-scale models.
The shifted goal has outpaced the existing benchmarks, which cannot provide a comprehensive and intuitive comparison for efficient methods. In the absence of a proper benchmark, measures of efficiency in different studies cannot be standardized, and different methods cannot be fairly compared. As a result, it is difficult to say whether and how much a method achieves Pareto improvement. To that end, we aim to build the Pareto frontier for various tasks with standard evaluation for both performance and efficiency. Our motivation can be briefly illustrated by Figure 1.
Need for a standard evaluation As the goal has shifted, a new benchmark is urgently needed to
Human Language Technologies, pages 3288 - 3303
comprehensively compare the NLP models in multiple dimensions. Currently, this multi-dimensional comparison is done in the individual papers, resulting in the following issues: (a) Incomplete comparison. The comparison is usually point-to-point, e.g. comparing model performance under the same FLOPs. The comparison in a broader range is usually missed, especially for works in conditional computation where the model performance varies with FLOPs. (b) Unaccessible results. Even if the comprehensive line-to-line comparison is conducted, the results are usually presented in form of figure, in which the data points are not accessible for the following work. As a result, the following work has to reproduce or estimate the results (e.g. Xin et al. (2021) estimate values from the figures of Zhou et al. (2020a)). (c) Non-standard measurements. Different works may adopt different metrics such as physical elapsed time, FLOPs, and executed model layers, making them hard to directly compare. Even if the adopted metrics are the same, there is no guarantee that they will be calculated in the same way (e.g. the hardware infrastructure, or the software to calculate FLOPs can be very different $^{1}$ ). (d) Inconvenience. Recent studies usually choose GLUE (Wang et al., 2019b) as the main benchmark, which, however, is not suitable for dynamic methods due to its submission limitation that is designed to avoid overfitting on test sets.
Need for a strong baseline Currently, there are roughly two branches of efficient methods in NLP: static methods (e.g. distillation, pruning, quantization, etc.) and dynamic methods (e.g. early exiting). (a) Static models are obtained given an expected number of parameters or inference latency. These methods often use the first few layers (to keep the same number of parameters or FLOPs) of some pre-trained model followed by a classification head as their baseline, which, however, is too weak to serve as a baseline. (b) Dynamic models usually add multiple internal classifiers to the pre-trained LMs, and therefore allow flexible inference conditioned on the input. Nevertheless, the injected internal classifiers introduce a gap between pre-training and fine-tuning. Training the internal classifiers on downstream tasks often degenerates the perfor
mance of the entire model (Xin et al., 2021). Thus, static models need a strong baseline, and dynamic models need a strong backbone.
Contributions In this work, we address the above needs by contributing the following:
- ELUE(Efficient Language Understanding Evaluation) – a standard benchmark for efficient NLP models. (1) ELUE supports online evaluation for model performance, FLOPs, and number of parameters. (2) ELUE is also an open-source platform that can facilitate future research. We reproduce and evaluate multiple compressed and early exiting methods on ELUE. All of the results are publicly accessible on ELUE. (3) ELUE provides an online leaderboard that uses a specific metric to measure how much a model oversteps the current Pareto frontier. ELUE leaderboard also maintains several separate tracks for models with different sizes. (4) ELUE covers six NLP datasets spanning sentiment analysis, natural language inference, similarity and paraphrase tasks. The ELUE benchmark is publicly available at http://eluebenchmark.fastnlp.top/.
- ElasticBERT - a strong baseline (backbone) for static (dynamic) models. ElasticBERT is a multi-exit Transformer (Vaswani et al., 2017) pre-trained on  $\sim 160\mathrm{GB}$  corpus. The pretraining objectives, MLM and SOP (Lan et al., 2020), are applied to multiple Transformer layers instead of only the last layer. Gradient equilibrium (Li et al., 2019) is adopted to alleviate the conflict of the losses at different layers. For static models, ElasticBERT is a strong baseline that can reach or even outperform distilled models. For dynamic models, ElasticBERT is a robust backbone that closes the gap between pre-training and fine-tuning. We release the pre-trained model weights of ElasticBERT² as well as code³.
NLP Benchmarks Evaluating the quality of language representations on multiple downstream tasks has become a common practice in the community. These evaluations have measured and
<sup>1</sup>We find that the FLOPs of Transformers calculated by different libraries (thop, ptflops, and torchstat) can be different. And besides, all of them missed FLOPs in some operations such as self-attention and layer normalization.
$^{2}$ https://huggingface.co/fnlp
<sup>3</sup>https://github.com/fastnlp/ElasticBERT
pushed the progress of NLP in recent years. SentEval (Conneau and Kiela, 2018) introduces a standard evaluation toolkit for multiple NLP tasks. Further, GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) provide a set of more difficult datasets for model-agnostic evaluation. Another line of work is multi-dimensional evaluations. EfficientQA (Min et al., 2020) is an open-domain question answering challenge that evaluates both accuracy and system size. The system size is measured as the number of bytes required to store a Docker image that contains the submitted system. Dynabench (Kiela et al., 2021), an open-source benchmark for dynamic dataset creation and model evaluation, also supports multidimensional evaluation. In particular, Dynabench measures model performance, throughput, memory use, fairness, and robustness. Both EfficientQA and Dynabench require the user to upload the model along with the required environment to the server, which is costly for users to upload and also for the server to evaluate. In contrast, ELUE adopts a cheaper way to evaluate performance and efficiency of the model. Recently, Long-Range Arena (LRA) (Tay et al., 2021) is proposed to evaluate models under the long-context scenario. Different from ELUE, LRA mainly focuses on Xformers (Lin et al., 2021). Besides, some tasks included in LRA are not NLP tasks, or even not real-world tasks, while ELUE consists of common language understanding tasks. In addition, ELUE is also inspired by other well-known benchmarks, such as SQuAD (Rajpurkar et al., 2016), MultiNLI (Williams et al., 2018), DecaNLP (McCann et al., 2018), CLUE (Xu et al., 2020b), HotpotQA (Yang et al., 2018), GEM (Gehrmann et al., 2021), etc.
Efficient NLP Models Current efficient NLP models can be roughly categorized as two streams: model compression (static methods) and conditional computation (dynamic methods). Model compression is to reduce the number or precision of model parameters to achieve faster training and inference. Currently, there are several ways to achieve model compression: (1) Knowledge Distillation, which is to learn a compact student model that learns from the output distribution of a largescale teacher model (Sanh et al., 2019; Jiao et al., 2020) (2) Model Pruning, which is to remove parts of parameters that are less important (Gordon et al., 2020), (3) Weight Sharing across different parts
of the model (Lan et al., 2020) is also a common technique to significantly reduce parameters, (4) Quantization, which is to use low bit precision to store parameter and accelerate inference with low bit hardware operations (Shen et al., 2020), and (5) Module Replacing, which is to replace the modules of a big model with more compact substitutes (Xu et al., 2020a). In contrast, conditional computation is to selectively execute only parts of the model conditioned on a given input (Bengio et al., 2013; Davis and Arel, 2014). As a representative, an end-to-end halting approach, Adaptive Computation Time (ACT) (Graves, 2016), is developed to perform input-adaptive computation for recurrent networks. The idea of ACT is later adopted in Universal Transformer (Dehghani et al., 2019). Recently, as the rising of deep models for natural language processing, early exiting is widely used to speedup inference of transformer models (Liu et al., 2020a; Xin et al., 2020; Schwartz et al., 2020; Zhou et al., 2020b; Elbayad et al., 2020; Liao et al., 2021; Xin et al., 2021; Sun et al., 2021b; Zhu, 2021; Li et al., 2021a).
3 ELUE: A Standard Benchmark for Efficient NLP Models
ELUE aims to offer a standard evaluation for various efficient NLP models, such that they can be fairly and comprehensively compared. In Section 3.1, we list the design considerations to achieve this motivation. In Section 3.2, we describe the tasks and datasets included in ELUE. In Section A.1, we illustrate how to make a submission on ELUE, and how the submission is evaluated. In Section 3.3, we discuss the design of our leaderboard.
3.1 Design Considerations
Now we enumerate main considerations in the design of ELUE to ensure that it meets the needs mentioned early.
Multi-dimensional Evaluation The evaluation of ELUE should be multi-dimensional for comprehensive comparison. Instead of point-to-point comparison, methods can be compared in a line-to-line style in ELUE, where the "line" is a performance-efficiency trade-off curve.
Public Accessible All data points in ELUE should be publicly accessible such that the following work does not need to reproduce or estimate
results from previous work. To facilitate future research, some representative methods should be reproduced and evaluated in ELUE.
Standard Evaluation The measurement of model efficiency should be standardized in ELUE such that this line of methods can be fairly compared. Current studies usually use number of parameters (Lan et al., 2020; Jiao et al., 2020), FLOPs (Jiao et al., 2020; Liu et al., 2020a; Li et al., 2021b), actual inference time (Sanh et al., 2019; Schwartz et al., 2020), or number of executed layers (Zhou et al., 2020a; Sun et al., 2021b) to measure model efficiency. Among these metrics, measuring actual inference time is costly for both users and the server, and highly depends on the computation infrastructure and software implementation, while number of executed layers ignores the shape of input and hidden layers, therefore is inaccurate. Thus, ELUE adopts number of parameters and FLOPs as the metrics for model efficiency.
Easy-to-Use ELUE should be friendly to users, which means that the submission should be as simple as possible. Roughly speaking, there are currently two ways of submissions: (1) submitting the trained model such as SQuAD (Rajpurkar et al., 2016), Dynabench (Kiela et al., 2021), and (2) submitting the predicted test files such as GLUE (Wang et al., 2019b), SuperGLUE (Wang et al., 2019a), and CLUE (Xu et al., 2020b). The submission of ELUE lies in the latter way. Nevertheless, to evaluate number of parameters and FLOPs, the submitted test files should conform to a specific format, and besides, a Python file to define the used model is also required. For more details about submission and evaluation, see Appendix A.1.
3.2 Task and Dataset Selection
Following GLUE (Wang et al., 2019b), SuperGLUE (Wang et al., 2019a), and CLUE (Xu et al., 2020b), we collect tasks that can be formatted as single sentence classification or sentence pair classification. Since ELUE mainly focuses on efficient models, the difficulty of dataset is not a primary consideration. Instead, we collect tasks and datasets that are commonly used and publicly available in the community. The statistics of the collected datasets are listed in Table 1.
Sentiment Analysis Sentiment analysis, which is to classify the polarity of a given text, is a fundamental task in NLP. We select two well-known
Tasks	Datasets	|Train|	|Dev|	|Test|
Sentiment	SST-2	8,544	1,101	2,208
Analysis	IMDb	20,000	5,000	25,000
Natural Language	SNLI	549,367	9,842	9,824
Inference	SciTail	23,596	1,304	2,126
Similarity and	MRPC	3,668	408	1,725
Paraphrase	STS-B	5,749	1,500	1,379
Table 1: Statistics of datasets in ELUE.
movie review datasets, Stanford Sentiment Treebank (SST) (Socher et al., 2013) and IMDb (Maas et al., 2011). For SST, we use the two-way class split, i.e. SST-2. Different from GLUE, SST-2 samples in ELUE are complete sentences instead of phrases. For IMDb, we randomly select  $2.5\mathrm{k}$  positive samples and  $2.5\mathrm{k}$  negative samples from training set to construct a development set.
Natural Language Inference Natural language inference (NLI) is a task to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither. NLI is often formulated as a sentence pair classification task (Devlin et al., 2019; Sun et al., 2021a). We select two NLI datasets, SNLI (Bowman et al., 2015) and SciTail (Khot et al., 2018). SNLI is a crowd-sourced collection of sentence pairs with balanced labels: entailment, contradiction, and neutral. We use the spell-checked version of the test and development sets<sup>4</sup>. The hard samples, which do not have golden labels due to the disagreement of annotators, are removed from the dataset and left for model diagnostic. SciTail is a two-way (entail or neutral) entailment classification dataset, which is derived from multiple-choice science exams and web sentences.
Similarity and Paraphrase For similarity and paraphrase tasks, we also select two datasets, Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005), and Semantic Textual Similarity Benchmark (STS-B) (Cer et al., 2017), both of which are also included in GLUE. MRPC is a collection of automatically extracted sentence pairs, each manually-labeled with a judgment to indicate whether the pair constitutes a paraphrase. STS-B is a corpus of sentence pairs, each of which is labeled with a score from 0 to 5 to represent the degree to which two sentences are semantically equivalent.
4https://nlp.stanford.edu/projects/snli/
Figure 2: An illustration to show how ELUE score is computed.
3.3 Leaderboard
Following prior work (Yang et al., 2018; Wang et al., 2019b; Xu et al., 2020b), we also integrate a leaderboard in ELUE. For dynamic models that have multiple performance-FLOPs coordinates on each dataset, we need to sum up these coordinates as a score. A critical problem is to measure how good a coordinate is. In other words, to measure a coordinate  $(p,f)$ , where  $p$  is performance and  $f$  is FLOPs, we need a baseline performance under the same FLOPs. We choose ElasticBERT as the baseline curve. We evaluate different layers of ElasticBERT, and obtained 12 coordinates  $(p_i^{EB}, f_i^{EB})_{i=1}^{12}$ , which are then used to interpolate to get a performance-FLOPs function  $p^{EB}(f)$ . With the baseline curve at hand, we can score a submission curve as
$$
\text {E L U E S c o r e} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ p _ {i} - p ^ {E B} \left(f _ {i}\right) \right]. \tag {1}
$$
Note that the coordinates of ElasticBERT are separately interpolated on different datasets. The final ELUE score is an unweighted average of the scores on all the 6 datasets. Figure 2 gives an illustration of how ELUE score is computed. The ELUE score reflects the extent to which the submission oversteps the ElasticBERT.
In addition, following EfficientQA (Min et al., 2020), ELUE leaderboard also maintains four additional separate tracks, corresponding to models below 40M, 55M, 70M, 110M parameters. Models in these tracks are ranked by the average performance on all the datasets.
4 ElasticBERT: A Strong Baseline for Efficient Inference
Despite the encouraging results achieved by existing efficient models, we argue that a strong baseline (backbone) is needed for both static methods and
dynamic methods. Static methods often choose the first few layers of some pre-trained models as their baseline (e.g. Sun et al. (2019); Jiao et al. (2020)), which can be weak. Dynamic methods that enable early exiting by training multiple internal classifiers usually introduce a gap between pre-training and fine-tuning, and therefore hurt the performance of the entire model (Xin et al., 2021). Thus, as illustrated in Figure 3, we present the ElasticBERT that bridges the gap between static and dynamic methods, and therefore can serve as a strong baseline for static methods and also a strong backbone for dynamic methods.
ElasticBERT is a multi-exit pre-trained language model with the following training objective:
$$
\mathcal {L} = \sum_ {l = 1} ^ {L} \left(\mathcal {L} _ {l} ^ {\mathrm {M L M}} + \mathcal {L} _ {l} ^ {\mathrm {S O P}}\right), \tag {2}
$$
where  $L$  is the total number of layers,  $\mathcal{L}^{\mathrm{MLM}}$  is the n-gram masked language modeling loss,  $\mathcal{L}^{\mathrm{SOP}}$  is the sentence order prediction loss (Lan et al., 2020). The two losses are applied to each layer of the model, such that the number of layers can be flexibly scaled on downstream tasks, and therefore it is named "ElasticBERT".
Bridge the Gap Between Static and Dynamic Methods As a baseline for static methods, the depth of ElasticBERT can be flexibly reduced on demand. Compared with the first  $l$  layer of BERT (Devlin et al., 2019), the  $l$ -layered ElasticBERT is a complete model (Turc et al., 2019; Li et al., 2021a) and can achieve better performance. It is worth noticing that ElasticBERT can be regarded as a special instance of LayerDrop (Fan et al., 2020) where the dropped layers are constrained to the top consecutive layers. As a backbone for dynamic methods, training classifiers injected in intermediate layers would be consistent with pre-training. Therefore, ElasticBERT can not only be used as a static complete model, but also be used as a backbone model of dynamic early exiting.
Gradient Equilibrium Pre-training with the simply summed loss in Eq. (2) could lead to a gradient imbalance issue (Li et al., 2019). In particular, due to the overlap of subnetworks, the variance of the gradient may grow overly large, leading to unstable training. To address this issue, we follow Li et al. (2019) and adopt the gradient equilibrium
Figure 3: ElasticBERT is pre-trained with multiple pre-training heads attached at the intermediate layers. For static usage (left), it can be pruned on demand while outperforming previous pre-trained models with the same size. For dynamic usage (right), it can serve as the backbone for early exiting methods, achieving better performance-efficiency trade-off than early exiting models with other backbones.
(GE) strategy $^{5}$  in the pre-training of ElasticBERT.
Grouped Training In our preliminary experiments, we found that summing up losses at all layers could slow down pre-training and increase memory footprints. To alleviate this, we divide  $L$  exits into  $G$  groups. During training, we optimize the losses of the exits within each group by cycling alternately between different batches:
$$
\mathcal {L} = \sum_ {l \in \mathcal {G} _ {i}} \left(\mathcal {L} _ {l} ^ {\mathrm {M L M}} + \mathcal {L} _ {l} ^ {\mathrm {S O P}}\right). \tag {3}
$$
In Section B.3 we explore the performance of different grouping methods. As a result, we group the 12 exits of ElasticBERT<sub>BASE</sub> into  $\mathcal{G}_1 = \{1, 3, 5, 7, 9, 11, 12\}$  and  $\mathcal{G}_2 = \{2, 4, 6, 8, 10, 12\}$ , and group the 24 exits of ElasticBERT<sub>LARGE</sub> into  $\mathcal{G}_1 = \{1, 4, 7, \dots, 22, 24\}$ ,  $\mathcal{G}_2 = \{2, 5, 8, \dots, 23, 24\}$ , and  $\mathcal{G}_3 = \{3, 6, 9, \dots, 21, 24\}$ . Our experiments demonstrate that grouped training can significantly speedup the process of pre-training without a loss in performance.
Pre-training Setup Following BERT (Devlin et al., 2019), we train ElasticBERT in two different configurations: ElasticBERT<sub>BASE</sub> and ElasticBERT<sub>LARGE</sub>, which have the same model sizes with BERT<sub>BASE</sub> and BERT<sub>LARGE</sub>, respectively. The detailed description can be found in Appendix B.1.
Downstream Evaluation We evaluate ElasticBERT on the ELUE benchmark, as a static model and as a dynamic model. As a static
(b) LARGE models.
Figure 4: Comparison of the average performance on ELUE test sets between ElasticBERT and baselines.