SlowGuess/NAACL_2022_OCR huggingface dataset:
model, we evaluate different layers of ElasticBERT, denoted as ElasticBERT- $n$ L. As a dynamic model, we inject and train internal classifiers in ElasticBERT $_{\text{BASE}}$  and adopt two strategies, entropy (Xin et al., 2020) and patience (Zhou et al., 2020a), to enable early exiting, denoted as ElasticBERT $_{\text{entropy}}$  and ElasticBERT $_{\text{patience}}$ . To compare with previous work, we also evaluate ElasticBERT on the GLUE benchmark (Wang et al., 2019b). The comparison results is shown in Appendix B.2. For static usage, we fine-tune ElasticBERT and our baseline models for 10 epochs with early stopping using AdamW optimizer (Loshchilov and Hutter, 2019) with learning rates of  $\{1e-5, 2e-5, 3e-5\}$  and batch size of 32, and warm up the learning rate for the first 6 percent of total steps. In dynamic usage, for the models using two-stage training methods, we train for 3 epochs for each stage, and we train for 5 epochs for other models. Other optimization configurations are the same as those in static scenario.
5The reader is referred to the original paper for more details. In brief, the gradients of  $\mathcal{L}_j$  w.r.t. the parameters of the  $i$ -th layer  $(i < j)$  would be properly rescaled.
Model	#Params	#FLOPs	SST-2	IMDb	MRPC	STS-B	SNLI	SciTail	Average
BASE Models
BERTBASE	109M	13399M	85.1	93.0	83.1	84.2	90.4	93.2	88.2
ALBERTBASE	12M	13927M	86.6	92.9	87.8	88.3	90.1	93.4	89.9
RoBERTaBASE	125M	13103M	88.3	94.9	88.0	89.6	91.3	92.8	90.8
LayerDropBASE	125M	13103M	88.5	94.2	88.2	87.1	90.7	92.8	90.3
ElasticBERTBASE	109M	13399M	88.6	93.9	87.9	87.6	91.3	93.8	90.5
BERTBASE-6L	67M	6700M	83.3	91.0	82.6	82.5	88.9	90.7	86.5
ALBERTBASE-6L	12M	6972M	84.7	92.0	85.3	83.5	89.3	92.3	87.9
RoBERTaBASE-6L	82M	6552M	86.8	92.6	86.7	84.5	90.2	91.3	88.7
LayerDropBASE-6L	82M	6552M	86.3	92.9	86.3	86.1	89.5	90.3	88.6
HeadPrune-BERTBASE	86M	9249M	84.8	84.7	77.8	74.8	87.8	88.3	83.0
DistilBERT	67M	6700M	84.8	92.0	83.8	81.7	89.2	89.7	86.9
TinyBERT-6L	67M	6700M	85.3	89.0	86.2	85.7	89.3	90.0	87.6
BERT-of-Theseus	67M	6700M	84.4	90.7	82.4	85.0	89.4	92.1	87.3
ElasticBERTBASE-6L	67M	6700M	87.0	92.7	87.3	86.9	90.1	92.5	89.4
LARGE Models
BERTLARGE	335M	47214M	87.9	94.0	85.9	86.7	90.8	93.9	89.9
ALBERTLARGE	18M	48876M	87.7	93.8	88.1	89.3	90.2	93.6	90.5
RoBERTaLARGE	355M	46042M	90.5	95.7	89.9	90.5	91.6	95.8	92.3
LayerDropLARGE	355M	46042M	90.4	95.3	89.5	91.0	91.4	95.2	92.1
ElasticBERTLARGE	335M	47214M	89.8	95.0	89.8	90.9	91.4	95.7	92.1
BERTLARGE-6L	108M	11922M	80.4	89.6	74.3	70.5	87.4	84.4	81.1
ALBERTLARGE-6L	18M	12397M	84.5	92.0	84.7	85.1	89.4	90.8	87.8
RoBERTaLARGE-6L	129M	11664M	83.5	91.7	77.9	72.7	88.6	84.7	83.2
LayerDropLARGE-6L	129M	11664M	85.4	92.5	77.3	75.9	88.8	84.1	84.0
ElasticBERTLARGE-6L	108M	11922M	86.8	92.9	86.2	86.3	89.8	92.4	89.1
Table 2: ElasticBERT and static baseline performance on ELUE task test sets. We report the mean of Accuracy and F1 for MRPC, Pearson and Spearman correlation for STS-B and Accuracy for other tasks. The reported FLOPs is the average over all the datasets.
Baselines We compare ElasticBERT with three types of baselines: (1) Directly fine-tuning pretrained models and their first  $n$  layers. We choose BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), RoBERTa (Liu et al., 2019) and LayerDrop (Fan et al., 2020) as our baselines. For the use of the first  $n$  layers, we simply add a linear classifier on top of the truncated model. (2) Compressed models. We choose two distilled models, DistilBERT (Sanh et al., 2019) and TinyBERT (Jiao et al., 2020), one pruned model, HeadPrune (Michel et al., 2019), and one model obtained by using module replacing, BERT-of-Theseus (Xu et al., 2020a) as our baseline models. (3) Dynamic early exiting models. To verify the effectiveness of ElasticBERT as a strong backbone of dynamic early exiting methods, we also compare ElasticBERTentropy and ElasticBERTpatience which have the same early exiting strategy as DeeBERT (Xin et al., 2020) and PABEE (Zhou et al., 2020a) with four representative early exiting models: DeeBERT (Xin et al., 2020), FastBERT (Liu et al., 2020b), PABEE (Zhou et al., 2020a), and CascadeBERT (Li et al., 2021a).
5.2 Evaluating ElasticBERT on ELUE
ElasticBERT and our baselines are evaluated on ELUE tasks. For the BASE version of ElasticBERT, BERT, ALBERT, RoBERTa and LayerDrop, we evaluate the first 3/4/6/12 layers. For the LARGE version of the models, we evaluate the first 6/8/12/24 layers. For dynamic methods, we fine-tune ElasticBERT along with the injected internal classifiers using the gradient equilibrium (GE) strategy (Li et al., 2019), and adopt two different early exiting strategies: entropy-based strategy (Xin et al., 2020) and patience-based strategy (Zhou et al., 2020a).
Results of Static Models The performance of ElasticBERT and our baseline models on ELUE task test sets is shown in Table 2, where we find that ElasticBERT<sub>BASE</sub> and ElasticBERT<sub>LARGE</sub> outperform BERT and ALBERT with the same number of layers, but are slightly weaker than RoBERT<sub>BASE</sub> and RoBERT<sub>LARGE</sub>. Besides, we find that the superiority of ElasticBERT over its baselines can be significant with fewer layers (See Figure 4 for the
(a) SST-2
(b)IMDb
(d) SciTail
(f) STS-B
Figure 5: Performance-FLOPs trade-offs on ELUE task test sets. Because STS-B is a regression task, for which the entropy-based methods are not applicable, we only evaluate patience-based methods, i.e., PABEE and ElasticBERT $_{\text{patience}}$ .
Model	SST-2	IMDb	MRPC	STS-B	SNLI	SciTail	Average
ElasticBERTBASE	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Static Models
BERTBASE	-4.55	-2.15	-5.88	-4.75	-1.50	-3.35	-3.70
ALBERTBASE	-2.41	-1.08	-2.34	-2.81	-1.55	-1.50	-1.95
RoBERTaBASE	-0.89	-0.11	-2.95	-5.38	-0.66	-3.32	-2.22
LayerDropBASE	-1.17	-0.13	-2.17	-2.98	-1.36	-4.14	-1.99
HeadPrune-BERTBASE	-3.81	-8.61	-9.73	-11.9	-2.89	-4.18	-6.85
DistilBERT	-2.20	-0.70	-3.50	-5.20	-0.90	-2.80	-2.55
TinyBERT-6L	-1.70	-3.70	-2.60	-1.90	-0.80	-2.50	-2.20
BERT-of-Theseus	-4.21	-2.61	-5.13	-1.67	-1.29	-0.38	-2.55
Dynamic Models
PABEE	-1.33	-0.23	-2.93	-2.13	-0.85	-0.43	-1.50
DeeBERT	-12.1	-14.0	-4.88	-	-8.35	-6.19	-
FastBERT	-1.51	0.16	-3.70	-	-0.22	-1.23	-
CascadeBERT	-2.13	-0.12	-4.05	-	-0.23	0.14	-
ElasticBERTpatience	0.40	0.20	-1.00	-0.44	0.03	0.36	-0.08
ElasticBERTentropy	0.97	1.02	-0.14	-	0.02	0.64	-
Table 3: ELUE scores calculated using Eq. (1) for static and dynamic baseline models. '-' denotes that the dataset/metric is not applicable to the model.
results of 3/4 (6/8) layers of the BASE (LARGE) models).
Results of Dynamic Models We compare ElasticBERTentropy and ElasticBERTpatience with four dynamic models: DeeBERT (Xin et al., 2020), FastBERT (Liu et al., 2020b), PABEE (Zhou et al., 2020a), and CascadeBERT (Li et al., 2021a). The performance-FLOPs trade-off of the dynamic models on ELUE task test sets are shown in Figure 5, which demonstrates that ElasticBERT can achieve
better performance-FLOPs trade-off.
Evaluating ELUE Scores According to Eq. (1), we also evaluate the ELUE scores of these baselines. As shown in Table 3, the ELUE score of ElasticBERT<sub>BASE</sub> is natural to be zero on all tasks. Among the other baselines, we find that ElasticBERT<sub>patience</sub> achieves the best ELUE score, while HeadPrune achieves the worst ELUE score. In addition, we find that dynamic models perform better than static models on average.
In this work, we present ELUE, which is a public benchmark and platform for efficient models, and ElasticBERT, which is a strong baseline (backbone) for efficient static (dynamic) models. Both of the two main contributions are aimed to build the Pareto frontier for NLU tasks, such that the position of existing work can be clearly recognized, and future work can be easily and fairly measured.
Our future work is mainly in four aspects: (1) Including more baselines in ELUE, (2) Supporting the evaluation for more frameworks such as TensorFlow (Abadi et al., 2016), (3) Supporting diagnostics for submissions, (4) Supporting the evaluation of more different types of tasks.
This work was supported by the National Key Research and Development Program of China (No.2020AAA0106702) and National Natural Science Foundation of China (No.62022027).
The proposed ELUE benchmark aims to standardize efficiency measurement of NLP models. The collected datasets are widely used in previous work and, to our knowledge, do not have any attached privacy or ethical issues. Our proposed ElasticBERT is a pre-trained model to reduce computation cost and carbon emission. The pre-training data is public resources adopted in previous work, and therefore would not introduce new ethical concerns.
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016, pages 265-283. USENIX Association.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642. The Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Daniel M. Cer, Mona T. Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation. CoRR, abs/1708.00055.
Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).
Andrew S. Davis and Itamar Arel. 2014. Low-rank approximations for conditional feedforward computation in deep neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop
on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. Asian Federation of Natural Language Processing.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. CoRR, abs/2102.01672.
Aaron Gokaslan and Vanya Cohen. 2019. Openweb-text corpus.
Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2020, Online, July 9, 2020, pages 143-155. Association for Computational Linguistics.
Alex Graves. 2016. Adaptive computation time for recurrent neural networks. CoRR, abs/1603.08983.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling BERT for natural language understanding. In Proceedings of EMNLP 2020, pages 4163-4174. Association for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence,
(AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5189-5197. AAAI Press.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 4110-4124. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, and Gao Huang. 2019. Improved techniques for training adaptive deep networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 1891-1900. IEEE.
Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021a. Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade. In Findings of EMNLP.
Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2021b. Accelerating BERT inference for sequence labeling via early-entry. In Proceedings of ACL/IJCNLP 2021, pages 189-199. Association for Computational Linguistics.
Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. 2021. A global past-future early exit method for accelerating inference of pretrained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2013-2023. Association for Computational Linguistics.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2021. A survey of transformers. CoRR, abs/2106.04554.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020a. Fastbert: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6035-6044. Association for Computational Linguistics.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020b. Fastbert: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6035-6044. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 142-150. The Association for Computer Linguistics.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14014-14024.
Sewon Min, Jordan L. Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick S. H. Lewis, Yuxiang Wu, Heinrich Kuttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun
Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Sejr Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen-tau Yih. 2020. Neurips 2020 efficientqa competition: Systems, analyses and lessons learned. In NeurIPS 2020 Competition and Demonstration Track, 6-12 December 2020, Virtual Event / Vancouver, BC, Canada, volume 133 of Proceedings of Machine Learning Research, pages 86-111. PMLR.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. SCIENCE CHINA Technological Sciences.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6640-6651. Association for Computational Linguistics.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-BERT: hessian based ultra low precision quantization of BERT. In The Thirty-Fourth AAAI Conference on Artificial Intelligence,