SlowGuess/NAACL_2022_OCR huggingface dataset:
AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8815-8821. AAAI Press.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1631-1642. ACL.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4322-4331. Association for Computational Linguistics.
Tianxiang Sun, Xiangyang Liu, Xipeng Qiu, and Xu-anjing Huang. 2021a. Paradigm shift in natural language processing. CoRR, abs/2109.12575.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge embedding. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 3660-3670. International Committee on Computational Linguistics.
Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021b. Early exiting with ensemble internal classifiers. CoRR, abs/2105.13792.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908.08962.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP 2020 - Demos, pages 38-45. Association for Computational Linguistics.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2246-2251. Association for Computational Linguistics.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. Berxit: Early exiting for BERT with better fine-tuning and extension to regression. In Proceedings of EACL 2021, pages 91-104. Association for Computational Linguistics.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020a. Bert-of-theseus: Compressing BERT by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7859-7869. Association for Computational Linguistics.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020b. CLUE: A chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762-4772. International Committee on Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotq: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369-2380. Association for Computational Linguistics.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley, Ke Xu, and Furu Wei. 2020a. BERT loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley, Ke Xu, and Furu Wei. 2020b. BERT loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Wei Zhu. 2021. Leebert: Learned early exit for BERT with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2968-2980. Association for Computational Linguistics.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching
movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19-27. IEEE Computer Society.
A Details of Evaluation
A.1 Submission and Evaluation
ELUE supports two kinds of submissions: submitting test files, or submitting from a paper.
Submit test files Users are required to submit two kinds of files: (1) predicted test files, and (2) a model definition file in Python. The predicted test files can be multiple, each indicates the prediction under a certain efficiency. The submitted test files should be in the following format:
index	pred	modules
0	1	(10),emb; (10,768),layer_1; (768),exit_1
1	0	(15),emb; (15,768),layer_1; (768),exit_1; (15,768),layer_2; (768),exit_2
2	1	(12),emb; (12,768),layer_1; (768),exit_1
...	...	...
Different from traditional predicted test files as in GLUE, an additional column "modules" is required to indicate the activated modules to predict each sample. The numbers before each module represent the input shape of that module, e.g. the "(10)" before "emb" indicates that the input of "emb" is a sequence of length 10. Note that this format is also compatible with token-level early exiting methods (Li et al., 2021b), where the sequence length is progressively reduced as the processing of layers.
Along with the test files, a Python file to define the model is also required. Figure 6 is an example Python file using PyTorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020).
With the submitted Python file, ELUE is able to evaluate the average FLOPs on a dataset, and the number of parameters of the model.
In cases that the evaluation is not applicable, e.g. the programming language, or dependencies of the submitted Python file is not supported in ELUE, the user is allowed to evaluate FLOPs and number of parameters by themselves and upload their results along with the predictions to the ELUE website.
Submit from a paper Inspired by Paper with Code<sup>6</sup>, we also expect that ELUE can serve as an open-source platform that can facilitate future research. Therefore, there is a track for the authors of
<sup>6</sup>https://paperswithcode.com/
Figure 6: An example Python file for submission.
published papers to share their experimental results on ELUE datasets.
Performance Metrics Since the classes in MRPC are imbalanced, we report the unweighted average of accuracy and F1 score. For STS-B, we evaluate and report the Pearson and Spearman correlation coefficients. For other datasets, we simply adopt accuracy as the metric.
B.1 Details of Training ElasticBERT
The parameters of ElasticBERT are initialized with BERT, and therefore it has the same vocabulary and tokenizer as BERT. ElasticBERT is pre-trained on  $\sim 160\mathrm{GB}$  uncompressed English text corpora, which is comprised of English Wikipedia (12GB), BookCorpus (4GB) (Zhu et al., 2015), OpenWebText (38GB) (Gokaslan and Cohen, 2019), and part of the C4 corpus (110GB) (Raffel et al., 2020). We use Adam optimizer (Kingma and Ba, 2015) to pre-train ElasticBERT<sub>BASE</sub> and ElasticBERT<sub>LARGE</sub> and other hyperparameters are listed in Table 4. Our implementation is based on Huggingface's Transformers (Wolf et al., 2020) and the Megatron-LM toolkit (Shoeybi et al., 2019). ElasticBERT is trained on 64 32G NVIDIA Tesla V100 GPUs.
Hyperparameter	ElasticBERTBASE	ElasticBERTLARGE
Adam β1	0.9	0.9
Adam β2	0.999	0.999
Peak Learning Rate	2e-4	2e-4
Warm Up Type	Linear	Linear
Warm Up Rate	0.04	0.04
Weight Decay	0.01	0.01
Batch Size	4096	4096
Training Steps	125k	125k
Number of Layers	12	24
Hidden Size	768	1024
Attention Heads	12	16
FFN Intermediate Size	3072	4096
B.2 Evaluating ElasticBERT on GLUE
To verify the effectiveness and the elasticity of ElasticBERT, we also evaluate ElasticBERT and our static baselines on the GLUE benchmark. We evaluate the first 6/12 layers of the BASE models, and the first 6/24 layers of the LARGE models.
Experimental results of ElasticBERT and our baseline models on GLUE are presented in Table 6, from which we find that ElasticBERT outperforms BERT and ALBERT with the same number of layers, but is weaker than RoBERTa in the 12/24 layers configuration. Compared with ElasticBERT that is trained for  $125\mathrm{K}$  steps with batch size of 4K, RoBERTa is trained for  $500\mathrm{K}$  steps with batch size of 8K, which makes its number of training samples 8 times larger than that of ElasticBERT. When using fewer layers (6 layers of BASE and LARGE models), ElasticBERT achieves the best performance among the static baselines, confirming its great elasticity.
Table 4: Hyperparameters for ElasticBERT pretraining
Grouping	Accuracy
w/o Grouping	76.7
G1={1,3,5,7,9,11,12}	76.7
G2={2,4,6,8,10,12}
G1={1,4,7,10,12}	75.7
G2={2,5,8,11,12}
G3={3,6,9,12}
G1={1,2,3,4,12}	75.5
G2={5,6,7,8,12}
G3={9,10,11,12}
G1={1,2,3,4,5,6,12}	75.9
G2={7,8,9,10,11,12}
Table 5: The average accuracy acorsss all the BERT exits on the MNLI dataset with different grouping.
Model	#Params	#FLOPs	CoLA	MNLI-m/mm	MRPC	QNLI	QQP	RTE	SST-2	STS-B	Average
BASE Models
BERTBASE	109M	6615M	56.5	84.6/84.9	87.6	91.2	89.6	69.0	92.9	89.4	82.9
ALBERTBASE	12M	6861M	56.8	84.9/85.6	90.5	91.4	89.2	78.3	92.8	90.7	84.5
RoBERTaBASE	125M	6727M	63.6	87.5/87.2	90.8	92.7	90.3	77.5	94.8	90.9	86.1
LayerDropBASE	125M	6727M	64.5	86.4/86.5	91.6	92.2	89.9	71.1	93.7	88.6	84.9
ElasticBERTBASE	109M	6615M	64.3	85.3/85.9	91.0	92.0	90.2	76.5	94.3	90.7	85.6
BERTBASE-6L	67M	3308M	44.6	81.4/81.4	84.9	87.4	88.7	65.7	90.9	88.1	79.2
ALBERTBASE-6L	12M	3435M	52.4	82.6/82.2	89.0	89.8	88.7	70.4	90.8	89.6	81.7
RoBERTaBASE-6L	82M	3364M	44.4	84.2/84.6	87.9	90.5	89.8	60.6	92.1	89.0	80.3
LayerDropBASE-6L	82M	3364M	53.7	83.8/83.8	87.6	89.8	89.4	64.3	91.3	88.1	81.3
HeadPrune-BERTBASE	87M	4744M	48.7	71.0/79.7	80.2	86.1	84.7	62.5	89.4	85.2	76.4
DistilBERT	67M	3308M	55.6	82.1/82.0	86.5	89.2	88.8	63.9	91.3	86.7	80.7
TinyBERT-6L	67M	3308M	46.3	83.6/83.8	88.7	90.6	89.1	73.6	92.0	89.4	81.9
BERT-of-Theseus	67M	3308M	45.1	81.4/81.9	88.1	88.1	88.9	70.1	91.4	88.8	80.4
ElasticBERTBASE-6L	67M	3308M	53.7	84.3/84.2	89.7	90.8	89.7	74.0	92.7	90.2	83.3
Test Set Results
TinyBERT-6L	67M	3308M	42.5	83.2/82.4	86.2	89.6	79.6	73.0	91.8	85.7	79.3
ElasticBERTBASE-6L	67M	3308M	49.1	83.7/83.4	87.3	90.4	79.7	68.7	92.9	86.9	80.3
LARGE Models
BERTLARGE	335M	23446M	61.6	86.2/86	90.1	92.2	90.1	72.9	93.5	90.4	84.8
ALBERTLARGE	18M	24296M	60.1	86/86.1	90.4	91.6	89.6	83.0	95.2	91.4	85.9
RoBERTaLARGE	355M	23840M	66.4	89/89.6	91.6	94.2	90.7	86.6	95.4	92.3	88.4
LayerDropLARGE	355M	23840M	66.6	89.7/89.6	91.2	93.9	88.5	86.6	95.5	92.6	88.2
ElasticBERTLARGE	335M	23446M	66.3	88/88.5	92.0	93.6	90.9	83.1	95.3	91.7	87.7
BERTLARGE-6L	108M	5863M	20.2	76.5/76.5	76.4	84.3	87.3	58.5	89.7	77.3	78.5
ALBERTLARGE-6L	18M	6083M	51.7	82.2/82.9	86.5	89.4	88.6	66.4	92.2	89.4	81.0
RoBERTaLARGE-6L	129M	5962M	43.3	80.4/80.9	80.0	86.1	88.9	54.9	90.1	80.5	76.1
LayerDropLARGE-6L	129M	5962M	44.3	81.4/81.0	79.8	87.1	88.5	53.1	91.4	83.0	76.6
ElasticBERTLARGE-6L	108M	5863M	53.9	83.5/84.3	89.6	90.8	90.1	71.1	91.9	90.1	82.8
Test Set Results
ALBERTLARGE-6L	18M	6083M	46.5	81.9/82.2	84.7	88.5	78.9	62.3	91.3	85.1	77.9
ElasticBERTLARGE-6L	108M	5863M	47.2	83.2/82.6	86.2	90.4	80.2	67.0	92.5	86.3	79.5
Table 6: ElasticBERT and static baseline performance on GLUE tasks. For MRPC, we report the mean of accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy.
B.3 Ablation Study
About the Training Strategy ElasticBERT adopts the gradient equilibrium (GE) to alleviate the conflict between the losses at different exits. Here, we compare GE with two other existing training strategies, two-stage training (Xin et al., 2020) and weighted training (Zhou et al., 2020a). Two-stage training is that, first training the top classifier along with the backbone model, and then freeze the parameters of the backbone model and train the injected internal classifiers. By this, two-stage training maintains the performance of the top classifier. Weighted training is to weight the loss of each exit according to the corresponding layer, which is
$$
\mathcal {L} = \frac {\sum_ {l = 1} ^ {L} l \cdot \mathcal {L} _ {l}}{\sum_ {l = 1} ^ {L} l}. \tag {4}
$$
Experimental results of ElasticBERT with the three training strategies are shown in Figure 7. It
(b) MRPC
Figure 7: Performance of the ElasticBERT exits at different layers with different training strategies.
can be observed that training with GE strategy performs the best on both SST-2 and MRPC.
About the Grouped Exits As shown in Eq. (3), we divide the  $L$  exits into different groups to speedup the pre-training. Therefore, how to group these exits needs to be explored. Here we evaluate four different grouping methods, as described in Table 5. To keep the overall performance of the entire model, the exit classifier on the top of the model is included in each group. According to
Method	MNLI-m/mm	MRPC	QNLI	QQP	Average
12 Layers
ElasticBERTBASE	85.4/85.7	89.2	91.9	89.8	88.4
w/o Grouped Training	85.3/85.1	89.3	92.0	89.8	88.3
w/o Grouped Training + GE	85.5/85.6	89.0	91.8	89.5	88.3
6 Layers
ElasticBERTBASE	83.9/83.6	87.1	90.6	89.6	87.0
w/o Grouped Training	84.5/83.6	87.5	90.7	89.4	87.1
w/o Grouped Training + GE	83.5/83.3	87.2	90.7	88.3	86.6
Table 7: Comparison between ElasticBERT<sub>BASE</sub> with and without GE and Grouped Training. Here, ElastiBERT<sub>BASE</sub> is pre-trained using Wikipedia and BookCorpus data. ElasticBERT<sub>BASE</sub> denotes that model is pre-trained with GE and Grouped Training.
Method	Training Time (h)
ElasticBERTBASE	106.0
-w/o Grouped Training	186.0
-w/o Grouped Training + GE	174.5
Table 8: The training time for different training strategies. All models are trained on the same GPU servers.
the experimental results in Table 5, we choose the odd/even grouping method for ElasticBERT<sub>BASE</sub>. Similarly, our experiments demonstrate that grouping 24 exits into  $\mathcal{G}_1 = \{1, 4, 7, \dots, 22, 24\}$ ,  $\mathcal{G}_2 = \{2, 5, 8, \dots, 23, 24\}$ , and  $\mathcal{G}_3 = \{3, 6, 9, \dots, 21, 24\}$  works well for ElasticBERT<sub>LARGE</sub>.
Effect of Gradient Equilibrium To verify that GE can enhance performance, we pre-train ElasticBERT<sub>BASE</sub> with and without GE using Wikipedia and BookCorpus data. As shown in Table 7, ElasticBERT with GE outperforms that without GE in two different configurations.
Effect of Grouped Training If we divide  $L$  exits into  $G$  groups, the number of samples used for training the remaining exits is  $1 / G$  of the last exit. To verify that this does not degrade the performance of the internal exits, we compare the performances of pre-training ElasticBERT<sub>BASE</sub> with and without Grouped Training. As shown in Table 7, we can observe that ElasticBERT with Grouped Training and GE does not suffer much performance loss compared with that training with only GE. In addition, as shown in the Table 8, using Grouped Training reduces  $\sim 43\%$  training time compared with that without Grouped Training.
(a) ElasticBERTBASE
(b) ElasticBERTLARGE
Figure 8: Comparison of average performance on ELUE test sets between the ElasticBERT and other pretrained models.
B.4 Overall Comparison
We compare ElastiBERT with other large scale pretrained models in Figure 8, from which we find that ElasticBERT is more robust to depth reduction. As the number of layers decreases, ElasticBERT offers greater advantages over other pre-trained models.
C ELUE Website
The ELUE website is built using Vue and Spring Boot. We use MySQL for data storage and our private servers to run the scoring script for each submission.