SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1: Model performance on the GLUE test set with different PLMs as backbone. The speed-up ratio (Spd-up) is approximately  $2.00 \times$  and our method significantly outperforms previous early exit methods.
Model	MNLI-m	MNLI-mm	QQP	QNLI	SST-2	MRPC	RTE	Macro
Acc	Spd-up	Acc	Spd-up	F1/Acc	Spd-up	Acc	Spd-up	Acc	Spd-up	F1/Acc	Spd-up	Acc	Spd-up
BERT
BERT-base (Devlin et al., 2019)	84.6	1.00×	83.4	1.00×	71.2/ -	1.00×	90.5	1.00×	93.5	1.00×	88.9/ -	1.00×	66.4	1.00×	-
BERT-4L	77.6	3.00×	77.2	3.00×	67.7/87.5	3.00×	85.4	3.00×	88.7	3.00×	82.9/74.9	3.00×	63.0	3.00×	78.4
DeeBERT	61.0	2.80×	59.8	2.84×	66.1/86.9	3.19×	80.8	2.88×	84.7	2.71×	83.5/75.5	2.61×	60.5	2.90×	71.8
PABEE	75.9	2.70×	75.3	2.71×	69.5/88.2	2.57×	82.6	3.04×	85.2	3.15×	82.6/73.1	2.72×	60.5	2.38×	76.6
Ours	78.4	2.99×	77.4	3.02×	70.4/89.2	3.16×	87.3	2.78×	91.1	2.97×	84.5/77.7	2.87×	63.0	2.88×	79.7
RoBERTa
RoBERTa-base (Xin et al., 2020)	87.0	1.00×	86.3	1.00×	71.8/ -	1.00×	92.4	1.00×	94.3	1.00×	90.4/ -	1.00×	67.5	1.00×	-
RoBERTa-4L	80.3	3.00×	79.6	3.00×	69.8/88.4	3.00×	86.0	3.00×	91.3	3.00×	85.0/78.1	3.00×	53.2	3.00×	80.2
DeeBERT	55.1	2.31×	56.6	2.27×	67.1/88.1	3.24×	76.0	2.82×	72.3	2.67×	85.9/79.4	2.87×	-	-	-
Ours	81.4	2.97×	80.5	3.02×	71.9/89.3	3.12×	89.2	2.83×	93.5	2.67×	87.1/82.2	2.75×	54.1	3.01×	80.6
ALBERT
ALBERT-base	85.2	1.00×	84.7	1.00×	70.5/88.7	1.00×	92.0	1.00×	93.3	1.00×	89.0/84.8	1.00×	72.0	1.00×	84.8
ALBERT-4L	80.1	3.00×	79.2	3.00×	68.9/88.1	3.00×	87.6	3.00×	89.5	3.00×	84.4/78.9	3.00×	61.2	3.00×	79.7
PABEE	79.6	2.95×	78.9	2.96×	70.8/88.8	2.61×	87.9	3.25×	91.9	2.64×	83.6/75.1	2.66×	64.6	2.69×	80.3
Ours	82.5	2.93×	82.0	2.95×	70.3/88.6	3.17×	91.0	2.92×	92.5	2.88×	87.6/82.8	2.72×	68.1	2.92×	83.0
Table 2: Model performance on the GLUE test set with different PLMs as backbone. The speed-up ratio (Spd-up) is approximately  $3.00 \times$  and our method significantly outperforms previous early exit methods.
method maintains a comparable result with the original models on most datasets. We also notice that directly reducing layers performs well and serves as a strong baseline. Nevertheless, our proposal significantly outperforms such a method as well as the other two early exit methods.
We then adopt a more aggressive  $3.00 \times$  speed-up ratio to verify the effectiveness of our method. According to Table 2, the performance of PABEE and DeeBERT deteriorates badly. In contrast, our model exhibits more robust and stable performance, showing its superiority over previous early exit methods. Particularly, ALBERT is already very efficient in model size owing to its layer-sharing mechanism. Results shown in the bottom of Table 2 suggest that our model can obtain a good result with minimum performance loss on such a parameter-efficient model.
The success of our proposal might be attributed to the global perspective for prediction. DeeBERT makes prediction with the help of the state of a single branch classifier, leading to less reliable results. Although PABEE employs cross-layer prediction to prevent error from one single classifier, they ignore much available information of past states as well as the high-level semantic features captured by future layers. Different from those methods, our method jointly takes into consideration the hierarchical linguistic information embedded in all layers and thus is able to produce more accurate results.
4.4 Performance-Efficiency Trade-Off
To further verify the robustness and efficiency of our method, we visualize the performance-efficiency trade-off curves in Figure 3 on a representative subset of the GLUE dev set. The backbone
(b)QQP
(c)QNLI
Figure 3: Performance and efficiency trade-off for early exit methods with BERT as backbone. Our method outperforms previous early exit methods by a large margin especially under high speed-up ratios.
(d) SST-2
Method	MNLI-m Acc	QNLI Acc	SST-2 Acc	MRPC F1/Acc	Macro
Naive global strategies
Voting Ensemble	71.33	87.19	89.56	87.71/82.35	83.28
70.92	87.85	90.37	87.44/81.13	83.41
Our global strategies
Avg-Pooling	81.11	90.43	92.43	88.48/83.09	87.44
Max-Pooling	82.86	90.18	92.32	87.85/82.11	87.59
SequentialNN	82.52	90.17	92.09	89.35/85.05	88.00
Attn-Pooling	83.02	90.37	93.00	87.83/81.86	87.81
Concatenation	83.30	90.46	92.89	88.44/83.08	88.10
Table 3: The performance of different strategies to incorporate multiple states on the GLUE dev set. The speed-up ratio is approximately  $2.00 \times (\pm 4\%)$ .
model is BERT. Please refer to the Appendix A for results of RoBERTa and ALBERT. As can be seen from Figure 3, the performance of previous state-of-the-art early exit methods drops dramatically when the speed-up ratio increases, which limits their practicality for higher acceleration requirements. By comparison, our method demonstrates more tolerance of speed-up ratio. It significantly improves performance compared to previous best-performing early exit models under the same speed-up ratio, especially in the case that the speed-up ratio is high, indicating that it can be applied in a wider range of acceleration scenarios.
4.5.1 Effect of Global Strategies
The results of different global strategies on a representative subset of GLUE dev are shown in Table 3. The naive global strategies including voting and ensemble perform poorly, which demonstrates that existing global strategies can only achieve suboptimal performance. In contrast, we design simple yet effective global strategies to incorporate past states which bring significant improvement compared to baselines. In addition, we empirically find
that the concatenation strategy works best from an overall point of view. We assume that such a strategy allows interaction among different states, yielding better performance. In addition, the effect of the merging gate can be found in Appendix B.
4.5.2 Analysis of Future Information
To assess whether and how future information contributes to the prediction, we first evaluate the Global Future version of our early exit method where all the approximations of futures states are incorporated through the concatenation strategy. Effect of future information is backed with the results shown in Table 4. We observe that the Global Future mechanism brings improvement on most datasets for both  $2 \times$  speed-up ratio and  $3 \times$  speed-up ratio, which confirms that the approximations of future states help enhance the model ability in prediction. Beyond that, the future states can be especially advantageous for the models with a higher speed-up ratio. Recall that approximations of future states complement the high-level semantic information and the exit at shallow layers loses more semantic information in comparison with the exit at deep layers. Therefore, the benefit of future information is more significant compared to the exit at shallow layers, which is validated by the larger improvement gap with a  $3 \times$  speed-up ratio.
We also investigate the effect of future information on exit time. Figure 4 demonstrates the distribution of exit layers with and without future information. When future information is engaged, we observe that the proportion of exit at shallow layers increases. The observation conforms with our intuition: with the approximations of future states supplemented for prediction, the merged state at a shallow layer is able to make a confident and correct prediction. Thus the exit time is earlier compared to situations without future states, result
Model	MNLI-m	MNLI-mm	QQP	QNLI	SST-2	MRPC	RTE	Macro
Acc	Spd-up	Acc	Spd-up	F1/Acc	Spd-up	Acc	Spd-up	Acc	Spd-up	F1/Acc	Spd-up	Acc	Spd-up
≈ 2.00× speed-up
BERT-local	81.97	1.96×	82.47	1.96×	88.18/91.21	1.85×	89.90	2.00×	92.09	2.00×	86.84/80.39	2.08×	66.43	1.96×	83.73
+Global Future	82.14	2.04×	82.93	1.96×	88.01/91.12	1.89×	90.04	2.04×	92.32	2.08×	87.19/80.64	2.08×	66.78	1.92×	83.96
≈ 3.00× speed-up
BERT-local	76.72	2.78×	77.64	2.78×	85.80/89.53	3.03×	86.62	2.86×	90.25	2.94×	85.35/77.45	2.94×	62.82	2.86×	80.45
+Global Future	79.06	2.70×	78.86	2.70×	85.65/89.61	3.03×	86.93	2.86×	91.40	2.94×	86.12/78.43	2.86×	62.45	2.86×	81.23
(a) MNLI
Figure 4: The distribution of exit layers with and without future states on the MNLI and MRPC tasks. The exit threshold for the same task is fixed. When future states are engaged for prediction, we observe an increase of exit at shallow (1-4) layers as well as a performance boost.
ing in a higher speed-up ratio. To be more specific, for MRPC, the speed-up ratios with and without future states are 1.69 and 1.99, and are 1.92 and 2.04 for MNLI, respectively. Meanwhile, we observe a performance boost with future states involved. It confirms our assumption that the high-level semantic features embedded in future states help improve performance in early exit framework.
4.5.3 Comparison with Distillation Methods
As an alternative method to accelerate inference, knowledge distillation also exhibits promising performance for NLP tasks. We provide comparison with typical knowledge distillation methods in Table 5. Existing model TinyBERT (Jiao et al., 2019) exerts multiple elaborate strategies to achieve the state-of-the-art results, including the expensive general distillation process and a vast amount of augmented data for fine-tuning. We remove these two techniques to exclude the effect of extra training data. Under the same settings, we observe that our method outperforms the distillation methods with the same speed-up ratio.
In general, early exit and distillation methods improve inference efficiency from different perspec
Table 4: Effect of the approximated future states. BERT-local denotes the early exit method using only current state and Global Future represents the incorporation of future states. Results are on the GLUE dev set.
Method	MNLI-m Acc	QQP F1/Acc	QNLI Acc	SST-2 Acc	Macro
DistilBERT	81.9	70.0/88.4	88.2	92.1	85.4
BERT-PKD (Sun et al., 2019)	81.5	70.7/88.9	89.0	92.0	85.6
PD-BERT (Turc et al., 2019)	82.8	70.4/88.9	88.9	91.8	85.8
BERT-of-Theseus (Xu et al., 2020)	82.4	71.6/89.3	89.6	92.2	86.2
TinyBERT‡	81.9	70.0/88.6	88.6	92.0	85.5
Ours	83.3	71.2/89.4	89.8	92.8	86.6
Table 5: Comparison with distillation methods on the GLUE test set. TinyBERT‡ is our implementation that removes general distillation and additional fine-tuning resources to match the settings of other methods. The speed-up ratio is approximately  $2.00 \times (\pm 4\%)$ .
tives. The distillation methods are more efficient in saving memory usage, but the downside is that such static methods suffer from high computation cost to adapt to different speed-up ratios. A new student model has to be trained from scratch if the speed-up requirement changes. By contrast, dynamic methods are more flexible to meet different acceleration requirements. Concretely, simple instances will be processed by passing through fewer layers and complex instances may require more layers. Moreover, the speed-up ratio can be easily adjusted depending on the acceleration requests. Nevertheless, early exit and distillation accelerate inference from different perspectives and these two kinds of techniques can be integrated to further compress the model size and accelerate the inference time.
We propose a novel Past-Future early exit method from a global perspective. Unlike previous work using only local states for prediction, our model employs all available past states for prediction and proposes a novel approach to engage the future states which are originally inaccessible for prediction. Experiments illustrate that our method achieves significant improvement over baseline methods with different models as backbones, suggesting the superiority of our early exit method.
We thank all the anonymous reviewers for their constructive comments. This work is partly supported by National Key R&D Program of China No. 2019YFC1521200 and Beijing Academy of Artificial Intelligence (BAAI). Xu Sun is the corresponding author.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4565-4573.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3651-3657. Association for Computational Linguistics.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling BERT for natural language understanding. CoRR, abs/1909.10351.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. Shallow-deep networks: Understanding and mitigating network overthinking. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3301-3310. PMLR.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942.
Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020a. BERT-EMD: many-to-many layer mapping for BERT compression with earth mover's distance. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3009-3018. Association for Computational Linguistics.
Lei Li, Yankai Lin, Shuhuai Ren, Deli Chen, Xuancheng Ren, Peng Li, Jie Zhou, and Xu Sun. 2020b. Accelerating pre-trained language models via calibrated cascade. CoRR, abs/2012.14682.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Khanh Nguyen. 2016. Imitation learning with recurrent neural networks. CoRR, abs/1607.05241.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pages 627-635. JMLR.org.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6640-6651. Association for Computational Linguistics.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4322-4331. Association for Computational Linguistics.
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, Cancún, Mexico, December 4-8, 2016, pages 2464-2469. IEEE.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-Read Students Learn Better:
On the Importance of Pre-training Compact Models. arXiv e-prints, page arXiv:1908.08962.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. CoRR, abs/2002.10957.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2246-2251. Association for Computational Linguistics.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020. Bert-of-theseus: Compressing BERT by progressive module replacing. CoRR, abs/2002.02925.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 5754-5764.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. BERT loses patience: Fast and robust inference with early exit. CoRR, abs/2006.04152.
A More Performance-Efficiency Trade-Off Curves
Performance-efficiency curves with RoBERTa and ALBERT as backbones are shown in Figure 5 and Figure 6 respectively. Similar to the observation with BERT as backbone, the performance of DeeBERT and PABEE becomes progressively worse as the speed-up ratio increases. In contrast, our past-future early exit method shows more robust results.
(b) QQP
(c)QNLI
(d) SST-2
Figure 5: Performance-efficiency trade-off for early exit method DeeBERT with RoBERTa as backbone.
(c)QNLI
Figure 6: Performance-efficiency trade-off for early exit method PABEE with ALBERT as backbone.
B Effect of Merging Gate
Method	MNLI-m Acc	QNLI Acc	SST-2 Acc	MRPC F1/Acc	Macro
Ours	83.30	90.46	92.89	88.44/83.08	88.10
-merging gate	83.15	90.61	92.43	86.86/80.64	87.49
Table 6: Ablation study of the merging gate. The speed-up ration is approximately  $2.00 \times$  and the model implementation is based on BERT.
We conduct ablation study to show the effect of the merging gate and report the result in Table 6. We can see that the performance drops when we remove the merging gate from our model, suggesting that the merging gate plays an important role in keeping the balance between past information and future information.