SlowGuess/NAACL_2022_OCR huggingface dataset:
Putting performance comparisons aside, one benefit of PCEE-BERT is that it is flexible since by adjusting the threshold and the patience parameter, it can easily control the average inference layers and cover (or achieve values close to) any speedup ratios.2
Figure 4: This figure demonstrates that PCEE-BERT can work with other confidence measures.
4.6 Ablation studies
4.6.1 Ablation on the confidence measures
Note that our PCEE-BERT is a novel combination of PABEE and BranchyNet. Thus PCEE-BERT mainly uses the entropy of predicted distributions as the confidence measure of an intermediate layer. However, can PCEE-BERT work with the other confidence measures, such as Shallow-Deep? We switch the entropy-based confidence level  $C^{(M)}$  (Equation 2) with that from Shallow-Deep (Kaya et al., 2019):
$$
C ^ {(M)} = \operatorname {A r g m a x} _ {k} p _ {k} ^ {(m)}, \tag {5}
$$
and we will call this version of PCEE-BERT as PCEE-BERT-v1. Note that PCEE-BERT-v1 does not require a newly fine-tuned model.
With BERxiT, we can come up with PCEE-BERT-v2. Following BERxiT, PCEE-BERT-v2 fine-tunes the multi-exit BERT with a fully connected layer right after each transformer block designated to evaluate the confidence score  $C^{(M)}$  for early exiting at that layer.  $C^{(M)}$  is learned along with the training of intermediate classifiers. Note that PCEE-BERT-v2 can not reuse the fine-tuned checkpoints used in PCEE-BERT and requires one to fine-tune the BERT backbones on the task at hand.
We conduct the experiments on the QNLI tasks, and the results are reported in Figure 4. We can see that PCEE-BERT-v1 and PCEE-BERT-v2 perform comparably to PCEE-BERT. The results show that the proposed PCEE-BERT early exiting mechanism is off-the-shelf, and the reason for the success of our PCEE-BERT is its early exiting mechanism,
<sup>1</sup>The score-speedup curves for the other five GLUE tasks can be found in the appendix.
See the Appendix for demonstration on MRPC.
(a) ALBERT base as backbone
(b)  $\mathrm{TinyBERT}_6$  as backbone
Figure 5: Ablation study on alternative PLMs.
Figure 6: Results for ablation study of whether PCEE-BERT should apply the cross-layer ensemble.
that is, early exit if a group of consecutive exits is confident for their predictions.
4.6.2 Ablation of PLM backbones
In the main experiments, we use BERT as the pretrained backbone model. However, PCEE-BERT can also work with the other types of pre-trained backbones, such as ALBERT base (Lan et al., 2020) and TinyBERT $_6$  (Jiao et al., 2020b). We conduct the experiments on the QNLI task with these two backbone models, and results are presented in Figure 5(a) and 5(b). We can see that when using the other pre-trained backbones, PCEE-BERT also performs better than the baseline methods.
The results for PCEE-BERT on the TinyBERT also convey an important message: as an inference speedup method, our PCEE method can work alongside the model compression methods to further reduce the latency of BERT.
4.6.3 Ablation of cross-layer ensemble
Since we have a prediction module at each layer of BERT, we can conduct model ensemble across layers that the forward pass has gone through already. In Figure 6, we conduct the ablation studies on the RTE and QNLI tasks. According to Figure 6, cross-layer ensemble leads to performance degradation when the speedup ratio is large, while when the average inference layers is close to the number of BERT's transformer blocks  $M$ , cross-layer ensemble results in slight improvements. In conclusion, the cross-layer ensemble does not result in consistent performance improvements.
A possible application of the above results is to apply the cross-layer ensemble when a low speedup ratio is applied. And when we ask the model to exit early in the shallow layers, the cross-layer ensemble is not used.
4.6.4 PCEE-BERT are effective for image classification
Our main experiments are conducted on BERT, a pre-trained language model, and the GLUE benchmark, a series of natural language understanding tasks. However, our PCEE-BERT method is a plug-and-play early exiting and can be applied to models and tasks of different modalities. To demonstrate the effectiveness of PCEE-BERT on the image classification task, we follow the experimental settings in PABEE (Zhou et al., 2020). We conduct experiments on two image classification datasets, CIFAR-10 and CIFAR-100 (Krizhevsky, 2009). The ResNet-56 model (He et al., 2016) serves as the backbone, and we compare PCEE-BERT with PABEE. We place an exiting classifier
Method	CIFAR-10	CIFAR-100
speed-up	Acc.	speed-up	Acc.
ResNet-56	0.0	91.8	0.0	68.6
PABEE	77%	78.3	76%	51.2
52%	86.7	48%	62.5
26%	91.9	24%	69.2
PCEE-BERT	76%	81.2	74%	55.6
51%	87.3	49%	64.8
25%	92.1	24%	69.4
Table 3: Experimental results of PCEE-BERT when applied in the image classification tasks.
at every two convolutional layers. We set the batch size to 128 and use an SGD optimizer with a learning rate of 0.1.
Table 3 reports the results. PCEE-BERT outperforms PABEE when early exiting at different speedup ratios. In addition, the performance advantages of PCEE-BERT are larger when the speedup ratio is large, which is also observed in the NLP tasks. And PCEE-BERT outperforms the original ResNet-56 on both tasks even when it provides around  $25\%$  speedup.
In this work, we propose PCEE-BERT, a novel efficient inference method that can yield a better performance-speed trade-off than the existing early exiting methods. PCEE-BERT adopts BERT as the backbone model and makes the exiting decision if there are enough intermediate layers to make confident predictions. The confidence level is measured by the entropy of the predicted distributions. Experiments on the GLUE benchmark demonstrate that our method outperforms the previous SOTA early exiting methods, especially when the speedup ratio is large. In addition, PCEE-BERT can achieve different speedup ratios by adjusting the patience parameter and the confidence threshold, which makes it more flexible in industrial usage. Ablation studies show that: (a) our PCEE-BERT can adopt different confidence measures, such as maximum probability mass; (b) our method performs consistently well on different PLMs and can work together with model compression methods to speed up the BERT's inference; (c) our PCEE-BERT also performs well on computer vision tasks.
This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC
(Information Technology Research Center) support program (IITP-2021-0-02051) supervised by the IITP (Institute for Information Communications Technology Planning Evaluation) and the BK21 FOUR program of the National Research Foundation of Korea funded by the Ministry of Education (NRF5199991014091).
Haoli Bai, Wei Zhang, L. Hou, L. Shang, Jing Jin, X. Jiang, Qun Liu, Michael R. Lyu, and Irwin King. 2020. Binarybert: Pushing the limit of bert quantization. ArXiv, abs/2012.15701.
Tolga Bolukbasi, J. Wang, O. Dekel, and Venkatesh Saligrama. 2017. Adaptive neural networks for efficient inference. In ICML.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. ArXiv, abs/1910.10073.
Angela Fan, E. Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. *ArXiv*, abs/1909.11556.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Gao Huang, Danlu Chen, T. Li, Felix Wu, L. V. D. Maaten, and Kilian Q. Weinberger. 2017. Multi-scale dense convolutional networks for efficient prediction. ArXiv, abs/1703.09844.
Heinrich Jiang, Been Kim, and Maya R. Gupta. 2018. To trust or not to trust a classifier. In NeurIPS.
Xiaoqi Jiao, Y. Yin, L. Shang, Xin Jiang, X. Chen, Linlin Li, F. Wang, and Qun Liu. 2020a. Tinybert: Distilling bert for natural language understanding. ArXiv, abs/1909.10351.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020b.
TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
Y. Kaya, Sanghyun Hong, and T. Dumitras. 2019. Shallow-deep networks: Understanding and mitigating network overthinking. In ICML.
Se-Hoon Kim, Amir Gholami, Zhewei Yao, M. W. Mahoney, and K. Keutzer. 2021. I-bert: Integer-only bert quantization. ArXiv, abs/2101.01321.
A. Krizhevsky. 2009. Learning multiple layers of features from tiny images.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942.
Weijie Liu, P. Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Q. Ju. 2020. Fastbert: a self-distilling bert with adaptive inference time. ArXiv, abs/2004.02178.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In NeurIPS.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. ArXiv, abs/2003.08271.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. *ArXiv*, abs/1910.01108.
Roy Schwartz, Gabi Stanovsky, Swabha Swayamdipta, Jesse Dodge, and N. A. Smith. 2020a. The right tool for the job: Matching model and instance complexities. In ACL.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020b. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640-6651, Online. Association for Computational Linguistics.
S. Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression. In EMNLP/IJCNLP.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, D. Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. CoRR, abs/1312.6199.
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464-2469.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Black-boxNLP@EMNLP.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
J. Xin, Raphael Tang, J. Lee, Y. Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. ArXiv, abs/2004.12993.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. BERxiT: Early exiting for BERT with better fine-tuning and extension to regression. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 91-104, Online. Association for Computational Linguistics.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and M. Zhou. 2020. Bert-of-theseus: Compressing bert by progressive module replacing. In EMNLP.
Z. Yang, Zihang Dai, Yiming Yang, J. Carbonell, R. Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
W. Zhang, L. Hou, Y. Yin, L. Shang, X. Chen, X. Jiang, and Qun Liu. 2020. Ternarybert: Distillation-aware ultra-low bit bert. ArXiv, abs/2009.12812.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. ArXiv, abs/2006.04152.
M. Zhu and S. Gupta. 2018. To prune, or not to prune: exploring the efficacy of pruning for model compression. ArXiv, abs/1710.01878.
Wei Zhu, Xiaoling Wang, Yuan Ni, and Guotong Xie. 2021. GAML-BERT: Improving BERT early exiting by gradient aligned mutual learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3033-3044, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A.1 Quality-efficiency trade-offs on GLUE benchmark tasks.
In the main content, we present the quality-efficiency trade-offs curves for four GLUE tasks. And here we put the results of the other three tasks in Figure 7.
A.2 Demonstrating PCEE-BERT can cover (or achieve values close to) any speedup ratios
PCEE-BERT's speedup ratio can be conveniently adjusted by setting different values for the patience parameter and the confidence threshold. To validate our claim, we alternate the threshold among 100 points between 0.0 to 1.0 when the patience parameter takes the value of 1, 2, 3, 6. The average numbers of inference layers are reported in the scatter plot (Figure 8). We can see that by adjusting the threshold and the patience parameter, one can easily control the average inference layers and cover (or achieve values close to) any speedup ratios.
(a) CoLA task
(b) RTE task
(c) SST-2 task
Figure 7: Performance-efficiency trade-offs using different exiting strategies. We can see that our PCEE-BERT consistently outperforms the strong baseline, PABEE, especially when the speed-up ratio is large.
Figure 8: This figure demonstrates that PCEE-BERT can cover (or achieve values close to) any speedup ratios.