SlowGuess/ACL_2020_OCR huggingface dataset:
Results: From Table 3, we observe all strategies that share information across languages to work better (D0-S vs. D0) with soft logits adding more value than hard targets (D1 vs. D0-S). Interestingly, we observe simply combining representation loss with logits (D3.1 vs. D2) hurts the model. We observe this strategy to be vulnerable to the hyperparameters  $(\alpha, \beta, \gamma$  in Eqn. 7) used to combine multiple loss functions. We vary hyper-parameters in multiples of 10 and report best numbers.
Stage-wise optimizations remove these hyperparameters and improve performance. We also observe the gradual unfreezing scheme to improve both stage-wise distillation strategies significantly.
Table 3: Comparison of several strategies with average  $F_{1}$ -score (and standard deviation) across 41 languages over different transfer data size.  $S_{i}$  depicts separate stages and corresponding optimized loss functions.
Stage	Unfreezing Layer	F1	Std. Dev.
2	Linear (⟨Wr, br⟩)	0	0
2	Projection (⟨Wf, bf⟩)	2.85	3.9
2	BiLSTM (θb)	81.64	5.2
2	Word Emb (θw)	85.99	4.4
3	Softmax (Ws)	86.38	4.2
3	Projection (⟨Wf, bf⟩)	87.65	3.9
3	BiLSTM (θb)	88.08	3.9
3	Word Emb (θw)	88.64	3.8
Table 4: Gradual  $F_{1}$ -score improvement over multiple distillation stages in XtremeDistil.
Model	F1	Std. Dev.
mBERT-single (Devlin et al., 2019)	90.76	3.1
mBERT (Devlin et al., 2019)	91.86	2.7
MMNER (Rahimi et al., 2019)	89.20	2.8
XtremeDistil (ours)	88.64	3.8
Table 5:  $F_{1}$ -score comparison of different models with standard deviation across 41 languages.
Focusing on the data dimension, we observe all models to improve as more and more unlabeled data is used for transferring teacher knowledge to student. However, we also observe the improvement to slow down after a point where additional unlabeled data does not yield significant benefits. Table 4 shows the gradual performance improvement in XtremeDistil after every stage and unfreezing various neural network layers.
6.2 Performance, Compression and Speedup
Performance: We observe XtremeDistil in Table 5 to perform competitively with other models. mBERT-single models are fine-tuned per language with corresponding labels, whereas mBERT is fine-tuned with data across all languages. MMNER results are reported from Rahimi et al. (2019).
Figure 2 shows the variation in  $F_{1}$ -score across different languages with variable amount of training data for different models. We observe all the models to follow the general trend with some aber
(a) Parameter compression vs.  $F_{1}$ -score.
(b) Inference speedup vs.  $F_{1}$ -score.
Figure 1: Variation in XtremeDistil  $F_{1}$ -score with parameter and latency compression against mBERT. Each point in the linked scatter plots depict a setting with corresponding embedding dimension and BiLSTM hidden states as  $(E, H)$ . Data point (50, 200) in both figures correspond to  $35x$  compression and  $51x$  speedup.
rations for languages with less training labels.
Parameter compression: XtremeDistil performs at par with MMNER in terms of  $F_{1}$ -score while obtaining at least  $41x$  compression. Given  $L$  languages, MMNER learns  $(L - 1)$  ensembled and distilled models, one for each target language. Each of the MMNER language-specific models is comparable in size to our single multilingual model. We learn a single model for all languages, thereby, obtaining a compression factor of at least  $L = 41$ .
Figure 1a shows the variation in  $F_{1}$ -scores of XtremeDistil and compression against mBERT with different configurations corresponding to the embedding dimension  $(E)$  and number of BiLSTM hidden states  $(2 \times H)$ . We observe that reducing the embedding dimension leads to great compression with minimal performance loss. Whereas, reducing the BiLSTM hidden states impacts the performance more and contributes less to the compression.
Inference speedup: We compare the runtime inference efficiency of mBERT and our model in a single P100 GPU for batch inference (batch size  $= 32$ ) on 1000 queries of sequence length 32. We average the time taken for predicting labels for all the queries for each model aggregated over 100 runs. Compared to batch inference, the speedups are less for online inference (batch size  $= 1$ ) at  $17x$  on Intel(R) Xeon(R) CPU (E5-2690 v4 @2.60GHz) (refer to Appendix for details).
Model	#Transfer Samples	F1
MMNER	-	62.1
mBERT	-	79.54
XtremeDistil	4.1K	19.12
705K	76.97
1.3MM	77.17
7.2MM	77.26
Table 6:  $F_{1}$ -score comparison for low-resource setting with 100 labeled samples per language and transfer set of different sizes for XtremeDistil.
Figure 1b shows the variation in  $F_{1}$ -scores of XtremeDistil and inference speedup against mBERT with different (linked) parameter configurations as before. As expected, the performance degrades with gradual speedup. We observe that parameter compression does not necessarily lead to an inference speedup. Reduction in the word embedding dimension leads to massive model compression, however, it does not have a similar effect on the latency. The BiLSTM hidden states, on the other hand, constitute the real latency bottleneck. One of the best configurations leads to  $35x$  compression,  $51x$  speedup over mBERT retaining nearly  $95\%$  of its performance.
6.3 Low-resource NER and Distillation
Models in all prior experiments are trained on  $705K$  labeled instances across all languages. In this setting, we consider only 100 labeled samples for each language with a total of  $4.1K$  instances. From Table 6, we observe mBERT to outperform MMNER by more than 17 percentage points with XtremeDistil closely following suit.
Furthermore, we observe our model's performance to improve with the transfer set size depicting the importance of unlabeled transfer data for knowledge distillation. As before, a lot of additional data has marginal contribution.
6.4 Word Embeddings
From Table 7 we observe randomly initialized word embeddings to work quite well. Multilingual FastText embeddings (Bojanowski et al., 2016) lead to minor improvement due to  $38\%$  overlap between FastText tokens and mBERT wordpieces. English Glove does much better. We experiment with dimensionality reduction techniques and find SVD to work better leading to marginal improvement over mBERT embeddings before reduction. As expected, fine-tuned mBERT embeddings perform better than that from pre-trained checkpoints.
Figure 2:  $F_{1}$ -score comparison for different models across 41 languages. The y-axis on the left shows the scores, whereas the axis on the right (plotted against blue dots) shows the number of training labels (in thousands).
Word Embedding	F1	Std. Dev.
SVD + mBERT (fine-tuned)	88.64	3.8
mBERT (fine-tuned)	88.60	3.9
SVD + mBERT (pre-trained)	88.54	3.9
PCA + PPA (d=14) (Raunak et al., 2019)	88.35	3.9
PCA + PPA (d=17) (Raunak et al., 2019)	88.25	4.0
Glove (Pennington et al., 2014)	88.16	4.0
FastText (Bojanowski et al., 2016)	87.91	3.9
Random	87.43	4.1
6.5 Architectural Considerations
Which teacher layer to distil from? The topmost teacher layer captures more task-specific knowledge. However, it may be difficult for a shallow student to capture this knowledge given its limited capacity. On the other hand, the less-deep representations at the middle of teacher model are easier to mimic by shallow student. From Table 8 we observe the student to benefit most from distilling the  $6^{th}$  or  $7^{th}$  layer of the teacher.
Table 7: Impact of using various word embeddings for initialization on multilingual distillation. SVD, PCA, FastText and Glove use 300-dim. word embeddings.
Layer (l)	F1-score	Std. Dev.	Layer (l)	F1-score	Std. Dev.
11	88.46	3.8	4	88.19	4
9	88.31	3.8	2	88.50	4
7	88.64	3.8	1	88.51	4
6	88.64	3.8
Table 8: Comparison of XtremeDistil performance on distilling representations from  $l^{th}$  mBERT layer.
Comparison of student architecture. Recent works leverage both BiLSTM and Transformer as students. In this experiment, we vary the embedding dimension and hidden states for BiLSTM-, and embedding dimension and depth for Transformer-based students to obtain configurations with similar inference latency. Each of 13 configurations in Figure 3 depict  $F_{1}$ -scores obtained
Figure 3: BiLSTM and Transformer  $F_{1}$ -score (left y-axis) vs. inference latency (right y-axis) in 13 different settings with corresponding embedding dimension and width / depth of the student as  $(E, W / D)$ .
by students of different architecture but similar latency (refer to Table 15 in Appendix for statistics) – for strategy D0-S in Table 3. We observe that for low-latency configurations BiLSTMs with hidden states  $\{2 \times 100, 2 \times 200\}$  work better than 2-layer Transformers. Whereas, the latter starts performing better with more than 3-layers although with a higher latency compared to the aforementioned BiLSTM configurations.
6.6 Distillation for Text Classification
We switch gear and focus on classification tasks. In contrast to sequence tagging, we use the last hidden state of the BiLSTM as the final sentence representation for projection, regression and softmax.
Table 9 shows the distillation performance of XtremeDistil with different teachers on four benchmark text classification datasets. We observe the student to almost match the teacher performance for all of the datasets. The performance also improves with a better teacher, although the improvement is marginal as the student capacity saturates.
Table 10 shows the distillation performance with only 500 labeled samples per class. The distilled student improves over the non-distilled version by 19.4 percent and matches the teacher performance for all of the tasks demonstrating the impact of distillation for low-resource settings.
Data	Student no distil.	Distil (Base)	Distil (Large)	BERT (Base)	BERT (Large)
AG	89.71	92.33	94.33	92.12	94.63
IMDB	89.37	91.22	91.70	91.70	93.22
Elec	90.62	93.55	93.56	93.46	94.27
DB	98.64	99.10	99.06	99.26	99.20
Table 9: Distillation performance with BERT.
Dataset	Student no distil.	Student with distil.	BERT Large
AG News	85.85	90.45	90.36
IMDB	61.53	89.08	89.11
Elec	65.68	91.00	90.41
DBpedia	96.30	98.94	98.94
Table 10: Distillation with BERT Large on 500 labeled samples per class.
Comparison with other distillation techniques: SST-2 (Socher et al., 2013) from GLUE (Wang et al., 2018) has been used as a test bed for other distillation techniques for single instance classification tasks (as in this work). Table 11 shows the accuracy comparison of such methods reported in SST-2 development set with the same teacher.
We extract 11.7MM sentences from all IMDB movie reviews in Table 1 to form the unlabeled transfer set for distillation. We obtain the best performance on distilling with BERT Large (uncased, whole word masking model) than BERT Base - demonstrating a better student performance with a better teacher and outperforming other methods.
Teacher hidden representation and distillation schedule: Internal teacher representations help in distillation, although a naive combination hurts the student model. We show that a distillation schedule with stagewise optimization, gradual unfreezing with a cosine learning rate scheduler (D4.1 + D4.2 in Table 3) obtains the best performance. We also show that the middle layers of the teacher are easier to distil by shallow students and result in the best performance (Table 8). Additionally, the student performance improves with bigger and better teachers (Tables 9 and 11).
Model	Transfer Set	Acc.
BERT Large Teacher	-	94.95
XtremeDistil	SST+Imdb	93.35
BERT Base Teacher	-	92.78
XtremeDistil	SST+Imdb	92.89
Sun et al. (2019)	SST	92.70
Turc et al. (2019)	SST+IMDB	91.10
Table 11: Model accuracy on of SST-2 (dev. set).
Student architecture: We compare different student architectures like BiLSTM and Transformer in terms of configuration and performance (Figure 3, Table 15 in Appendix), and observe BiLSTM to perform better at low-latency configurations, whereas the Transformer outperforms the former with more depth and higher latency budget.
Unlabeled transfer data: We explored data dimension in Tables 3 and 6 and observed unlabeled data to be the key for knowledge transfer from pretrained teachers to shallow students and bridge the performance gap. We observed a moderate amount of unlabeled transfer samples (0.7-1.5 MM) lead to the best student, whereas larger amounts of transfer data does not result in significant gains. This is particularly helpful for low-resource NER (with only 100 labeled samples per language as in Table 6).
Performance trade-off: Parameter compression does not necessarily reduce inference latency, and vice versa. We explored model performance with parameter compression, inference latency and  $F_{1}$  to show trade-off in Fig. 1 and Table 16 in Appendix.
Multilingual word embeddings: Random initialization of word embeddings work well. A better initialization, which is also parameter-efficient, is given by Singular Value Decomposition (SVD) over fine-tuned mBERT word embeddings with the best performance for downstream task (Table 7).
Generalization: The outlined distillation techniques and strategies are model-, architecture-, and language-agnostic and can be easily extended to arbitrary tasks and languages, although we only focus on NER and classification in this work.
Massive compression: Our techniques demonstrate massive compression (35x for parameters) and inference speedup (51x for latency) while retaining 95% of the teacher performance allowing deep pre-trained models to be deployed in practice.
We develop XtremeDistil for massive multi-lingual NER and classification that performs close to huge pre-trained models like MBERT but with massive compression and inference speedup. Our distillation strategy leveraging teacher representations agnostic of its architecture and stage-wise optimization schedule outperforms existing ones. We perform extensive study of several distillation dimensions like the impact of unlabeled transfer set, embeddings and student architectures, and make interesting observations outlined in summary.
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Edward Guo. 2019. Knowledge distillation from internal representations.
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2654-2662.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017. A survey of model compression and acceleration for deep neural networks. CoRR, abs/1710.09282.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019. Bam! born-again multi-task networks for natural language understanding. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. 2014. Compressing deep convolutional networks using vector quantization. CoRR, abs/1412.6115.
Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. *ICLR*.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328-339.
Peter Izsak, Shira Guskin, and Moshe Wasserblat. 2019. Training compact models for low resource entity tagging using pre-trained language models.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. CoRR, abs/1904.09482.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 2011, Portland, Oregon, USA, pages 142-150.
Julian J. McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In *Seventh ACM Conference on Recommender Systems*, RecSys '13, Hong Kong, China, October 12-16, 2013, pages 165-172.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958, Vancouver, Canada. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar, A meeting of SIG-DAT, a Special Interest Group of the ACL, pages 1532-1543.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy. Association for Computational Linguistics.
Vikas Raunak, Vivek Gupta, and Florian Metze. 2019. Effective dimensionality reduction for word embeddings. Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019).
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In 3rd International Conference on Learning Representations, ICLR2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Victor Sanh. 2019. Introducing distilbert, a distilled version of bert. https://medium.com/huggingface/distilbert-8cf3380435b5.
Yangyang Shi, Mei-Yuh Hwang, Xin Lei, and Haoyu Sheng. 2019. Knowledge distillation for recurrent neural network language modeling with trust regularization. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Mohammad Shoeybi, Mostofa Ali Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. ArXiv, abs/1909.08053.
Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455-465, Sofia, Bulgaria. Association for Computational Linguistics.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903.12136.
Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical bert models for sequence labeling. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019. Extreme language model compression with optimal subwords and shared projections.
Wei Zhu, Xiaofeng Zhou, Keqiang Wang, Xun Luo, Xiepeng Li, Yuan Ni, and Guotong Xie. 2019. PANLP at MEDIQA 2019: Pre-trained language models, transfer learning and knowledge distillation. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 380-388, Florence, Italy. Association for Computational Linguistics.
XtremeDistil uses Tensorflow. Code and resources available at: https://aka.ms/XtremeDistil.
A.2 Parameter Configurations
All the analyses in the paper — except compression and speedup experiments that vary embedding dimension  $E$  and BiLSTM hidden states  $H$  — are done with the following model configuration in Table 12 with the best  $F_{1}$ -score. Optimizer Adam is used with cosine learning rate scheduler (lr_high = 0.001, lr_low = 1e-8).
The model corresponding to the  $35x$  parameter compression and  $51x$  speedup for batch inference uses  $E = 50$  and  $H = 2 \times 200$ .
Parameter	Value
SVD + MBERT word emb. dim.	E=300
BiLSTM hidden states	H=2×600
Dropout	0.2
Batch size	512
Teacher layer	7
Optimizer	Adam
Following hyper-parameter tuning was done to select dropout rate and batch size.
Table 12: XtremeDistil config. with best  $F_{1} = 88.64$
Dropout Rate	F1-score
1e-4	87.94
0.1	88.36
0.2	88.49
0.3	88.46
0.6	87.26
0.8	85.49
Table 13: Impact of dropout.
Batch size	F1-score
128	87.96
512	88.4
1024	88.24
2048	88.13
4096	87.63
Table 14: Impact of batch size.
BiLSTM	Transformer
Emb. Dim.	Hidden States	F1-Score	Params (MM)	Latency	Emb. Dim.	Depth	Params (MM)	Latency	F1-Score
50	100	80.26	4.7	0.311	48	2	4.4	0.307	76.67
200	100	79.21	18.1	0.354	144	1	13.4	0.357	78.49
300	100	79.63	27	0.385	72	2	6.7	0.388	77.98
50	200	81.22	5.1	0.472	96	2	9	0.47	79.19
300	200	80.04	27.7	0.593	132	2	12.5	0.6	80
50	400	81.98	6.5	0.892	204	2	19.7	0.88	80.96
200	400	80.61	20.2	0.978	228	2	22.1	0.979	80.87
100	400	81.54	11.1	1	240	2	23.3	1.03	80.79
300	400	80.16	29.4	1.06	252	2	24.6	1.075	80.84
50	600	81.78	8.5	1.5	228	3	22.7	1.448	83.75
100	600	81.94	13.1	1.53	240	3	24	1.498	84.07
200	600	80.7	22.5	1.628	252	3	25.3	1.591	84.08
300	600	81.42	31.8	1.766	276	3	28	1.742	84.06