SlowGuess/NAACL_2022_OCR huggingface dataset:
First, compare the Original with the Naive KD Finetuned of each model respectively. It's very interesting that the accuracy of the student mPLM is consistently higher than its teacher. The XLM-RoBERTa students gain  $2.8\%$ ,  $2.4\%$  and  $2.0\%$  improvement from the Original teachers as Transformer, BiLSTM and CNN respectively. The phenomenon implies the general effectiveness of language modeling of mPLMs. We conjecture the improvement comes from two aspects: 1) mPLMs'
Models	Original en	Finetuned en	Zero-Shot Cross-Lingual Transfer
de	es	fr	hi	ja	pt	tr	zh	Avg
Transformer	XLM-RoBERTa	94.4	97.2	96.9	96.8	96.2	90.8	90.1	95.6	84.3	91.3	92.8
mBERT	96.9	88.4	92.4	93.8	81.1	85.7	94.0	73.7	83.2	86.5
BiLSTM	XLM-RoBERTa	90.8	93.2	93.5	93.5	93.2	90.4	83.4	93.5	77.2	85.0	88.7
mBERT	92.3	80.4	87.5	82.5	79.2	79.4	82.3	76.5	75.0	80.3
CNN	XLM-RoBERTa	92.7	94.7	91.8	94.1	93.2	90.3	90.9	94.2	83.5	90.3	91.0
mBERT	93.3	82.6	86.9	87.9	78.1	78.7	88.2	72.9	80.9	82.0
Table 5: Classification accuracy of XLM-RoBERTa and mBERT. Step-1 KD: off-the-shelf English model -> mPLM. The mPLMs are finetuned and evaluated on MultiATIS++.
Models	Original en	Transferred en	Transferred Target Languages
de	es	fr	hi	ja	pt	tr	zh	Avg	Δ
Transformer	XLM-RoBERTa	94.4	94.0	94.4	93.3	90.8	85.7	81.3	92.4	76.5	87.1	87.7	-5.1
mBERT	95.6	86.2	91.8	92.8	79.8	81.7	90.9	72.9	80.3	84.6	-1.9
BiLSTM	XLM-RoBERTa	90.8	89.8	89.8	89.7	89.5	84.1	78.1	84.9	71.5	83.1	83.8	-4.9
mBERT	89.6	80.9	82.9	81.0	78.5	76.0	83.7	71.7	78.6	79.2	-1.1
CNN	XLM-RoBERTa	92.7	90.7	87.9	88.1	87.8	85.0	83.2	86.3	75.2	86.7	85.0	-6.0
mBERT	89.1	79.4	82.9	81.5	76.0	78.2	82.4	72.0	79.5	79.0	-3.0
Table 6: Classification accuracy of the target models, distilled from XLM-RoBERTa and mBERT respectively. Step-2 KD: mPLM -> target model.  $\Delta$ : changes w.r.t Table 5. The target models are transferred and evaluated on MultiATIS++.
generalizability learn from the large-scale pretraining, 2) the large model scale of mPLMs, which enhances its NLU capability. Besides, the improvement with respect to the Original varies across model architectures. Especially when compare Transformer  $(+2.8\%)$  to CNN  $(+2.0\%)$ , although the Transformer's student XLM-RoBERTa performs much closer to the Gold-supervised, it still improves greater than the CNN's student. Since the XLM-RoBERTa is Transformer-based network, it implies that the knowledge distillation performs better if the architectures of the teacher and the student are more similar.
Second, under the cross-lingual transfer condition, although the Gold-supervised outperforms the Naive KD on most target languages, it performs weaker on Turkish (tr). It demonstrates the better generalizability and few-shot performance of the Naive KD, since Turkish is a low-resource language in MultiATIS++. The number of training examples of Turkish (578) is less than other languages (4488).
Third, the effectiveness of the proposed language balanced distillation is very clear. In the comparison between the Naive KD and + Balanced distillation, the accuracy is highly boosted almost on all the target languages. This improvement is particularly significant on the languages that is not in the same family of English: Hindi (hi), Japanese (ja), Turkish (tr) and Chinese (zh). A future research topic is to improve language balanced dis
tillation on the languages similar to the source language, e.g. European languages to English. Data selection algorithms may have potentials.
In sum, the proposed framework and distillation method is effective and of strong generalizability. Future researches on heterogeneous distillation and data selection may benefit the proposed framework.
4.2 Effects of mPLM Models
Table 5 and Table 6 respectively reports accuracy of the step-1 KD and step-2 KD in the proposed framework. According to Table 6, the choice of mPLM is critical to the target models' performance. Performance with XLM-RoBERTa as the mPLM is stronger than with mBERT. However, there are interesting observations we should notice.
First, observe the performance changes  $(\Delta)$  of the Step-2 KD: from the mPLM teacher to the target model student. We notice that the performance drop of mBERT is slighter than the XLM-RoBERTa's, based on the results of the average score in Table 6 minus those in Table 5. It implies that as the capability of mPLM increases, the KD dissipation tends to increase as well. Similar to the analysis in Section 4.1, the KD dissipation may come from: 1) the pre-trained language model that the target models lack of, 2) discrepancy between the model size of the mPLM and the target models. Hence, performance based on XLM-RoBERTa drops more due to its gap to the target models is greater than mBERT's in both the two aspects of
discrepancy. To reduce the KD dissipation, researches should focus on how to reduce the model discrepancy between mPLM and the target model, e.g. improve the language modeling capability of the target model. Besides, the performance difference among model architectures is consistent, either based on XLM-RoBERTa or mBERT. It further evidences that the proposed framework is general and works well for different model architectures.
4.3 Cross-Architecture Transfer
To analyze the proposed framework in a more general setting, we free the architecture ties of the off-the-shelf English models and the target models to be heterogeneous, that the source and target models can be different.
Figure 4: Classification accuracy of the target models via cross-architecture transfer, averaged over all target languages. Transfer from rows to columns. Row: architectures of source English models, Column: architectures of the target models. Experimented on MultiATIS++.
As depicted in Figure 4, the transfer performs the best when taking Transformer as both the source and target models. The worst comes to the transfer between BiLSTM models. On one side, the advantage of the Transformer architecture may be a reason. On the other side, it reconfirms the observation that the more similar teacher and student models are, the better transfer performance comes.
Besides, taking the BiLSTM as the source or target model consistently result in lower accuracy, no matter what the corresponding target or source models are. Hence, we guess the architecture similarity between BiLSTM and the Transformer-based mPLM is lower than that between CNN and the mPLM. We leave this for future work.
In addition, we study the accuracy drop from the source English models to the target models, as shown in Figure 5. From the perspective of the source model, the drop is the least when BiLSTM
Figure 5: Accuracy drop from the source English models to the target models, averaged over all target languages. Transfer from rows to columns. Row: architectures of source English models, Column: architectures of the target models. Experimented on MultiATIS++.
is the source. From the perspective of the target model, the drop is the least when Transformer is the target. It reveals an asymmetry between the two KD steps with respect to the mPLM. To reduce the KD dissipation to the largest extent, it seems mPLM should be distilled from a weaker teacher architecture (e.g. BiLSTM) before teaching a stronger student architecture (e.g. Transformer).
In brief, the proposed framework works for heterogeneous cross-lingual transfer. The future work may focus on how to define the similarity between model architectures and how to evaluate the source-target model pairs.
In this paper, we define a novel cross-lingual transfer (CLT) problem - FreeTransfer-X, especially for CLT in private scenarios such as medical and business. The FreeTransfer-X is defined to transfer knowledge from off-the-shelf models in rich-resource languages to poor-resource languages, without labeled corpora. To address the FreeTransfer-X, we propose a 2-step knowledge distillation (2-step KD) framework based on multilingual pre-trained language models. In addition, two data augmentation methods for cross-lingual KD are proposed to boost the performance of the 2-step KD framework. Experimental results clearly demonstrate the effectiveness of the proposed framework. It's worth noting that the proposed KD framework can be applied between heterogeneous models, which benefits the deployment in different environment. Further analyses point out various research directions for future work.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. Multilingual bert.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1-48.
Yinpeng Guo, Yi Liao, Xin Jiang, Qing Zhang, Yibo Zhang, and Qun Liu. 2019. Zero-shot paraphrase generation with multilingual language models. CoRR, abs/1911.03597.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
James M. Joyce. 2011. Kullback-Leibler Divergence, pages 720-722. Springer Berlin Heidelberg, Berlin, Heidelberg.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2950-2962, Online. Association for Computational Linguistics.
Qianchu Liu, Diana McCarthy, Ivan Vulic, and Anna Korhonen. 2019. Investigating cross-lingual alignment methods for contextualized embeddings with token-level evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 33–43, Hong Kong, China. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
M. Schuster and K.K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Leslie N. Smith. 2015. No more pesky learning rate guessing games. CoRR, abs/1506.01186.
Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2670-2680, Copenhagen, Denmark. Association for Computational Linguistics.
Shyam Upadhyay, Manaal Faruquli, Gokhan Tur, Hakkani-Tur Dilek, and Larry Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6034-6038.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime G. Carbonell. 2020. Cross-lingual alignment vs joint training: A comparative study and a simple unified framework. In International Conference on Learning Representations.
Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for crosslingual NLU. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5052-5063, Online. Association for Computational Linguistics.
A Language-Wise Results
Here we list the detailed language-wise experimental results of Table 1 and Table 3 for reference.
Models	Source en	Targets
en	de	es	fr	hi	ja	pt	tr	zh	Avg
Reference	Gold-supervised	-	88.5	89.7	89.6	91.9	81.9	79.5	86.3	73.4	89.1	85.2
Transformer	Baselines	Translate-test	94.4	92.5	90.0	88.1	90.6	83.7	86.8	88.1	75.1	88.7	86.4
Translate-train-pseudo	92.7	89.4	90.0	90.5	83.1	74.8	90.1	74.1	85.9	84.7
Ours	2-step KD	94.0	94.4	93.3	90.8	85.7	81.3	92.4	76.5	87.1	87.7
+ Balanced distillation	94.3	93.5	92.9	95.0	84.0	83.0	93.3	78.5	90.7	88.9
+ Data augmentation	94.7	93.6	93.3	94.7	84.2	83.7	93.2	77.6	89.7	88.7
BiLSTM	Baselines	Translate-test	94.4	87.1	84.2	83.4	85.8	77.9	81.5	84.4	64.1	84.2	80.7
Translate-train-pseudo	87.8	85.0	85.4	86.9	80.4	73.5	86.1	72.0	80.4	81.2
Ours	2-step KD	89.8	89.8	89.7	89.5	84.1	78.1	84.9	71.5	83.1	83.8
+ Balanced distillation	90.7	89.4	88.7	88.6	82.5	81.7	86.1	76.2	88.6	85.2
+ Data augmentation	89.1	90.6	90.9	88.7	83.4	82.5	86.7	77.9	90.3	86.4
CNN	Baselines	Translate-test	94.4	90.7	86.6	86.3	88.7	86.9	85.3	88.2	81.0	86.3	86.2
Translate-train-pseudo	86.3	84.4	85.6	86.5	83.8	79.8	82.2	77.5	85.9	83.2
Ours	2-step KD	90.7	87.9	88.1	87.8	85.0	83.2	86.3	75.2	86.7	85.0
+ Balanced distillation	89.1	88.5	87.8	88.8	85.7	83.0	86.2	79.2	90.6	86.2
+ Data augmentation	89.6	89.8	89.2	89.0	86.0	83.8	87.1	79.4	90.9	86.9
Table 7: Sentence classification accuracy on MultiATIS++.