SlowGuess/NAACL_2022_OCR huggingface dataset:
FreeTransfer-X: Safe and Label-Free Cross-Linguual Transfer from Off-the-Shelf Models
Yinpeng Guo and Liangyou Li and Xin Jiang and Qun Liu  
Huawei Noah's Ark Lab
{guo.yinpeng, liliangyou, jiang.xin, qun.liu}@huawei.com
Cross-lingual transfer (CLT) is of various applications. However, labeled cross-lingual corpus is expensive or even inaccessible, especially in the fields where labels are private, such as diagnostic results of symptoms in medicine and user profiles in business. Although being lack of labels, there are off-the-shelf models in these sensitive fields. Instead of pursuing the original labels, a workaround for CLT is to transfer knowledge from the off-the-shelf models without labels. To this end, we define a novel CLT problem named FreeTransfer-X that aims to achieve knowledge transfer from the off-the-shelf models in rich-resource languages. To address the problem, we propose a 2-step knowledge distillation (KD, Hinton et al., 2015) framework based on multilingual pre-trained language models  $(\mathrm{mPLM})^{1}$ . The significant improvement over strong neural machine translation (NMT) baselines demonstrates the effectiveness of the proposed method. In addition to reducing annotation cost and protecting private labels, the proposed method is compatible with different networks and easy to be deployed. Finally, a range of analyses indicate the great potential of the proposed method.
Cross-lingual transfer (CLT) is a critical topic for natural language processing due to the data imbalance between languages. While models of rich-resource languages (e.g. English) have been applied on various real-world tasks, the progress on poor-resource languages lags behind. CLT researches enable the knowledge transfer from the rich-resource languages to the poor-resource languages.
Although the application of CLT is valuable, data labels are expensive or even inaccessible in
private and sensitive domains, such as medicine and business. For example, the diagnostic results of a user's symptoms are private and a company's internal description of users are confidential. Since short of labels for CLT, even though there are excellent applications in rich-resource languages, it is difficult to benefit the people using poor-resource languages. Previous CLT researches have not well studied how to leverage knowledge of rich-resource languages without labels. To define and tackle this problem will benefit both the community and the industry.
In order to reduce the demand of labels, existing works mainly fall into two paradigms as follows. One paradigm focuses on learning language-agnostic representation and model parameters. CLT is realized by either aligning parameters of monolingual models or sharing parameters among different languages (Liu et al., 2019; Devlin et al., 2019b; Conneau et al., 2020; Wang et al., 2020). The objective is to build a unified representation, which is used by downstream tasks, for all the languages. In this paradigm, although the demand of labels is reduced, it still requires a certain number of labels to adapt the model to a particular language and task. Besides, models in this paradigm are usually large-scale Transformers (Vaswani et al., 2017) based on mPLMs, which limits their deployment in real-world. Another paradigm is to leverage machine translation (MT) systems to generate training or testing pseudo-corpus for a specific language (Conneau et al., 2018). For simplicity, we take English as the rich-resource languages in this paper. Translate-train translates annotated training corpus from English to other languages. Gold labels are directly applied to the translated data. Although labels in poor-resource languages are not required, gold labels in English are still necessary. On the contrary, Translate-test translates testing corpus from poor-resource languages to English. This method can directly lever
$^{1}$ Source code are available at https://github.com/huawei-noah/noah-research/tree/master/NLP/FreeTransfer-X
Findings of the Association for Computational Linguistics: NAACL 2022, pages 229 - 240 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: Overview of the proposed 2-step knowledge distillation (KD) framework. KD-(1) distills knowledge from the off-the-shelf English model to the mPLM. KD-(2) distills knowledge from the mPLM to the model in the target language. Blue modules: in the source language src, green modules: in the target language tgt.
age off-the-shelf English models, but it runs a 2-pass inference which highly limits its efficiency. Both the two CLT paradigms mentioned above require language-specific and task-specific labels, except for the 2-pass Translate-test. The demand of labels highly limits the reuse of the English knowledge in private and sensitive domains. Then a question comes up: Is it possible to perform CLT totally without labels?
In this paper, we define a novel problem: safe and label-free cross-lingual transfer from off-the-shelf models (FreeTransfer-X). The FreeTransfer-X asks researchers to achieve CLT only with off-the-shelf English models but any labels, as formally defined in Section 2.1. To the best knowledge of the authors, it's the first time that the FreeTransfer-X is clearly defined.
To address the FreeTransfer-X, we propose a 2-step knowledge distillation (KD, Hinton et al., 2015) framework based on mPLM, as shown in Figure 1. Given an off-the-shelf model  $\theta_{src}$  in the source language (e.g. English), first we take  $\theta_{src}$  as the teacher and an mPLM model  $\theta_{src}^{m}$  as the student, then train  $\theta_{src}^{m}$  on unlabeled corpus  $\mathcal{D}_{src}$ . Second, we take  $\theta_{tgt}^{m}$  as the teacher and train a student  $\theta_{tgt}$  on unlabeled corpus  $\mathcal{D}_{tgt}$ . This cross-lingual transfer framework is label-free and applicable for any model architecture. Experimental results demonstrate the effectiveness of the proposed framework on both sentence classification and sequence tagging.
In short, the major contributions of this work include:
- A novel cross-lingual transfer problem FreeTransfer-X is defined. The FreeTransfer-X asks researchers to achieve CLT from off-
the-shelf models without using labels. It reduces the labeling cost and protects the labels in private domains such as medicine and business.
- We propose a 2-step knowledge distillation framework based on mPLMs, e.g. XLM-RoBERTa (Conneau et al., 2020), to address the FreeTransfer-X. It significantly outperforms the NMT baselines on sentence classification and sequence tagging tasks. Besides, it's compatible with heterogeneous networks.
Further analysis indicates abundant research potentials of the proposed framework. To improve the two distillation steps and the mPLM may benefit the framework.
Denote the source language and the target language as  $src$  and  $tgt$  respectively. Given an off-the-shelf model  $\theta_{src}$  (e.g. English intent classifier), unlabeled in-domain corpus  $\mathcal{D}_{src}$  and unlabeled indomain corpus  $\mathcal{D}_{tgt}$ , the objective is to output a model  $\theta_{tgt}$  in the target language  $tgt$ . For simplicity in this paper, we constrain the target model  $\theta_{tgt}$  to be of the same network architecture to the off-the-shelf source model  $\theta_{src}$ .
2.2 Basic Framework
We propose to adopt knowledge distillation (KD, Hinton et al., 2015) to address the FreeTransfer-X, since it can transfer knowledge from teacher models without knowing original labels. In addition, knowledge distillation is free from network architectures and can be applied between heterogeneous
networks, which benefits the deployment in various environment.
2.2.1 Two-Step Knowledge Distillation
For a specific natural language processing (NLP) task, given a model  $\theta_{src}$  and the unlabeled data  $\mathcal{D}_{src}$  in the source language src and the unlabeled data  $\mathcal{D}_{tgt}$  in the target language tgt. As shown in Figure 1, we propose to train a model  $\theta_{tgt}$  in the target language tgt via 2 KD steps:
1. Leverage the NLP capability of the off-the-shelf model  $\theta_{src}$ , e.g. an English sentence classifier  $\theta_{en,cls}$ . We distill knowledge from the teacher  $\theta_{src}$  to the student mPLM  $\theta_{src}^{m}$  on data  $\mathcal{D}_{src}$ .
2. Due to the zero-shot cross-lingual transfer capability of the mPLMs,  $\theta_{src}^{m}$  implicitly achieve the NLP capability on the target language  $\theta_{tgt}^{m}$ . Then similar to the step 1, we distill knowledge from the teacher  $\theta_{tgt}^{m}$  to the student  $\theta_{tgt}$  in the target language tgt on data  $\mathcal{D}_{tgt}$ .
The proposed framework works for arbitrary network including but not limited to Transformers (Vaswani et al., 2017), BiLSTM (Schuster and Paliwal, 1997) and CNN (Kim, 2014).
2.2.2 Training Objectives
The training is purely based on KD that no other training objectives is included. We only apply KD between the classification distribution  $P_{T}(\cdot)$  and  $P_{S}(\cdot)$  of the teacher and the student respectively, which is compatible to arbitrary model architecture. Freezing the parameters of the teacher, we train the student by minimizing the Kullback-Leibler Divergence  $(Div_{KL},\mathrm{Joyce},2011)$  between them. Denote the prediction category as  $\mathcal{C} = [c_0,c_1,\dots ,c_k]$ , then the  $Div_{KL}$  can be formalized as,
$$
\begin{array}{l} D i v _ {K L} \left(P _ {T} (\mathcal {C} | \cdot) \| P _ {S} (\mathcal {C} | \cdot) \right. \\ = \sum_ {c _ {i} \in \mathcal {C}} P _ {T} (c _ {i} | \cdot) \log \left(\frac {P _ {T} (c _ {i} | \cdot)}{P _ {\mathcal {S}} (c _ {i} | \cdot)}\right) \tag {1} \\ \end{array}
$$
However, KD objectives of different NLU tasks varies a lot. We classify NLU tasks into two categories: 1) sentence-level tasks like sentence classification, 2) word-level tasks like sequence tagging. Given an input example  $\mathcal{X} \in \mathcal{D}$  as a sequence of words  $\mathcal{X} = [x_0, x_1, \ldots, x_n]$ . For sentence-level tasks,  $\mathcal{X}$ 's sentence-level category is  $\mathcal{C}_{\mathcal{X}}$ . The teacher model and student model respectively output sentence-level prediction distribution
$P_{T}(\mathcal{C}_{\mathcal{X}}|\mathcal{X})$  and  $P_{S}(\mathcal{C}_{\mathcal{X}}|\mathcal{X})$ . For word-level tasks,  $\mathcal{X}$ 's word-level category is  $\mathcal{C}_{x_i}, i \in [0,n]$ . Then the KD objective can be written as,
$$
\mathcal {L} = \operatorname {D i v} _ {K L} \left(P _ {T} (\mathcal {C} | \mathcal {X}) \| P _ {S} (\mathcal {C} | \mathcal {X})\right)
$$
$$
\text {w h e r e} \mathcal {C} = \left\{ \begin{array}{l l} \mathcal {C} _ {\mathcal {X}} & , \text {s e n t e n c e - l e v e l} \\ \mathcal {C} _ {x _ {i}} & , \text {w o r d - l e v e l} \end{array} \right. \tag {2}
$$
It's worth noting that word-level  $Div_{KL}$  cannot be directly applied for heterogeneous teacher and student models since their tokenizations are different. In order to align the predictions of teacher and student, we only adopt the prediction on the first sub-word of each word.
2.3 Enhanced Cross-Lingual Distillation
To explore the potentials of improving the two KD steps, we propose to enhance them with machine translation (MT) and paraphrase generation (PG).
2.3.1 Language Balanced Distillation
During the first KD step that training the mPLM from an English (i.e. the source language) classifier, to leverage the cross-lingual transferarability of mPLM, the conventional method is to train the mPLM only on the English corpus. However, in our preliminary experiments, we notice that the mPLM's accuracy gap between English and the target languages are very huge. It's over  $5\%$  between the English target model (94.0) and the average of all target models (88.4), as reported by 2-step KD in Table 7, Appendix A.
Figure 2: Language balanced distillation. Leverage the MT model to translate unlabeled English  $\mathcal{D}_{en}$  into target languages  $D_{tgt}^{trans}$ . Perform KD on the translated  $D_{tgt}^{trans}$  with  $\theta_{en}$ 's predicted distribution  $P_{en}(\mathcal{C}|\mathcal{X})$ .
Hence, we propose to translate the unlabeled English corpus  $\mathcal{D}_{en}$  to target languages  $\mathcal{D}_{tgt}^{trans}$ , as depicted by Figure 2. Since  $\mathcal{D}_{en}$  and  $\mathcal{D}_{tgt}^{trans}$  are aligned, source English model's predicted distribution  $P_{en}(\mathcal{C}|\mathcal{X})$  of  $\mathcal{D}_{en}$  can be directly applied to  $\mathcal{D}_{tgt}^{trans}$ . In this way, KD is able to be performed on not only the source language but also the target languages.
As shown in the lower left of Figure 1, the translated  $\mathcal{D}_{tgt}^{trans}$  is incorporated in the training of KD step one.
2.3.2 Language-Specific Data Augmentation
Inspired by data augmentation for KD (Jiao et al., 2020) and multilingual paraphrase generation (Guo et al., 2019), we augment the unlabeled target corpus  $\mathcal{D}_{tgt}$  via paraphrasing.
Figure 3: Language-specific data augmentation. We paraphrase the target corpus  $\mathcal{D}_{tgt}$  into  $\mathcal{D}_{tgt}^{para}$  as the augmented training data. KD is then performed on the mixture of  $\mathcal{D}_{tgt}$  and  $\mathcal{D}_{tgt}^{para}$ .
3.1 Datasets and Preprocessing
MultiATIS++ (Xu et al., 2020) extends the Multilingual ATIS corpus (Upadhyay et al., 2018) to 9 languages across 4 language families, including Indo-European (English, Spanish, German, French, Portuguese and Hindi), Sino-Tibetan (Chinese), Japonic (Japanese) and Altaic (Turkish). It provides annotations for intent recognition (sentence classification) and slot filling (sequence tagging) for each languages. The utterances are professionally translated from English and manually annotated. MultiATIS++ includes 37,084 training examples and 7,859 testing examples.
MTOP (Li et al., 2021) is a recently released multilingual NLU dataset covering 6 languages: English, German, French, Spanish, Hindi, Thai. It's also manually annotated for intent recognition (sentence classification) and slot filling (sequence tagging). MTOP provides a larger corpus consisting of 104,445 examples, of which  $10\%$  is validation set and  $20\%$  is testing set.
For each language, we randomly split both MultiATIS++ and MTOP into two balanced parts: annotated and unannotated. The annotated parts are used to train and simulate the off-the-shelf source models while the unannotated parts are used for training the baselines and the proposed 2-step distillation model. We tokenize Chinese, Japanese and Thai utterances using Jieba $^2$ , MeCab $^3$  and
pythainlp $^4$  respectively.
Translate-Test (Conneau et al., 2018) is a machine translation based method. It performs two-pass inferences to tackle the FreeTransfer-X problem: 1) translate the testing utterances into English (i.e. the source language) from the target language, 2) predict on the translated English utterances with the off-the-shelf English model.
Translate-Train-Pseudo is also based on machine translation. It's a variant of the TranslateTrain (Conneau et al., 2018), which translates English training examples into target languages and applies English annotations to the translated examples. However, annotations are not provided in the FreeTransfer-X problem. Hence, Translate-Train-Pseudo utilizes the prediction of the off-the-shelf English model to pseudoly annotates the translated examples.
Gold-Supervised is for reference since it's trained with annotations. It replaces the first distillation step of the proposed framework with gold-supervised training, in other words, the mPLM is supervised by gold annotations instead of the off-the-shelf English model. It's supposed to be very strong.
3.3 Experiment Settings
3.3.1 Model Architectures
We experiment with three mainstream NLU model architectures to verify the universality of the proposed framework. They are used as the backbones of the off-the-shelf models  $\theta_{src}$  and the output models  $\theta_{tgt}$  in target language.
Transformer encoder (Vaswani et al., 2017) models input sequences fully with Attention mechanism. We follow the language modeling method of BERT (Devlin et al., 2019a). We adopt absolute positional encoding. The contextual representation vector of the first word is used for sentence classification. Sequence tagging is based on the contextual representation of each word.
Bidirectional LSTM (BiLSTM) (Schuster and Paliwal, 1997) models input sequences via leveraging two stacked LSTM layers respectively from backward and forward directions. We take the representation vector of the last word for sentence classification. Word-level representation is used for sequence tagging like Transformer.
<sup>3</sup>https://github.com/polm/fugashi
<sup>4</sup>https://github.com/PyThaiNLP/pythainlp
Models	MTOP	MultiATIS++	Avg
Transformer	BiLSTM	CNN	Transformer	BiLSTM	CNN
Reference	Off-the-shelf En source	88.3	86.2	90.5	94.4	90.8	92.7	90.5
Gold-supervised target	78.4	65.0	79.2	84.6	85.2	86.6	79.8
Baselines	Translate-test	69.6	66.0	73.8	86.4	80.7	86.2	77.1
Translate-train-pseudo	64.2	57.9	67.4	84.7	81.2	83.2	73.1
Ours	2-step KD	75.1	72.3	75.6	87.7	83.8	85.0	79.9
+ Balanced distillation	79.3	75.9	77.8	88.9	85.2	86.2	82.2
+ Data augmentation	79.6	79.1	78.8	88.7	86.4	86.9	83.3
Table 1: Classification accuracy averaged over target languages. MTOP: de, es, fr, hi, th. MultiATIS++: de, es, fr, hi, ja, pt, tr, zh. For simplicity, the architecture of a target model is identical to its corresponding English source model.
Convolutional Neural Networks (CNN) (Kim, 2014) encodes input sequences with CNN modules. We adopt three kinds of 1-D kernels with kernel size of 3, 4 and 5. Output vectors from all kernels and channels are concatenated as the representation for sentence classification. Dilated CNN (Strubell et al., 2017) is adopted for sequence tagging.
3.3.2 Training Details
English is regarded as the source language in all the experiments. Off-the-shelf English models are trained on the hold-out annotated English corpus as described in Section 3.1. All the experimented models are controlled in comparable model scale. AdamW (Loshchilov and Hutter, 2019) is adopted as the optimizer with  $\epsilon = 1e - 8$ . We train the models for 50 epochs and take the checkpoint of the best validation accuracy as the final model. Table 2 reports the hyper-parameters of the model architectures.
Model	Embed size	Hidden size	#Layers	#Params
Transformer	256	256	4	5.3M
BiLSTM	256	512	2	5.3M
CNN	256	768	2	5.0M
Table 2: Hyper-parameters of the experimented models.
Initial learning rate is decided based on a gradient-based searching heuristics proposed by Smith (2015), since in our preliminary experiments Smith (2015) stably finds better learning rates than manual searching. We build vocabularies of 10k words for each language via Byte Pair Encoding (BPE) Sennrich et al. (2016). Experiments are implemented with PyTorch (Paszke et al., 2019) and conducted on a single Nvidia V100 32GB GPU.
3.3.3 Auxiliary Models
M2M-100 (Fan et al., 2021) is adopted as the MT system in our experiments. We apply the 418M model checkpoint from Huggingface<sup>5</sup>.
XLM-RoBERTa (Conneau et al., 2020) is adopted as the mPLM in the proposed 2-step distillation framework.
Average accuracy across languages and models is given in Table 1 and Table 3. Language-wise results are provided in Appendix A.
3.4.1 Sentence Classification
As shown in Table 1, the proposed 2-step KD framework significantly outperforms the MT baselines on most model architectures, except for the CNN of Translate-test. Although Translate-test is strong in a very few cases, it requires 2-pass inference (MT and classification) that results in a high latency. On the contrary, the proposed framework directly produces classification models in the target languages, which is more efficient. In addition, the language-balanced distillation and language-specific data augmentation further enhance our model to a large extent,  $+2.3\%$  and  $+1.1\%$  respectively. Language-wise results in Table 7 demonstrate the robustness of our method across various languages.
To our surprise, the naive 2-step  $KD$  model even performs on par with the Gold-supervised reference on average. We guess it's due to the regularization effects of knowledge distillation that brings a good generalizability to the proposed model. It implies the proposed framework may be an annotation-free alternative to current zero-shot cross-lingual transfer framework.
<sup>5</sup>https://huggingface.co/facebook/m2m100_418M
Models	MTOP	MultiATIS++	Avg
Transformer	BiLSTM	CNN	Transformer	BiLSTM	CNN
Reference	Off-the-shelf En source	74.8	81.1	72.1	88.4	94.0	89.1	83.3
Gold-supervised target	64.6	68.6	63.3	71.5	76.5	74.1	69.8
Baselines	Translate-test	37.2	41.4	34.2	24.8	38.8	40.8	36.2
Translate-train-pseudo	34.4	40.4	28.6	53.9	63.1	61.8	47.0
Ours	2-step KD	63.7	67.6	55.7	71.7	76.9	73.5	68.2
Table 3: Sequence tagging F1 score averaged over target languages. MTOP: de, es, fr, hi, th. MultiATIS++: de, es, fr, hi, ja, pt, tr, zh. For simplicity, the architecture of a target model is identical to its corresponding English source model.
Models	Original en	Finetuned en	Zero-Shot Cross-Lingual Transfer
de	es	fr	hi	ja	pt	tr	zh	Avg
Gold-supervised	-	97.9	97.6	97.4	97.4	92.4	90.6	97.3	83.8	92.8	93.7
Transformer	Naive KD	94.4	97.2	96.9	96.8	96.2	90.8	90.1	95.6	84.3	91.3	92.8
+ Balanced distillation	97.5	97.6	96.9	96.6	95.4	96.3	96.1	90.8	97.5	95.9
BiLSTM	Naive KD	90.8	93.2	93.5	93.5	93.2	90.4	83.4	93.5	77.2	85.0	88.7
+ Balanced distillation	92.4	93.3	93.7	92.4	91.8	91.9	93.1	86.6	92.8	92.0
CNN	Naive KD	92.7	94.7	91.8	94.1	93.2	90.3	90.9	94.2	83.5	90.3	91.0
+ Balanced distillation	92.8	93.2	92.9	92.4	91.8	91.6	93.2	89.2	92.9	92.2
Table 4: Classification accuracy of the finetuned mPLM models, i.e. XLM-RoBERTa. Evaluated on MultiATIS++. Gold-supervised is trained with gold annotations. Bold languages is not in the Indo-European language family as English.
However, comparing the results of the English source model and those of the target models in Table 7, the cross-lingual transferred models still lag far behind the original English models. There is a great potential of the proposed framework.
3.4.2 Sequence Tagging
On the sequence tagging task, the proposed model beats the baselines by a wide margin. The MT-based baselines perform very poor on this task due to the error from word-level annotation alignment. Also because of the alignment error, we do not apply language balanced distillation and language-specific data augmentation on this task.
As to the comparison with the Gold-supervised reference, our model performs slightly worse than it. It may due to the insufficient knowledge distillation from the teacher to the student, which comes from the discrepancy between teacher's and student's tokenizations. Although, as described in Section 2.2.2, we perfectly align their prediction at word-level, only the first subword of each word is used for distillation. More informative subword-level aligning and distillation methods can be explored. We leave this problem for the future research. Besides, similar to sentence classification, gap between the English source model and the transferred target models is huge, as shown in Table 3.
In sum, both experimental results on sentence
classification and sequence tagging demonstrate that the proposed model is significantly stronger than MT-based cross-lingual transfer methods. Furthermore, the proposed model only slightly lags behind or even performs on par with the strong Gold-supervised reference, which is not able to address the FreeTransfer-X problem.
In order to explore the potential of the proposed framework, we analyze it in more details. For simplicity, experiments in this Section are conducted only on the MultiATIS++ sentence classification task.
4.1 Effects of the Distillation
Table 4 reports the accuracy of the XLM-RoBERTa finetuned from gold annotations, Transformer teacher, BiLSTM teacher and CNN teacher.