SlowGuess/ACL_2020_OCR huggingface dataset:
Xinyu Wang $^{\diamond}$ , Yong Jiang $^{\dagger}$ , Nguyen Bach $^{\dagger}$ , Tao Wang $^{\dagger}$ , Fei Huang $^{\dagger}$ , Kewei Tu $^{\diamond*}$
$^{\diamond}$ School of Information Science and Technology, ShanghaiTech University
Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences
{wangxyl,tukw}@shanghaiitech.edu.cn
{yongjiang.jy,nguyen.bach,leeo.wangt,f.huang}@alibaba-inc.com
Multilingual sequence labeling is a task of predicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, using a multilingual model has the benefit of a smaller model size, easier in online serving, and generalizability to low-resource languages. However, current multilingual models still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance between the student's and the teachers' structure-level probability distributions, (2) aggregates the structure-level knowledge to local distributions and minimizes the distance between two local probability distributions. Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.
Sequence labeling is an important task in natural language processing. Many tasks such as named entity recognition (NER) and part-of-speech (POS) tagging can be formulated as sequence labeling problems and these tasks can provide extra information to many downstream tasks and products such as searching engine, chatbot and syntax parsing (Jurafsky and Martin, 2009). Most of the previ
ous work on sequence labeling focused on monolingual models, and the work on multilingual sequence labeling mainly focused on cross-lingual transfer learning to improve the performance of low-resource or zero-resource languages (Johnson et al., 2019; Huang et al., 2019a; Rahimi et al., 2019; Huang et al., 2019b; Keung et al., 2019), but their work still trains monolingual models. However, it would be very resource consuming considering if we train monolingual models for all the  $7,000+$  languages in the world. Besides, there are languages with limited labeled data that are required for training. Therefore it is beneficial to have a single unified multilingual sequence labeling model to handle multiple languages, while less attention is paid to the unified multilingual models due to the significant difference between different languages. Recently, Multilingual BERT (M-BERT) (Devlin et al., 2019) is surprisingly good at zero-shot cross-lingual model transfer on tasks such as NER and POS tagging (Pires et al., 2019). M-BERT bridges multiple languages and makes training a multilingual sequence labeling model with high performance possible (Wu and Dredze, 2019). However, accuracy of the multilingual model is still inferior to monolingual models that utilize different kinds of strong pretrained word representations such as contextual string embeddings (Flair) proposed by Akbik et al. (2018).
To diminish the performance gap between monolingual and multilingual models, we propose to utilize knowledge distillation to transfer the knowledge from several monolingual models with strong word representations into a single multilingual model. Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) is a technique that first trains a strong teacher model and then trains a weak student model through mimicking the output probabilities (Hinton et al., 2015; Lan et al., 2018; Mirzadeh et al., 2019) or hidden states (Romero
* Kewei Tu is the corresponding author. This work was conducted when Xinyu Wang was interning at Alibaba DAMO Academy.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3317-3330 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
et al., 2014; Seunghyun Lee, 2019) of the teacher model. The student model can achieve an accuracy comparable to that of the teacher model and usually has a smaller model size through KD. Inspired by KD applied in neural machine translation (NMT) (Kim and Rush, 2016) and multilingual NMT (Tan et al., 2019), our approach contains a set of monolingual teacher models, one for each language, and a single multilingual student model. Both groups of models are based on BiLSTM-CRF (Lample et al., 2016; Ma and Hovy, 2016), one of the state-of-the-art models in sequence labeling. In BiLSTM-CRF, the CRF layer models the relation between neighboring labels which leads to better results than simply predicting each label separately based on the BiLSTM outputs. However, the CRF structure models the label sequence globally with the correlations between neighboring labels, which increases the difficulty in distilling the knowledge from the teacher models. In this paper, we propose two novel KD approaches that take structure-level knowledge into consideration for multilingual sequence labeling. To share the structure-level knowledge, we either minimize the difference between the student's and the teachers' distribution of global sequence structure directly through an approximation approach or aggregate the global sequence structure into local posterior distributions and minimize the difference of aggregated local knowledge. Experimental results show that our proposed approach boosts the performance of the multilingual model in 4 tasks with 25 datasets. Furthermore, our approach has better performance in zero-shot transfer compared with the baseline multilingual model and several monolingual teacher models.
BiLSTM-CRF (Lample et al., 2016; Ma and Hovy, 2016) is one of the most popular approaches to sequence labeling. Given a sequence of  $n$  word tokens  $\mathbf{x} = \{x_{1},\dots ,x_{n}\}$  and the corresponding sequence of gold labels  $\mathbf{y}^{*} = \{y_{1}^{*},\dots ,y_{n}^{*}\}$ , we first feed the token representations of  $\mathbf{x}$  into a BiLSTM to get the contextual token representations  $\mathbf{r} = \{\mathbf{r}_1,\dots ,\mathbf{r}_n\}$ . The conditional probability  $p(\mathbf{y}|\mathbf{x})$  is defined by:
$$
\psi \left(y ^ {\prime}, y, \mathbf {r} _ {i}\right) = \exp \left(\mathbf {W} _ {y} ^ {T} \mathbf {r} _ {i} + \mathbf {b} _ {y ^ {\prime}, y}\right) \tag {1}
$$
$$
p (\mathbf {y} | \mathbf {x}) = \frac {\prod_ {i = 1} ^ {n} \psi \left(y _ {i - 1} , y _ {i} , \mathbf {r} _ {i}\right)}{\sum_ {\mathbf {y} ^ {\prime} \in \mathcal {Y} (\mathbf {x})} \prod_ {i = 1} ^ {n} \psi \left(y _ {i - 1} ^ {\prime} , y _ {i} ^ {\prime} , \mathbf {r} _ {i}\right)} \tag {2}
$$
where  $\mathcal{V}(\mathbf{x})$  denotes the set of all possible label sequences for  $\mathbf{x}$ ,  $\psi$  is the potential function,  $\mathbf{W}_y$  and  $\mathbf{b}_{y',y}$  are parameters and  $y_0$  is defined to be a special start symbol.  $\mathbf{W}_y^T r_i$  and  $\mathbf{b}_{y',y}$  are usually called emission and transition scores respectively. During training, the negative log-likelihood loss for an input sequence is defined by:
$$
\mathcal {L} _ {\mathrm {N L L}} = - \log p (\mathbf {y} ^ {*} | \mathbf {x})
$$
BiLSTM-Softmax approach to sequence labeling reduces the task to a set of label classification problem by disregarding label transitions and simply feeding the emission scores  $\mathbf{W}^T\mathbf{r}_i$  into a softmax layer to get the probability distribution of each variable  $y_{i}$
$$
p \left(y _ {i} \mid \mathbf {x}\right) = \operatorname {s o f t m a x} \left(\mathbf {W} ^ {T} \mathbf {r} _ {i}\right) \tag {3}
$$
The loss function then becomes:
$$
\mathcal {L} _ {\mathrm {N L L}} = - \sum_ {i = 1} ^ {n} \log p \left(y _ {i} ^ {*} | \mathbf {x}\right)
$$
In spite of its simplicity, this approach ignores correlations between neighboring labels and hence does not adequately model the sequence structure. Consequently, it empirically underperforms the first approach in many applications.
A typical approach to KD is training a student network by imitating a teacher's predictions (Hinton et al., 2015). The simplest approach to KD on BiLSTM-Softmax sequence labeling follows Eq. 3 and performs token-level distillation through minimizing the cross-entropy loss between the individual label distributions predicted by the teacher model and the student model:
$$
\begin{array}{l} \mathcal {L} _ {\text {T o k e n}} = \\ - \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {| \mathcal {V} |} p _ {t} \left(y _ {i} = j \mid \mathbf {x}\right) \log p _ {s} \left(y _ {i} = j \mid \mathbf {x}\right) \tag {4} \\ \end{array}
$$
where  $p_t(y_i = j|\mathbf{x})$  and  $p_s(y_i = j|\mathbf{x})$  are the label distributions predicted by the teacher model and the student model respectively and  $|\mathcal{V}|$  is the number of possible labels. The final loss of the student
Figure 1: Structure-level knowledge distillation approaches. Mono/Multi represents Monolingual and Multilingual, respectively. Pos. represents the posterior distribution.
model combines the KD loss and the negative log-likelihood loss:
$$
L = \lambda \mathcal {L} _ {\text {T o k e n}} + (1 - \lambda) \mathcal {L} _ {\text {N L L}}
$$
where  $\lambda$  is a hyperparameter. As pointed out in Section 2.1, however, sequence labeling based on Eq. 3 has the problem of ignoring structure-level knowledge. In the BiLSTM-CRF approach, we can also apply an Emission distillation through feeding emission scores in Eq. 3 and get emission probabilities  $\tilde{p}(y_i|\mathbf{x})$ , then the loss function becomes:
$$
\begin{array}{l} \mathcal {L} _ {\text {E m i s s i o n}} = \\ - \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {| \mathcal {V} |} \tilde {p} _ {t} \left(y _ {i} = j \mid \mathbf {x}\right) \log \tilde {p} _ {s} \left(y _ {i} = j \mid \mathbf {x}\right) \tag {5} \\ \end{array}
$$
In this section, we propose two approaches to learning a single multilingual sequence labeling model (student) by distilling structure-level knowledge from multiple mono-lingual models. The first approach approximately minimizes the difference between structure-level probability distributions predicted by the student and teachers. The second aggregates structure-level knowledge into local posterior distributions and then minimizes the difference between local distributions produced by the student and teachers. Our approaches are illustrated in Figure 1.
Both the student and the teachers are BiLSTM-CRF models (Lample et al., 2016; Ma and Hovy, 2016), one of the state-of-the-art models in sequence labeling. A BiLSTM-CRF predicts the distribution of the whole label sequence structure, so token-level distillation is no longer possible and structure-level distillation is required.
3.1 Top-K Distillation
Inspired by Kim and Rush (2016), we propose to encourage the student to mimic the teachers' global structural probability distribution over all possible label sequences:
$$
\mathcal {L} _ {\mathrm {S t r}} = - \sum_ {\mathbf {y} \in \mathcal {Y} (\mathbf {x})} p _ {t} (\mathbf {y} | \mathbf {x}) \log p _ {s} (\mathbf {y} | \mathbf {x}) \tag {6}
$$
However,  $|\mathcal{V}(\mathbf{x})|$  is exponentially large as it represents all possible label sequences. We propose two methods to alleviate this issue through efficient approximations of  $p_t(\mathbf{y}|\mathbf{x})$  using the  $k$ -best label sequences.
Top-K Eq. 6 can be seen as computing the expected student log probability with respect to the teacher's structural distribution:
$$
\mathcal {L} _ {\mathrm {S t r}} = - \mathbb {E} _ {p _ {t} (\mathbf {y} | \mathbf {x})} [ \log p _ {s} (\mathbf {y} | \mathbf {x}) ] \tag {7}
$$
The expectation can be approximated by sampling from the teacher's distribution  $p_t(\mathbf{y}|\mathbf{x})$ . However, unbiased sampling from the distribution is difficult. We instead apply a biased approach that regards the  $k$ -best label sequences predicted by the
ψ(yk-1,yk,rk)	LABEL SEQ.PROBS.	STRUCTURAL KNOWLEDGE
y1	y2	y3	Prob.	y1	y2	y3	Weights
k=2	F	F	F	0.035	Top-2	T	T	F	0.57
yk-1\yk	y2=F	y2=T	F	F	T	0.316	F	F	T	0.43
y1=F	2	1/2	F	T	F	0.105	α(yk=F)	1.00	2.50	10.83
y1=T	1/2	2	F	T	T	0.007	α(yk=T)	1.00	2.50	8.13
k=3	T	F	F	0.009	β(yk=F)	8.79	3.33	1.00
yk-1\yk	y3=F	y3=T	T	F	T	0.079	β(yk=T)	10.17	4.25	1.00
y2=F	1/3	3	T	T	F	0.422	q(yk=F|x)	0.46	0.44	0.57
y2=T	4	1/4	T	T	T	0.026	q(yk=T|x)	0.54	0.56	0.43
Table 1: Example of computing the structural knowledge for a sequence of 3 tokens with a label set of  $\{T,F\}$ .  $\psi(y_{k-1},y_k,\mathbf{r}_k)$  represents the potential formulated in Eq. 1. Each Label Seq. Probs. is defined in Eq. 2 for the corresponding label sequence. Top-2 represents the two label sequences with the highest scores and Weights are their corresponding weights for KD (Eq. 8, 9).  $\alpha(y_k),\beta(y_k)$  and the posterior distribution  $q(y_k|\mathbf{x})$  are computed based on Eq. 11, 12 and 10 respectively. We assume that  $\psi(y_0,y_1,\mathbf{r}_1) = 1$  regardless of whether  $y_1$  is  $T$  or  $F$ .
teacher model as our samples. We use a modified Viterbi algorithm to predict the  $k$ -best label sequences  $\mathcal{T} = \{\hat{\mathbf{y}}^1, \dots, \hat{\mathbf{y}}^k\}$ . Eq. 7 is then approximated as:
$$
\mathcal {L} _ {\text {T o p - K}} = - \frac {1}{k} \sum_ {\hat {\mathbf {y}} \in \mathcal {T}} \log p _ {s} (\hat {\mathbf {y}} | \mathbf {x}) \tag {8}
$$
This can also be seen as data augmentation through generating  $k$  pseudo target label sequences for each input sentence by the teacher.
Weighted Top-K The Top-K method is highly biased in that the approximation becomes worse with a larger  $k$ . A better method is to associate weights to the  $k$  samples to better approximate  $p_t(\mathbf{y}|\mathbf{x})$ .
$$
p _ {t} ^ {\prime} (\mathbf {y} | \mathbf {x}) = \left\{ \begin{array}{l l} \frac {p _ {t} (\mathbf {y} | \mathbf {x})}{\sum_ {\hat {\mathbf {y}} \in \mathcal {T}} p _ {t} (\hat {\mathbf {y}} | \mathbf {x})} & \mathbf {y} \in \mathcal {T} \\ 0 & \mathbf {y} \notin \mathcal {T} \end{array} \right.
$$
Eq. 7 is then approximated as:
$$
\mathcal {L} _ {\text {T o p - W K}} = - \sum_ {\mathbf {y} \in \mathcal {T}} p _ {t} ^ {\prime} (\mathbf {y} | \mathbf {x}) \log p _ {s} (\mathbf {y} | \mathbf {x}) \tag {9}
$$
This can be seen as the student learning weighted pseudo target label sequences produced by the teacher for each input sentence.
The Top-K approach is related to the previous work on model compression in neural machine translation (Kim and Rush, 2016) and multilingual neural machine translation (Tan et al., 2019). In neural machine translation, producing  $k$ -best label sequences is intractable in general and in practice, beam search decoding has been used to approximate the  $k$ -best label sequences. However, for linear-chain CRF model,  $k$ -best label sequences can be produced exactly with the modified Viterbi algorithm.
3.2 Posterior Distillation
The Top-K is approximate with respect to the teacher's structural distribution and still is slow on large  $k$ . Our second approach tries to distill structure-level knowledge based on tractable local (token-wise) distributions  $q(y_{k}|\mathbf{x})$ , which can be exactly computed.
$$
\begin{array}{l} q \left(y _ {k} | \mathbf {x}\right) = \sum_ {\left\{y _ {1}, \dots , y _ {n} \right\} \backslash y _ {k}} p \left(y _ {1}, \dots , y _ {n} | \mathbf {x}\right) \\ = \frac {\sum_ {\left\{y _ {1} , \dots , y _ {n} \right\} \backslash y _ {k}} \prod_ {i = 1} ^ {n} \psi \left(y _ {i - 1} , y _ {i} , \mathbf {r} _ {i}\right)}{\mathcal {Z}} \tag {10} \\ \propto \alpha (y _ {k}) \times \beta (y _ {k}) \\ \end{array}
$$
$$
\alpha \left(y _ {k}\right) = \sum_ {\left\{y _ {0}, \dots , y _ {k - 1} \right\}} \prod_ {i = 1} ^ {k} \psi \left(y _ {i - 1}, y _ {i}, \mathbf {r} _ {i}\right) \tag {11}
$$
$$
\beta \left(y _ {k}\right) = \sum_ {\left\{y _ {k + 1}, \dots , y _ {n} \right\}} \prod_ {i = k + 1} ^ {n} \psi \left(y _ {i - 1}, y _ {i}, \mathbf {r} _ {i}\right) \tag {12}
$$
where  $\mathcal{Z}$  is the denominator of Eq. 2 that is usually called the partition function and  $\alpha(y_k)$  and  $\beta(y_k)$  are calculated in forward and backward pass utilizing the forward-backward algorithm. We assume that  $\beta(y_n) = 1$ .
Given the local probability distribution for each token, we define the KD loss function in a similar manner with the token-level distillation in Eq. 5.
$$
\mathcal {L} _ {\text {P o s .}} = - \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {| \mathcal {V} |} q _ {t} \left(y _ {i} = j \mid \mathbf {x}\right) \log q _ {s} \left(y _ {i} = j \mid \mathbf {x}\right) \tag {13}
$$
The difference between token-level distillation and posterior distillation is that posterior distillation is based on BiLSTM-CRF and conveys global
structural knowledge in the local probability distribution.
Posterior distillation has not been used in the related research of knowledge distillation in neural machine translation because of intractable computation of local distributions. In sequence labeling, however, local distributions in a BiLSTM-CRF can be computed exactly using the forward-backward algorithm.
An example of computing the structural knowledge discussed in this and last subsections is shown in Table 1.
3.3 Multilingual Knowledge Distillation
Let  $\mathcal{D} = \{D^1,\ldots ,D^l\}$  denotes a set of training data with  $l$  languages.  $D^{i}$  denotes the corpus of the  $i$ -th language that contains multiple sentence and label sequence pairs  $D^{i} = \{(\mathbf{x}_{j}^{i},\mathbf{y}_{j}^{i})\}_{j = 1}^{m_{i}}$ . To train a single multilingual student model from multiple monolingual pretrained teachers, for each input sentence, we first use the teacher model of the corresponding language to predict the pseudo targets ( $k$ -best label sequences or posterior distribution for posterior distillation). Then the student jointly
learns from the gold targets and pseudo targets in training by optimizing the following loss function:
$$
\mathcal {L} _ {\text {A L L}} = \lambda \mathcal {L} _ {\text {K D}} + (1 - \lambda) \mathcal {L} _ {\text {N L L}}
$$
where  $\lambda$  decreases from 1 to 0 throughout training following Clark et al. (2019),  $\mathcal{L}_{\mathrm{KD}}$  is one of the Eq. 5, 8, 9, 13 or an averaging of Eq. 9, 13. The overall distillation process is summarized in Algorithm 1.
Dataset We use datasets from 4 sequence labeling tasks in our experiment.
- CoNLL NER: We collect the corpora of 4 languages from the CoNLL 2002 and 2003 shared task (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003)
- WikiAnn NER (Pan et al., 2017): The dataset contains silver standard NER tags that are annotated automatically on 282 languages that exist in Wikipedia. We select the data of 8 languages from different language families or from different language subgroups of Indo-European languages. We randomly choose 5000 sentences from the dataset for each language except English, and choose 10000 sentences for English to reflect the abundance of English corpora in practice. We split the dataset by 8:1:1 for training/development/test.
- Universal Dependencies (UD) (Nivre et al., 2016): We use universal POS tagging annotations in the UD datasets. We choose 8 languages from different language families or language subgroups and one dataset for each language.
- Aspect Extraction: The dataset is from an aspect-based sentiment analysis task in SemEval-2016 Task 5 (Pontiki et al., 2016). We choose subtask 1 of the restaurants domain which has the most languages in all domains $^{1}$ , and split  $10\%$  of the training data as the development data.
<sup>1</sup>Subtask 1 of the restaurants domain contains 6 languages but we failed to get the French dataset as the dataset is not accessible from the provided crawling toolkit.
Task	CoNLL NER	SemEval 2016 Aspect Extraction
Approach	English	Dutch	Spanish	German	Avg.	Turkish	Spanish	Dutch	English	Russian	Avg.
REF	TEACHERS	92.43	91.90	89.19	84.00	89.38	59.29	74.29	72.85	72.80	71.77	70.20
SOFTMAX	90.08	88.99	87.72	81.40	87.05	52.39	71.54	68.86	65.87	66.85	65.10
TOKEN	90.02	88.87	88.24	81.30	87.11	52.56	72.12	69.33	66.81	67.20	65.61
BASE	BASELINE	90.13	89.11	88.06	82.16	87.36	55.79	72.02	69.35	67.54	68.02	66.54
EMISSION	90.28	89.31	88.65	81.96	87.55	51.52	72.60	69.10	67.21	68.52	65.79
OURS	TOP-K	90.57	89.33	88.61	81.99	87.62	55.74	73.13	69.81	67.99	69.21	67.18
TOP-WK	90.52	89.24	88.64	82.15	87.64	56.40	72.81	69.33	68.16	69.42	67.22
POSTERIOR	90.68	89.41	88.57	82.22	87.72	56.69	73.47	69.98	68.11	69.22	67.49
POS.+TOP-WK	90.53	89.58	88.66	82.31	87.77	55.00	73.97	70.15	67.83	69.76	67.34
Table 2: Results in F1 score of CoNLL 2002/2003 NER task and Aspect Extraction of SemEval 2016 Task 5.
Approach	English	Tamil	Basque	Hebrew	Indonesian	Persian	Slovenian	French	Avg.
REF	TEACHERS	83.80	86.72	94.68	83.72	90.48	90.37	91.66	90.29	88.97
SOFTMAX	81.86	80.72	93.72	77.11	90.64	90.03	91.05	88.18	86.66
TOKEN	81.33	80.88	93.56	77.47	90.50	89.83	91.08	87.93	86.57
BASE	BASELINE	82.56	82.39	94.13	78.89	91.11	90.23	91.62	88.92	87.48
EMISSION	82.54	82.23	94.37	78.45	90.92	89.92	91.56	89.47	87.43
OURS	TOP-K	82.39	82.94	94.13	78.93	90.93	90.12	91.56	89.25	87.53
TOP-WK	82.55	82.71	94.44	78.79	91.18	90.22	91.37	89.32	87.57
POSTERIOR	83.03	83.02	94.35	78.77	91.75	90.11	91.95	89.65	87.83
Pos.+Top-WK	82.77	82.81	94.47	78.87	91.18	90.31	91.84	89.42	87.71
Table 3: F1 scores in the WikiAnn NER task.
Approach	English	Hebrew	Japanese	Slovenian	French	Indonesian	Persian	Tamil	Avg.
REF	TEACHERS	96.94	97.54	96.81	95.01	99.10	94.02	98.07	93.01	96.31
SOFTMAX	95.61	96.25	96.59	90.66	97.94	92.56	96.62	86.58	94.10
TOKEN	95.66	96.28	96.47	90.82	97.95	92.70	96.58	86.41	94.11
BASE	BASELINE	95.71	96.18	96.60	90.64	97.89	92.62	96.63	86.19	94.06
EMISSION	95.63	96.21	96.52	90.76	97.98	92.64	96.61	86.66	94.13
OURS	TOP-K	95.74	96.27	96.56	90.66	97.96	92.58	96.64	86.57	94.12
TOP-WK	95.68	96.23	96.58	90.73	97.89	92.62	96.62	86.74	94.14
POSTERIOR	95.71	96.34	96.59	90.91	97.99	92.72	96.69	87.36	94.29
POS.+TOP-WK	95.74	96.27	96.47	90.84	98.02	92.58	96.73	86.97	94.20
Table 4: Accuracies in UD POS tagging.
Model Configurations In our experiment, all the word embeddings are fixed and M-BERT token embeddings are obtained by average pooling. We feed the token embeddings into the BiLSTM-CRF for decoding. The hidden size of the BiLSTM layer is 256 for the monolingual teacher models and 600 or 800 for the multilingual student model depending on the dataset as larger hidden size for the multilingual model results in better performance in our experiment. The settings of teacher and student models are as follows:
- Monolingual Teachers: Each teacher is trained with a dataset of a specific language. We use M-BERT concatenated with language-specific Flair (Akbik et al., 2018) embeddings and fastText (Bojanowski et al., 2017) word embeddings as token embeddings² for all the
monolingual teacher models.
- Multilingual Student: The student model is trained with the datasets of all the languages combined. We only use M-BERT as token embeddings for the multilingual student model.
Training For model training, the mini-batch size is set to 2000 tokens. We train all models with SGD optimizer with a learning rate of 0.1 and anneal the learning rate by 0.5 if there is no improvements on the development set for 10 epochs. For all models, we use a single NVIDIA Tesla V100 GPU for training including the student model. We tune the loss interpolation anneal rate in  $\{0.5, 1.0\}$  and the  $k$  value of Top-K ranging from [1, 10].
We report results of the following approaches.
2We use fastText + M-BERT instead if the Flair embedding is not available for a certain language.
Tamil	Basque	Hebrew	Indonesian	Persian	Slovenian	French	Avg.
TEACHERS	24.98	40.51	25.39	35.54	11.05	59.95	60.54	36.85
BASELINE	37.83	47.80	47.96	38.71	16.23	61.22	59.34	44.15
EMISSION	37.99	46.69	47.34	38.52	16.11	60.75	59.81	43.89
POSTERIOR	38.93	47.52	48.33	38.76	16.69	62.04	60.77	44.72
POSTERIOR+TOP-WK	38.23	47.49	48.79	39.32	16.19	62.03	60.34	44.63