SlowGuess/ACL_2020_OCR huggingface dataset:
Qianhui Wu $^{1}$ , Zijia Lin $^{2}$ , Börje F. Karlsson $^{2}$ , Jian-Guang Lou $^{2}$ , and Biqing Huang $^{1}$
<sup>1</sup>Beijing National Research Center for Information Science and Technology (BNRist) Department of Automation, Tsinghua University, Beijing 100084, China
wuqianhui@tsinghua.org.cn, hbq@tsinghua.edu.cn
$^{2}$ Microsoft Research, Beijing 100080, China
{zijlin,borje.karlsson,jlou}@microsoft.com
To better tackle the named entity recognition (NER) problem on languages with little/no labeled data, cross-lingual NER must effectively leverage knowledge learned from source languages with rich labeled data. Previous works on cross-lingual NER are mostly based on label projection with pairwise texts or direct model transfer. However, such methods either are not applicable if the labeled data in the source languages is unavailable, or do not leverage information contained in unlabeled data in the target language. In this paper, we propose a teacher-student learning method to address such limitations, where NER models in the source languages are used as teachers to train a student model on unlabeled data in the target language. The proposed method works for both single-source and multi-source cross-lingual NER. For the latter, we further propose a similarity measuring method to better weight the supervision from different teacher models. Extensive experiments for 3 target languages on benchmark datasets well demonstrate that our method outperforms existing state-of-the-art methods for both single-source and multisource cross-lingual NER.
Named entity recognition (NER) is the task of identifying text spans that belong to pre-defined categories, like locations, person names, etc. It's a fundamental component in many downstream tasks, and has been greatly advanced by deep neural networks (Lample et al., 2016; Chiu and Nichols, 2016; Peters et al., 2017). However, these approaches generally require massive manually labeled data, which prohibits their adaptation to low-resource languages due to high annotation costs.
One solution to tackle that is to transfer knowledge from a source language with rich labeled data to a target language with little or even no labeled
Figure 1: Comparison between previous cross-lingual NER methods (a/b) and the proposed method (c). (a): direct model transfer; (b): label projection with pairwise texts; (c): proposed teacher-student learning method.  $\mathcal{M}_{src / tgt}$ : learned NER model for source/target language;  $\{X,Y\}_{src}$ : labeled data in source language;  $\{X'\}_{tgt}$ : unlabeled data in target language;  $\{X',Y'\}_{tgt} / \{X',P'\}_{tgt}$ : pseudo-labeled data in target language with hard labels / soft labels.
data, which is referred to as cross-lingual NER (Wu and Dredze, 2019; Wu et al., 2020). In this paper, following Wu and Dredze (2019) and Wu et al. (2020), we focus on the extreme scenario of cross-lingual NER where no labeled data is available in the target language, which is challenging in itself and has attracted considerable attention from the research community in recent years.
Previous works on cross-lingual NER are mostly based on label projection with pairwise texts or direct model transfer. Label-projection based methods focus on using labeled data in a source language to generate pseudo-labelled data in the target language for training an NER model. For example, Ni et al. (2017) creates automatically labeled NER data for the target language via label projection on comparable corpora and develops a heuristic scheme to select good-quality projection-labeled data. Mayhew et al. (2017) and Xie et al. (2018)
6505
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6505-6514 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
translate the source language labeled data at the phrase/word level to generate pairwise labeled data for the target language. Differently, model-transfer based methods (Wu and Dredze, 2019; Wu et al., 2020) focus on training a shared NER model on the labeled data in the source language with language-independent features, such as cross-lingual word representations (Devlin et al., 2019), and then directly testing the model on the target language.
However, there are limitations in both label-projection based methods and model-transfer based methods. The former relies on labeled data in the source language for label projection, and thus is not applicable in cases where the required labeled data is inaccessible (e.g., due to privacy/sensitivity issues). Meanwhile, the later does not leverage unlabeled data in the target language, which can be much cheaper to obtain and probably contains very useful language information.
In this paper, we propose a teacher-student learning method for cross-lingual NER to address the mentioned limitations. Specifically, we leverage multilingual BERT (Devlin et al., 2019) as the base model to produce language-independent features. A previously trained NER model for the source language is then used as a teacher model to predict the probability distribution of entity labels (i.e., soft labels) for each token in the non-pairwise unlabeled data in the target language. Finally, we train a student NER model for the target language using the pseudo-labeled data with such soft labels. The proposed method does not rely on labelled data in the source language, and it also leverages the available information from unlabeled data in the target language, thus avoiding the mentioned limitations of previous works. Note that we use the teacher model to predict soft labels rather than hard labels (i.e., one-hot labelling vector), as soft labels can provide much more information (Hinton et al., 2015) for the student model. Figure 1 shows the differences between the proposed teacher-student learning method and the typical label-projection or model-transfer based methods.
We further extend our teacher-student learning method to multi-source cross-lingual NER, considering that there are usually multiple source languages available in practice and we would prefer transferring knowledge from all source languages rather than a single one. In this case, our method still enjoys the same advantages in terms of data availability and inference efficiency, compared with
existing works (Täckström, 2012; Chen et al., 2019; Enghoff et al., 2018; Rahimi et al., 2019). Moreover, we propose a method to measure the similarity between each source language and the target language, and use this similarity to better weight the supervision from the corresponding teacher model.
We evaluate our proposed method for 3 target languages on benchmark datasets, using different source language settings. Experimental results show that our method outperforms existing state-of-the-art methods for both single-source and multi-source cross-lingual NER. We also conduct case studies and statistical analyses to discuss why teacher-student learning reaches better results.
- We propose a teacher-student learning method for single-source cross-lingual NER, which addresses limitations of previous works w.r.t data availability and usage of unlabeled data.
- We extend the proposed method to multi-source cross-lingual NER, using a measure of the similarities between source/target languages to better weight teacher models.
- We conduct extensive experiments validating the effectiveness and reasonableness of the proposed methods, and further analyse why they attain superior performance.
Single-Source Cross-Lingual NER: Such approaches consider one single source language for knowledge transfer. Previous works can be divided into two categories: label-projection and model-transfer based methods.
Label-projection based methods aim to build pseudo-labeled data for the target language to train an NER model. Some early works proposed to use bilingual parallel corpora and project model expectations (Wang and Manning, 2014) or labels (Ni et al., 2017) from the source language to the target language with external word alignment information. But obtaining parallel corpora is expensive or even infeasible. To tackle that, recent methods proposed to firstly translate source-language labeled data at the phrase level (Mayhew et al., 2017) or word level (Xie et al., 2018), and then directly copy labels across languages. But translation introduces extra noise due to sense ambiguity and word order differences between languages, thus hurting the trained model.
6506
Model-transfer based methods generally rely on language-independent features (e.g., crosslingual word embeddings (Ni et al., 2017; Huang et al., 2019; Wu and Dredze, 2019; Moon et al., 2019), word clusters (Täckström et al., 2012), gazetteers (Zirikly and Hagiwara, 2015), and wikifier features (Tsai et al., 2016)), so that a model trained with such features can be directly applied to the target language. For further improvement, Wu et al. (2020) proposed constructing a pseudotraining set for each test case and fine-tuning the model before inference. However, these methods do not leverage any unlabeled data in the target language, though such data can be easy to obtain and benefit the language/domain adaptation.
Multi-Source Cross-Linguial NER: Multi-source cross-lingual NER considers multiple source languages for knowledge transfer.
Täckström (2012) and Moon et al. (2019) concatenated the labeled data of all source languages to train a unified model, and performed cross-lingual NER in a direct model transfer manner. Chen et al. (2019) leveraged adversarial networks to learn language-independent features, and learns a mixture-of-experts model (Shazeer et al., 2017) to weight source models at the token level. However, both methods straightly rely on the availability of labeled data in the source languages.
Differently, Enghoff et al. (2018) implemented multi-source label projection and studied how source data quality influence performance. Rahimi et al. (2019) applied truth inference to model the transfer annotation bias from multiple source-language models. However, both methods make predictions via an ensemble of source-language models, which is cumbersome and computationally expensive, especially when a source-language model has massive parameter space.
Teacher-Student Learning: Early applications of teacher-student learning targeted model compression (Bucilu et al., 2006), where a small student model is trained to mimic a pre-trained, larger teacher model or ensemble of models. It was soon applied to various tasks like image classification (Hinton et al., 2015; You et al., 2017), dialogue generation (Peng et al., 2019), and neural machine translation (Tan et al., 2019), which demonstrated the usefulness of the knowledge transfer approach.
Figure 2: Framework of the proposed teacher-student learning method for single-source cross-lingual NER.
In this paper, we investigate teacher-student learning for the task of cross-lingual NER, in both single-source and multi-source scenarios. Different from previous works, our proposed method does not rely on the availability of labelled data in source languages or any pairwise texts, while it can also leverage extra information in unlabeled data in the target language to enhance the cross-lingual transfer. Moreover, compared with using an ensemble of source-language models, our method uses a single student model for inference, which can enjoy higher efficiency.
Named entity recognition can be formulated as a sequence labeling problem, i.e., given a sentence  $\boldsymbol{x} = \{x_{i}\}_{i=1}^{L}$  with  $L$  tokens, an NER model is supposed to infer the entity label  $y_{i}$  for each token  $x_{i}$  and output a label sequence  $\boldsymbol{y} = \{y_{i}\}_{i=1}^{L}$ . Under the paradigm of cross-lingual NER, we assume there are  $K$  source-language models previously trained with language-independent features. Our proposed teacher-student learning method then uses those  $K$  source-language models as teachers to train an effective student NER model for the target language on its unlabeled data  $D_{tgt}$ .
3.1 Single-Source Cross-Linguual NER
Here we firstly consider the case of only one source language ( $K = 1$ ) for cross-lingual NER. The overall framework of the proposed teacher-student learning method for single-source cross-lingual NER is illustrated in Figure 2.
6507
3.1.1 NER Model Structure
As shown in Figure 2, for simplicity, we employ the same neural network structure for both teacher (source-language) and student (target-language) NER models. Note that the student model is flexible and its structure can be determined according to the trade-off between performance and training/inference efficiency.
Here the adopted NER model consists of an encoder layer and a linear classification layer. Specifically, given an input sequence  $\boldsymbol{x} = \{x_{i}\}_{i=1}^{L}$  with  $L$  tokens, the encoder layer  $f_{\theta}$  maps it into a sequence of hidden vectors  $\boldsymbol{h} = \{h_{i}\}_{i=1}^{L}$ :
$$
\boldsymbol {h} = f _ {\theta} (\boldsymbol {x}) \tag {1}
$$
Here  $f_{\theta}(\cdot)$  can be any encoder model that produces cross-lingual token representations, and  $h_i$  is the hidden vector corresponding to the  $i$ -th token  $x_i$ .
With each  $h_i$  derived, the linear classification layer computes the probability distribution of entity labels for the corresponding token  $x_i$ , using a softmax function:
$$
p (x _ {i}, \Theta) = \operatorname {s o f t m a x} \left(W h _ {i} + b\right) \tag {2}
$$
where  $p(x_{i},\Theta)\in \mathbb{R}^{|C|}$  with  $C$  being the entity label set, and  $\Theta = \{f_{\theta},W,b\}$  denotes the to-belearned model parameters.
3.1.2 Teacher-Student Learning
Training: We train the student model to mimic the output probability distribution of entity labels by the teacher model, on the unlabeled data in the target language  $D_{tgt}$ . Knowledge from the teacher model is expected to transfer to the student model, while the student model can also leverage helpful language-specific information available in the unlabeled target-language data.
Given an unlabeled sentence  $\pmb{x}^{\prime} \in D_{tgt}$  in the target language, the teacher-student learning loss w.r.t  $\pmb{x}^{\prime}$  is formulated as the mean squared error (MSE) between the output probability distributions of entity labels by the student model and those by the teacher model, averaged over tokens. Note that here we follow Yang et al. (2019) and use the MSE loss, because it is symmetric and mimics all probabilities equally. Suppose that for the  $i$ -token in  $\pmb{x}^{\prime}$ , i.e.,  $x_{i}^{\prime}$ , the probability distribution of entity labels output by the student model is denoted as  $\hat{p}(x_{i}^{\prime}, \Theta_{S})$ , and that output by the teacher model as  $\tilde{p}(x_{i}^{\prime}, \Theta_{T})$ . Here  $\Theta_{S}$  and  $\Theta_{T}$ , respectively, denote
Figure 3: Framework of the proposed teacher-student learning method for multi-source cross-lingual NER.
the parameters of the student and the teacher models. The teacher-student learning loss w.r.t  $\pmb{x}^{\prime}$  is then defined as:
$$
\mathcal {L} \left(\boldsymbol {x} ^ {\prime}, \Theta_ {S}\right) = \frac {1}{L} \sum_ {i = 1} ^ {L} \operatorname {M S E} \left(\hat {p} \left(x _ {i} ^ {\prime}, \Theta_ {S}\right), \tilde {p} \left(x _ {i} ^ {\prime}, \Theta_ {T}\right)\right) \tag {3}
$$
And the whole training loss is the summation of losses w.r.t all sentences in  $D_{tgt}$ , as defined below.
$$
\mathcal {L} \left(\Theta_ {S}\right) = \sum_ {\boldsymbol {x} ^ {\prime} \in D _ {t g t}} \mathcal {L} \left(\boldsymbol {x} ^ {\prime}, \Theta_ {S}\right) \tag {4}
$$
Minimizing  $\mathcal{L}(\Theta_S)$  will derive the student model.
Inference: For inference in the target language, we only utilize the learned student model to predict the probability distribution of entity labels for each token  $x_{i}$  in a test sentence  $\pmb{x}$ . Then we take the entity label  $c \in C$  with the highest probability as the predicted label  $y_{i}$  for  $x_{i}$ :
$$
y _ {i} = \arg \max  _ {c} \hat {p} \left(x _ {i}, \Theta_ {S}\right) _ {c} \tag {5}
$$
where  $p(x_{i},\Theta_{S})_{c}$  denotes the predicted probability corresponding to the entity label  $c$  in  $p(x_{i},\Theta_{S})$
3.2 Multi-Source Cross-Lingual NER
The framework of the proposed teacher-student learning method for multi-source  $(K > 1)$  crosslingual NER is illustrated in Figure 3.
3.2.1 Extension to Multiple Teacher Models
As illustrated in Figure 3, we extend the single-teacher framework in Figure 2 into a multi-teacher one, while keeping the student model unchanged.
Note that, for simplicity, all teacher models and the student model use the same model structure as
6508
3.1.1. Take the  $k$ -th teacher model for example, and denote its parameters as  $\Theta_T^{(k)}$ . Given a sentence  $\pmb{x}^{\prime} = \{x_{i}^{\prime}\}_{i=1}^{L}$  with  $L$  tokens from the unlabeled data  $D_{tgt}$  in the target language, the output probability distribution of entity labels w.r.t the  $i$ -th token  $x_{i}$  can be derived as Eq. 1 and 2, which is denoted as  $\tilde{p}(x_{i}', \Theta_{T}^{(k)})$ . To combine all teacher models, we add up their output probability distributions with a group of weights  $\{\alpha_{k}\}_{k=1}^{K}$  as follows.
$$
\tilde {p} (x _ {i} ^ {\prime}, \Theta_ {T}) = \sum_ {k = 1} ^ {K} \alpha_ {k} \cdot \tilde {p} (x _ {i} ^ {\prime}, \Theta_ {T} ^ {(k)}) \qquad (6)
$$
where  $\tilde{p}(x_i', \Theta_T)$  is the combined probability distribution of entity labels,  $\Theta_T = \{\Theta_T^{(k)}\}_{k=1}^K$  is the set of parameters of all teacher models, and  $\alpha_k$  is the weight corresponding to the  $k$ -th teacher model, with  $\sum_{k=1}^K \alpha_k = 1$  and  $\alpha_k \geq 0, \forall k \in \{1, \ldots, K\}$ .
3.2.2 Weighting Teacher Models
Here we elaborate on how to derive the weights  $\{\alpha_{k}\}_{k = 1}^{K}$  in cases  $w/$  or  $w/o$  unlabeled data in the source languages. Source languages more similar to the target language should generally be assigned higher weights to transfer more knowledge.
Without Any Source-Language Data: It is straightforward to average over all teacher models:
$$
\alpha_ {k} = \frac {1}{K}, \forall k \in \{1, 2, \dots , K \} \tag {7}
$$
With Unlabeled Source-Language Data: As no labeled data is available, existing supervised language/domain similarity learning methods for a target task (i.e., NER) (McClosky et al., 2010) are not applicable here. Inspired by Pinheiro (2018), we propose to introduce a language identification auxiliary task for calculating similarities between source and target languages, and then weight teacher models based on this metric.
In the language identification task, for the  $k$ -th source language, each unlabeled sentence  $\pmb{u}^{(k)}$  in it is associated with the language index  $k$  to build its training dataset, denoted as  $D_{src}^{(k)} = \{(\pmb{u}^{(k)}, k)\}$ . We also assume that in the  $m$ -dimensional language-independent feature space, sentences from each source language should be clustered around the corresponding language embedding vector. We thus introduce a learnable language embedding vector  $\mu^{(k)} \in \mathbb{R}^m$  for the  $k$ -th source language, and then utilize a bilinear operator to measure similarity between a given sentence
$\pmb{u}$  and the  $k$  -th source language:
$$
s (\boldsymbol {u}, \mu^ {(k)}) = g ^ {T} (\boldsymbol {u}) M \mu^ {(k)} \tag {8}
$$
where  $g(\cdot)$  can be any language-independent model that outputs sentence embeddings, and  $M \in \mathbb{R}^{m \times m}$  denotes the parameters of the bilinear operator.
By building a language embedding matrix  $P \in \mathbb{R}^{m \times K}$  with each  $\mu^{(k)}$  column by column, and applying a softmax function over the bilinear operator, we can derive language-specific probability distributions w.r.t  $\pmb{u}$  as below.
$$
q (\boldsymbol {u}, M, P) = \operatorname {s o f t m a x} \left(g ^ {T} (\boldsymbol {u}) M P\right) \tag {9}
$$
Then the parameters  $M$  and  $P$  are trained to identify the language of each sentence in  $\{D_{src}^{(k)}\}_{k = 1}^{K}$  via minimizing the cross-entropy (CE) loss:
$$
\begin{array}{l} \mathcal {L} (P, M) = - \frac {1}{Z} \sum_ {\left(\boldsymbol {u} ^ {(k)}, k\right) \in D _ {s t c}} \operatorname {C E} \left(q \left(\boldsymbol {u} ^ {(k)}, M, P\right), k\right) \\ + \gamma \| P P ^ {T} - I \| _ {F} ^ {2} \tag {10} \\ \end{array}
$$
where  $D_{src}$  is the union set of  $\{D_{src}^{(k)}\}_{k = 1}^{K}, Z = |D_{src}|, \| \cdot \|_F^2$  denotes the squared Frobenius norm, and  $I$  is an identity matrix. The regularizer in  $\mathcal{L}(P,M)$  is to encourage different dimensions of the language embedding vectors to focus on different aspects, with  $\gamma \geq 0$  being its weighting factor.
With learned  $M$  and  $P = [\mu^{(1)},\mu^{(2)},\dots ,\mu^{(K)}]$  we compute the weights  $\{\alpha_{k}\}_{i = 1}^{K}$  using the unlabeled data in the target language  $D_{tgt}$ :
$$
\alpha_ {k} = \frac {1}{| D _ {t g t} |} \sum_ {\boldsymbol {x} ^ {\prime} \in D _ {t g t}} \frac {\exp (s (\boldsymbol {x} ^ {\prime} , \mu^ {(k)}) / \tau)}{\sum_ {i = 1} ^ {K} \exp (s (\boldsymbol {x} ^ {\prime} , \mu^ {(i)}) / \tau)} \tag {11}
$$
where  $\tau$  is a temperature factor to smooth the output probability distribution. In our experiments, we set it as the variance of all values in  $\{s(\pmb{x}',\mu^{(k)})\}, \forall \pmb{x}' \in D_{tgt}, \forall k \in \{1,\dots,K\}$ , so that  $\alpha_{k}$  would not be too biased to either 0 or 1.
3.2.3 Teacher-Student Learning
Training: With the combined probability distribution of entity labels from multiple teacher models, i.e.,  $\tilde{p}(x_i', \Theta_T)$  in Eq. 6, the training loss for the student model is identical to Eq. 3 and 4.
Inference: For inference on the target language, we only use the learned student model and make predictions as in the single-source scenario (Eq. 5).
6509
Language	Type	Train	Dev	Test
English-en
(CoNLL-2003)	Sentence	14,987	3,466	3,684
Entity	23,499	5,942	5,648
German-de
(CoNLL-2003)	Sentence	12,705	3,068	3,160
Entity	11,851	4,833	3,673
Spanish-es
(CoNLL-2002)	Sentence	8,323	1,915	1,517
Entity	18,798	4,351	3,558
Dutch-nl
(CoNLL-2002)	Sentence	15,806	2,895	5,195
Entity	13,344	2,616	3,941
We conduct extensive experiments for 3 target languages (i.e., Spanish, Dutch, and German) on standard benchmark datasets, to validate the effectiveness and reasonableness of our proposed method for single- and multi-source cross lingual NER.
Datasets We use two NER benchmark datasets: CoNLL-2002 (Spanish and Dutch) (Tjong Kim Sang, 2002); CoNLL-2003 (English and German) (Tjong Kim Sang and De Meulder, 2003). Both are annotated with 4 entity types: PER, LOC, ORG, and MISC. Each language-specific dataset is split into training, development, and test sets. Table 1 reports the dataset statistics. All sentences are tokenized into sequences of subwords with WordPiece (Wu et al., 2016). Following Wu and Dredze (2019), we also use the BIO entity labelling scheme.
In our experiments, for each source language, an NER model is trained previously with its corresponding labeled training set. As for the target language, we discard the entity labels from its training set, and use it as unlabeled target-language data  $D_{tgt}$ . Similarly, unlabeled source-language data for learning language similarities (Eq. 10) is simulated via discarding the entity labels of each training set.
Network Configurations We leverage the cased multilingual BERT<sub>BASE</sub> (Wu and Dredze, 2019) for both  $f(\cdot)$  in Eq. 1 and  $g(\cdot)$  in Eq. 8, with 12 Transformer blocks, 768 hidden units, 12 self-attention head, GELU activations (Hendrycks and Gimpel, 2016), and learned positional embeddings. We use the final hidden vector of the first [CLS] token as the sentence embedding for  $g(\cdot)$ , and use the mean value of sentence embeddings w.r.t the  $k$ -th source language to initialize  $\mu^{(k)}$  in Eq. 8.