SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1: Statistics of the benchmark datasets.
es	nl	de
Täckström et al. (2012)	59.30	58.40	40.40
Tsai et al. (2016)	60.55	61.56	48.12
Ni et al. (2017)	65.10	65.40	58.50
Mayhew et al. (2017)	65.95	66.50	59.11
Xie et al. (2018)	72.37	71.25	57.76
Wu and Dredze (2019)†	74.50	79.50	71.10
Moon et al. (2019)†	75.67	80.38	71.42
Wu et al. (2020)	76.75	80.44	73.16
Ours	76.94	80.89	73.22
Table 2: Performance comparisons of single-source cross-lingual NER.  $\dagger$  denotes the reported results w.r.t. freezing the bottom three layers of BERT $_{\text{BASE}}$  as in this paper.
Network Training We implement our proposed method based on huggingface Transformers<sup>1</sup>. Following Wolf et al. (2019), we use a batch size of 32, and 3 training epochs to ensure convergence of optimization. Following Wu and Dredze (2019), we freeze the parameters of the embedding layer and the bottom three layers of BERT<sub>BASE</sub>. For the optimizers, we use AdamW (Loshchilov and Hutter, 2017) with learning rate of  $5e - 5$  for teacher models (Wolf et al., 2019), and  $1e - 4$  for the student model (Yang et al., 2019) to converge faster. As for language similarity measuring (i.e., Eq. 10), we set  $\gamma = 0.01$  following Pinheiro (2018). Besides, we use a low-rank approximation for the bilinear operator  $M$ , i.e.,  $M = U^T V$  where  $U, V \in \mathbb{R}^{d \times m}$  with  $d \ll m$ , and we empirically set  $d = 64$ .
Performance Metric We use phrase level F1-score as the evaluation metric, following Tjong Kim Sang (2002). For each experiment, we conduct 5 runs and report the average F1-score.
Single-Source Cross-Lingual NER Table 2 reports the results of different single-source crosslingual NER methods. All results are obtained with English as the source language and others as target languages.
It can be seen that our proposed method outperforms the previous state-of-the-art methods. Particularly, compared with the remarkable Wu and Dredze (2019) and Moon et al. (2019), which use nearly the same NER model as our method but is based on direct model transfer, our method obtains significant and consistent improvements in
<sup>1</sup>https://github.com/huggingface/transformers
es	nl	de
Täckström (2012)	61.90	59.90	36.40
Rahimi et al. (2019)	71.80	67.60	59.10
Chen et al. (2019)	73.50	72.40	56.00
Moon et al. (2019)†	76.53	83.35	72.44
Ours-avg	77.75	80.70	74.97
Ours-sim	78.00	81.33	75.33
F1-scores, ranging from 0.51 for Dutch to 1.80 for German. That well demonstrates the benefits of teacher-student learning over unlabeled target-language data, compared to direct model transfer. Moreover, compared with the latest meta-learning based method (Wu et al., 2020), our method requires much lower computational costs for both training and inference, meanwhile reaching superior performance.
Multi-Source Cross-Lingual NER Here we select source languages in a leave-one-out manner, i.e., all languages except the target one are regarded as source languages. For fair comparisons, we take Spanish, Dutch, and German as target languages, respectively.
Table 3 reports the results of different methods for multi-source cross-lingual NER. Both our teacher-student learning methods, i.e., Ours-avg (averaging teacher models, Eq. 7) and Ours-sim (weighting teacher models with learned language similarities, Eq. 11), outperform previous state-of-the-art methods on Spanish and German by a large margin, which well demonstrates their effectiveness. We attribute the large performance gain to the teacher-student learning process to further leverage helpful information from unlabeled data in the target language. Though Moon et al. (2019) achieves superior performance on Dutch, it is not applicable in cases where the labeled source-language data is inaccessible, and thus it still suffers from the aforementioned limitation w.r.t. data availability.
Moreover, compared with Ours-avg, Ours-sim brings consistent performance improvements. That means, if unlabeled data in source languages is available, using our proposed language similarity measuring method for weighting different teacher
Table 3: Performance comparisons of multi-source cross-lingual NER. Ours-avg: averaging teacher models (Eq. 7). Ours-sim: weighting teacher models with learned language similarities (Eq. 11).  $\dagger$  denotes the reported results w.r.t. freezing the bottom three layers of BERT $_{\text{BASE}}$ .
es	nl	de
Single-source:
Ours	76.94	80.89	73.22
HL	76.60 (-0.34)	80.43 (-0.46)	72.98 (-0.24)
MT	75.60 (-1.34)	79.99 (-0.90)	71.76 (-1.46)
Multi-source:
Ours-avg	77.75	80.70	74.97
HL-avg	77.65 (-0.10)	80.39 (-0.31)	74.31 (-0.66)
MT-avg	77.25 (-0.50)	80.53 (-0.17)	74.18 (-0.79)
Ours-sim	78.00	81.33	75.33
HL-sim	77.81 (-0.19)	80.27 (-1.06)	74.63 (-0.70)
MT-sim	77.12 (-0.88)	80.24 (-1.09)	74.33 (-1.00)
Table 4: Ablation study of the proposed teacher-student learning method for cross-lingual NER. HL: Hard Label; MT: Direct Model Transfer;  $*$ -avg: averaging source-language models;  $*$ -sim: weighting source-language models with learned language similarities.
models can be superior to simply averaging them.
Analyses on Teacher-Student Learning To validate the reasonableness of our proposed teacher-student learning method for cross-lingual NER, we introduce the following baselines. 1) Hard Label (HL), which rounds the probability distribution of entity labels (i.e., soft labels output by teacher models) into a one-hot labelling vector (i.e., hard labels) to guide the learning of the student model. Note that in multi-source cases, we use the combined probability distribution of multiple teacher models (Eq. 6) to derive the hard labels. To be consistent with Eq. 3, we still adopt the MSE loss here. In fact, both MSE loss and cross-entropy loss lead to the same observation described in this subsection. 2) Direct Model Transfer (MT), where NO unlabeled target-language data is available to perform teacher-student learning, and thus it degenerates into: a) directly applying the source-language model in single-source cases, or b) directly applying a weighted ensemble of source-language models in multi-source cases, with weights derived via Eq. 6 and Eq. 11.
Table 4 reports the ablation study results. It can be seen that using hard labels (i.e.,  $\mathrm{HL - }^{*}$ ) would result in consistent performance drops in all crosslingual NER settings, which validates using soft labels in our proposed teacher-student learning method can convey more information for knowledge transfer than hard labels. Moreover, we can also observe that, using direct model transfer (i.e.,
6511
#1 Spanish	Source-Language Model: ...Etchart [I-PER, 1.00] Sydney [B-LOC, 0.98] (Australia [B-LOC, 1.00]), 23 may ( EFE [O, 0.53 ]). Ours: Por Mario [B-PER] Etchart [I-PER] Sydney [B-LOC] ( Australia [B-LOC] ), 23 may ( EFE [B-ORG] ). Examples in D_tgt: Asi lo annunció a EFE [B-ORG, 1.00] Hans Gaasbek, el abogado de Murillo, argumentando que ...
#2 Dutch	Source-Language Model: Vanderpoorten [O, 0.87]: ' Dit is een eerste stap in de herwaardering van het beroepsonderwijts " Ours: Vanderpoorten [B-PER]: ' Dit is een eerste stap in de herwaardering van het beroepsonderwijts " Examples in D_tgt: Vanderpoorten [B-PER, 0.99] stond op het punt die reputatie te bezwadderen.
#3 German	Source-Language Model: ... tatsäch berücksichtigt werden mûsse , forderte Hof [B-ORG, 0.85] eine " Transparenz" ... Ours: Weil die Altersstruktur darauf berücksichtigt werden mûsse , forderte Hof [B-PER] eine " Transparenz" ... Examples in D_tgt: ... meint Hof [B-PER, 0.99], den der " erstaunliche Pragmatismus der Jugendlichen " beeindruckt.
Figure 4: Case study on why teacher-student learning works. The GREEN (RED) highlight indicates a correct (incorrect) label. The real-valued numbers indicate the predicted probability corresponding to the entity label.
es	nl	de
Ours	78.00	81.33	75.33
cosine	77.86 (-0.14)	79.94 (-1.39)	75.24 (-0.09)
l2	77.72 (-0.28)	79.74 (-1.59)	75.09 (-0.24)
Table 5: Comparison between the proposed language similarity measuring method and the commonly used cosine/  $\ell_2$  metrics for multi-source cross-lingual NER.
MT-*) would lead to even more significant performance drops in all cross-lingual NER settings (up to 1.46 F1-score). Both demonstrate that leveraging unlabeled data in the target language can be helpful, and that the proposed teacher-student learning method is capable of leveraging such information effectively for cross-lingual NER.
Analyses on Language Similarity Measuring We further compare the proposed language similarity measuring method with other commonly used unsupervised metrics, i.e., cosine similarity and  $\ell_2$  distance. Specifically,  $s(\pmb{x}',\mu^{(k)})$  in Eq. 11 is replaced by cosine similarity or negative  $\ell_2$  distance between  $\pmb{x}'$  and the mean value of sentence embeddings w.r.t the  $k$ -th source language.
As shown in Table 5, replacing the proposed language similarity measuring method with either cosine  $/ \ell_2$  metrics leads to consistent performance drops across all target languages. This further demonstrates the benefits of our language identification based similarity measuring method.
4.4 Why Teacher-Student Learning Works?
By analyzing which failed cases of directly applying the source-language model are corrected by the proposed teacher-student learning method, we try to bring up insights on why teacher-student learning works, in the case of single-source cross-lingual NER.
Figure 5: Percentage of corrected mispredictions, in different probability intervals.
Firstly, teacher-student learning can probably help to learn label preferences for some specific words in the target language. Specifically, if a word appears in the unlabeled target-language data and the teacher model consistently predicts it to be associated with an identical label with high probabilities, the student model would learn the preferred label w.r.t that word, and predict it in cases where the sentence context may not provide enough information. Such label preference can help the predictions for tokens that are less ambiguous and generally associated with an identical entity label. As illustrated in Figure 4, in example #1, the source-language (teacher) model, fails to identify "EFE" as an ORG in the test sentences, while the student model (i.e., Ours) can correctly label it, because it has seen "EFE" labeled as ORG by the teacher model with high probabilities in the unlabeled target-language data  $D_{tgt}$ . Similar results can also be observed in example #2 and #3.
Moreover, teacher-student learning may help to find a better classifying hyperplane for the student NER model with unlabelled target-language data. Actually, we notice that the source-language model generally makes correct label predictions with higher probabilities, and makes mispredictions with relatively lower probabilities. By calcu
6512
lating the proportion of its mispredictions that are corrected by our teacher-student learning method in different probability intervals, we find that our method tends to correct the low-confidence mispredictions, as illustrated in Figure 5. We conjecture that, with the help of unlabeled target-language data, our method can probably find a better classifying hyperplane for the student model, so that the low-confidence mispredictions, which are closer to the classifying hyperplane of the source-language model, can be clarified.
In this paper, we propose a teacher-student learning method for single-/multi-source cross-lingual NER, via using source-language models as teachers to train a student model on unlabeled data in the target language. The proposed method does not rely on labelled data in the source languages and is capable of leveraging extra information in the unlabelled target-language data, which addresses the limitations of previous label-projection based and model-transfer based methods. We also propose a language similarity measuring method based on language identification, to better weight different teacher models. Extensive experiments on benchmark datasets show that our method outperforms the existing state-of-the-art approaches.
Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535-541. ACM.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098-3112.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Jan Vium Enghoff, Søren Harrison, and Željko Agić. 2018. Low-resource named entity recognition via multi-source projection: Not quite there yet? In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 195-201.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Lifu Huang, Heng Ji, and Jonathan May. 2019. Cross-lingual multi-level adversarial transfer to enhance low-resource name tagging. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3823-3833, Minneapolis, Minnesota.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101.
Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognition. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2536-2545.
David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 28-36.
Taesun Moon, Parul Awasthy, Jian Ni, and Radu Florian. 2019. Towards lingua franca named entity recognition with bert. arXiv preprint arXiv:1912.01389.
Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1470–1480.
Shuke Peng, Xinjing Huang, Zehao Lin, Feng Ji, Haiqing Chen, and Yin Zhang. 2019. Teacher-student framework enhanced multi-domain dialogue generation. arXiv preprint arXiv:1908.07137.
6513
Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1756-1765.
Pedro O Pinheiro. 2018. Unsupervised domain adaptation with similarity learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8004-8013.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
Oscar Tackström. 2012. Nudging the envelope of direct transfer methods for multilingual named entity recognition. In Proceedings of the NAACL-HLT Workshop on the Induction of Linguistic Structure, pages 55-63.
Oscar Tackström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 477-487.
Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In *COLING-02: The 6th Conference on Natural Language Learning* 2002 (CoNLL-2002).
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wikification. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 219–228.
Mengqiu Wang and Christopher D. Manning. 2014. Cross-lingual projected expectation regularization for weakly supervised learning. Transactions of the Association for Computational Linguistics, 2:55-66.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtopicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen, Borje F Karlsson, Biqing Huang, and Chin-Yew Lin. 2020. Enhanced meta-learning for cross-lingual named entity recognition with minimal resources. In Proceedings of the AAAI Conference on Artificial Intelligence.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 369-379.
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2019. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. arXiv preprint arXiv:1910.08381.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. 2017. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285-1294. ACM.
Ayah Zirikly and Masato Hagiwara. 2015. Crosslingual transfer of named entity recognizers without parallel corpora. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 390-396.
6514