SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 1: (a) The count of errors made by the LORELEI Byte-CRF monolingual and polyglot models, compared to the fine-tuned (FT) models (across all languages), (b) shows the CoNLL setting. Deltas (Errors minus FT Errors) are displayed on top. Polyglot models tend to make more errors on O-tagged tokens (precision errors) than monolingual models. However, fine-tuning tends to recover these errors to nearly monolingual performance. In the CoNLL regime, polyglot models make fewer errors on PER and ORG tags, and fine-tuned models generally maintain that error rate.
fine-tuning tends to correct this trend back towards monolingual performance. We additionally find that, compared to monolingual models, fine-tuned models do much better PER and ORG tags (in both LORELEI and CoNLL settings). However, the same is not true for polyglot LORELEI models, indicating that some of this transfer comes from the combination of polyglot and fine-tune training.
One reason that polyglot fine-tuned models may perform better than monolingual models is the larger number of entities they see during training. Many languages contain entities in their validation set, which appear in the training sets of other languages. We identify such "common entities" as entities in the validation set of a language  $l$  which share some level of surface form overlap (either n-gram or exact match) and type with an entity appearing in the training set of lan
<sup>8</sup>We explore n-gram overlap with  $n = 4,5,6,7,8$  and exact name overlap. We report the average rate across each granularity.
8098
Figure 2: The rate of errors containing surface forms that overlap with an entity of the same type in other languages' training set. We report the harmonic mean between the rate in precision and recall errors, for the monolingual, polyglot, and fine-tuned byte-CRF models. We find that polyglot models have a lower rate of errors on entities which appear in other languages' training sets, indicating that they are benefiting from the higher quantity of entities seen.
guage  $l' \neq l$ . We plot the average error rate (defined as the harmonic mean between the rate of precision errors and the rate of recall errors) of the CoNLL Byte-CRF model in Figure 2. We find that polyglot models have a lower error rate on "common entities" than monolingual models, indicating that such entities are a source of transfer in polyglot NER. We also see that language-specific fine-tuning tends to increase the error rate, either due to forgetting or simply to decreasing errors on "non-common entities" during fine-tuning.
5.2 Polyglot Parameter Efficiency
Many studies have demonstrated that modern neural models have enormous capacity, and that not all parameters are needed to model the target function (LeCun et al., 1990; Hinton et al., 2015; Frankle and Carbin, 2019; Sanh et al., 2019). Let us assume that it takes  $M^l$  parameters to learn a monolingual NER model for language  $l$ . If we sought to train monolingual models for each language in  $L$ , we would need  $\hat{M} = \sum_{l \in L} M^l$  parameters. Does a polyglot model trained on these languages need  $\hat{M}$  parameters? Perhaps the polyglot NER model is partitioning its parameters by language, and little sharing occurs across languages, so the full  $\hat{M}$  parameters are needed. In this case, the negative results for polyglot learning could be explained by the under-parameterization of the model. Conversely, the model could be sharing parameters across many languages, effectively learning cross
lingual representations. In this case, we would expect the model to need much fewer than  $\hat{M}$  parameters, and the over-sharing of parameters across languages could explain the poor polyglot performance.
Model Compression To explore polyglot model behavior, we utilize model compression techniques, which have the goal of compressing a large number of parameters into a smaller amount with minimal loss in overall model accuracy. We use magnitude weight pruning (Han et al., 2015) to answer two questions: (1) How many more parameters do polyglot models require than monolingual models? (2) Does fine-tuning learn an equally compact solution to that of monolingual training?
We analyze the byte-level CRF because they are stronger than, or comparable to, all other models with no pretraining, and have the same number of parameters across all languages and settings (monolingual, polyglot, and fine-tuned). We perform our analysis on models without pretraining, as we wish to isolate the effects of polyglot learning on our models from external polyglot resources. We prune the lowest magnitude weights of each model in  $10\%$  increments and plot the average performance over time in Figure 3. Additionally, we define "over-pruning" to occur for language  $l$  and model  $m$  when pruning causes the performance of model  $m$  on language  $l$  to decrease by more than 1 F1 from model  $m$  's original performance. We plot the pruning threshold for each language and model<sup>10</sup> before "over-pruning" occurs in Figure 3 as well.
We find that polyglot models require more parameters than monolingual models to maintain their performance, but are significantly more efficient, i.e. they need much fewer than  $\hat{M}$  parameters. For example, the CoNLL polyglot model needs  $60\%$  of its parameters to maintain performance on all languages; English, Spanish, and Dutch require fewer parameters still. Compared to the total number of parameters needed by the four individual monolingual models combined  $(\hat{M})$ , the polyglot model needs only  $30\%$  of that, although this is paid for by an average decrease of 0.33 F1. This suggests that polyglot performance suffers due to over-sharing parameters, rather than
$^{9}$ Averaged across all CoNLL or LORELEI languages. For polyglot models we report the percentage required to maintain performance on each individual language using the same model.
8099
Figure 3: (a & b) Average F1 of Byte-CRF models as the pruning threshold increases. We find that monolingual models learn much more sparse solutions than polyglot models. Interestingly, fine-tuning does not recover the sparsity of the monolingual models. (c & d) The pruning thresholds before language performance drops by more than 1 F1 for each model. In the CoNLL setting, languages share nearly equally sparse solutions. However, in the LORELEI setting, the sparsity across all languages exhibits high variance, even in the fully shared polyglot model.
under-sharing, during joint optimization.
Additionally, we find that fine-tuning the polyglot models does not recover as sparse a solution as monolingual training. This finding suggests that either fine-tuning utilizes polyglot parameters to learn a denser solution than monolingual models, or that fine-tuning retains several high-magnitude polyglot weights not crucial to the target language. In the latter case, more sophisticated pruning criteria may be better suited to determining the sparsity of fine-tuned models, despite recent evidence indicating the strength of simple magnitude pruning (Gale et al., 2019).
5.3 Important Weights Across Languages
In addition to measuring the parameter efficiency of the polyglot models, we are interested in knowing how much overlap exists between the parameters which are most important for different languages, and how those parameters change during fine-tuning. This answers two important questions: 1) How do languages utilize shared polyglot parameters? 2) Does fine-tuning benefit from many or few polyglot weights?
To measure overlap between important weights for each language in a polyglot model, we compare the language-specific Fisher information matrix diagonals of the polyglot model. The Fisher information matrix has been used in this way
to measure individual parameter importance on a specific task, and has been shown to be effective for retaining important information across tasks during sequential learning (Kirkpatrick et al., 2016; Thompson et al., 2019).
For a given language  $l$  with  $N$  training examples we estimate the Fisher information matrix  $F^l$  with the empirical Fisher information matrix  $\bar{F}^l$ .  $F^l$  is computed via
$$
\frac {1}{N} \sum_ {i = 1} ^ {N} \underset {y \sim p _ {\theta}} {\mathbb {E}} \left[ \nabla_ {\theta} \log p _ {\theta} (y | x _ {i}) \nabla_ {\theta} \log p _ {\theta} (y | x _ {i}) ^ {T} \right]
$$
We take the diagonal values  $\bar{F}_{i,i}$  as an assignment of importance to  $\theta_{i}$ .
To compute the overlap of important weights shared between two tasks, we take the top  $5\%$ ,  $25\%$ , and  $50\%$  of weights from each layer for each task (given by the tasks' Fishers) and calculate the percentage overlap between them. We do this for two settings: First, we consider the percentage of weights shared between a specific language and all other languages in a polyglot model. Second, we examine the percentage of weights that remain important to a particular language after finetuning. We plot the average overlap across all lan
11The expectation over  $y\sim p_{\theta}$  is approximated by sampling exactly from the posterior of each  $x_{i}$ . We take 1,000 samples for each example.
Figure 4: (a) Percentage of important weight overlap between a single language and all other languages in the polyglot Byte-CRF LORELEI model (averaged over all languages). The top  $5\%$  of parameters for each language share little overlap with other languages, implying that the most important weights for each language are uniquely important to that language. (b) Overlap of important weights between the polyglot and fine-tuned Byte-CRF LORELEI model, for a given language (averaged over all languages). Only  $30\%$  of the top  $5\%$  of weights important to a given language are retained after fine-tuning, suggesting that fine-tuning targets the most important parameters for a language.
guages for each setting with our LORELEI Byte-CRF models in Figure 4.
We find that languages share a high number of important weights between each other in the polyglot model (40% overlap in the top 25% of weights of the LSTM layers), which helps explain how polyglot models are competitive, with fewer parameters, than multiple monolingual models. Interestingly, however, we find that the most important weights (top 5%) for each language share little overlap, implying that in polyglot learning, each language acquires parameters that are uniquely important to that language.
We additionally find that fine-tuning does not shift the importance of a significant number of weights (more than half of the top  $25\%$  important weights for a language in the polyglot model remain similarly important after fine-tuning). Surprisingly, the parameters that were most important to a language in the polyglot model are the
parameters that are the most affected during finetuning for that language. Thus, we see that language-specific fine-tuning retains the importance of many shared parameters, but the most important weights to that language are significantly affected.[12]
We explore the benefits of polyglot training for NER across a range of models. We find that, while not all models can benefit in performance from polyglot training, the parameters learned by those models can be leveraged in a language-specific way to consistently outperform monolingual models. We probe properties of polyglot NER models, and find that they are much more efficient than monolingual models in terms of the parameters they require, while generally maintaining a competitive performance across all languages. We show that the high amount of parameter sharing in polyglot models partially explains this, and additionally find that language-specific fine-tuning may use a large portion of those shared parameters. In future work, we will explore whether the observed trends hold in much larger polyglot settings, e.g. the Wikiann NER corpus (Pan et al., 2017b).
Finally, regarding the sharing of weights between languages in polyglot models, our key conclusion is that standard training objectives are unable to find an optimum which simultaneously achieves high task performance across all languages. With this in mind, exploring different training strategies, such as multi-objective optimization, may prove beneficial (Sener and Koltun, 2018). On the other hand, when the objective is to maximize performance on a single target language it may be possible to improve the proposed fine-tuning approach further using methods such as elastic weight consolidation (Kirkpatrick et al., 2016).
We would like to thank the anonymous reviewers for their helpful comments.
<sup>12</sup>Note that typically it is not reasonable to compare the weights of two different neural networks, as they are unidentifiable (Goodfellow and Vinyals, 2015). However, since one model is initialized from the other, we believe it is reasonable to characterize how weights shift during language-specific fine-tuning.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multisource cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098–3112, Florence, Italy. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale.
Ryan Cotterell and Kevin Duh. 2017. Low-resource named entity recognition with cross-lingual, character-level neural conditional random fields. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 91–96. Asian Federation of Natural Language Processing.
Silviu Cucerzan and David Yarowsky. 1999. Language independent named entity recognition combining morphological and contextual evidence. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora.
Arjun Das, Debasis Ganguly, and Utpal Garain. 2017. Named entity recognition with word embeddings and wikipedia categories for a low-resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(3):18.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Jacob Devlin. 2018. Multilingual bert readme document.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Greg Durrett and Dan Klein. 2015. Neural crf parsing. In Proceedings of the Association for Computational
Linguistics, Beijing, China. Association for Computational Linguistics.
Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multilingual named entity-annotated corpus using annotation projection. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pages 118-124, Hissar, Bulgaria. Association for Computational Linguistics.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Trevor Gale, Erich Elsen, and Sara Hooker. 2019. The state of sparsity in deep neural networks. *ArXiv*, abs/1902.09574.
Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language processing from bytes. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1296-1306. Association for Computational Linguistics.
Ian J. Goodfellow and Oriol Vinyals. 2015. Qualitatively characterizing neural network optimization problems. CoRR, abs/1412.6544.
Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1135-1143, Cambridge, MA, USA. MIT Press.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Lifu Huang, Heng Ji, and Jonathan May. 2019. Crosslingual multi-level adversarial transfer to enhance low-resource name tagging. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3823-3833, Minneapolis, Minnesota. Association for Computational Linguistics.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objective for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 36-44.
8102
Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017. Cross-lingual transfer learning for POS tagging without cross-lingual resources. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2832–2838, Copenhagen, Denmark. Association for Computational Linguistics.
Sungchul Kim, Kristina Toutanova, and Hwanjo Yu. 2012. Multilingual named entity recognition using parallel data and metadata from wikipedia. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL '12, pages 694-702, Stroudsburg, PA, USA. Association for Computational Linguistics.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114 13:3521-3526.
Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing universal dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2779-2795, Hong Kong, China. Association for Computational Linguistics.
Onur Kuru, Ozan Arkan Can, and Deniz Yuret. 2016. Charner: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 911-921, Osaka, Japan. The COLING 2016 Organizing Committee.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270. Association for Computational Linguistics.
Yann LeCun, John S. Denker, and Sara A. Solla. 1990. Optimal brain damage. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems 2, pages 598-605. Morgan-Kaufmann.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions
of the Association for Computational Linguistics, 5:365-378.
Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation, pages 76-79.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074. Association for Computational Linguistics.
Lluis Marquez, Adrià de Gispert, Xavier Carreras, and Lluis Padro. 2003. Low-cost named entity classification for Catalan: Exploiting multilingual resources and unlabeled data. In Proceedings of the ACL 2003 Workshop on Multilingual and Mixed-language Named Entity Recognition, pages 25-32, Sapporo, Japan. Association for Computational Linguistics.
Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap Translation for Cross-Linguual Named Entity Recognition. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019. Polyglot contextual representations improve crosslingual transfer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3912-3918, Minneapolis, Minnesota. Association for Computational Linguistics.
Rudra Murthy, Anoop Kunchukuttan, and Pushpak Bhattacharyya. 2018. Judicious selection of training data in assisting language for multilingual neural ner. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 401-406. Association for Computational Linguistics.
Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1470–1480, Vancouver, Canada. Association for Computational Linguistics.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017a. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1946–1958.
8103