SlowGuess/ACL_2020_OCR huggingface dataset:
David Mueller $^{1,2}$  Nicholas Andrews $^{2}$  Mark Dredze $^{1,2}$
$^{1}$ Center for Language and Speech Processing, Johns Hopkins University  $^{2}$ Human Language Technology Center of Excellence, Johns Hopkins University
dam, noa@jhu.edu mdredze@cs.jhu.edu
Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and significantly outperform their monolingual counterparts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolingual counterparts. We find that polyglot models efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters.
Multilingual learning—using data from multiple languages to train a single model—can take many forms, such as adapting a model from a high-resource to low-resource language (Xie et al., 2018; Ni et al., 2017; Mayhew et al., 2017; Cotterell and Duh, 2017; Wu and Dredze, 2019; Marquez et al., 2003), taking advantage of beneficial multilingual features or datasets (Kim et al., 2012; Ehrmann et al., 2011; Tackstrom, 2012), and unsupervised representation learning (Devlin et al., 2018a). We adopt the term “Polyglot” from Tsvetkov et al. (2016) to refer to models that are trained on and applied to multiple languages. There are several advantages to training a single
polyglot model across languages. Single models ease production requirements; only one model need be maintained. They can be more efficient, using fewer parameters than multiple monolingual models. Additionally, they can enable multilingual transfer (Devlin, 2018; Wu and Dredze, 2019; Pires et al., 2019).
However, a key goal of polyglot learning concerns producing a single model that does better on each language than a monolingual model. In the context of named entity recognition, we may expect aspects of the task to transfer across languages. For example, since entity names tend to be transliterated or directly used across languages, even distant languages may see benefit from training a single model, e.g. "Apple" (company) is rendered as such in French rather than as "Pomme." Intuitively, the more similar and the larger the set of languages, the more we should expect to see a benefit from considering them jointly. These polyglot models can take advantage of different sets of labeled corpora in different languages (Gillick et al., 2016; Mulcaire et al., 2019).
Nevertheless, progress towards this goal remains mixed; polyglot models often do not improve results in each language (Mulcaire et al., 2019; Kondratyuk and Straka, 2019; Upadhyay et al., 2018; Conneau et al., 2019). Models trained across all languages come close but typically fail to outperform monolingual models. Thus, while multilingual learning can benefit low resource languages through transfer and simplify models by sharing one across all languages, it fails to realize a key goal: improving results in each language. Our experiments in §4 confirm this negative result in two different multilingual settings for 4 different neural NER models.
Our first contribution is a technique in which a polyglot NER model can be adapted to a target language by fine-tuning on monolingual data.
8093
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8093-8104 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
A similar continued training approach to transfer has been explored for domain adaptation in neural machine translation (Luong and Manning, 2015; Khayrallah et al., 2018); we show that it works with polyglot models for NER, improving performance by up to  $3F_{1}$  over monolingual baselines.
Our second contribution is an explanation of the surprising effectiveness of this technique through an extensive empirical study of polyglot models for NER. We compare several types of neural NER models, including three character (or byte) level architectures, and evaluate transfer across a small (4) and large (10) set of languages. In particular, we find that:
- §4 Other than Byte-to-Span (BTS; Gillick et al., 2016), most NER architectures do not benefit from polyglot training. Still, simpler models than BTS, with more inductive bias, can outperform BTS in both monolingual and polyglot settings.
- §5.2 Polyglot models are more efficient than monolingual models in that for a given level of performance, they require vastly fewer parameters. This suggests that many parameters are shared cross-lingually.
- §4.2 Polyglot weights transfer to unseen languages with mixed results. In particular, transfer can occur when there is high lexical overlap or closely related languages in the polyglot training set.
- §5.3 Languages share a large number of important parameters between each other in polyglot models, and fine-tuning may utilize those parameters to strengthen its performance.
To our knowledge, ours is the first systematic study of polyglot NER models.
There is a long history of multilingual learning for NER (Kim et al., 2012; Ehrmann et al., 2011; Tackström, 2012). This work has is driven by an interest in learning NER models for many languages (Cucerzan and Yarowsky, 1999; Pan et al., 2017a) and the relative lack of data for many languages of interest (Das et al., 2017).
Polyglot Models Johnson et al. (2017) and Lee et al. (2017) showed that a single neural MT model could benefit from being trained in a multilingual setting. Gillick et al. (2016) showed similar results for NER, presenting a model that benefited
from learning to perform NER on 4 languages at once. We find that other polyglot NER models are rarely better than monolingual models in terms of absolute performance.
Mulcaire et al. (2019) showed that polyglot language model pretraining can help improve performance on NER tasks, although polyglot NER training hurts. However, multilingual BERT (Devlin et al., 2018b), when compared to monolingual BERT performance on NER, shows that polyglot pretraining is not always beneficial for downstream tasks.
Finally, most recently, Kondratyuk and Straka (2019) showed how to train a single model on 75 languages for dependency parsing while retaining competitive performance or improving performance, mostly on low-resource languages. This work is closely related to ours, although we are predominantly interested in how we can leverage polyglot learning to improve performance across all languages.
Cross-lingual Models Cross-lingual transfer leverages labeled data from different source languages to augment data for a target language. Rahimi et al. (2019) do this on a massive scale for NER, leveraging over 40 languages for cross-lingual transfer. Xie et al. (2018) employed self-attention to combat word-order differences when transferring parameters from high-resource languages to low-resource.
Much work in this space has looked at how to leverage a mixture of shared features and language-specific features (Kim et al., 2017), similar to domain adaptation techniques Daumé III (2007). Recently, a lot of this work has focused on using adversarial models to force models to learn language-agnostic feature spaces (Chen et al., 2019; Huang et al., 2019). These works show, similar to our work, that it is possible to leverage multilingual data to increase performance across languages.
We evaluate three polyglot NER neural models.
3.1 Word Level CRF
The Neural (BiLSTM) CRF is a standard model for sequence labeling tasks (Ma and Hovy, 2016; Durrett and Klein, 2015). Our implementation
We release the code for these models at https://github.com/davidandym/multilingual-NER
8094
Model	Eng	Deu	Nld	Spa	Avg	Amh	Ara	Fas	Hin	Hun	Ind	Som	Swa	Tgl	Vie	Avg
Character CRF
Monolingual	84.91	71.39	78.96	82.60	79.45	60.62	43.22	45.11	62.12	60.47	62.14	61.75	68.04	84.13	47.31	59.49
Polyglot	83.38	70.86	79.38	81.64	77.85	59.39	43.25	43.20	62.88	60.86	64.59	65.45	68.32	84.80	49.71	59.87
Finetuned	86.49	72.95	80.91	82.72	80.82	59.86	44.69	46.85	68.30	65.21	67.15	66.11	70.07	87.03	51.80	62.71
Byte CRF
Monolingual	85.75	71.42	78.36	81.19	79.18	59.13	44.95	44.76	65.89	57.91	61.46	61.05	67.09	84.46	48.73	59.54
Polyglot	83.79	71.54	79.43	80.25	78.75	57.03	42.88	41.88	65.10	60.46	61.07	62.22	68.40	82.75	47.27	58.90
Finetuned	86.68	73.02	80.09	82.95	80.69	59.37	42.69	45.25	67.68	63.91	64.38	64.92	70.78	86.25	51.14	61.64
CharNER
Monolingual	83.83	69.30	79.60	79.46	78.05	54.33	36.31	40.68	62.03	53.04	58.05	56.88	63.70	81.04	39.64	54.53
Polyglot	84.14	69.19	78.94	79.39	77.92	49.64	36.98	37.41	60.02	49.37	55.51	58.56	63.49	79.36	44.50	53.48
Finetuned	85.23	70.60	81.00	82.00	79.70	53.46	40.15	39.20	65.57	59.84	60.70	59.09	68.85	84.61	45.47	57.70
Byte To Span
Monolingual	87.91	63.92	71.34	73.07	74.06	48.23	39.41	26.76	19.01	44.51	54.32	58.81	54.27	71.76	26.90	44.50
Polyglot	86.43	71.10	76.11	74.26	76.98	46.41	41.59	40.09	55.69	60.53	57.58	62.30	54.78	74.52	43.95	53.64
Multilingual BERT
Monolingual	90.94	81.50	88.62	88.16	87.31	-	48.36	56.42	72.52	66.99	78.32	62.69	72.18	86.13	54.18	66.75
Polyglot	90.67	80.96	87.48	87.04	86.53	-	48.33	56.92	74.81	68.16	77.56	59.29	71.92	87.59	57.06	66.84
Finetuned	91.08	81.27	88.74	86.87	86.99	-	49.94	54.67	76.83	69.52	80.14	62.70	73.16	88.05	56.74	69.97
Table 1: Performance for monolingual, multilingual, and finetuned models trained on either CoNLL (left) or LORELEI (right) data sets. The results are taken from the best model out of 5 random seeds, as measured by dev performance. Almost every model achieves the best performance in the finetuned setting, indicating that multilingual pretraining is learning transferable parameters, but multilingual models are not able to use them effectively across all languages simultaneously. Note that we do not evaluate Amharic with mBERT, because the Amharic script is not a part of mBERT's vocabulary.
broadly follows the description in Lample et al. (2016), and we consider three different variants of this model.
The first two are character- and byte-level models. We consider these since Gillick et al. (2016) showed that multilingual transfer could occur across byte-level representations and we were interested in whether characters produced similar results when more diverse languages were involved. Each word passes through a multi-layer BiLSTM as a sequence of characters or bytes to produce word-level representations. Word-level representations feed into a sentence-level BiLSTM, which outputs, for each time step, logits for all possible labels. The logits are then fed into a CRF model (Lafferty et al., 2001) trained to maximize the log-likelihood of the gold label sequences.
The third variant of this model uses contextualized representations from multilingual BERT (mBERT) (Devlin et al., 2018b). This model is similar to the one described above, with the key difference being that word-level representation are obtained using a pretrained subword-level BERT model, as opposed to being built from raw characters/bytes. As is done in the original BERT paper,
we treat the representation of the first subword of each word as a representation for that word, and take the concatenation of the outputs of the last 4 layers at that subword position as our final word representation.
3.2 CharNER
CharNER (Kuru et al., 2016) is a deep neural sequence labeling architecture which operates strictly at the character level during training, but uses word-level boundaries during inference. The model runs a 5-layer BiLSTM over sequences of characters, and is trained to predict the NER tag for each character of the sequence (without BIO labels). During inference a Viterbi decoder with untrained transition parameters enforces consistent character level tags across each word; no heuristics and little post-processing is necessary to obtain word-level BIO labels.
To compare with the other architectures, we apply this model to bytes and evaluate its polyglot performance. Intuitively, we expect this model to do better than a word-level CRF at seeing beneficial transfer across languages, as it is closer to the model of Gillick et al. (2016): a deep, byte-level model that performs inference at the level of individual bytes.
$^{2}$ Early experiments found these models suffered much less from multilingual training than subword/word models.
3.3 Byte to Span (BTS)
BTS is a sequence-to-sequence model operating over byte sequences (Gillick et al., 2016). The input consists of a window of UTF-8 bytes, and the output is sequences with sufficient statistics of labeled entity spans occurring in the input sequence. Because byte sequences are long BTS operates over a sliding window of 60 bytes, treating each segment independently; the model's entire context is always limited to 60 bytes. By consuming bytes and producing byte annotations, it has the attractive quality of being truly language-agnostic, without any language specific preprocessing.
Despite obviating the need for language-specific preprocessing, BTS achieves comparable results to more standard model architectures with no pretraining information. Additionally, it showed significant improvement in monolingual CoNLL performance after being trained on all 4 CoNLL languages. In this paper, we find that this trend holds in our multilingual settings, although our results show lower overall numbers to those reported in Gillick et al. (2016).
3.4 Hyperparameters
All experiments are run on GeForce RTX 2080 Ti GPUs, using Tensorflow (Abadi et al., 2016).
CRF The character- and byte-level neural CRF use a sub-token BiLSTM encoder with 2-layers and 256 hidden units. The sentence-level BiLSTM has 1-layer with 256 hidden units. All characters and bytes have randomly initialized embeddings of size 256. We optimized these parameters with grid-search over 1-3 layers at each level and hidden sizes of {128, 256, 512}. We train using Adam with a learning rate of 0.001 and tune the early stop parameter for each model based on development set F1 performance.
CharNER Our CharNER model operates over bytes rather than characters. It uses the same hyperparameters reported in Kuru et al. (2016), (5
Language	Code	Family	Genus	Script	# Train Sent.
CoNLL
English	eng	Indo-European	Germanic	Latin	11,663
Spanish	spa	Indo-European	Romance	Latin	8,323
German	deu	Indo-European	Germanic	Latin	12,152
Dutch	nld	Indo-European	Germanic	Latin	15,806
LORELEI
Amharic	amh	Afroasiatic	Semitic	Ge'ez	4,923
Arabic	ara	Afroasiatic	Semitic	Arabic	4,990
Farsi	fas	Indo-Iranian	-	Arabic	3,849
Hindi	hin	Indo-European	Indo-Aryan	Devanagari	4,197
Hungarian	hun	Uralic	Ugric	Latin	4,846
Indonesian	ind	Austronesian	Malayo-Polynesian	Latin	4,605
Somali	som	Afroasiatic	Cushitic	Latin	3,253
Swahili	swa	Niger-Congo	Bantu	Latin	3,318
Tagalog	tgl	Austronesian	-	Latin	4,780
Vietnamese	vie	Austroasiatic	Vietic	Latin (Viet.)	4,042
LORELEI - held out for zeroshot
Russian	rus	Indo-European	Slavic	Cyrillic	6,480
Bengali	ben	Indo-European	Indo-Aryan	Bengali	7,538
Uzbek	uzb	Turkic	-	Arabic	11,323
Yoruba	yor	Niger-Congo	-	Latin	1,753
Table 2: Different sets of languages we used, their sources, family and genus, script, and training set size.
layers with hidden size 128, Adam Optimizer) with a byte dropout of 0.2, and dropout rates of 0.8 on the final layer, and 0.5 on the other layers. We also train our models using a learning rate of 0.001 and early stop based on development set F1 performance.
BTS For BTS we use the same training scheme and hyperparameters reported in Gillick et al. (2016). Since we do not have document-level information in LORELEI, we treat each separate language dataset as its a whole document and slide a window across the entire dataset at once. We train using SGD (Adam performed much worse), with a learning rate of 0.3, and similarly, early stop based on development set F1 performance.
Each LORELEI language has less than half the data of a CoNLL language, but in total, the two datasets are roughly equal in size. The CoNLL setting consists of European languages in the same alphabet, and prior work has shown beneficial transfer in this setting (Gillick et al., 2016). LORELEI is more challenging because it contains more distantly related languages.
We train a monolingual NER model for each language (14 models) and two polyglot models: CoNLL and LORELEI. For polyglot training we concatenate each annotated language-specific dataset into one combined corpus. Because our language-specific datasets are comparable in size
54 layers with 320 hidden units, byte dropout of 3.0 and layer dropout of 5.0.
3For a PER span at bytes 5-10, the correct output sequence is  $y = S:5$ ,  $L:5$ , PER, STOP
<sup>4</sup>We reimplemented BTS based on correspondence with the model authors. We matched the published results on CoNLL English, and the same overall trends, but could not match the other three CoNLL languages. Despite significant effort, two differences remained: the authors could not share their proprietary implementation or deep learning library, and reported using more byte segments than is available in our CoNLL dataset.
8096
we do not correct for minor size differences. All models were trained over 5 random seeds, with the best model selected by development performance. For polyglot models, we select the best model using the average development performance across all languages.
Results Table 1 reports test performance. With few exceptions, polyglot training does worse than monolingual. In some cases, the two settings do nearly the same (such as Character and mBERT CRFs on LORELEI) but we do not see improved results from a polyglot model.
Murthy et al. (2018) found that languages with different label distributions do worse for transfer. We find large label distribution changes in CoNLL, but not LORELEI. To determine if this could explain polyglot NER failures in CoNLL, we allow our CRF models to learn language-specific label distributions via language-specific CRF transition parameters. However, we saw little difference in the results for either CoNLL or LORELEI (no more than 0.5 F1 on any language). This suggests that other factors are preventing more language transfer.
The exception to these observations is the BTS model, which showed significant improvements in the polyglot settings, matching the conclusion of Gillick et al. (2016). However, our implementation failed to match the higher numbers of the original paper, and so the model is significantly worse overall compared to the other NER models. Perhaps the unique architecture of BTS enables it to improve in the polyglot setting. However, if BTS requires more training data to achieve results similar to the other models, the polyglot improvements may not hold up.
Conclusion Polyglot NER models fail to improve over their monolingual counterparts, despite using 4 (CoNLL) or 10 (LORELEI) times more labeled data. Discrepancies of label priors between languages do not, by themselves, account for this.
4.1 Target Language Polyglot Adaptation
While polyglot models perform worse than monolingual models, they are competitive. This suggests that polyglot models may be successfully learning multilingual representations, but that the optimization procedure is unable to find a global
Language	Monoling.	Poly. (Zero-shot)	Poly. (Fine-tuned)
Russian	43.97	1.61	41.55
Bengali	76.10	2.08	76.63
Uzbek	65.39	14.54	61.10
Yoruba	62.66	29.02	64.95
Table 3: F1 of a Byte-level CRF on 4 different lorelei language datasets, compared to the performance of the multilingual model which was not trained on any of these 4 languages, as well as the multilingual model after finetuning. The results are mixed - moreover, zero-shot performance does not seem to be a good indicator of transferability.
minimum for all languages. To test this theory, we fine-tune the polyglot model separately for each language. We treat the parameters of the polyglot NER models as initializations for monolingual models of each language, and we train these models in the same fashion as the monolingual models, with the exception of using a different initial step size. With few exceptions, fine-tuned polyglot models surpass their monolingual counterparts (Table 1), improving up to  $3F_{1}$  over monolingual baselines.
Conclusion This demonstrates that the polyglot models are in fact learning more from observing multiple languages, and that this information can transfer to each language. Additionally, this indicates that the ideal optima for a monolingual model may not be achievable using standard training objectives without observing other languages; we found more regularization did not help the monolingual models. However, jointly optimizing all languages naively may provide too challenging an optimization landscape to obtain that optima for each language simultaneously.
4.2 Novel language transfer
Finally, since the polyglot models demonstrate the ability to transfer information between languages, we ask: can these models generalize to unseen languages? We consider a similar approach to the previous section, except we now fine-tune the polyglot model on a novel language for which we have supervised NER data. In this setting, we only consider byte-level models, since byte vocabularies mean we can use the same parameters on unseen languages with different character sets. We select 4 additional LORELEI languages: Rus
We use the Adam optimizer settings saved from multilingual training.
6 A uniform sampling strategy is recommended for language combinations with significant size discrepancies.
8097
sian, Yoruba, Bengali, and Uzbek. For comparison, we train monolingual Byte CRF models (from scratch), following the same optimization protocols, as described above.
Table 3 shows results for the monolingual model, polyglot fine-tuned, and the polyglot model evaluated without any fine-tuning (zero-shot). Unsurprisingly, the polyglot model does poorly in the zero-shot setting as it has never seen the target language. However, sharing a script with some languages in the polyglot training set can lead to significantly better than random performance (as in the case of Yoruba and Uzbek). In the fine-tuning setting, the results are mixed. Yoruba, which enjoys high script overlap with the polyglot training set, sees a large boost in performance from utilizing the polyglot parameters, whereas Uzbek, which has moderate script overlap but no family overlap, is hurt by it. Russian and Bengali have no script overlap with the polyglot training set, but Bengali, which is closely related to Hindi (sharing family and genus) sees a moderate amount of transfer, while Russian, which is not closely related to any language in the training set, is negatively impacted from using the polyglot weights.
Conclusion The transferability of the polyglot parameters to unseen languages depends on a variety of factors. We conjecture that these factors are partially connected to relatedness to languages in the original polyglot training set.
5 How do Polyglot Models Learn?
We now turn our attention towards understanding how polyglot models are transferring information across languages. We examine the types of errors made in each setting, as well as how polyglot models efficiently use parameters and how parameter weights are shared across languages.
5.1 Error Analysis
We broadly examine the types of errors made across each of our regimes, focusing on results from the Byte-CRF model. To explore what kinds of errors polyglot fine-tuning targets we plot, in Figure 1, the counts of recall errors (including O-tags) on validation data made by the monolingual and polyglot models, compared to the fine-tuned model. We find that polyglot models tend to make more errors on O-tags, indicating a tendency towards making precision errors, but that