SlowGuess/NAACL_2021_OCR huggingface dataset:
Benjamin Muller†* Antonios Anastasopoulos‡ Benoit Sagot† Djamé Seddah†
†Inria, Paris, France *Sorbonne Université, Paris, France ‡Department of Computer Science, George Mason University, USA firstname.lastname@inria.fr antonis@gmu.edu
Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. We show that transliterating those languages significantly improves the potential of large-scale multilingual language models on downstream tasks. This result provides a promising direction towards making these massively multilingual models useful for a new set of unseen languages. $^{1}$
Language models are now a new standard to build state-of-the-art Natural Language Processing (NLP) systems. In the past year, monolingual language models have been released for more than 20 languages including Arabic, French, German, and Italian (Antoun et al., 2020; Martin et al., 2020; de Vries et al., 2019; Canete et al., 2020; Kuratov and Arkhipov, 2019; Schweter, 2020, inter alia). Additionally, large-scale multilingual models covering more than 100 languages are now available (XLM-R by Conneau et al. (2020) and mBERT by Devlin et al. (2019)). Still, most of the  $6500+$  spoken languages in the world (Hammarstrom, 2016) are not covered—remaining unseen—by
those models. Even languages with millions of native speakers like Sorani Kurdish (about 7 million speakers in the Middle East) or Bambara (spoken by around 5 million people in Mali and neighboring countries) are not covered by any available language models at the time of writing.
Even if training multilingual models that cover more languages and language varieties is tempting, the curse of multilinguality (Conneau et al., 2020) makes it an impractical solution, as it would require to train ever larger models. Furthermore, as shown by Wu and Dredze (2020), large-scale multilingual language models are sub-optimal for languages that are under-sampled during pretraining.
In this paper, we analyze task and language adaptation experiments to get usable language model-based representations for under-studied low resource languages. We run experiments on 15 typologically diverse languages on three NLP tasks: part-of-speech (POS) tagging, dependency parsing (DEP) and named-entity recognition (NER).
Our results bring forth a diverse set of behaviors that we classify in three categories reflecting the abilities of pretrained multilingual language models to be used for low-resource languages. We dub those categories Easy, Intermediate and Hard.
Hard languages include both stable and endangered languages, but they predominantly are languages of communities that are majorly underserved by modern NLP. Hence, we direct our attention to these Hard languages. For those languages, we show that the script they are written in can be a critical element in the transfer abilities of pretrained multilingual language models. Transliterating them leads to large gains in performance outperforming non-contextual strong baselines. To sum up, our contributions are the following:
- We propose a new categorization of the low-resource languages that are unseen by available language models: the Hard, the Intermediate and the Easy languages.
$^{1}$ Code available at https://github.com/benjaminmlr/mbert-unseen-languages.git
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448-462 June 6-11, 2021. ©2021 Association for Computational Linguistics
- We show that Hard languages can be better addressed by transliterating them into a better-handled script (typically Latin), providing a promising direction towards making multilingual language models useful for a new set of unseen languages.
As Joshi et al. (2020) vividly illustrate, there is a large divergence in the coverage of languages by NLP technologies. The majority of the  $6500+$  of the world's languages are not studied by the NLP community, since most have few or no annotated datasets, making systems' development challenging.
The development of such models is a matter of high importance for the inclusion of communities, the preservation of endangered languages and more generally to support the rise of tailored NLP ecosystems for such languages (Schmidt and Wiegand, 2017; Stecklow, 2018; Seddah et al., 2020). In that regard, the advent of the Universal Dependencies project (Nivre et al., 2016) and the WikiAnn dataset (Pan et al., 2017) have greatly increased the number of covered languages by providing annotated datasets for more than 90 languages for dependency parsing and 282 languages for NER.
Regarding modeling approaches, the emergence of multilingual representation models, first with static word embeddings (Ammar et al., 2016) and then with language model-based contextual representations (Devlin et al., 2019; Conneau et al., 2020) enabled transfer from high to low-resource languages, leading to significant improvements in downstream task performance (Rahimi et al., 2019; Kondratyuk and Straka, 2019). Furthermore, in their most recent forms, these multilingual models process tokens at the sub-word level (Kudo and Richardson, 2018). As such, they work in an open vocabulary setting, only constrained by the pretraining character set. This flexibility enables such models to process any language, even those that are not part of their pretraining data.
When it comes to low-resource languages, one direction is to simply train contextualized embedding models on whatever data is available. Another option is to adapt/fine-tune a multilingual pretrained model to the language of interest. We briefly discuss these two options.
Pretraining language models on a small amount of raw data Even though the amount
of pretraining data seems to correlate with downstream task performance (e.g. compare BERT and RoBERTa (Liu et al., 2020)), several attempts have shown that training a model from scratch can be efficient even if the amount of data in that language is limited. Indeed, Ortiz Suárez et al. (2020) showed that pretraining ELMo models (Peters et al., 2018) on less than 1GB of text leads to state-of-the-art performance while Martin et al. (2020) showed that pretraining a BERT model on as few as 4GB of diverse enough data results in state-of-the-art performance. Micheli et al. (2020) further demonstrated that decent performance was achievable with only 100MB of raw text data.
Adapting large-scale models for low-resource languages Multilingual language models can be used directly on unseen languages, or they can also be adapted using unsupervised methods. For example, Han and Eisenstein (2019) successfully used unsupervised model adaptation of the English BERT model to Early Modern English for sequence labeling. Instead of fine-tuning the whole model, Pfeiffer et al. (2020) recently showed that adapter layers (Houlsby et al., 2019) can be injected into multilingual language models to provide parameter efficient task and language transfer.
Still, as of today, the availability of monolingual or multilingual language models is limited to approximately 120 languages, leaving many languages without access to valuable NLP technology, although some are spoken by millions of people, including Bambara and Sorani Kurdish, or are an official language of the European Union, like Maltese.
What can be done for unseen languages? Unseen languages strongly vary in the amount of available data, in their script (many languages use non-Latin scripts such as Sorani Kurdish and Mingrelian), and in their morphological or syntactical properties (most largely differ from high-resource Indo-European languages). This makes the design of a methodology to build contextualized models for such languages challenging at best. In this work, by experimenting with 15 typologically diverse unseen languages, (i) we show that there is a diversity of behavior depending on the script, the amount of available data, and the relation to the pretraining languages; (ii) Focusing on the unseen languages that lag in performance compared to their easier-to-handle counterparts, we show that the script plays a
critical role in the transfer abilities of multilingual language models. Transliterating such languages to a script which is used by a related language seen during pretraining can lead to significant improvement in downstream performance.
We will refer to any languages that are not covered by pretrained language models as "unseen." We select a small portion of those languages within a large scope of language families and scripts. Our selection is constrained to 15 typologically diverse languages for which we have evaluation data for at least one of our three downstream tasks. Our selection includes low-resource Indo-European and Uralic languages, as well as members of the Bantu, Semitic, and Turkic families. None of these 15 languages are included in the pretraining corpora of mBERT. Information about their scripts, language families, and amount of available raw data can be found in the Appendix in Table 12.
To perform pretraining and fine-tuning on monolingual data, we use the deduplicated datasets from the OSCAR project (Ortiz Suárez et al., 2019). OSCAR is a corpus extracted from a Common Crawl Web snapshot. It provides a significant amount of data for all the unseen languages we work with, except for Buryat, Meadow Mari, Erzya and Livvi for which we use Wikipedia dumps and for Narabizi, Naija and Faroese, for which we use data collected by Seddah et al. (2020), Caron et al. (2019) and Biemann et al. (2007) respectively.
3.2 Non-contextual Baselines
For parsing and POS tagging, we use the UDPipe future system (Straka, 2018) as our baseline. This model is a LSTM-based (Hochreiter and Schmidhuber, 1997) recurrent architecture trained with pretrained static word embedding (Mikolov et al., 2013) (hence our non-contextual characterization) along with character-level embeddings. This system was ranked in the very first positions for parsing and tagging in the CoNLL shared task 2018 (Zeman and Hajic, 2018). For NER we use the LSTM-CRF model with character and word level embedding using Qi et al. (2020) implementation.
In all our study, we train our language models using the Transformers library (Wolf et al., 2020).
MLM from scratch The first approach we evaluate is to train a dedicated language model from scratch on the available raw data we have. To do so, we train a language-specific SentencePiece tokenizer (Kudo and Richardson, 2018) before training a Masked-Language Model (MLM) using the RoBERTa (base) architecture and objective functions (Liu et al., 2019). As we work with significantly smaller pretraining sets than in the original setting, we reduce the number of layers to 6 layers in place of the original 12 layers.
Multilingual Language Models We want to assess how large-scale multilingual language models can be used and adapted to languages that are not in their pretraining corpora. We work with the multilingual version of BERT (mBERT) trained on the concatenation of Wikipedia corpora in 104 languages (Devlin et al., 2019). We also ran experiments with the XLM-R base version (Conneau et al., 2020) trained on 100 languages using data extracted from the Web. As the observed behaviors are very similar between both models, we only report results using mBERT. Note that mBERT is highly biased toward Indo-Europeans languages written in the Latin script. More than  $77\%$  of the subword vocabulary are in the Latin script while only  $1\%$  are in the Georgian script (Ács, 2019).
Adapting Multilingual Language Models to unseen languages with MLM-TUNING Following previous work (Han and Eisenstein, 2019; Karthikeyan et al., 2019; Pfeiffer et al., 2020), we adapt large-scale multilingual models by fine-tuning them with their Mask-Language-Model objective directly on the available raw data in the unseen target language. We refer to this process as MLM-TUNING. We will refer to a MLM-tuned mBERT model as mBERT+MLM.
3.4 Downstream Tasks
We perform experiments on POS tagging, Dependency Parsing (DEP), and Name Entity Recognition (NER). We use annotated data from the Universal Dependency project (Nivre et al., 2016) for POS tagging and parsing, and the WikiAnn dataset (Pan et al., 2017) for NER. For POS tagging and NER, we append a linear classifier layer on top of
<sup>2</sup>http://commoncrawl.org/
Model	UPOS	LAS	NER
MBERT	MBERT+MLM	MLM	Baseline	MBERT	MBERT+MLM	MLM	Baseline	MBERT	MBERT+MLM	MLM	Baseline
Faroese	96.3	96.5	91.1	95.4	84.0	86.4	67.6	83.1	52.1	58.3	39.3	44.8
Naija	89.3	89.6	87.1	89.2	71.5	69.2	63.0	68.3	-	-	-	-
Swiss German	76.7	78.7	65.4	75.2	41.2	69.6	30.0	32.2	-	-	-	-
Mingrelian	-	-	-	-	-	-	-	-	53.6	68.4	42.0	48.2
Table 1: Easy Languages POS, Parsing and NER scores comparing mBERT, mBERT+MLM and monolingual MLM to strong non-contextual baselines when trained and evaluated on unseen languages. Easy Languages are the ones on which mBERT outperforms strong baselines out-of-the-box. Baselines are LSTM based models from UDPipe-future (Straka, 2018) for parsing and POS tagging and Stanza (Qi et al., 2020) for NER.
the language model. For parsing, following Kondratyuk and Straka (2019), we append a Bi-Affine Graph prediction layer (Dozat and Manning, 2017). We refer to the process of fine-tuning a language model in a task-specific way as TASK-TUNING. $^3$
3.5 Dataset Splits
For each task and language, we use the provided training, validation and test dataset split except for the ones that have less than 500 training sentences. In this case, we concatenate the training and test set and perform 8-folds cross-Validation and use the validation set for early stopping. If no validation set is available, we isolate one of the folds for validation and report the test scores as the average of the other folds. This enables us to train on at least 500 sentences in all our experiments (except for Swiss German for which we only have 100 training examples) and reduce the impact of the annotated dataset size on our analysis. Since cross-validation results in training on very limited number of examples, we refer to training in this cross-validation setting as few-shot learning.
4 The Three Categories of Unseen Languages
For each unseen language and each task, we experiment with our three modeling approaches: (a) Training a language model from scratch on the available raw data and then fine-tuning it on any available annotated data in the target language. (b) Fine-tuning mBERT with TASK-TUNING directly on the target language. (c) Finally, adapting mBERT to the unseen language using MLM-TUNING before fine-tuning it in a supervised way on the target language. We then compare all these experiments to our non-contextual strong baselines. By doing so, we can assess if language models are
Figure 1: Visualizing our Typology of Unseen Languages. (X,Y) positions are computed for each language and each task as follows: given the score of mBERT denoted  $s$  and  $s_0$  the baseline score:  $X = \frac{s_m BERT - s_0}{s_0}$ ,  $Y = \frac{s_m BERT + MLM - s_0}{s_0}$
Easy Languages are the ones on which mBERT performs better than the baseline without MLM-TUNING and Intermediate languages are the ones that require MLM-TUNING. For Hard languages, mBERT under-performs the baselines in all settings.
a practical solution to handle each of these unseen languages.
Interestingly we find a large diversity of behaviors across languages regarding those language model training techniques. As summarized in Figure 1, we observe three clear clusters of languages.
The first cluster, which we dub "Easy", corresponds to the languages that do not require extra MLM-TUNING for mBERT to achieve good performance. mBERT has the modeling abilities to process such languages without relying on raw data and can outperform strong non-contextual baselines as such. In the second cluster, the "Intermediate" languages require MLM-TUNING. mBERT is not able to beat strong non-contextual baselines using only TASK-TUNING, but MLM-TUNING enables it to do so. Finally, Hard languages are those on which mBERT fails to deliver any decent per
3Details about optimization can be found in Appendix B
Model	UPOS	LAS	NER
MBERT	MBERT+MLM	MLM	Baseline	MBERT	MBERT+MLM	MLM	Baseline	MBERT	MBERT+MLM	MLM	Baseline
Maltese	92.0	96.4	92.05	96.0	74.4	82.1	66.5	79.7	61.2	66.7	62.5	63.1
Narabizi	81.6	84.2	71.3	84.2	56.5	57.8	41.8	52.8	-	-	-	-
Bambara	90.2	92.6	78.1	92.3	71.8	75.4	46.4	76.2	-	-	-	-
Wolof	92.8	95.2	88.4	94.1	73.3	77.9	62.8	77.0	-	-	-	-
Erzya	89.3	91.2	84.4	91.1	61.2	66.6	47.8	65.1	-	-	-	-
Livvi	83.0	85.5	81.1	84.1	36.3	42.3	35.2	40.1	-	-	-	-
Mari	-	-	-	-	-	-	-	-	55.2	57.6	44.0	56.1
Table 2: Intermediate Languages POS, Parsing and NER scores comparing mBERT, mBERT+MLM and monolingual MLM to strong non-contextual baselines when trained and evaluated on unseen languages. Intermediate Languages are the ones for which mBERT requires MLM-TUNING to outperform the baselines.
formance even after MLM- and TASK- fine-tuning. mBERT simply does not have the capacity to learn and process such languages.
We emphasize that our categorization of unseen languages is only based on the relative performance of mBERT after fine-tuning compared to strong non-contextual baseline models. We leave for future work the analysis of the absolute performance of the model on such languages (e.g. analysing the impact of the fine-tuning data set size on mBERT's downstream performance).
In this section, we present our results in detail in each of these language clusters and provide insights into their linguistic properties.
4.1 Easy
Easy languages are the ones on which mBERT delivers high performance out-of-the-box, compared to strong baselines. We classify Faroese, Swiss German, Naija and Mingrelian as easy languages and report performance in Table 1.
We find that those languages match two conditions:
- They are closely related to languages used during MLM pretraining
- These languages use the same script as such closely related languages.
Such languages benefit from multilingual models, as cross-lingual transfer is easy to achieve and hence quite effective.
More details about those languages can be found in Appendix C.
4.2 Intermediate
The second type of languages (which we dub "Intermediate") are generally harder to process for pretrained MLMs out-of-the-box. In particular, pretrained multilingual language models are typically outperformed by a non-contextual strong baselines. Still, MLM-TUNING has an important impact and
leads to usable state-of-the-art models.
A good example of such an intermediate language is Maltese, a member of the Semitic language but using the Latin script. Maltese has not been seen by mBERT during pretraining. Other Semitic languages though, namely Arabic and Hebrew, have been included in the pretraining languages. As seen in Table 2, the non-contextual baseline outperforms mBERT. Additionally, a monolingual MLM trained on only 50K sentences matches mBERT performance for both NER and POS tagging. However, the best results are reached with MLM-TUNING: the proper use of monolingual data and the advantage of similarity to other pretraining languages render Maltese a tackle-able language as shown by the performance gain over our strong non-contextual baselines.
Our Maltese dependency parsing results are in line with those of Chau et al. (2020), who also showed that MLM-TUNING leads to significant improvements. They also additionally showed that a small vocabulary transformation allowed fine-tuning to be even more effective and gain 0.8 LAS points more. We further discuss the vocabulary adaptation technique of Chau et al. (2020) in section 6.
We consider Narabizi (Seddah et al., 2020), an Arabic dialect spoken in North-Africa written in the Latin script and code-mixed with French, to fall in the same Intermediate category, because it follows the same pattern. For both POS tagging and parsing, the multilingual models outperform the monolingual NarabiziBERT. In addition, MLM-TUNING leads to significant improvements over the non-language-tuned mBERT baseline, also outperforming the non-contextual dependency parsing baseline.
We also categorize Bambara, a Niger-Congo Bantu language spoken in Mali and surrounding countries, as Intermediate, relying mostly on the
Model	UPOS	LAS	NER
MBERT	MBERT+MLM	MLM	Baseline	MBERT	MBERT+MLM	MLM	Baseline	MBERT	MBERT+MLM	MLM	Baseline
Uyghur	77.0	88.4	87.4	90.0	45.5	48.9	57.3	67.9	24.3	34.6	41.4	53.8
Sindhi	-	-	-	-	-	-	-	-	42.3	47.9	45.2	51.4
Sorani Kurdish	-	-	-	-	-	-	-	-	70.4	75.6	80.6	80.5
Table 3: Hard Languages POS, Parsing and NER scores comparing mBERT, mBERT+MLM and monolingual MLM to strong non-contextual baselines when trained and evaluated on unseen languages. Hard Languages are the ones for which mBERT fails to reach decent performance even after MLM-TUNING.
POS tagging results which follow similar patterns as Maltese and Narabizi. We note that the BambaraBERT that we trained achieves notably poor performance compared to the non-contextual baseline, a fact we attribute to the extremely low amount of available data (1000 sentences only). We also note that the non-contextual baseline is the best performing model for dependency parsing, which could also potentially classify Bambara as a "Hard" language instead.
Our results in Wolof follow the same pattern. The non-contextual baseline achieves a 77.0 in LAS outperforming mBERT. However, MLM-TUNING achieves the highest score of 77.9.
The importance of script We now turn our focus to Uralic languages. Finnish, Estonian, and Hungarian are high-resource representatives of this language family that are typically included in multilingual LMs, also having task-tuning data available in large quantities. However, for several smaller Uralic languages, task-tuning data are generally very scarce.
We report in Table 2 the performance for two low-resource Uralic languages, namely Livvi and Erzya using 8-fold cross-validation, with each run only using around 700 training instances. Note the striking difference between the parsing performance (LAS) of mBERT on Livvi, written with the Latin script, and on Erzya that uses the Cyrillic script. This suggests that the script could be playing a critical role when transferring to those languages. We explore this hypothesis in detail in section 5.2.
4.3 Hard
The last category of the hard unseen language is perhaps the most interesting one, as these languages are very hard to process. mBERT is outperformed by non-contextual baselines as well as by monolingual language models trained from scratch on the available raw data. At the same time, MLM-TUNING on the available raw data has a minimal
impact on performance.
Uyghur, a Turkic language with about 10-15 million speakers in central Asia, is a prime example of a hard language for current models. In our experiments, outlined in Table 3, the noncontextual baseline outperforms all contextual variants, both monolingual and multilingual, in all the tasks with up to 20 points difference compared to mBERT for parsing. Additionally, the monolingual UyghurBERT trained on only 105K sentences outperforms mBERT even after MLM-TUNING.
We attribute this discrepancy to script differences: Uyghur uses the Perso-Arabic script, when the other Turkic languages that were part of mBERT pretraining use either the Latin (e.g. Turkish) or the Cyrillic script (e.g. Kazakh).