SlowGuess/NAACL_2021_OCR huggingface dataset:
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy. Association for Computational Linguistics.
Shruti Rijhwani, Jiateng Xie, Graham Neubig, and Jaime Carbonell. 2019. Zero-shot neural transfer for cross-lingual entity linking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6924-6931.
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1-10, Valencia, Spain. Association for Computational Linguistics.
Stefan Schweter. 2020. BERTurk - BERT models for Turkish.
Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoit Sagot, and Abhishek Srivastava. 2020. Building a user-generated content North-African Arabizi treebank: Tackling hell. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1139-1150, Online. Association for Computational Linguistics.
Steve Stecklow. 2018. Why Facebook is losing the war on hate speech in Myanmar, Reuters. https://www.reuters.com/investigates/spcial-report/myanmar-facebook-hate/.
Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197-207.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch BERT model. arXiv:1912.09582.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Computational Linguistics.
Daniel Zeman and Jan Hajic, editors. 2018. Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium.
A Languages
We list the 15 typologically diverse unseen languages we experiment with in Table 12 with information on their language family, script, origin and number of sentences available along with the categories we classified them in.
Data Sources We base our experiments on data originated from two sources. The Universal Dependency project (Nivre et al., 2016) downloadable here https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-2988 and the WikiNER dataset (Pan et al., 2017). We also use of the CoNLL-2003 shared task NER English dataset https://www.clips.uantwerpen.be/conll2003/
B Reproducibility
Infrastructure Our experiments were ran on a shared cluster on the equivalent of 15 Nvidia Tesla T4 GPUs. $^{8}$
Optimization For all pretraining and fine-tuning runs, we use the Adam optimizer (Kingma and Ba, 2015). For fine-tuning, following Devlin et al. (2019), we only back-propagate through the first
<sup>8</sup>https://www.nvidia.com/en-sg/data-center/tesla-t4/
Params.	Parsing	NER	POS	Bounds
batch size	32	16	16	[1,256]
learning rate	5e-5	3.5e-5	5e-5	[1e-6,1e-3]
epochs (best)	15	6	6	[1,+inf]
#grid	60	60	180	-
Run-time	32	24	75	-
Table 6: Fine-tuning best hyper-parameters for each task as selected on the validation set with bounds. #grid: number of grid search trial. Runtime is reported in average for training and evaluation. Run-time indicated in minutes.
Parameter	Value
batch size	64
learning rate	5e-5
optimizer	Adam
warmup	linear
warmup steps	10% total
epochs (best of)	10
token of each word. We select the hyperparameters that minimize the loss on the validation set. The reported results are the average score of 5 runs with different random seeds computed on the test splits. We report the hyperparameters in Table 6-7.
C Easy Languages
We describe here in more details languages that we classify as Easy in section 4.1.
In practice, one can obtain very high performance even in zero-shot settings for such languages, by performing task-tuning on related languages.
Table 7: Unsupervised fine-tuning hyperparameters
Model	UPOS	LAS	NER
Zero-Shot
(1)	FaroeseBERT	66.4	35.8	-
(2)	mBERT	79.4	67.5	-
(3)	mBERT +MLM	83.4	67.8	-
Few-Shot (CV with around 500 instances)
(4)	Baseline	95.36	83.02	44.8
(5)	FaroeseBERT	91.12	67.66	39.3
(6)	mBERT	96.31	84.02	52.1
(7)	mBERT +MLM	96.52	86.41	58.3
Perhaps the best example of such an "easy" set
ting is Faroese. mBERT has been trained on several languages of the north Germanic genus of the Indo-European language family, all of which use the Latin script. As a result, the multilingual mBERT model performs much better than the monolingual FaroeseBERT model that we trained on the available Faroese text (cf rows 1-2 and 5-6 in Table 8). Fine-tuning mBERT on the Faroese text is even more effective (rows 3 and 6 in Table 8), leading to further improvements, reaching more than  $96.5\%$  POS-tagging accuracy,  $86\%$  LAS for dependency parsing, and  $58\%$  NER F1 in the few-shot setting, surpassing the non-contextual baseline. In fact, even in zero-shot conditions, where we task-tune only on related languages (Danish, Norwegian, and Swedish), the model achieves remarkable performance of over  $83\%$  POS-tagging accuracy and  $67.8\%$  LAS dependency parsing.
Table 8: Faroese is an "easy" unseen language: a multilingual model (+ language-specific MLM) easily outperforms all baselines. Zero-shot performance, after task-tuning only on related languages (Danish, Norwegian, Swedish) is also high.
Model	UPOS	LAS
Zero-Shot
(1)	SwissGermanBERT	64.7	30.0
(2)	mBERT	62.7	41.2
(3)	mBERT +MLM	87.9	69.6
Few-Shot (CV with around 100 instances)
(4)	Baseline	75.22	32.18
(5)	SwissGermanBERT	65.42	30.0
(6)	mBERT	76.66	41.2
(7)	mBERT +MLM	78.68	69.6
Table 9: Swiss German is an "easy" unseen language: a multilingual model (+ language-specific MLM) outperforms all baselines in both zero-shot (task-tuning on the related High German) and few-shot settings.
Swiss German is another example of a language for which one can easily adapt a multilingual model and obtain good performance even in zero-shot settings. As in Faroese, simple MLM fine-tuning of the mBERT model with 200K sentences leads to an improvement of more than 25 points in both POS tagging and dependency parsing (Table 9) in zero-shot settings, with similar improvement trends in the few-shot setting.
The potential of similar-language pretraining along with script similarity is also showcased in the case of Naija (also known as Nigerian English or Nigerian Pidgin), an English creole spoken by millions in Nigeria. As Table 10 shows, with results after language- and task-tuning on 6K training examples, the multilingual approach surpasses the monolingual baseline.
On a side note, we can rely on Han and Eisen-
Model	UPOS	LAS
NaijaBERT	87.1	63.02
mBERT	89.3	71.6
mBERT +MLM	89.6	69.2
stein (2019) to also classify Early Modern English as an easy language. Similarly, the work of Chau et al. (2020) allows us to also classify Singlish (Singaporean English) as an easy language. In both cases, these languages are technically unseen by mBERT, but the fact that they are variants of English allows them to be easily handled by mBERT.
D Additional Uralic languages experiments
Following a similar procedure as in the Appendix C, we start with mBERT, perform task-tuning on Finnish and Estonian (both of which use the Latin script) and then do zero-shot experiments on Livvi, and Komi, all low-resource Uralic languages (results on the top part of Table 11). We also report results on the Finnish treebanks after task-tuning, for better comparison. The difference in performance on Livvi (which uses the Latin script) and the other languages that use the Cyrillic script is striking.
Although they are not easy enough to be tackled in a zero-shot setting, we show that the low-resource Uralic languages fall in the "Intermediate" category, since mBERT has been trained on similar languages: a small amount of annotated data are enough to improve over mBERT using task-tuning.
For both Livvi and Erzya, the multilingual model along with MLM-TUNING achieves the best performance, outperforming the non-contextual baseline by more than 1.5 point for parsing and POS tagging.
E Controlled experiment: Transliterating High-Resource Languages
To have a broader view on the effect of transliteration when using mBERT (section 5.2), we study the impact of transliteration to the Latin script on high resource languages seen during mBERT pretraining such as Arabic, Japanese and Russian. We compare the performance of mBERT fine-tuned
Table 10: Performance on Naija, an English creole, is very high, so we also classify it as an "easy" unseen language.
Language	UPOS	LAS
Task-tuned – Latin script
Finnish (FTB)	93.1	77.5
Finnish (TDT)	95.0	78.9
Finnish (PUD)	96.8	83.5
Zero-Shot Experiments
Latin script
Livvi	72.3	40.3
Cyrillic script
Erzya	51.5	18.6
Few-Shot Experiments (CV)
Livvi – Latin script
Baseline	84.1	40.1
mBERT	83.0	36.3
mBERT +MLM	85.5	42.3
Erzya – Cyrillic script
Baseline	91.1	65.1
mBERT	89.3	61.2
mBERT +MLM	91.2	66.6
Table 11: The script matters for the efficacy of cross-lingual transfer. The zero-shot performance on Livvi, which is written in the same script as the task-tuning languages (Finnish, Estonian), is almost twice as good as the performance on the Uralic languages that use the Cyrillic script.
and evaluated on the original script with mBERT fine-tuned and evaluated on the transliterated text. As reported in Table 5, transliterating those languages to the Latin script leads to large drop in performance for all the three tasks.
Language (iso)	Script	Family	#sents	source	Category
Faroese (fao)	Latin	North Germanic	297K	(Biemann et al., 2007)	Easy
Mingrelian (xmf)	Georg.	Kartvelian	29K	Wikipedia	Easy
Naija (pcm)	Latin	English Pidgin	237K	(Caron et al., 2019)	Easy
Swiss German (gsw)	Latin	West Germanic	250K	OSCAR	Easy
Bambara (bm)	Latin	Niger-Congo	1K	OSCAR	Intermediate
Wolof (wo)	Latin	Niger-Congo	10K	OSCAR	Intermediate
Narabizi (nrz)	Latin	Semitic*	87K	(Seddah et al., 2020)	Intermediate
Maltese (mlt)	Latin	Semitic	50K	OSCAR	Intermediate
Buryat (bxu)	Cyrillic	Mongolic	7K	Wikipedia	Intermediate
Mari (mgr)	Cyrillic	Uralic	58K	Wikipedia	Intermediate
Erzya (myv)	Cyrillic	Uralic	20K	Wikipedia	Intermediate
Livvi (olo)	Latin	Uralic	9.4K	Wikipedia	Intermediate
Uyghur (ug)	Arabic	Turkic	105K	OSCAR	Hard
Sindhi (sd)	Arabic	Indo-Aryan	375K	OSCAR	Hard
Sorani (ckb)	Arabic	Indo-Iranian	380K	OSCAR	Hard
Table 12: Unseen Languages used for our experiments. #sents indicates the number of sentences used for training from scratch Monolingual Language Models as well as for MLM-TUNING mBERT *code-mixed with French