Introduction
In the last few years, developing massively multilingual Pre-trained Language Models (PLMs) to scale to several written languages is an active area of research—e.g. covering 100 languages Devlin et al. ( 2019 ); Conneau et al. ( 2020 ); Liu et al. ( 2020 ); Xue et al. ( 2021 ); He et al. ( 2023 ) . However, evaluation is often limited to a few tens of languages with benchmark datasets (Conneau et al., 2018 ; Hu et al., 2020 ; Ruder et al., 2021 ; Zhang et al., 2022 ) , thus, limiting the large-scale evaluation of current multilingual language models on many languages, especially those truly low-resource languages.

While there is evidence from previous works that languages not covered during pre-training often lead to lower performance, such analysis is also limited to a small selection of languages with annotated datasets Ponti et al. ( 2020 ); Pfeiffer et al. ( 2020 ); Adelani et al. ( 2022b ); Lee et al. ( 2022 ) .

Recently, there is a push to scale evaluation datasets to more than 100 languages, but this requires a very expensive annotation effort in terms of money and time. Often, this scaling is only carried out by a large community effort that spans many years like the Universal Dependency (UD) project (Nivre et al., 2017 , 2020 ; de Marneffe et al., 2021 ) or financed by BigTech companies (Goyal et al., 2022 ; NLLB-Team et al., 2022 ; Federmann et al., 2022 ; Conneau et al., 2022 ; Pratap et al., 2023 ) . Despite these investments in data curation, there are only few benchmarks for natural language understanding (NLU) tasks that cover all the languages seen during the pre-training of multilingual PLMs (ImaniGooghari et al., 2023 ) .

The largest benchmark datasets that are available for NLU are UD, Taxi1500 (Ma et al., 2023 ) , WikiANN Pan et al. ( 2017 ) , and Belebele (Bandarkar et al., 2023 ) for dependency parsing, text classification, named entity recognition, and reading comprehension, respectively. The largest is Taxi-1500 for 1500 languages—but it is biased to the religious domain, and some languages are not publicly available due to copyright. WikiANN on the other hand, was automatically annotated and with few instances for low-resource languages. UD and Belebele were manually annotated and covered between 100-125 languages. However, many languages are still missing in the above datasets.

In this paper, we create SIB-200 —a large-scale open-sourced benchmark dataset for topic classification to address the lack of evaluation datasets for NLU. The dataset is based on Flores-200 (NLLB-Team et al., 2022 ) —a multi-way parallel corpus (i.e. same sentences are available in 205 languages). We annotated the English portion of the Flores-200 dataset and extend the sentence-level annotation to the remaining 204 languages covered in Flores-200.

Our evaluation shows that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to
numerous world languages. Languages unseen during the pre-training of multilingual PLMs, languages from under-represented families (like Nilotic and Altantic-Congo), and languages from the regions of Africa, Americas, Oceania and South East Asia, often have the lowest performance on our dataset. We also find that simply scaling up the number of languages without scaling up the domains in the pre-training is unhelpful (e.g., Glot-500 pre-trained on 500 languages largely under-performs XLM-R pre-trained on 100 languages). It is crucial to mix text from various domains. For languages unseen during pre-training, we show the potential of multilingual language adaptive fine-tuning (MAFT) 2 2 2 adaptation of an existing multilingual PLM to multiple or new sets of languages simultaneously. (Tang et al., 2020 ; Alabi et al., 2022 ) in improving the performance of these languages by leveraging synthetic data for languages with tiny monolingual data (i.e. language with less than 10MB of data). Evaluation of this approach on African languages results in significant improvement (up to +5\% in accuracy on average) for the previously unseen languages.

Finally, we extend our evaluation to the zero-shot settings by training individually on English, French, Arabic and Chinese (Simplified) languages using XLM-R (Conneau et al., 2020 ) , and performing zero-shot evaluation on other languages. We compared these results with prompting GPT-4 large language models (LLMs). Our results show that LLMs perform poorly on over 64.7% (or 132 out of 205 languages) of the languages with less than 70% in accuracy while zero-shot adaptation from the English model only leads to performance less than 70% accuracy in 81 languages (or 39.3% of languages) 3 3 3 Performance of XLM-R on English is 92.1% in accuracy while prompting GPT-4 in English gave 76.6% in accuracy. . This shows that leveraging cross-lingual transfer from high-resource languages is much better than prompting LLMs for many languages.